adi_py issueshttps://code.arm.gov/adi/adi_py/-/issues2023-08-19T17:59:58Zhttps://code.arm.gov/adi/adi_py/-/issues/16Unable to set output coordinate length2023-08-19T17:59:58ZMaxwell LevinUnable to set output coordinate lengthIn `adi_py` `v3.13.0` I had written the following code block to set the coordinate values on an output coordinate variable that was previously empty. In `v3.21.0` this fails with an error message `Could Not Allocate Memory For Dataset Va...In `adi_py` `v3.13.0` I had written the following code block to set the coordinate values on an output coordinate variable that was previously empty. In `v3.21.0` this fails with an error message `Could Not Allocate Memory For Dataset Variable`.
<details><summary>set_output_coord_data()</summary>
```python
# Used in the process_data hook; this is a function on the `Process` class
def set_output_coord_data(
self, datastream: str, coord_name: str, data: NDArray[Any]
) -> xr.Dataset:
"""Convenience method to update a specified coordinate's values.
The shape of the data does not need to match the existing coordinate's shape;
the coordinate dimension length will be set to the length of the data array
provided. It is recommended to call this method before modifiying any data
variables dimensioned by the coordinate, as the data values may not be preserved
if the dimension length changes.
Args:
datastream (str): The output datastream to update.
coord_name (str): The name of the coordinate to update.
data (NDArray[Any]): The new data values that should be set.
Returns:
xr.Dataset: The updated output dataset.
"""
ds_name, ds_level = datastream.split(".")
# Use core dsproc library to set the dimension length
dsid_out = dsproc.get_output_datastream_id(ds_name, ds_level) # type: ignore
out_ds = dsproc.get_output_dataset(dsid_out, 0) # type: ignore
dsproc.set_dim_length(out_ds, coord_name, len(data)) # type: ignore
# Use core dsproc library to set the coordinate data values
out_var = dsproc.get_output_var(dsid_out, coord_name, 0) # type: ignore
coord_data = dsproc.alloc_var_data_index(out_var, 0, len(data)) # type: ignore
coord_data[:] = data[:]
return self.get_output_dataset(datastream)
```
</details>
And the error message
<details><summary>Current Error Message</summary>
```
/home/levin/.conda/envs/mergedsmpsaps/lib/python3.9/site-packa **************** Starting process_data_hook ****************
----- LOG MESSAGE ---------------------
/home/levin/.conda/envs/mergedsmpsaps/lib/python3.9/site-packa No length set for merged_diameter_mobility, setting length to 1
----- END LOG MESSAGE -----------------
----- ERROR MESSAGE -------------------
../../src/cds_dims.c:225 Could not change dimension length for: /sgpmergedsmpsapsE13.c1/_dims_/merged_diameter_mobility
-> data has been added for a variable using this dimension
----- END ERROR MESSAGE ---------------
../../src/dsproc.c:4013 Setting status to: 'Could Not Set Dimension Length In Dataset'
----- ERROR MESSAGE -------------------
../../src/cds_var_data.c:575 Could not allocate memory for variable data: /sgpmergedsmpsapsE13.c1/_vars_/merged_diameter_mobility
-> start sample (0) + sample count (212) > dimension length (1)
----- END ERROR MESSAGE ---------------
../../src/dsproc.c:4013 Setting status to: 'Could Not Allocate Memory For Dataset Variable'
../../src/dsproc.c:4013 Setting status to: 'Hook process_data_hook failed.'
----- ERROR MESSAGE -------------------
/home/levin/.conda/envs/mergedsmpsaps/lib/python3.9/site-packa Error Message: Hook process_data_hook failed.
Error Type: TypeError
Exception Message: 'NoneType' object does not support item assignment
Stack Trace:
File "/home/levin/.conda/envs/mergedsmpsaps/lib/python3.9/site-packages/adi_py/utils.py", line 91, in wrapper_function
hook_func(*args, **kwargs)
File "/home/levin/code.arm.gov/vap/mergedsmpsaps/mergedsmpsaps/__main__.py", line 150, in process_data_hook
ds = self.set_output_coord_data(
File "/home/levin/code.arm.gov/vap/mergedsmpsaps/mergedsmpsaps/__main__.py", line 133, in set_output_coord_data
coord_data[:] = data[:]
----- END ERROR MESSAGE ---------------
```
</details>
What do I need to do in order to set the output coordinate data & length in this new version of adi_py?https://code.arm.gov/adi/adi_py/-/issues/14Fix default values for qc variables if data array is empty2023-01-19T00:53:24ZCarina LansingFix default values for qc variables if data array is emptyBrian confirmed that we should be initializing qc values to zero if the corresponding data variable has values. If the corresponding data variable has no values, then initialize qc value to missing value bit.Brian confirmed that we should be initializing qc values to zero if the corresponding data variable has values. If the corresponding data variable has no values, then initialize qc value to missing value bit.Carina LansingCarina Lansinghttps://code.arm.gov/adi/adi_py/-/issues/12ADILogger.exception should not call dsproc.error2022-07-21T01:41:37ZMaxwell LevinADILogger.exception should not call dsproc.errorCurrently `ADILogger.exception()` calls `dsproc.error()` which results in the process exiting with a non-zero exit code. This is not the documented behavior of this method and its behavior not consistent with the python standard library ...Currently `ADILogger.exception()` calls `dsproc.error()` which results in the process exiting with a non-zero exit code. This is not the documented behavior of this method and its behavior not consistent with the python standard library method `logging.exception()` method, so it is not intuitive either.
To make `ADILogger.exception()` consistent with its docstring and with python built-ins it should call `dsproc.warning()` instead. This would allow users to both log and handle exceptions in their code instead of having to pick one or the other.
The `ADILogger.exception()` docstring is below:
```
"""-----------------------------------------------------------------------
Use this method to log the stack trace of any raised exception to the process's
ADI log file.
Args:
- message: str
An optional additional message to log, in addition to the stack trace.
-----------------------------------------------------------------------"""
```https://code.arm.gov/adi/adi_py/-/issues/10Add file name to 'hidden' metadata on Xarray retrieved dataset2022-05-25T23:15:23ZCarina LansingAdd file name to 'hidden' metadata on Xarray retrieved datasetWhen pulling the special ADI metadata into the xarray retrieved dataset, I will also get the file name where the data came from so Krista has access to it and add an accessor method.When pulling the special ADI metadata into the xarray retrieved dataset, I will also get the file name where the data came from so Krista has access to it and add an accessor method.https://code.arm.gov/adi/adi_py/-/issues/9create_adi_project C template2022-05-25T23:13:15ZCarina Lansingcreate_adi_project C templateAdd a .vscode folder to the create_adi_project template for C VAPs and ingests. Send the C link to Sherman so he can put it on i.arm.gov
The .vscode folder should include required workspace settings and a default launch configs for run...Add a .vscode folder to the create_adi_project template for C VAPs and ingests. Send the C link to Sherman so he can put it on i.arm.gov
The .vscode folder should include required workspace settings and a default launch configs for running and attaching to running. See how I set up VSCode for debugging the data consolidator.
May also need to add a README file to explain what to do, including instructions for connecting remotely.https://code.arm.gov/adi/adi_py/-/issues/8create_adi_project python template2022-05-25T23:11:25ZCarina Lansingcreate_adi_project python templateAdd the following changes to the python templates
1. In the method docs, make sure to add the 'raises' descriptor and explain how to raise the SkipProcessingInterval exception in order to return 0 or any other exception to return -1.
1....Add the following changes to the python templates
1. In the method docs, make sure to add the 'raises' descriptor and explain how to raise the SkipProcessingInterval exception in order to return 0 or any other exception to return -1.
1. Add the post_retrieval_hook method to the transform templatehttps://code.arm.gov/adi/adi_py/-/issues/7Update Readthedocs2022-05-25T23:06:03ZCarina LansingUpdate ReadthedocsUpdate the readthedocs documentation to make sure that it is clear that if you want to change variable data, you need to just change the data directly. Do not replace the variable unless you make a copy of the variable. This is so that...Update the readthedocs documentation to make sure that it is clear that if you want to change variable data, you need to just change the data directly. Do not replace the variable unless you make a copy of the variable. This is so that you don’t lose special metadata attributes that are needed for synchronization.https://code.arm.gov/adi/adi_py/-/issues/6Improve README file2022-05-25T23:04:57ZCarina LansingImprove README fileIn the README file created by create_adi_project, add instructions for setting up an 'attach' launch config. This can be used to attach to code when running dsproc_test.In the README file created by create_adi_project, add instructions for setting up an 'attach' launch config. This can be used to attach to code when running dsproc_test.https://code.arm.gov/adi/adi_py/-/issues/5Type hinting2022-05-25T23:26:52ZMaxwell LevinType hintingThis library has some type hints, but this information is not exported when the module is packaged, resulting in `mypy` errors and a lack of intellisense / autocompletion when writing VAP code that uses adi_py.
Screenshots below show my...This library has some type hints, but this information is not exported when the module is packaged, resulting in `mypy` errors and a lack of intellisense / autocompletion when writing VAP code that uses adi_py.
Screenshots below show mypy and pylance type checking errors in a VS Code window with the ErrorLens extension installed and mypy installed (pip install mypy).
![image](/uploads/def3c8b2b4fc81e54a62fe958bc2a1bb/image.png)
![image](/uploads/dacee0be90cc4018398169afc6fd68e3/image.png)
![image](/uploads/fe0b7e18df520a9592efb01eb14f5c06/image.png)Maxwell LevinMaxwell Levinhttps://code.arm.gov/adi/adi_py/-/issues/4Cannot install with Python 3.102022-05-23T20:07:57ZMaxwell LevinCannot install with Python 3.10The [ARM-DOE/adi_py](https://anaconda.org/ARM-DOE/adi_py) conda distribution does not work with Python 3.10. I think it probably just needs to be recompiled/rebuilt since the last build of adi_py was from 7 months ago, which is just abou...The [ARM-DOE/adi_py](https://anaconda.org/ARM-DOE/adi_py) conda distribution does not work with Python 3.10. I think it probably just needs to be recompiled/rebuilt since the last build of adi_py was from 7 months ago, which is just about when python 3.10 was officially released, but probably before conda fully supported it.
Here's a minimal example you can run to check this:
`environment.yml`:
```yaml
name: cloudphase
channels:
- conda-forge
dependencies:
- python=3.10
- arm-doe::adi_py
```
Then run:
```
conda env create -f environment.yml
```
I see output like this (this is on emerald):
```
Collecting package metadata (repodata.json): done
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package python conflicts for:
python=3.10
arm-doe::adi_py -> cython -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.10,<3.11.0a0|>=3.8,<3.9.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0|3.4.*|3.9.*']
arm-doe::adi_py -> python[version='>=3.7,<3.8.0a0|>=3.9,<3.10.0a0']The following specifications were found to be incompatible with your system:
- feature:/linux-64::__glibc==2.17=0
- arm-doe::adi_py -> libgcc-ng[version='>=7.3.0'] -> __glibc[version='>=2.17']
- python=3.10 -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
Your installed version is: 2.17
```https://code.arm.gov/adi/adi_py/-/issues/3Xarray has issues if height dimension changes in the middle of a netcdf file2022-05-25T22:59:52ZCarina LansingXarray has issues if height dimension changes in the middle of a netcdf filePer Tami's email:
Hi Carina,
The issue that xarray can't handle with time-resolved heights is exemplified in cell 3 of this notebook:
https://code.arm.gov/lasso/containers/run-lasso-o_shcu/-/blob/master/notebooks/plot_cloud_fraction.i...Per Tami's email:
Hi Carina,
The issue that xarray can't handle with time-resolved heights is exemplified in cell 3 of this notebook:
https://code.arm.gov/lasso/containers/run-lasso-o_shcu/-/blob/master/notebooks/plot_cloud_fraction.ipynb
If, instead of using netCDF4's Dataset, I use xr.open_dataset, I get the following error message:
MissingDimensionsError: 'height' has more than 1-dimension and the same name as one of its dimensions ('time', 'height'). xarray disallows such variables because they conflict with the coordinates used to label dimensions.
Looks like this is a known issue: https://github.com/pydata/xarray/issues/2233
Don't stress about it though because I don't know how often this type of data structure comes up with VAPs. When I've seen it, it's in model data (i.e., not observations), so maybe just keep it in the back of your mind.
-Tamihttps://code.arm.gov/adi/adi_py/-/issues/2Add more support for qc variables2022-05-25T22:59:12ZCarina LansingAdd more support for qc variablesAdd adi qc methods similar to ACT library qcfilter accessor. Since ACT uses different attributes for qc variables, we can't call these methods directly, so we should see about fixing ACT library so it would support either.Add adi qc methods similar to ACT library qcfilter accessor. Since ACT uses different attributes for qc variables, we can't call these methods directly, so we should see about fixing ACT library so it would support either.