GitLab

Working but... having non-critical logger issues FIXME. Removed hard cocded python3.5 dependencies. Drop multidimensional variables, don't do datetime decode.

main wrapper script is_arial function now uses input datastream instead of parsing file for deciding if it will call the icartt converter

version 2.0 for nc2csv, no more variable extraction, no dqr filtering, just cdf conversion to csv. The nc2csv file is significantly different, the icartt one is basically the same.

added checks to handle issues identified durring processing of datastream sgp1swfanalbrs1longC1.c1. Issues were, empty variables with no values, fill value set to a string when the values were floats or ints. Added in slow method of parsing because of index out of bounds error.

Lots of modifications. The one I just made is to deal with null values in the time_offset variable, and by deal with... I mean skip that file.

Fixed issue where variables were getting setup many times and slowing down processing. Vectorized calculating datetime index. Import netCDF4 instead of only netCDF4.Dataset

getting recent dev branch back into the fold.

See merge request !9

added several unit tests, modified the return value of a couple functions when errors happen and modified the main processing algorithm to raise the actual error. Much of the logging modifications are also present in this commit.

Comms config

See merge request !5

# Conflicts:
#   nc2csv.py

Changed main to only send errors. Updated nc2csv to import comms and send message when dqr web service is down. Started changing to use requests module instead of urllib.

Deleted library for timed rotating file handler. The config.yaml file specifies the regular rotating file handler that is size limited with 10 backups.

Change methods so that the debug attribute is no longer used. Deleted large commented block that contained code written to handle an error when variables could not be parsed earlier that would query the dqrws for each variable in each file each time, very, very slow.

script uses logger module now with different console and file logging levels. no more debug logging flag, all debug logging is in rotating file. Dict config used to configure logger. Dict loaded from yaml file in current directory.

Removed unnecessary print and parseargs print help message that was causing a test condition to fail

Fixed bug in nc2csv.py where if no varlist was passed the netcdf variables were overwriting the date_time key in the beginning of process_one_file method. Also added a notes file for basic info.

Slack messaging

See merge request !4

removed an unused import, and added try catch block and comment around the imports that aren't in the repo. Comms module and config files are specific to my monitoring and should eventually be replaced with some enterprise monitoring system.

Removed the backup dir because it is no longer in the local repo. Added the .ipynb checkpoints becuase it is a new local file