GitLab

Lots of modifications. The one I just made is to deal with null values in the time_offset variable, and by deal with... I mean skip that file.

Fixed issue where variables were getting setup many times and slowing down processing. Vectorized calculating datetime index. Import netCDF4 instead of only netCDF4.Dataset

getting recent dev branch back into the fold.

See merge request giansiracumt/netcdf2ascii!9

added several unit tests, modified the return value of a couple functions when errors happen and modified the main processing algorithm to raise the actual error. Much of the logging modifications are also present in this commit.

Comms config

See merge request giansiracumt/netcdf2ascii!5

# Conflicts:
#   nc2csv.py

Changed main to only send errors. Updated nc2csv to import comms and send message when dqr web service is down. Started changing to use requests module instead of urllib.

Deleted library for timed rotating file handler. The config.yaml file specifies the regular rotating file handler that is size limited with 10 backups.

Change methods so that the debug attribute is no longer used. Deleted large commented block that contained code written to handle an error when variables could not be parsed earlier that would query the dqrws for each variable in each file each time, very, very slow.

script uses logger module now with different console and file logging levels. no more debug logging flag, all debug logging is in rotating file. Dict config used to configure logger. Dict loaded from yaml file in current directory.

Removed unnecessary print and parseargs print help message that was causing a test condition to fail

Fixed bug in nc2csv.py where if no varlist was passed the netcdf variables were overwriting the date_time key in the beginning of process_one_file method. Also added a notes file for basic info.

Slack messaging

See merge request giansiracumt/netcdf2ascii!4

removed an unused import, and added try catch block and comment around the imports that aren't in the repo. Comms module and config files are specific to my monitoring and should eventually be replaced with some enterprise monitoring system.

Removed the backup dir because it is no longer in the local repo. Added the .ipynb checkpoints becuase it is a new local file

Fixed mismatch var columns list

See merge request giansiracumt/netcdf2ascii!3

fixing mismatch var columns list. Passed input vars to trans_header and only added new var to cleaned var columns if it is in vars. Then passed cleaned var columns back from trans_header method and used that list for the rest of the variables.

Single2pool

See merge request giansiracumt/netcdf2ascii!2

modified ProcessManager to user multiprocessing pool and debuged. added some file sorting for better file dates in output files.

first attempt at a process pool with a partial function at the end of ProcessManager.process_files()