GitLab

Lots of modifications. The one I just made is to deal with null values in the time_offset variable, and by deal with... I mean skip that file.

Fixed issue where variables were getting setup many times and slowing down processing. Vectorized calculating datetime index. Import netCDF4 instead of only netCDF4.Dataset

added several unit tests, modified the return value of a couple functions when errors happen and modified the main processing algorithm to raise the actual error. Much of the logging modifications are also present in this commit.

Changed main to only send errors. Updated nc2csv to import comms and send message when dqr web service is down. Started changing to use requests module instead of urllib.

Deleted library for timed rotating file handler. The config.yaml file specifies the regular rotating file handler that is size limited with 10 backups.

Change methods so that the debug attribute is no longer used. Deleted large commented block that contained code written to handle an error when variables could not be parsed earlier that would query the dqrws for each variable in each file each time, very, very slow.

script uses logger module now with different console and file logging levels. no more debug logging flag, all debug logging is in rotating file. Dict config used to configure logger. Dict loaded from yaml file in current directory.

Removed unnecessary print and parseargs print help message that was causing a test condition to fail

Fixed bug in nc2csv.py where if no varlist was passed the netcdf variables were overwriting the date_time key in the beginning of process_one_file method. Also added a notes file for basic info.

modified ProcessManager to user multiprocessing pool and debuged. added some file sorting for better file dates in output files.

first attempt at a process pool with a partial function at the end of ProcessManager.process_files()

moded all df processing into ProcessManager.process_one_file to make using multiprocessing pool easier

modified print statement from PostProc dev create_output_path. Haing issue with print statement for output path.

removed self.delimiter reference in the merge_output def of PostProc. Removed some commented out class attributes from Setup and modified the maxProcesses so that it is just the default. changed the wording on the dqr extracting -> filtering

changed this script to be linear processing, no more multiprocessing. Also added a significant amount of documentation and print statements.

I've changed the program to be single threaded. It is significantly slower but much easier to debug and maintain. There was an issue with the sgpbsrnC1.a1 datastream pre 1997 where the year on the file was a two digit year

no longer print to console for each var timeblock request. now write dqr for each var to file in output directory

parse_dqrs: dqr filtering uses web service. This only works for variable level resolution if a complete farlist is passed in the beginning. Otherwise it has to be done in process_one_thread for every variable in every file which is much slower. I have implemented the second option as a fallback in the event of an empty varlist. I also changed this method to always print which variables it is getting dqr timeblocks for but only prints the web link in debug mode. If debug mode is on it alos writes a dqr_results.txt to the csv folder. This feature may be useful for users and will have to be discussed further.
merge_output: only print tmpfiles if in debug mode.
ProcessManager def __init__: added self.datastream variable to work with the ugly/inefficient fix for dqr filtering if an empty varlist is passed. Also, consider getting rid of the ProcessManager.verbose attribute.
process_one_thread: Lines 614-647 are the bad fix for empty varlist in dqr-filtering
dqr_filtering: modified the time checks so that all time ranges aren't checked, only if the dqr start is after the file end or the dqr end is before the file start.
parse_arguments: turned auto debug off.

When variable list is not present then dqr_ranges does not have variable keys
and will need to be fixed. Problem occurs in parse_dqrs() on line 218 with the
all key grouping.

modified send_email to get host name from os module, separated validate input from contructor, added some comments

is time. If it isn't then it's likely a different size dimension like
range or power. This will cause a column index mismatch and would crash
the program because of pandas Dataframe restrictions. Also added a setup
program that will create the filename and random varlist files. It will
need to be modified for server use, so far it has only been tested on
a local machine.

The following code will be in active development with a more complete
readme file, more comments and evolving functionality.