Working on updating the notebooks folder & documentation so notebooks work out of the box.

README-SINGULARITY.md

@@ -5,10 +5,9 @@ slurm scheduler.  Actual commands may be different at your institution
 depending upon the scheduler used.
 
 ## Setting up Singularity environment on your HPC cluster
-#### Prerequisite:  Singularity 3+
-Singularity 3+ has been tested and approved for use with the LASSO-O
-container.  It might run with an older version of Singularity, but this
-has not been verified.
+#### Prerequisite:  Singularity 3.6+
+Singularity 3.6+ is required for use with the LASSO-O
+container.
 
 Usually Singularity is available via a module load.  To see if
 Singularity is available on your cluster, type the following:

README.md

@@ -47,6 +47,15 @@ sgpwrfout10C1.00
 sgpwrfstat10C1.00
 ```
 
+---
+**NOTE**
+
+You do not have to copy any observational data (e.g., sgpcldfracset15mC1.c1, sgplassodiagobsC1.c1)
+as this data is embedded in the container image.
+
+---
+
+
 ### 4) Edit config.yml file 
 Edit the config.yml file to provide parameters about your run.  The config.yml file contains
 detailed descriptions for each parameter.  If you run following these instructions, you

notebooks/README.md

@@ -78,13 +78,6 @@ notebooks folder:
 
 #### Using the notebooks
 
-You should replace the paths to the sample data used in these examples
-with the corresponding paths to the output data from your
-`run-lasso-o_shcu/data/outputs` folder.  For example:
-
-![Jupyter Notebooks](https://code.arm.gov/lasso/containers/lasso-o_shcu/-/raw/master/images/data-dir.png)
-
-
 The available notebooks include:
 * plot_1D.ipynb for plotting time series
 * plot_cloud_fraction.ipynb for plotting time-height cloud fraction plots

notebooks/link_output_data.py

+#!/usr/bin/env python
+"""---------------------------------------------------------------------------------------
+  Python script for copying LASSO-O container outputs to the notebooks/DATA via
+  symbolic links so that they can be easily visualized.
+---------------------------------------------------------------------------------------"""
+import os
+import subprocess
+import sys
+import ssl
+
+
+def create_sym_link(source_file, linked_file):
+    # make sure the target does not exist
+    if os.path.exists(linked_file):
+        os.remove(linked_file)
+
+    # Make a symbolic link
+    print("Creating symbolic link: " + linked_file)
+    os.symlink(source_file, linked_file)
+
+
+def main():
+    """-------------------------------------------------------------------
+    Main function.
+    -------------------------------------------------------------------"""
+
+    run_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+    outputs_dir = os.path.join(run_dir, "data/outputs")
+    data_dir = os.path.join(run_dir, "notebooks/DATA")
+
+    for root, dirs, files in os.walk(outputs_dir):
+        for name in files:
+            print(os.path.join(root, name))
+        for name in dirs:
+            print(os.path.join(root, name))
+
+
+if __name__ == "__main__":
+    main()
\ No newline at end of file

notebooks/plot_1D.ipynb

@@ -40,7 +40,7 @@
     "- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassodiagobsmod1C1.m1/sgplassodiagobsmod1C1.m1.20180710.120000.nc\")\n",
     "\n",
     "\n",
-    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
+    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
     "\n",
     "\n",
     "- **simids** is a list of strings representing simulation IDs to be plotted\n",

notebooks/plot_cloud_fraction.ipynb

@@ -41,7 +41,7 @@
     "- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassodiagobsmodz3C1.m1/sgplassodiagobsmodz3C1.m1.20180710.120000.nc\")\n",
     "\n",
     "\n",
-    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
+    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
     "\n",
     "\n",
     "- **simid** is a string representing the single simulation ID to be plotted\n"

notebooks/plot_profiles.ipynb

@@ -37,7 +37,7 @@
     "- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassomod3C1.m1/sgplassomod3C1.m1.20180710.120000.nc\")\n",
     "\n",
     "\n",
-    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
+    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
     "\n",
     "\n",
     "- **simid** is a string representing the single simulation ID to be plotted\n"

notebooks/plot_scores.ipynb

@@ -34,7 +34,7 @@
     "- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassoscoreC1.m1/sgplassoscoreC1.m1.20180710.120000.nc\")\n",
     "\n",
     "\n",
-    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
+    "- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
     "\n",
     "\n",
     "- **vars_to_plot** is a list of strings representing the variables to be plotted. Possible variables are as follows:\n",

run.sh

@@ -8,6 +8,9 @@
 GREEN='\033[0;32m' # Green text
 NC='\033[0m'       # No Color
 
+# Exit immediately if a command exits with non-zero status
+set -e
+
 show_help() {
     echo ""
     echo -e "$GREEN--------------------------------------------------------------------------$NC"
@@ -45,14 +48,24 @@ run() {
 
   if [ $container_runtime == "docker" ]; then
     run_docker
+    $lasso_ran=true
+
   elif [ $container_runtime == "singularity" ]; then
     run_singularity
+    $lasso_ran=true
+
   elif [ $container_runtime == "shifter" ]; then
     run_shifter
-  else
-    show_help
+    $lasso_ran=true
+
   fi
 
+  if [ -z "$lasso_ran" ]; then
+        show_help
+  else
+    echo -e "${GREEN}===>Updating links to output data in notebooks/DATA dir.${NC}"
+    $cwd/notebooks/link_output_data.sh
+  fi
 }
 
 run_docker() {
@@ -85,19 +98,27 @@ run_singularity() {
    --bind $input_folder:/data/lasso/inputs \
    --bind $output_folder:/data/lasso/outputs \
    --writable-tmpfs docker://registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu
-
 }
 
 run_shifter() {
   # Note that NERSC id not have gitlab.com on their allowed registries, so they
   # had to pull the image the first time in order to make this work.  They
   # are adding gitlab.com so that in the future, we can pull updates
-   shifter --image=docker:registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu \
-   --entrypoint \
+   shifter \
    --env=BEGIN_DATETIME=$begin_datetime \
    --volume=$input_folder:/data/lasso/inputs \
-   --volume=$output_folder:/data/lasso/outputs
+   --volume=$output_folder:/data/lasso/outputs \
+   --image=docker:registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu -- /apps/base/python3.6/bin/python /bin/run_lasso.py
+
+
+   shifter \
+   --env=BEGIN_DATETIME=20180710.115900 \
+   --volume=/global/homes/c/carinal/run-lasso-o_shcu/data/inputs:/data/lasso/inputs \
+   --volume=/global/homes/c/carinal/run-lasso-o_shcu/data/outputs:/data/lasso/outputs \
+   --image=docker:registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu -- /apps/base/python3.6/bin/python /bin/run_lasso.py
+
 
+# /apps/base/python3.6/bin/python /bin/run_lasso.py
 #srun -C gpu shifter run --mount=type=bind,source=$HOME,destination=$HOME debian ls -l $HOME
 
 }