Commit 3b11675b authored by Carina Lansing's avatar Carina Lansing
Browse files

Working on updating the notebooks folder & documentation so notebooks work out of the box.

parent e8025aeb
......@@ -5,10 +5,9 @@ slurm scheduler. Actual commands may be different at your institution
depending upon the scheduler used.
## Setting up Singularity environment on your HPC cluster
#### Prerequisite: Singularity 3+
Singularity 3+ has been tested and approved for use with the LASSO-O
container. It might run with an older version of Singularity, but this
has not been verified.
#### Prerequisite: Singularity 3.6+
Singularity 3.6+ is required for use with the LASSO-O
container.
Usually Singularity is available via a module load. To see if
Singularity is available on your cluster, type the following:
......
......@@ -47,6 +47,15 @@ sgpwrfout10C1.00
sgpwrfstat10C1.00
```
---
**NOTE**
You do not have to copy any observational data (e.g., sgpcldfracset15mC1.c1, sgplassodiagobsC1.c1)
as this data is embedded in the container image.
---
### 4) Edit config.yml file
Edit the config.yml file to provide parameters about your run. The config.yml file contains
detailed descriptions for each parameter. If you run following these instructions, you
......
......@@ -78,13 +78,6 @@ notebooks folder:
#### Using the notebooks
You should replace the paths to the sample data used in these examples
with the corresponding paths to the output data from your
`run-lasso-o_shcu/data/outputs` folder. For example:
![Jupyter Notebooks](https://code.arm.gov/lasso/containers/lasso-o_shcu/-/raw/master/images/data-dir.png)
The available notebooks include:
* plot_1D.ipynb for plotting time series
* plot_cloud_fraction.ipynb for plotting time-height cloud fraction plots
......
#!/usr/bin/env python
"""---------------------------------------------------------------------------------------
Python script for copying LASSO-O container outputs to the notebooks/DATA via
symbolic links so that they can be easily visualized.
---------------------------------------------------------------------------------------"""
import os
import subprocess
import sys
import ssl
def create_sym_link(source_file, linked_file):
# make sure the target does not exist
if os.path.exists(linked_file):
os.remove(linked_file)
# Make a symbolic link
print("Creating symbolic link: " + linked_file)
os.symlink(source_file, linked_file)
def main():
"""-------------------------------------------------------------------
Main function.
-------------------------------------------------------------------"""
run_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
outputs_dir = os.path.join(run_dir, "data/outputs")
data_dir = os.path.join(run_dir, "notebooks/DATA")
for root, dirs, files in os.walk(outputs_dir):
for name in files:
print(os.path.join(root, name))
for name in dirs:
print(os.path.join(root, name))
if __name__ == "__main__":
main()
\ No newline at end of file
......@@ -40,7 +40,7 @@
"- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassodiagobsmod1C1.m1/sgplassodiagobsmod1C1.m1.20180710.120000.nc\")\n",
"\n",
"\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"\n",
"\n",
"- **simids** is a list of strings representing simulation IDs to be plotted\n",
......
......@@ -41,7 +41,7 @@
"- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassodiagobsmodz3C1.m1/sgplassodiagobsmodz3C1.m1.20180710.120000.nc\")\n",
"\n",
"\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"\n",
"\n",
"- **simid** is a string representing the single simulation ID to be plotted\n"
......
......@@ -37,7 +37,7 @@
"- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassomod3C1.m1/sgplassomod3C1.m1.20180710.120000.nc\")\n",
"\n",
"\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"\n",
"\n",
"- **simid** is a string representing the single simulation ID to be plotted\n"
......
......@@ -34,7 +34,7 @@
"- **data_dir** is a string representing the directory where the input data is stored. The notebook expects files to be within a directory off of data_dir with the name of the data stream (e.g., data_dir = \"./DATA\", file = \"./DATA/sgplassoscoreC1.m1/sgplassoscoreC1.m1.20180710.120000.nc\")\n",
"\n",
"\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](./lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"- **yrmody** is a string representing the LASSO case date in the form YYYYMMDD. See [lasso_dates.txt](../lasso_dates.txt) for dates for which the LASSO-O Shallow Convection container is configured to run. Note: the sample data included with this notebook only includes 20180710.\n",
"\n",
"\n",
"- **vars_to_plot** is a list of strings representing the variables to be plotted. Possible variables are as follows:\n",
......
......@@ -8,6 +8,9 @@
GREEN='\033[0;32m' # Green text
NC='\033[0m' # No Color
# Exit immediately if a command exits with non-zero status
set -e
show_help() {
echo ""
echo -e "$GREEN--------------------------------------------------------------------------$NC"
......@@ -45,14 +48,24 @@ run() {
if [ $container_runtime == "docker" ]; then
run_docker
$lasso_ran=true
elif [ $container_runtime == "singularity" ]; then
run_singularity
$lasso_ran=true
elif [ $container_runtime == "shifter" ]; then
run_shifter
else
show_help
$lasso_ran=true
fi
if [ -z "$lasso_ran" ]; then
show_help
else
echo -e "${GREEN}===>Updating links to output data in notebooks/DATA dir.${NC}"
$cwd/notebooks/link_output_data.sh
fi
}
run_docker() {
......@@ -85,19 +98,27 @@ run_singularity() {
--bind $input_folder:/data/lasso/inputs \
--bind $output_folder:/data/lasso/outputs \
--writable-tmpfs docker://registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu
}
run_shifter() {
# Note that NERSC id not have gitlab.com on their allowed registries, so they
# had to pull the image the first time in order to make this work. They
# are adding gitlab.com so that in the future, we can pull updates
shifter --image=docker:registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu \
--entrypoint \
shifter \
--env=BEGIN_DATETIME=$begin_datetime \
--volume=$input_folder:/data/lasso/inputs \
--volume=$output_folder:/data/lasso/outputs
--volume=$output_folder:/data/lasso/outputs \
--image=docker:registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu -- /apps/base/python3.6/bin/python /bin/run_lasso.py
shifter \
--env=BEGIN_DATETIME=20180710.115900 \
--volume=/global/homes/c/carinal/run-lasso-o_shcu/data/inputs:/data/lasso/inputs \
--volume=/global/homes/c/carinal/run-lasso-o_shcu/data/outputs:/data/lasso/outputs \
--image=docker:registry.gitlab.com/gov-doe-arm/docker/lasso-o_shcu -- /apps/base/python3.6/bin/python /bin/run_lasso.py
# /apps/base/python3.6/bin/python /bin/run_lasso.py
#srun -C gpu shifter run --mount=type=bind,source=$HOME,destination=$HOME debian ls -l $HOME
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment