Commit 9d0b53aa authored by Carina Lansing's avatar Carina Lansing
Browse files

Moving from other repo.

parents
.DS_Store
.idea/
.history/
.vscode/
.vagrant/
# LASSO-O Docker Instructions
## Setting up Docker on Your Local Computer
Follow these instructions if you will be running LASSO-O on a user's
desktop machine or a machine for which you have full root privileges.
#### Install Docker
* [Click here to install Docker](https://docs.docker.com/install/)
* Test that your docker installation works by running the simple Docker image, [hello-world](https://hub.docker.com/_/hello-world/)
```
docker run hello-world
```
You should see the following output:
```
docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Already exists
Digest: sha256:6540fc08ee6e6b7b63468dc3317e3303aae178cb8a45ed3123180328bcc1d20f
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
```
* Check your docker version:
```
docker --version
Docker version 19.03.1, build 74b1e89
```
_For this demo, your docker version needs to be at least **`18.06.0`**. If your docker verion is older, update to the latest version._
More info about getting started with Docker can be found at: [https://docs.docker.com/get-started/](https://docs.docker.com/get-started/)
## Running LASSO-O via Docker
Start LASSO-O via the `run.sh` script from the run-lasso-o_shcu folder:
```bash
$ ./run.sh
```
* LASSO-O will take a while to run depending upon the number of simulations
you are processing.
The first time you run the lasso-o_shcu container, the container runtime
will download the container image, which will take a few minutes. Then,
LASSO-O will run a series of processes for each simulation provided as
input. The console will log which processes are running, so you can check
your progress. For example:
```bash
$ Running command: ['wrfstat_ingest', '-n', 'wrfstat3', '-s', 'sgp', '-f', 'C1', '-R']
$ Running command: ['wrfout_ingest', '-n', 'wrfout3', '-s', 'sgp', '-f', 'C1', '-R']
$ Running command: ['lassomod', '-n', 'lassomod3', '-s', 'sgp', '-f', 'C1', '-b', '20180710.115900', '-R']
$ Running command: ['lassodiagmod', '-n', 'lassodiagmod3', '-s', 'sgp', '-f', 'C1', '-b', '20180710', '-R']
$ Running command: ['lassodiagobsmodz', '-n', 'lassodiagobsmodz3', '-s', 'sgp', '-f', 'C1', '-b', '20180710', '-R']
$ Running command: ['lassodiagobsmod', '-n', 'lassodiagobsmod3', '-s', 'sgp', '-f', 'C1', '-b', '20180710', '-R']
$ Running command: ['wrfstat_ingest', '-n', 'wrfstat4', '-s', 'sgp', '-f', 'C1', '-R']
$ Running command: ['wrfout_ingest', '-n', 'wrfout4', '-s', 'sgp', '-f', 'C1', '-R']
$ Running command: ['lassomod', '-n', 'lassomod4', '-s', 'sgp', '-f', 'C1', '-b', '20180710.115900', '-R']
$ Running command: ['lassodiagmod', '-n', 'lassodiagmod4', '-s', 'sgp', '-f', 'C1', '-b', '20180710', '-R']
$ Running command: ['lassodiagobsmodz', '-n', 'lassodiagobsmodz4', '-s', 'sgp', '-f', 'C1', '-b', '20180710', '-R']
$ Running command: ['lassodiagobsmod', '-n', 'lassodiagobsmod4', '-s', 'sgp', '-f', 'C1', '-b', '20180710', '-R']
$ Running command: ['lassoscorez', '-s', 'sgp', '-f', 'C1', '-b', '20180710.115900', '-R']
$ Running command: ['lassoscore', '-s', 'sgp', '-f', 'C1', '-b', '20180710.115900', '-R']
```
When your job has completed, you may view the outputs created in the
`run-lasso-o_shcu/data/outputs` folder using the notebooks provided.
See the [notebooks/README.md](notebooks/README.md) file for more
information.
\ No newline at end of file
# LASSO-O Shifter Instructions
Follow these instructions if you will be running LASSO-O at NERSC.
## Running LASSO-O via Shifter
Start LASSO-O via the `run.sh` script from the **run-lasso-o_shcu** folder:
```bash
$ srun -N 1 -n 1 ./run.sh
```
LASSO-O will take a while to run depending upon the number of simulations
you are processing.
The first time you run the lasso-o_shcu container, the container runtime
will download the container image, which will take a few minutes. Then,
LASSO-O will run a series of processes for each simulation provided as
input.
When your job has completed, you may view the outputs created in the
`run-lasso-o_shcu/data/outputs` folder using the notebooks provided.
See the [notebooks/README.md](notebooks/README.md) file for more
information.
# LASSO-O Singularity Instructions
Follow these instructions if you will be running LASSO-O on an HPC cluster.
For the purposes of these instructions, we are using commands for the
slurm scheduler. Actual commands may be different at your institution
depending upon the scheduler used.
## Setting up Singularity environment on your HPC cluster
#### Prerequisite: Singularity 3+
Singularity 3+ has been tested and approved for use with the LASSO-O
container. It might run with an older version of Singularity, but this
has not been verified.
Usually Singularity is available via a module load. To see if
Singularity is available on your cluster, type the following:
```bash
$ module avail
```
#### Load singularity CLI on your HPC system
Use `module load` to load the Singularity command-line client. For
example, at PNNL:
```bash
$ module load singularity/3.7.1
```
Once in your compute node shell with the singularity module loaded,
you can run LASSO-O using the following instructions.
## Running LASSO-O via Singularity
Start LASSO-O via the `run.sh` script from the run-lasso-o_shcu folder:
```bash
$ srun ./run.sh
```
* Note that this example assumes you are using the Slurm scheduler. This
will invoke the Singularity container via the scheduler. If you do not
use the scheduler to invoke the script, it will run on the login node,
which may be killed per host policy if it runs for too long.
* LASSO-O will take a while to run depending upon the number of simulations
you are processing. The first time you run the lasso-o_shcu container, the container runtime
will download the container image, which will take a few minutes. Then,
LASSO-O will run a series of processes for each simulation provided as
input.
#### Checking the status of your job
To check the status of your job, list the queue for your user ID, as shown in the
following slurm example:
```bash
[d3k339@marianas run-lasso-o_shcu]$ squeue -u d3k339
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
27540 slurm run.sh d3k339 R 0:08 1 dc014
```
You can also get more information about the running job using the scontrol
command:
```bash
[d3k339@marianas run-lasso-o_shcu]$ scontrol show jobid -dd 27540
JobId=27540 JobName=run.sh
UserId=d3k339(20339) GroupId=users(100) MCS_label=N/A
Priority=290 Nice=0 Account=br21_d3k339 QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=0 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
DerivedExitCode=0:0
RunTime=00:00:39 TimeLimit=08:00:00 TimeMin=N/A
SubmitTime=2021-05-14T11:56:01 EligibleTime=2021-05-14T11:56:01
AccrueTime=Unknown
StartTime=2021-05-14T11:56:01 EndTime=2021-05-14T19:56:01 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2021-05-14T11:56:01
Partition=slurm AllocNode:Sid=marianas.pnl.gov:21213
ReqNodeList=(null) ExcNodeList=(null)
NodeList=dc014
BatchHost=dc014
NumNodes=1 NumCPUs=64 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=64,node=1,billing=64
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
JOB_GRES=(null)
Nodes=dc014 CPU_IDs=0-63 Mem=0 GRES=
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=./run.sh
WorkDir=/qfs/people/d3k339/lasso-test/run-lasso-o_shcu
Power=
NtasksPerTRES:0
```
When your job has completed, you may view the outputs created in the
`run-lasso-o_shcu/data/outputs` folder using the notebooks provided.
See the [notebooks/README.md](notebooks/README.md) file for more
information.
# Running LASSO-O Shallow Convection via Container
This project helps users to run the LASSO-O workflow software \<insert repository link>
in a local container environment - either Docker,Singularity, or Shifter depending upon the host policy.
LASSO stands for the Large-Eddy Simulation
(LES) Atmospheric Radiation Measurement (ARM) Symbiotic Simulation and Observation
(LASSO) activity. More information about LASSO can be found
[here](https://arm.gov/capabilities/modeling/lasso).
Questions about LASSO and this software can be directed to <lasso@arm.gov>.
## Getting Started
Follow these instructions to run LASSO-O at your local institution:
#### 1) Check out the LASSO-O Shallow Convection runtime helper
```bash
$ git clone https://gitlab.com/gov-doe-arm/docker/run-lasso-o_shcu.git
$ cd run-lasso-o_shcu
```
#### 2) Set up your container runtime environment
See the appropriate README file for setup instructions for your respective
environment:
* [README-SHIFTER.md](./README-SHIFTER.md). Use these instructions if you are running at NERSC.
* [README-DOCKER.md](./README-DOCKER.md). Use these instructions if you are running from your local desktop. **NOTE: this should be used for small test runs only**
* [README-SINGULARITY.md](./README-SINGULARITY.md). Use these instructions if you are running from another HPC cluster.
#### 3) Prepare Simulation Data
Copy your wrf simulation(s) output to the `data/inputs folder`. You may copy over 1 - 10 different
simulation outputs. Each simulation output should be placed in two separate folders, one for wrfout files and
one for wrfstat files. The wrfout and wrfstat folders for each simulation should be named accordingly:
```bash
$ ls -l data/inputs
# Simulation 1
sgpwrfout1C1.00/
sgpwrfstat1C1.00/
# Simulation 2
sgpwrfout3C1.00
sgpwrfstat3C1.00
# ....
# Simulation 10
sgpwrfout10C1.00
sgpwrfstat10C1.00
```
### 4) Edit config.yml file
Edit the config.yml file to provide parameters about your run. The config.yml file contains
detailed descriptions for each parameter. If you run following these instructions, you
should be able to use the default parameters with the exception of `begin_datetime`.
This must be set to correspond to the begin time for your wrf simulation, as shown below:
``` yaml
#-----------------------------------------------------------------
# Begin Datetime
#
# Enter the UTC start date and time for your simulation data
# in the format YYYYMMDD.HHMMSS
#
# The entered date MUST be one listed in the adjacent lasso_dates.txt
# file.
#-----------------------------------------------------------------
begin_datetime: 20180710.115900
```
### 5) Run LASSO-O
Start LASSO-O via the `run.sh` script from the run-lasso-0_shcu folder.
See the appropriate README file for setup instructions for your respective
environment:
* [README-SHIFTER.md](./README-SHIFTER.md). Use these instructions if you are running at NERSC.
* [README-DOCKER.md](./README-DOCKER.md). Use these instructions if you are running from your local desktop. **NOTE: this should be used for small test runs only**
* [README-SINGULARITY.md](./README-SINGULARITY.md). Use these instructions if you are running from another HPC cluster.
### 6) Plotting output and skill scores via Jupyter Notebook
See the [notebooks/README.md](notebooks/README.md) file for instructions on how to start
up a Jupter Notebook for visualizing the output of your LASSO-O run.
####################################################################
# Configuration file for running LASSO-O container.
####################################################################
#-----------------------------------------------------------------
# Input Folder
#
# Provide the parent folder where your sgpwrfout[1-10]C1.00 and
# sgpwrfstat[1-10]C1.00 folders are located. Your wrf netcdf files
# should be located directly under the sgpwrfout[1-10]C1.00 and
# sgpwrfstat[1-10]C1.00 folders.
#-----------------------------------------------------------------
input_folder: ./data/inputs
#-----------------------------------------------------------------
# Output Folder
#
# Provide a path to a folder where the output of the LASSO-O run
# and the log files will be stored.
#-----------------------------------------------------------------
output_folder: ./data/outputs
#-----------------------------------------------------------------
# Begin Datetime
#
# Enter the UTC start date and time for your simulation data
# in the format YYYYMMDD.HHMMSS
#
# The entered date MUST be one listed in the adjacent lasso_dates.txt
# file.
#-----------------------------------------------------------------
begin_datetime: 20180710.115900
#-----------------------------------------------------------------
# Container Runtime
#
# Identify whether to run via Docker or Singularity. Use 'docker'
# for Docker and 'singularity' for singularity
#
# (Note, Docker is intended for small-scale runs only.)
#-----------------------------------------------------------------
#container_runtime: docker
#container_runtime: shifter
container_runtime: singularity
# The following dates (YYYYMMDD) are available for processing
# via the LASSO-O Shallow Convection container in Docker or
# Singularity
20170403
20170405
20170509
20170524
20170527
20170605
20170609
20170614
20170626
20170627
20170629
20170630
20170704
20170705
20170709
20170712
20170716
20170717
20170719
20170720
20170721
20170725
20170728
20170802
20170826
20170828
20170830
20170920
20170922
20170923
20170924
20180514
20180522
20180523
20180529
20180530
20180531
20180606
20180618
20180619
20180704
20180705
20180707
20180709
20180710
20180711
20180712
20180731
20180805
20180809
20180811
20180901
20180902
20180909
20180911
20180914
20180916
20180917
20180918
20181001
20181002
20190404
20190506
20190512
20190517
20190607
20190612
20190617
20190626
20190701
20190704
20190707
20190709
20190714
20190804
20190805
20190901
20190929
\ No newline at end of file
# LASSO-O Notebooks for Plotting and Skill Scores
This folder contains a collection of Jupyter notebooks used for generating
plots and skill scores from the output of your LASSO-O workflow run. The plots
generated by the notebooks in this repository mimic those released within LASSO
data bundles and focus on comparing results from multiple simulations.
Questions about LASSO and this software can be directed to <lasso@arm.gov>.
These notebooks have been designed by Tami Fairless from Brookhaven National Laboratory.
## Installation
Use the following steps to install and start up a Jupyter kernel on
your server for interacting with your LASSO-O output.
#### Prerequesite: Anaconda/Miniconda 3
These steps depend upon the Anaconda package manager for Python. If you
are running on an HPC system, an Anaconda3 or Miniconda3 environment should
be available via `module load`. For example:
```bash
$ module load python/miniconda3.9
```
#### Creating your 'lasso' conda environment
You can create a conda environment named 'lasso' using the following
command:
```bash
$ cd notebooks
$ conda env create -f environment.yml
```
This should take a few minutes to install all the required libraries and
their dependencies which include:
* CDAT (https://github.com/CDAT/cdat/wiki/install) is required for plotting the Taylor diagram in plot_1d.ipynb
* xarray is used for reading in most data.
* netCDF4 is used for reading in data files with time-resolved height bins (e.g., datastream lassodiagobsmodz.m1)
Once the conda environment is created, users can proceed with the normal process of starting the notebook server
and working with the notebooks in this repository.
#### Starting your notebook server
Start up a headless juptyer notebook kernel with the following command:
```bash
$ conda activate lasso
$ jupyter-notebook --ip=0.0.0.0 --no-browser --port=56269
```
* Note that we are binding to the arbitrary port 56269. If this port
is in use on your server, feel free to change the port number as needed.
When you start your juptyer notebook kernel, the console should output
the http url from which you can access your notebook. For example:
```bash
(lasso) [d3k339@marianas notebooks]$ jupyter-notebook --ip=0.0.0.0 --no-browser --port=56269
[I 17:38:25.677 NotebookApp] Serving notebooks from local directory: /qfs/people/d3k339/run-lasso-o_shcu/notebooks
[I 17:38:25.677 NotebookApp] Jupyter Notebook 6.3.0 is running at:
[I 17:38:25.677 NotebookApp] http://marianas.pnl.gov:56269/?token=f1b17fb68b91f6c80fbffacfd65478b38329e2c7a4df22b2
[I 17:38:25.677 NotebookApp] or http://127.0.0.1:56269/?token=f1b17fb68b91f6c80fbffacfd65478b38329e2c7a4df22b2
[I 17:38:25.677 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 17:38:25.699 NotebookApp]
```
#### Opening notebooks in your browser
Open your browser and enter the url output from the jupyter-notebook
command on your server. You should see a list of the notebooks from the
notebooks folder:
![Jupyter Notebooks](https://code.arm.gov/lasso/containers/lasso-o_shcu/-/raw/master/images/notebook-folder.png)
#### Using the notebooks
You should replace the paths to the sample data used in these examples
with the corresponding paths to the output data from your
`run-lasso-o_shcu/data/outputs` folder. For example:
![Jupyter Notebooks](https://code.arm.gov/lasso/containers/lasso-o_shcu/-/raw/master/images/data-dir.png)
The available notebooks include:
* plot_1D.ipynb for plotting time series
* plot_cloud_fraction.ipynb for plotting time-height cloud fraction plots
* plot_profiles.ipynb for plotting sounding profiles
* plot_scores.ipynb for plotting scatter plots of the LASSO skill scores
# This is the environment that was produced after following
# the CDAT installation instructions at: https://github.com/CDAT/cdat/wiki/install
# Following that installation, the xarray and netCDF4 packages
# were added to the environment.
name: lasso
channels:
- cdat/label/v8.2.1
- conda-forge
- defaults
dependencies:
- libnetcdf[build=nompi_*]
- python=3.7
- pip=21.1.1
- mesalib=18.3.1
- cdat
- pip:
- cdms2==3.1.5
- cftime==1.4.1
- netcdf4==1.5.6
- regrid2==3.1.5
- xarray==0.18.0
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
# Taken from https://matplotlib.org/stable/gallery/images_contours_and_fields/image_annotated_heatmap.html
# Used in plot_1D.ipynb
def heatmap(data, row_labels, col_labels, ax=None,
cbar_kw={}, cbarlabel="", **kwargs):
"""
Create a heatmap from a numpy array and two lists of labels.
Parameters
----------
data
A 2D numpy array of shape (N, M).
row_labels
A list or array of length N with the labels for the rows.
col_labels
A list or array of length M with the labels for the columns.
ax
A `matplotlib.axes.Axes` instance to which the heatmap is plotted. If
not provided, use current axes or create a new one. Optional.
cbar_kw
A dictionary with arguments to `matplotlib.Figure.colorbar`. Optional.
cbarlabel
The label for the colorbar. Optional.
**kwargs
All other arguments are forwarded to `imshow`.
"""
if not ax:
ax = plt.gca()
# Plot the heatmap
im = ax.imshow(data, **kwargs)
# Create colorbar
cbar = ax.figure.colorbar(im, ax=ax, **cbar_kw)
cbar.ax.set_ylabel(cbarlabel, rotation=-90, va="bottom")
# We want to show all ticks...
ax.set_xticks(np.arange(data.shape[1]))
ax.set_yticks(np.arange(data.shape[0]))
# ... and label them with the respective list entries.
ax.set_xticklabels(col_labels)
ax.set_yticklabels(row_labels)
# Let the horizontal axes labeling appear on top.
ax.tick_params(top=True, bottom=False,
labeltop=True, labelbottom=False)
# Rotate the tick labels and set their alignment.
plt.setp(ax.get_xticklabels(), rotation=-30, ha="right",
rotation_mode="anchor")
# Turn spines off and create white grid.
#ax.spines[:].set_visible(False)
ax.set_xticks(np.arange(data.shape[1]+1)-.5, minor=True)
ax.set_yticks(np.arange(data.shape[0]+1)-.5, minor=True)
ax.grid(which="minor", color="w", linestyle='-', linewidth=3)
ax.tick_params(which="minor", bottom=False, left=False)
return im, cbar