Install docker, increase memory (under preferences -> advanced) from default 2GB to at least 16 GB (I used 24 GB).
Download the latest topsApp docker container, and unzip it.
Create a .ssh folder in your home directory, then ask a team member to securely share with you their .pem SSH keys and a config file. Then put them in that folder.
Chmod those .pem files (using "chmod og-rwx *.pem") so their permissions look like this:
-rw-------@ 1 killett staff 1.6K Oct 31 15:35 ariamh.pem -rw-------@ 1 killett staff 1.7K Oct 31 15:35 grfn-v2-ops.pem
Unzip the tarball container, then run docker load:
docker load -i <tar_file_name>
Then check the images:
docker images
Find the image ID in that list and copy it to your clipboard.
Now use docker run to create the container. The simplest docker run command without any mapping is:
docker run -ti –rm <docker_image_id> bash
Replace <docker_image_id> with the image ID you copied in the previous step. This will take you inside the docker container. The ariamh directory should be in the home directory. Feel free to go inside it and explore.
You’ll need to bind a data/work/ directory to the docker container, which will be accomplished in the full docker run command much further down this page. I put data/work/ into my home directory, because if I put it in the root directory then docker wouldn’t bind it to the container.
I used input and output files from this job. I also created a _job.json file with these contents in the ~/data/work/ directory:
{
"retry_count": 1
}
Download the ariamh repository from github; I put it in /Users/YOURPATH/ARIA/repos/ariamh. It doesn’t matter where you put the ariamh directory on your machine as long as you change that part of the full docker run command below to reflect that change. The full docker run command much further down this page will bind that directory to ~/ariamh/ inside the docker container when you run it. This will replace the ~/ariamh directory in the docker container with the ariamh directory you downloaded. As a result, you’ll only have the make the following changes once rather than having to repeat them every time you run the docker container.
Create a file called settings.conf filled with the following lines, but ask a team member for the Mozart IP address and port. Also, you’ll have to ask a team member for the appropriate USERNAMEs and PASSWORDs - we can’t email them to you. I put my file in /Users/YOURPATH/ARIA/topsapp/settings.conf. It doesn’t matter where you put your settings.conf file as long as you change that part of the full docker run command below to reflect that change.
GRQ_URL=https://c-datasets.aria.hysds.io/es ARIA_DAV_URL=https://aria-dav.jpl.nasa.gov ARIA_DAV_U=USERNAME ARIA_DAV_P=PASSWORD ARIA_DEM_URL=http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/dem/SRTM1_v3/ ARIA_NED1_DEM_URL=http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/dem/ned1/ ARIA_NED13_DEM_URL=http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/dem/ned13/ ARIA_WBD_URL=http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/dem/usgs_mirror/SRTMSWBD.003/2000.02.11/ ARIA_DEM_U=USERNAME ARIA_DEM_P=PASSWORD ARIA_WBD_U=USERNAME ARIA_WBD_P=PASSWORD ARIA_PRD_URL=http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/ ARIA_DB_VERSION=v1.1 GRQ_INDEX_PREFIX=grq MOZART_URL=amqp://USERNAME:PASSWORD@IP_ADDRESS:PORT//
If you’re offsite, you’ll need to use Pulse Secure to set up a JPL full tunnel. Even still, the docker container will time out while requesting some files. So you’ll have to make some changes, starting in ~/ariamh/interferogram/sentinel/create_standard_product_s1.py . In the function get_dataset_by_hash(), change this line:
r = requests.post(search_url, data=json.dumps(query))
to this:
r = requests.post(search_url, data=json.dumps(query), verify=False)
In the function check_ifg_status_by_hash(), comment the second line shown here:
logger.info("Duplicate dataset found: %s" %found_id)
# sys.exit(0)
Here’s my full docker run command. You’ll have to replace “YOURPATH” with your path:
docker run -ti -v /Users/YOURPATH/.ssh:/home/ops/.ssh -v /Users/YOURPATH/ARIA/data/work:/home/ops/data/work -v /Users/YOURPATH/ARIA/data/work/jobs:/home/ops/data/work/jobs -v /Users/YOURPATH/ARIA/data/work/tasks:/home/ops/data/work/tasks -v /Users/YOURPATH/ARIA/data/work/workers:/home/ops/data/work/workers -v /Users/YOURPATH/ARIA/data/work/cache:/home/ops/data/work/cache:ro -v /Users/YOURPATH/ARIA/.netrc:/home/ops/.netrc:ro -v /Users/YOURPATH/ARIA/.aws:/home/ops/.aws:ro -v /Users/YOURPATH/ARIA/topsapp/settings.conf:/home/ops/ariamh/conf/settings.conf:ro -v /Users/YOURPATH/ARIA/repos/ariamh:/home/ops/ariamh -w /home/ops/data/work <docker_image_id>
Once all this works, make sure your ~/data/work directory is filled with the SLC zip files, orbit files, and _job.json and _context.json. For instance, my work directory has these contents:
S1-GUNW-A-R-064-tops-20191008_20190908-015107-37379N_35506N-PP-4488-v2_0_2.context.json S1A_IW_SLC__1SDV_20191008T015053_20191008T015120_029361_03567D_82B5.zip S1A_OPER_AUX_POEORB_OPOD_20191028T120831_V20191007T225942_20191009T005942.EOF S1B_IW_SLC__1SDV_20190908T015003_20190908T015030_017940_021C50_9EA4.zip S1B_IW_SLC__1SDV_20190908T015027_20190908T015054_017940_021C50_E9B1.zip S1B_OPER_AUX_POEORB_OPOD_20190928T110647_V20190907T225942_20190909T005942.EOF _context.json _job.json
Then run: ~/ariamh/interferogram/sentinel/create_standard_product_s1.sh
Then wait. For a while. Grab a coffee or three.
OPTIONAL BUT USEFUL:
While that is running, get details by tailing the log in a new tab in the same docker terminal:
open a new terminal tab.
docker ps
[Look for container ID, copy it]
docker exec -it <container ID> bash
tail -f create_standard_product_s1.log
Add Comment