...

...

...

...

...

...

...

...

...

Table of Contents

Related Github Repos and

...

Tickets

• Github Repo: https://github.com/aria-jpl/topsstack

  • Versions

    • GNU Parallel

    • Multiprocessing

• Tickets:

  • https://jira.jpl.nasa.gov/browse/ARIA-42

  • https://jira.jpl.nasa.gov/browse/ARIA-8

  • Jira Legacy
    server System JIRA serverId 88de5227-42b1-365b-8364-d731c8efaf35 key ARIA-47

    https://jira.jpl.nasa.gov/browse/ARIA-8

Job Runtime

• Depends on how many SLCs are being procssed

• 1+ hours for 8 SLCs

Objective

Creating a stack of SLCs

...

Prerequisite to STAMPS processing

How to

...

Set Up the

...

Inputs

• Facets to get SLC inputs

  • region (ex. Hawaii) (optional)

  • track_number or trackNumber (depends)

  • datatype: SLC

...

• Code Block
  raise Exception('Could not determine a suitable burst offset')

• There must only be one track in your SLC inputs

correct facet SLC inputs incorrect facet SLC inputs

...

Job Inputs

...

• Bbox ([*required in Multiprocessing and optional in GNU Parallel (MBR)]

  • min_lat max_lat min_lon max_lon

...

• ops-bcluster_container-builder_aria-jpl_topsstack_master

• Link: http://b-ci.grfn.hysds.io:8080/job/ops-bcluster_container-builder_aria-jpl_topsstack_master/

  • WARNING: If rebuilding on the same branch (master), make sure to remove docker image so that it reloads when restarting the job: docker rmi <topsStack docker image id>

HySDS-

...

IO and Jobspec-

...

IO

hysds-io.json.topsstack

{
  "label": "topsStack Processor",
  "submission_type": "individual",
  "allowed_accounts": [ "ops" ],
  "action-type":  "both",
  "params": [
    {
      "name": "min_lat",
      "from": "submitter",
      "type": "number",
      "optional": false
    },
    {
      "name": "max_lat",
      "from": "submitter",
      "type": "number",
      "optional": false
    },
    {
      "name": "min_lon",
      "from": "submitter",
      "type": "number",
      "optional": false
    },
    {
      "name": "max_lon",
      "from": "submitter",
      "type": "number",
      "optional": false
    },
    {
      "name":"localize_products",
      "from":"dataset_jpath:",
      "type":"text",
      "lambda" : "lambda met: get_partial_products(met['_id'],get_best_url(met['_source']['urls']),[met['_id']+'.zip'])"
    }
  ]
}

...

merged/
    baselines/
        20190506/
        20190518/
        20190530/
            20190530
            20190530.full.vrt
            20190530.vrt
            20190530.xml
    geom_master/
	    *.rdr.aux.xml
        *.rdr.full
        *.rdr.full.aux.xml
        *.rdr.full.vrt
        *.rdr.full.xml
    SLC/
        20190506/
        20190518/
        20190530/
            20190530.slc.full
            20190530.slc.full.aux.xml
            20190530.slc.full.vrt
            20190530.slc.full.xml
            20190530.slc.hdr

Datasets.json

...

Entry

• file located in .sds/files/datasets.json

{
  "ipath": "ariamh::data/STACK",
  "match_pattern": "/(?P<id>coregistered_slcs-(?P<year>\\d{4})(?P<month>\\d{2})(?P<day>\\d{2})(?P<time>\\d{6}).+)$",
  "alt_match_pattern": null,
  "extractor": null,
  "level": "NA",
  "type": "stack",
  "publish": {
    "s3-profile-name": "default",
    "location": "s3://s3-us-west-2.amazonaws.com:80/##BUCKET##/datasets/{type}/{version}/{year}/{month}/{day}/{id}",
    "urls": [
      "http://##WEBDAV_URL##/datasets/{type}/{version}/{year}/{month}/{day}/{id}",
      "s3://##S3_URL##:80/##BUCKET##/datasets/{type}/{version}/{year}/{month}/{day}/{id}"
    ]
  },
  "browse": {
    "location": "davs://##WEBDAV_USER##@##WEBDAV USER##/browse/{type}/{version}/{year}/{month}/{day}/{id}",
    "urls": [
      "https://##WEBDAV##/browse/{type}/{version}/{year}/{month}/{day}/{id}"
    ],
  }
}

STILL TODO:

...

Publish the contents from merged/ into a datasets directory (testing)

...

Create dataset file name template (testing)

...

ASG (Auto Scaling Group) Configurations

...

Assessment of Steps

https://docs.google.com/spreadsheets/d/1W2KzAWm8VjceB77jfc9kuQUrXtYDiwSPmqxcHuEoQIg/edit#gid=0

Job Runtime

• Depends on how many SLCs are being processed and number of bursts covered by Bbox.

GNU Parallel

Multiprocessing