Versions Compared

Old Version 33

changes.mady.by.user Charles Marshak

Saved on

compared with

New Version 34

changes.mady.by.user Charles Marshak

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The standard product pipeline performs radar interferometry and related processing to measure centimeter scale deformations on the ground using pairs of single-look complex images (SLCs). The final products S1-GUNWs are used by scientists for this analysis. For those operators curious about what radar interferometry is and its relationship to ground deformation, here is a brief introductory video from a leading expert in radar as well as a longer, more technical one from an expert at JPL.

This currently is a document that collects all the learnings related to the standard product pipeline on AWS. There are generally two ways to run this pipeline:

  1. Running all the steps as on-demand jobs

  2. Setting up trigger rules based on datasets downstream of the enumerator

The two methods are elaborated simultaneously.

...

Walkthrough

Create an AOI

...

  • once the scraper jobs have completed, facet on the AOI in Tosca and run the following job:

    • Action: AOI Enumerator Submitter [develop]

    • Queue: factotoum-job_worker-small

    • track_number: the track number of the AOI you are processing (provided by scientist/customer)

    • enumeration_job_version: develop

    • enumerator_queue: aria-standard_product-enumerator

      • note the default queue is stale

    • min_match: number of nearest neighbors (provided by scientist/customer)

    • acquisition_version: v2.0

    • skipDays: number of days to skip before pairing (provided by scientist/customer)

    • Result: this job will iterate over the S1-AUX_POEORBs that cover the AOI and submit individual enumeration jobs (aoi_track_acquisition_enumerator). The individual enumeration jobs produce products in the following datasets:

      • S1-GUNW-acq-list: I think of these as a shopping cart that carry the IDs of the SLCs needed to produce an S1-GUNW

        • each of these correspond to a unique ifg-cfg and an S1-GUNW

      • S1-GUNW-acqlist-audit_trail: these are evaluation assessments of each viable pair of SLCs evaluated by the enumerator

        • they are later used by data accountability tools are later used by data accountability tools

  • Note: If you plan to use trigger rules to improve operation efficiency, prior to running enumration, ensure that you have:

    1. a trigger rule for acq-list-evaluator (faceted on SLC) - typically this is turned on

    2. a trigger rule for slc-localizer (faceted on acq-lists) - typically this is turned on

    3. a trigger rule for topsApp (faceted on ifg-cfg, track number, and AOI extent) - we generally have to create this one because we increase the number of facets to ensure topsApp is not accidentally run due to it’s high cost.

Run AOI based enumeration job with periods (an alternative enumeration scheme)

(Alternative enumeration strategy to above where we want SLCs to be within month range)

...

  • at this point, we have completed the necessary processing to now run topsapp and generate an ifg

  • facet on the ifg-cfgs and run the following job

    • Action: TopsApp PGE in *Standard-Product* Pipeline for S1-GUNW Interferograms [develop]

    • Queue: topsapp jobs take a while and run on expensive machines – therefore, this PGE significantly drives up costs for the pipeline! We have designated queues to tag the jobs with different accounts so customers can pay for these charges.

      • Current Recommended Queues (last updated 3/2021):

        • aria-standard_product-s1gunw-topsapp-NSLCT_Bekaert

        • aria-standard_product-s1gunw-topsapp-Access_Bekaert

        • aria-standard_product-s1gunw-topsapp-Volcano_Lundgren

        • aria-standard_product-s1gunw-topsapp-Rise_Limonadi

        • Note the last token in the above queues indicate the project name but more tags can be seen in the Autoscaling group setup in AWS.

    • dataset_tag: this is a comma-delimited list of tags that will be added to the produced S1-GUNW metadata.dataset_tags field and can be used to facet on the product in the future

      • for the standard product pipeline on AWS, standard_product,aws should always be included in this parameter

    • Result: a S1-GUNW product will be produced

    • Notes on Trigger Rules:

      • General trigger rules with topsApp must be created with care because making a trigger rule that is too lenient can really run up costs. Here are some general rules. For topsApp trigger rule use the following facets:

        • Spatial extent of the AOI

        • The track number associated with the AOI

        • TODO: temporal spans associate with the enumerator

      • Due to the creation of the coseismic pipeline, there are some shared datasets. It is important to use NOT "Coseismic" in the query box to ensure coseismic datasets are ignored. More specific pipelines must ignore the machine tag called s1-gunw-coseismic.

    • Notes on Errors:

      • There are some error types that are worth mentioning as they can arise even if the pipeline has been run correctly. Make sure the errors match exactly to those examples found below as ISCE errors are very, very hard to catch and a slight difference in the error output can mean be the result of totally different sources (note both error examples below mention “burst”):

        • Burst overlap errors like this job - the SLCs (on two different dates) do not have an overlap. This occurs when the metadata used to enumerate the job and create the IFG-CFG was slightly off from what is on the ground and/or the overlap is just not sufficient for ISCE2 to do it’s processing. This means that the IFG-CFG is malformed and should be ignored.

        • DEM download errors like this job - this is likely a transient error and will go away on a re-run. Simply, the DEM was not downloaded successfully from our S3 bucket during processing. If problems persist, please reach out to Nicholas Arenas.

        • Clobber errors like this job - although there are “short circuits” within the topsApp PGE exist, the PGE checks the completed GUNW database. Therefore, if two identical topsApp jobs were called on the same ifg-cfg before either could complete, then we will get these clobber errors. Note the clobber errors will generally not all be identical because it depends what file is uploaded first. However, an easy way to determine if such an error was due to duplication in the operator faceting, facet on a single input ifg-cfg and check the related topsApp jobs. Here is an example of such faceting in figaro.

      • If the errors are beyond the scope of those listed above, the relevant logs will be saved on Tosca using triaging HySDS functionality which is currently running for the topsApp PGE; here is an example of triaged job datasets. Facet on one of the failing ifg-cfg’s and send to current topsApp maintainer (as of March 2021, this is charlie.z.marshak@jpl.nasa.gov).

    • Trigger Rule:

      • Generally, you want to set a trigger rule related to topsApp prior to running the enumerator.

Generate AOI-Tracks product

  • AOI tracks are often too large to be covered by 1 S1-GUNW for a given date-pair

  • once all the ifgs for a specific date-pair are generated an S1-GUNW-AOI_TRACK product is produced

  • to generate these products, facet on the S1-GUNW products and submit the following job:

    • Action: Standard Product S1-GUNW - S1-GUNW Completeness Evaluator [develop]

    • Queue: factotum-job_worker-large

    • Result: this job will look at an S1-GUNW and check if it “completes” the track for a given date-pair. If so S1-GUNW-AOI_TRACK product is generated. Otherwise, the evaluator silently completes without producing anything.

  • note that these jobs are automatically submitted by the trigger rule s1gunw-aws-s1gunw-completeness-evaluator and you should not normally need to submit them on-demand

  • once an S1-GUNW-AOI_TRACK product is produced, the S1-GUNW products will be published to ASF and ARIA-products via the following pipeline: TODO: delivery pipeline

  • Checking the Delivery to ASF

    • In addition to using the various ops reports, you can go directly to ASF: https://search.asf.alaska.edu/ and us their search by “list” feature. Copying the GUNW ids into this feature can illustrate the delivery publically! This is generally a good method of “delivering” the final AOI to science customers. Here is an example.

  • Notes on Errors:

    • This is not an error, but a confusing behavior. If any GUNW from an entire date pair is abset, then none of the GUNWs will deliver. So even if you facet on missing GUNWs, this process will complete without error, but not deliver your desired ASF.

Clean up

  • Purge localized SLCs as done here.

  • Delete trigger rules associated with TopsApp - clutters trigger rules.

...

Notes

Faceting in Tosca and Figaro

...