Page Comparison

...

• at this point, we have completed the necessary processing to now run topsapp and generate an ifg

• facet on the ifg-cfgs and run the following job

  • Action: TopsApp PGE in *Standard-Product* Pipeline for S1-GUNW Interferograms [develop]

  • Queue: topsapp jobs take a while and run on expensive machines – therefore, this PGE significantly drives up costs for the pipeline! We have designated queues to tag the jobs with different accounts so customers can pay for these charges.

    • Current Recommended Queues (last updated 3/2021):

      • aria-standard_product-s1gunw-topsapp-NSLCT_Bekaert

      • aria-standard_product-s1gunw-topsapp-Access_Bekaert

      • aria-standard_product-s1gunw-topsapp-Volcano_Lundgren

      • aria-standard_product-s1gunw-topsapp-Rise_Limonadi

      • Note the last token in the above queues indicate the project name but more tags can be seen in the Autoscaling group setup in AWS.

  • dataset_tag: this is a comma-delimited list of tags that will be added to the produced S1-GUNW metadata.dataset_tags field and can be used to facet on the product in the future

    • for the standard product pipeline on AWS, standard_product,aws should always be included in this parameter

  • Result: a S1-GUNW product will be produced

  • Notes on Trigger Rules:

    • General trigger rules with topsApp must be created with care because making a trigger rule that is too lenient can really run up costs. Here are some general rules. For topsApp trigger rule use the following facets:

      • Spatial extent of the AOI

      • The track number associated with the AOI

      • TODO: temporal spans associate with the enumerator

    • Due to the creation of the coseismic pipeline, there are some shared datasets. It is important to use NOT "Coseismic" in the query box to ensure coseismic datasets are ignored. More specific pipelines must ignore the machine tag called s1-gunw-coseismic.

  • Greylisting

    • This likely will require its own page at some point. Greylisting is important for the following reason: when we deliver products over an AOI, we only do so if all possible GUNWs for a date pair are completed. That means if a GUNW can be completed as completed as outlined by a given acq-list/ifg-cfg, then we cannot deliver all the other GUNWs that have been generated. There are two types of errors that we want to group together that will not be able to finish with topsApp when they crop up for an ifg-cfg that should not hamper our delivery of products:
      a. Exception: Could not determine a suitable burst offset
      b. No swaths contain any burst overlaps ... cannot continue for interferometry applications
      These occur when we have a GUNW that is trying to be produced over water or over an area with extremely low coherence e.g. here.

      • Caution!

        • The two relevant trigger rules are:  standard-product-greylist-failed-gunw-burst-overlap and standard-product-greylist-failed-gunw-swaths-overlap (these correspond to the errors above)

        • The trigger rules for grey listing are in figaro (since they facet on jobs)

        • The trigger rules (those cited above) require specification of a container - if a topsapp job is being modified be sure to update these trigger rules reference the correct container - we need to check this.

        Facet: On the jobs in

        • to ensure the trigger rules are correctly invoked - Charlie M. accidentally changed the name of the job-spec.<pge_container_name> exchanging a - with a _ and trigger rules were no longer valid.

      • Facet: On the jobs in figaro using the error codes above.

      • Action: Standard Product S1-GUNW - Greylist S1-GUNW from topsapp job [python3]

      • Result: You will get a greylist id associated with the ifg-cfg which can be identified using the hash-id.

  • Notes on Errors:

    • There are some error types that are worth mentioning as they can arise even if the pipeline has been run correctly. Make sure the errors match exactly to those examples found below as ISCE errors are very, very hard to catch and a slight difference in the error output can mean be the result of totally different sources (note both error examples below mention “burst”):

      • Burst overlap errors like this job - the SLCs (on two different dates) do not have an overlap. This occurs when the metadata used to enumerate the job and create the IFG-CFG was slightly off from what is on the ground and/or the overlap is just not sufficient for ISCE2 to do it’s processing. This means that the IFG-CFG is malformed and should be ignored.

      • DEM download errors like this job - this is likely a transient error and will go away on a re-run. Simply, the DEM was not downloaded successfully from our S3 bucket during processing. If problems persist, please reach out to Nicholas Arenas.

      • Clobber errors like this job - although there are “short circuits” within the topsApp PGE exist, the PGE checks the completed GUNW database. Therefore, if two identical topsApp jobs were called on the same ifg-cfg before either could complete, then we will get these clobber errors. Note the clobber errors will generally not all be identical because it depends what file is uploaded first. However, an easy way to determine if such an error was due to duplication in the operator faceting, facet on a single input ifg-cfg and check the related topsApp jobs. Here is an example of such faceting in figaro.

    • If the errors are beyond the scope of those listed above, the relevant logs will be saved on Tosca using triaging HySDS functionality which is currently running for the topsApp PGE; here is an example of triaged job datasets. Facet on one of the failing ifg-cfg’s and send to current topsApp maintainer (as of March 2021, this is charlie.z.marshak@jpl.nasa.gov).

  • Trigger Rule:

    • Generally, you want to set a trigger rule related to topsApp prior to running the enumerator.

    • Trigger rules that are so narrowly faceted can be hard to create if no existing dataset exists. Frequently, we have some existing trigger rule and simply edit that in the menu. For reference, here is a template (what would be in the final query window of the trigger rule):

...

• Purge localized SLCs as done here.

• Delete trigger rules associated with TopsApp - clutters trigger rules - or turn them off!

• Check stray instances.

  • In certain cases, RabbitMQ does not accurately capture all the instances that are running from a given queue in ASG. You can check the queues used for the standard product pipeline in EC2 > Auto Scaling Groups (sidebar) and check the queues there. Then, set the “desired capacity” to 0 in the topmost menu when you click on a given queue. Alternatively, to view all the running instances, AWS console > EC2 > Instances (sidebar) and check the instances that are “running”.

...

Notes

Faceting in Tosca and Figaro

...