...
once the AOI has been created, facet on the AOI in Tosca and run the following jobs:
acq scraper jobs
Action:
AOI based submission of acq scraper jobs [develop]Queue:
factotoum-job_worker-smallResult: this job will submit individual acq scraper jobs over the AOI
IPF scraper jobs
Action:
AOI based submission of IPF scraper jobs [develop]Queue:
factotoum-job_worker-smallResult: this job will submit individual IPF scraper jobs over the AOI
these update the acquisition-S1-IW_SLC dataset
Notes:
An “overloaded” term is
AOIandtrack. Here are the definitions:An
AOIis the area which we want to cover with GUNWs. The SLCs to be downloaded and paired will be according to the enumeration strategy determined by the enumerator submitter below.A
trackis the path in which the satellite follows and repeats during its orbits around Earth. Here is an image for Sentinel 6’s tracks. These track numbers are calledpathnumbers in ASF Search (see the Filters menu; the example in the linked search shows Track 48 in January)How are they related? We purposefully create AOIs that align with a particular track so that all the SLCs come from a given track. This is ensured in the enumeration because we have a minimum coverage threshold for SLCs and only those within the track will satisfy that threshold. Note that every SLC is collected on a given track so every SLC has a track number.
We can facet on tracks within Tosca using:
metadata.track_number:<track_number>ormetadata.trackNumber:<track_number>(depending on the ES dataset)
After creating AOIs, the spatial extent of the AOI dataset in tosca will be the single most important way to query the subsequent datasets related to an AOI. Here is the rough process to do so:
Select the recently created AOI in Tosca
Click
Query Regionwithin said datasetFacet on other dataset or query strings for further filtering.
...
once the scraper jobs have completed, facet on the AOI in Tosca and run the following job:
Action:
AOI Enumerator Submitter [develop]Queue:
factotoum-job_worker-smalltrack_number: the track number of the AOI you are processing (provided by scientist/customer)
enumeration_job_version:
developenumerator_queue:
aria-standard_product-enumeratornote the default queue is stale
min_match: number of nearest neighbors (provided by scientist/customer)
acquisition_version:
v2.0skipDays: number of days to skip before pairing (provided by scientist/customer)
Result: this job will iterate over the S1-AUX_POEORBs that cover the AOI and submit individual enumeration jobs (
aoi_track_acquisition_enumerator). The individual enumeration jobs produce products in the following datasets:S1-GUNW-acq-list: I think of these as a shopping cart that carry the IDs of the SLCs needed to produce an S1-GUNW
each of these correspond to a unique ifg-cfg and an S1-GUNW
S1-GUNW-acqlist-audit_trail: these are evaluation assessments of each viable pair of SLCs evaluated by the enumerator
they are later used by data accountability tools
run AOI based enumeration job with periods (Alternative enumeration strategy to above where we want SLCs to be within month range)
To Be Reviewed by Jack McNelis.
once the scraper jobs have completed, facet on the
starttimeofS1-AUX_POEORB. Here is a sample query to get January 1 to April 1:(starttime: {2014-01-01T00:00:00 TO 2014-04-01T00:00:00}) OR (starttime: {2015-01-01T00:00:00 TO 2015-04-01T00:00:00}) OR … OR (starttime: {2020-01-01T00:00:00 TO 2020-04-01T00:00:00}). You have to fill in those…! Here is an example.Action:
Standard Product S1-GUNW - aoi_track_acquisition_enumerator [develop]Queue:
TBDtrack_number: the track number of the AOI you are processing (provided by scientist/customer)
AOI Name: determined via customer
enumeration_job_version:
developenumerator_queue:
aria-standard_product-enumeratornote the default queue is stale
min_match: number of nearest neighbors (provided by scientist/customer)
acquisition_version:
v2.0skipDays: number of days to skip before pairing (provided by scientist/customer)
Result: this job will iterate over the S1-AUX_POEORBs that we faceted over AND cover the AOI. The individual enumeration jobs produce products in the following datasets:
S1-GUNW-acq-list: I think of these as a shopping cart that carry the IDs of the SLCs needed to produce an S1-GUNW
each of these correspond to a unique ifg-cfg and an S1-GUNW
S1-GUNW-acqlist-audit_trail: these are evaluation assessments of each viable pair of SLCs evaluated by the enumerator
they are later used by data accountability tools
download SLCs
once all acq-lists have been generated, facet on such acq-lists in Tosca
you can query by the AOI id and then facet on the S1-GUNW-acq-list dataset
then submit the localizer jobs:
Action:
Standard Product S1-GUNW slc_localizer [develop]Queue:
aria-standard_product-localizer(Perhaps create a dedicated queue on factotum like we did for the evaluator.)asf_ngap_download_queue:
factotum-job_worker-slc_sling-asfNote the other queue
slc-sling-extract-asfis an ASG
esa_download_queue:
factotum-job_worker-slc_sling-scihubNote the other queue
slc-sling-extract-scihubis an ASG
spyddder_sling_extract_version:
developResult: this job will iterate over the SLCs listed in the acq-list and submit a data sling job
these sling jobs take acquisition-S1-IW_SLCs as an input and will download the corresponding SLC from ASF (relatively old acquisition) or Scihub (acquisition is less than 2 weeks old) to s3 and register the SLC in the S1-IW_SLC dataset in GRQ
Notes:
Acquisition lists are in one-to-one correspondence with ifg-cfgs
SLCs can be shared among acquisition lists and ifg-cfg's within an AOI. Therefore, #SLCs < # acq-lists = #ifg-cfgs within your AOI. As an example, within an AOI, there were ~700 SLCs for 2300 ifg-cfgs.
Say you run the localizer and you see that you have a bunch ifg-cfgs haven’t been created even though most of the sling jobs have been completed successfully. You may have only a few SLCs to download (or much less than the ifg-cfgs that are missing). Check the unique SLCs in the ops report.
If you have the proper trigger rules set up and activated, every time a new SLC is slinged and put into the system, then an ifg-cfg is created. This is a helpful trigger rule to have. Currently it is called
acqlist_evaluator.
...
at this point, we have completed the necessary processing to now run topsapp and generate an ifg
facet on the ifg-cfgs and run the following job
Action:
TopsApp PGE in *Standard-Product* Pipeline for S1-GUNW Interferograms [develop]Queue: topsapp jobs take a while and run on expensive machines – therefore, this PGE significantly drives up costs for the pipeline! We have designated queues to tag the jobs with different accounts so customers can pay for these charges.
Current Recommended Queues (last updated 3/2021):
aria-standard_product-s1gunw-topsapp-NSLCT_Bekaertaria-standard_product-s1gunw-topsapp-Access_Bekaertaria-standard_product-s1gunw-topsapp-Volcano_Lundgrenaria-standard_product-s1gunw-topsapp-Rise_LimonadiNote the last token in the above queues indicate the project name but more tags can be seen in the Autoscaling group setup in AWS.
dataset_tag: this is a comma-delimited list of tags that will be added to the produced S1-GUNW
metadata.dataset_tagsfield and can be used to facet on the product in the futurefor the standard product pipeline on AWS,
standard_product,awsshould always be included in this parameter
Result: a S1-GUNW product will be produced
Notes on Trigger Rules:
General trigger rules with topsApp must be created with care because making a trigger rule that is too lenient can really run up costs. Here are some general rules. For topsApp trigger rule use the following facets:
Spatial extent of the AOI
The track number associated with the AOI
TODO: temporal spans associate with the enumerator
Due to the creation of the coseismic pipeline, there are some shared datasets. It is important to use
NOT "Coseismic"in the query box to ensure coseismic datasets are ignored. More specific pipelines must ignore themachine tagcalleds1-gunw-coseismic.
Notes on Errors:
There are some error types that are worth mentioning as they can arise even if the pipeline has been run correctly. Make sure the errors match exactly to those examples found below as ISCE errors are very, very hard to catch and a slight difference in the error output can mean be the result of totally different sources (note both error examples below mention “burst”):
Burst overlap errors like this job - the SLCs (on two different dates) do not have an overlap. This occurs when the metadata used to enumerate the job and create the IFG-CFG was slightly off from what is on the ground and/or the overlap is just not sufficient for ISCE2 to do it’s processing. This means that the IFG-CFG is malformed and should be ignored.
DEM download errors like this job - this is likely a transient error and will go away on a re-run. Simply, the DEM was not downloaded successfully from our S3 bucket during processing. If problems persist, please reach out to Nicholas Arenas.
Clobber errors like this job - although there are “short circuits” within the topsApp PGE exist, the PGE checks the completed GUNW database. Therefore, if two identical topsApp jobs were called on the same ifg-cfg before either could complete, then we will get these clobber errors. Note the clobber errors will generally not all be identical because it depends what file is uploaded first. However, an easy way to determine if such an error was due to duplication in the operator faceting, facet on a single input ifg-cfg and check the related topsApp jobs. Here is an example of such faceting in figaro.
If the errors are beyond the scope of those listed above, the relevant logs will be saved on Tosca using triaging HySDS functionality which is currently running for the topsApp PGE; here is an example of triaged job datasets. Facet on one of the failing ifg-cfg’s and send to current topsApp maintainer (as of March 2021, this is charlie.z.marshak@jpl.nasa.gov).
...