{"type":"doc","content":[{"type":"paragraph","content":[{"text":"This is a step by step procedure for the bulk processing of a new area of interest (AOI) to obtain all the interferograms (represented by S1-GUNW datasets). ","type":"text"}]},{"type":"paragraph","content":[{"text":"An ISCE super-user could obtain all GUNWs over a single AOI within 1 week on a JPL server. It is the job of the operator to consistently beat this time. ","type":"text"}]},{"type":"paragraph","content":[{"text":"The standard product pipeline performs radar interferometry and related processing to measure centimeter scale deformations on the ground using ","type":"text"},{"text":"pairs","type":"text","marks":[{"type":"em"}]},{"text":" of single-look complex images (SLCs). The final products S1-GUNWs are used by scientists for this analysis. For those operators curious about what radar interferometry is and its relationship to ground deformation, here is a brief introductory ","type":"text"},{"text":"video","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.youtube.com/watch?v=VSo-P1JHp_Y&ab_channel=EOCollege"}}]},{"text":" from a leading expert in radar as well as a longer, more technical ","type":"text"},{"text":"one","type":"text","marks":[{"type":"link","attrs":{"href":"https://jpltube.jpl.nasa.gov/watch?v=XN97lA"}}]},{"text":" from an expert at JPL. ","type":"text"}]},{"type":"paragraph","content":[{"text":"This currently is a document that collects all the learnings related to the standard product pipeline on AWS. There are generally two ways to run this pipeline:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Running all the steps as on-demand jobs","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setting up trigger rules based on datasets downstream of the enumerator","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The two methods are elaborated simultaneously.","type":"text"}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Walkthrough","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Create an AOI","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"walk through this ","type":"text"},{"text":"procedure","type":"text","marks":[{"type":"link","attrs":{"__confluenceMetadata":{"isRenamedTitle":true,"linkType":"page","contentTitle":"Create AOI","versionAtSave":"2"},"href":"/wiki/spaces/ARIA/pages/376307713"}}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Run AOI based scraper jobs","type":"text"}]},{"type":"paragraph","content":[{"text":"Running these jobs ensures that GRQ has the most recent metadata about the acqs that cover the AOI","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"once the AOI has been created, facet on the AOI in Tosca and run the following jobs:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"acq scraper jobs","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"AOI based submission of acq scraper jobs [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotoum-job_worker-small","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": this job will submit individual acq scraper jobs over the AOI","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"IPF scraper jobs","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"AOI based submission of IPF scraper jobs [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotoum-job_worker-small","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": this job will submit individual IPF scraper jobs over the AOI","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"these update the ","type":"text"},{"text":"acquisition-S1-IW_SLC","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJhY3F1aXNpdGlvbi1TMS1JV19TTEMifX1dfX0sInNvcnQiOlt7Il90aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIl90aW1lc3RhbXAiLCJfc291cmNlIl19"}}]},{"text":" dataset ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Notes:","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"An “overloaded” term is ","type":"text"},{"text":"AOI","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"track","type":"text","marks":[{"type":"code"}]},{"text":". Here are the definitions:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"An ","type":"text"},{"text":"AOI","type":"text","marks":[{"type":"code"}]},{"text":" is the area which we want to cover with GUNWs. The SLCs to be downloaded and paired will be according to the enumeration strategy determined by the enumerator submitter below.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"track","type":"text","marks":[{"type":"code"}]},{"text":" is the path in which the satellite follows and repeats during its orbits around Earth. Here is an ","type":"text"},{"text":"image","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.esa.int/ESA_Multimedia/Images/2020/11/Copernicus_Sentinel-6_orbital_tracks"}}]},{"text":" for Sentinel 6’s tracks. These track numbers are called ","type":"text"},{"text":"path","type":"text","marks":[{"type":"code"}]},{"text":" numbers in ","type":"text"},{"text":"ASF Search","type":"text","marks":[{"type":"link","attrs":{"href":"https://search.asf.alaska.edu/#/?path=48-48&start=2021-01-01T00:00:00Z&end=2021-02-01T23:59:00Z&resultsLoaded=true&granule=S1A_WV_SLC__1SSV_20210129T004249_20210129T010337_036345_0443C8_354F-SLC&maxResults=5000&zoom=3¢er=-33.412081,-27.743164"}}]},{"text":" (see the Filters menu; the example in the linked search shows Track 48 in January)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"How are they related? We purposefully create AOIs that align with a particular track so that all the SLCs come from a given track. This is ensured in the enumeration because we have a minimum coverage threshold for SLCs and only those within the track will satisfy that threshold. Note that every SLC is collected on a given track so every SLC has a track number.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"We can facet on tracks within Tosca using:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"metadata.track_number:","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"metadata.trackNumber:","type":"text","marks":[{"type":"code"}]},{"text":" (depending on the ES dataset)","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"After creating AOIs, the spatial extent of the AOI dataset in tosca will be the ","type":"text"},{"text":"single most important way to query the subsequent downstream datasets","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":" related to an AOI. Here is the rough process to do so:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Select the recently created AOI in Tosca","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click ","type":"text"},{"text":"Query Region","type":"text","marks":[{"type":"code"}]},{"text":" within said dataset","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Facet on other dataset or query strings for further filtering.","type":"text"}]}]}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Clean up Upstream Datasets (optional)","type":"text"}]},{"type":"paragraph","content":[{"text":"Context: ","type":"text","marks":[{"type":"em"}]},{"text":"Important datasets for monitoring are:","type":"text"}]},{"type":"paragraph","marks":[{"type":"indentation","attrs":{"level":1}}],"content":[{"text":"Acquisition Lists (created by the enumerator) : Ifg-cfgs (created by acq-list evaluator) : GUNWs (created by topsApp)","type":"text"}]},{"type":"paragraph","content":[{"text":"If the upstream datasets, namely acq-lists and ifg-cfgs, have been generated previous to the enumeration job (next step), then it will be difficult to track. Ultimately, the GUNWs are what we deliver and the acquisition lists and ifg-cfgs are ","type":"text"},{"text":"internal","type":"text","marks":[{"type":"em"}]},{"text":". Thus, it is safe and recommended to delete these upstream datasets that intersect the extent of an AOI prior to running the enumeration. ","type":"text"},{"text":"Do not delete GUNWs","type":"text","marks":[{"type":"em"}]},{"text":". More specifically, for the ops report discussed in this ","type":"text"},{"text":"link","type":"text","marks":[{"type":"link","attrs":{"__confluenceMetadata":{"isRenamedTitle":true,"linkType":"page","contentTitle":"Data Accountability And Delivery Verification","versionAtSave":"2"},"href":"/wiki/spaces/ARIA/pages/950796309"}}]},{"text":", we start at the upstream dataset (acquisition lists) and then go to datasets downstream to track where processing is failing using the one-to-one correspondence above. Therefore, if you facet on:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AOI extent","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Track number","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Upstream dataset: acq-lists or ifg-cfg","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"And you find acquisition lists and ifg-cfgs within your AOI extent, then it will be helpful to purge these datasets. Such datasets could be due to previous AOI processing or migration from previous AWS clusters.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Facet on (a) AOI extent, (b) Track number, and (c) acq-list or ifg-cfg","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Purge Dataset [develop-es]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"systems-jobs-queue","type":"text","marks":[{"type":"code"}]}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Run AOI based enumeration job","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"once the scraper jobs have completed, facet on the AOI in Tosca and run the following job: ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"AOI Enumerator Submitter [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotoum-job_worker-small","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"track_number","type":"text","marks":[{"type":"underline"}]},{"text":": the track number of the AOI you are processing (provided by scientist/customer)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"enumeration_job_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"develop","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"enumerator_queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"aria-standard_product-enumerator","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"note the default queue is stale","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"min_match","type":"text","marks":[{"type":"underline"}]},{"text":": number of nearest neighbors (provided by scientist/customer)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"acquisition_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"v2.0","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"skipDays","type":"text","marks":[{"type":"underline"}]},{"text":": number of days to skip before pairing (provided by scientist/customer)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": this job will iterate over the ","type":"text"},{"text":"S1-AUX_POEORB","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1BVVhfUE9FT1JCIn19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":"s that cover the AOI and submit individual enumeration jobs (","type":"text"},{"text":"aoi_track_acquisition_enumerator","type":"text","marks":[{"type":"code"}]},{"text":"). The individual enumeration jobs produce products in the following datasets:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"S1-GUNW-acq-list","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLWFjcS1saXN0In19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":": I think of these as a shopping cart that carry the IDs of the SLCs needed to produce an S1-GUNW","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"each of these correspond to a unique ifg-cfg and an S1-GUNW","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"S1-GUNW-acqlist-audit_trail","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLWFjcWxpc3QtYXVkaXRfdHJhaWwifX1dfX0sInNvcnQiOlt7Il90aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIl90aW1lc3RhbXAiLCJfc291cmNlIl19"}}]},{"text":": these are evaluation assessments of each viable pair of SLCs evaluated by the enumerator","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"they are later used by data accountability tools ","type":"text"}]}]}]}]}]}]}]},{"type":"paragraph"}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":":","type":"text","marks":[{"type":"em"}]},{"text":" If you plan to use ","type":"text"},{"text":"trigger rules ","type":"text","marks":[{"type":"underline"}]},{"text":"to improve operation efficiency, prior to running enumration, ensure that you have:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"a trigger rule for acq-list-evaluator (faceted on SLC) - typically this is turned on","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"a trigger rule for slc-localizer (faceted on acq-lists) - typically this is turned on","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"a trigger rule for topsApp (faceted on ifg-cfg, track number, and AOI extent) - we generally have to create this one because we increase the number of facets to ensure topsApp is not accidentally run due to it’s high cost.","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Run AOI based enumeration job with periods (an alternative enumeration scheme)","type":"text"}]},{"type":"paragraph","content":[{"text":"(Alternative enumeration strategy to above where we want SLCs to be within month range)","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"once the scraper jobs have completed, facet on the ","type":"text"},{"text":"starttime","type":"text","marks":[{"type":"code"}]},{"text":" of ","type":"text"},{"text":"S1-AUX_POEORB","type":"text","marks":[{"type":"code"}]},{"text":". Here is a sample query to get January 1 to April 1: ","type":"text"},{"text":"(starttime: {2014-01-01T00:00:00 TO 2014-04-01T00:00:00}) OR (starttime: {2015-01-01T00:00:00 TO 2015-04-01T00:00:00}) OR … OR (starttime: {2020-01-01T00:00:00 TO 2020-04-01T00:00:00})","type":"text","marks":[{"type":"code"}]},{"text":". You have to fill in those ","type":"text"},{"text":"…","type":"text","marks":[{"type":"code"}]},{"text":"! Here is an ","type":"text"},{"text":"example","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1BVVhfUE9FT1JCIn19LHsicXVlcnlfc3RyaW5nIjp7InF1ZXJ5IjoiKHN0YXJ0dGltZTogezIwMTctMDEtMDFUMDA6MDA6MDAgVE8gMjAxNy0wNC0wMVQwMDowMDowMH0pIE9SKHN0YXJ0dGltZTogezIwMTgtMDEtMDFUMDA6MDA6MDAgVE8gMjAxOC0wNC0wMVQwMDowMDowMH0pIE9SIChzdGFydHRpbWU6IHsyMDE5LTAxLTAxVDAwOjAwOjAwIFRPIDIwMTktMDQtMDFUMDA6MDA6MDB9KSBPUiAoc3RhcnR0aW1lOiB7MjAyMC0wMS0wMVQwMDowMDowMCBUTyAyMDIwLTA0LTAxVDAwOjAwOjAwfSkgT1IgKHN0YXJ0dGltZTogezIwMjEtMDEtMDFUMDA6MDA6MDAgVE8gMjAyMS0wNC0wMVQwMDowMDowMH0pIE9SIChzdGFydHRpbWU6IHsyMDE2LTAxLTAxVDAwOjAwOjAwIFRPIDIwMTYtMDQtMDFUMDA6MDA6MDB9KSBPUiAoc3RhcnR0aW1lOiB7MjAxNS0wMS0wMVQwMDowMDowMCBUTyAyMDE1LTA0LTAxVDAwOjAwOjAwfSkgIiwiZGVmYXVsdF9vcGVyYXRvciI6Ik9SIn19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":". ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Standard Product S1-GUNW - aoi_track_acquisition_enumerator [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"TBD","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"track_number","type":"text","marks":[{"type":"underline"}]},{"text":": the track number of the AOI you are processing (provided by scientist/customer)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AOI Name","type":"text","marks":[{"type":"underline"}]},{"text":": determined via customer","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"enumeration_job_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"develop","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"enumerator_queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"aria-standard_product-enumerator","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"note the default queue is stale","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"min_match","type":"text","marks":[{"type":"underline"}]},{"text":": number of nearest neighbors (provided by scientist/customer)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"acquisition_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"v2.0","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"skipDays","type":"text","marks":[{"type":"underline"}]},{"text":": number of days to skip before pairing (provided by scientist/customer)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": this job will iterate over the ","type":"text"},{"text":"S1-AUX_POEORB","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1BVVhfUE9FT1JCIn19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":"s that we faceted over AND cover the AOI. The individual enumeration jobs produce products in the following datasets:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"S1-GUNW-acq-list","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLWFjcS1saXN0In19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":": I think of these as a shopping cart that carry the IDs of the SLCs needed to produce an S1-GUNW","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"each of these correspond to a unique ifg-cfg and an S1-GUNW","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"S1-GUNW-acqlist-audit_trail","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLWFjcWxpc3QtYXVkaXRfdHJhaWwifX1dfX0sInNvcnQiOlt7Il90aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIl90aW1lc3RhbXAiLCJfc291cmNlIl19"}}]},{"text":": these are evaluation assessments of each viable pair of SLCs evaluated by the enumerator","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"they are later used by data accountability tools ","type":"text"}]}]}]}]}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Download SLCs","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"once all acq-lists have been generated, facet on such acq-lists in Tosca","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"you can query by the AOI id and then facet on the ","type":"text"},{"text":"S1-GUNW-acq-list","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLWFjcS1saXN0In19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":" dataset","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"then submit the localizer jobs:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Standard Product S1-GUNW slc_localizer [develop]","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"aria-standard_product-localizer","type":"text","marks":[{"type":"code"}]},{"text":" (this is an ASG and is now tagged with ","type":"text"},{"text":"Charlie","type":"text","marks":[{"type":"code"}]},{"text":" with the queue name)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"asf_ngap_download_queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotum-job_worker-slc_sling-asf","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note the other queue ","type":"text"},{"text":"slc-sling-extract-asf","type":"text","marks":[{"type":"code"}]},{"text":" is an ASG","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"esa_download_queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotum-job_worker-slc_sling-scihub","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note the other queue ","type":"text"},{"text":"slc-sling-extract-scihub","type":"text","marks":[{"type":"code"}]},{"text":" is an ASG","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"spyddder_sling_extract_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"develop","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": this job will iterate over the SLCs listed in the acq-list and submit a data sling job","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"these sling jobs take ","type":"text"},{"text":"acquisition-S1-IW_SLC","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJhY3F1aXNpdGlvbi1TMS1JV19TTEMifX1dfX0sInNvcnQiOlt7Il90aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIl90aW1lc3RhbXAiLCJfc291cmNlIl19"}}]},{"text":"s as an input and will download the corresponding SLC from ASF (relatively old acquisition) or Scihub (acquisition is less than 2 weeks old) to s3 and register the SLC in the ","type":"text"},{"text":"S1-IW_SLC","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1JV19TTEMifX1dfX0sInNvcnQiOlt7Il90aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIl90aW1lc3RhbXAiLCJfc291cmNlIl19"}}]},{"text":" dataset in GRQ","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Notes","type":"text","marks":[{"type":"underline"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Acquisition lists are in one-to-one correspondence with ifg-cfgs","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SLCs can be shared among acquisition lists and ifg-cfg's within an AOI. Therefore, #SLCs < # acq-lists = #ifg-cfgs within your AOI. As an example, within an AOI, there were ~700 SLCs for 2300 ifg-cfgs.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Say you run the localizer and you see that you have a bunch ifg-cfgs haven’t been created even though most of the sling jobs have been completed successfully. You may have only a few SLCs to download (or much less than the ifg-cfgs that are missing). Check the ","type":"text"},{"text":"unique SLCs","type":"text","marks":[{"type":"em"}]},{"text":" in the ops report. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If you have the proper trigger rules set up and activated, every time a new SLC is slinged and put into the system, then an ifg-cfg is created. This is a helpful trigger rule to have. Currently it is called ","type":"text"},{"text":"acqlist_evaluator","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"if you ever need to download a particular SLC, facet on the corresponding ","type":"text"},{"text":"acquisition-S1-IW_SLC","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJhY3F1aXNpdGlvbi1TMS1JV19TTEMifX1dfX0sInNvcnQiOlt7Il90aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIl90aW1lc3RhbXAiLCJfc291cmNlIl19"}}]},{"text":" (the SLC is a substring of the acquisition id) and submit the following job","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Data Sling and Extract for {asf, Scihub} [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotum-job_worker-{large,small}","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"sling jobs have a tendency to fail since certain products are archived in the DAACs","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"retrying/resubmitting the failed jobs a little later will usually complete","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Generate ifg-cfg products","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"facet on the SLCs in Tosca and submit the following job:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Standard Product S1-GUNW - acqlist_evaluator_ifgcfg [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotum-job_worker-standard_product-acqlist_evaluator_ifgcfg","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"acqlist_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"v2.0.0","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"acquisition_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"v2.0","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"slc_version","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"v1.1","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": this job will look at an SLC and check the acqlists to see if this SLC “completes” it (ie. was the last SLC needed to be ingested in order to proceed). If all SLCs have been downloaded, a ","type":"text"},{"text":"S1-GUNW-ifg-cfg","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLWlmZy1jZmcifX1dfX0sInNvcnQiOlt7Il90aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIl90aW1lc3RhbXAiLCJfc291cmNlIl19"}}]},{"text":" product will be produced. These produced are the input/configuration parameters for the topsapp PGE.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Notes","type":"text","marks":[{"type":"underline"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This is extremely fast. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Sometimes it’s helpful to facet on the AOI and run the SLCs that you have staged to see what new ifg-cfgs are ready to be processed.","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"note that these jobs are automatically submitted by the trigger rule ","type":"text"},{"text":"acqlist_evaluator","type":"text","marks":[{"type":"code"}]},{"text":" and you should not normally need to submit them on-demand","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"if you do need to facet on the SLCs over an AOI to submit jobs on-demand, do the following:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"query by the AOI region","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"refine your facet by adding ","type":"text"},{"text":"metadata.trackNumber: ","type":"text","marks":[{"type":"code"}]},{"text":" to the query box","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"note other data products have the field named ","type":"text"},{"text":"metadata.track_number","type":"text","marks":[{"type":"code"}]},{"text":" instead ","type":"text"}]}]}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Run topsapp","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"at this point, we have completed the necessary processing to now run topsapp and generate an ifg","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"facet on the ifg-cfgs and run the following job","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"TopsApp PGE in *Standard-Product* Pipeline for S1-GUNW Interferograms [develop]","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": topsapp jobs take a while and run on expensive machines – therefore, ","type":"text"},{"text":"this PGE ","type":"text","marks":[{"type":"em"}]},{"text":"significantly drives up costs for the pipeline","type":"text","marks":[{"type":"underline"},{"type":"em"}]},{"text":"! We have designated queues to tag the jobs with different accounts so customers can pay for these charges.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Current Recommended Queues (last updated 3/2021):","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"aria-standard_product-s1gunw-topsapp-NSLCT_Bekaert","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"aria-standard_product-s1gunw-topsapp-Access_Bekaert","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"aria-standard_product-s1gunw-topsapp-Volcano_Lundgren","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"aria-standard_product-s1gunw-topsapp-Rise_Limonadi","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note the last token in the above queues indicate the project name but more tags can be seen in the Autoscaling group setup in AWS.","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"dataset_tag","type":"text","marks":[{"type":"underline"}]},{"text":": this is a comma-delimited list of tags that will be added to the produced S1-GUNW ","type":"text"},{"text":"metadata.dataset_tags","type":"text","marks":[{"type":"code"}]},{"text":" field and can be used to facet on the product in the future","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"for the standard product pipeline on AWS, ","type":"text"},{"text":"standard_product,aws","type":"text","marks":[{"type":"code"}]},{"text":" should always be included in this parameter","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": a ","type":"text"},{"text":"S1-GUNW","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XIn19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":" product will be produced","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Notes on Trigger Rules:","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"General trigger rules with topsApp must be created with care because making a trigger rule that is too lenient can really run up costs. Here are some general rules. For topsApp trigger rule use the following facets:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Spatial extent of the AOI ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The track number associated with the AOI","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"TODO: temporal spans associate with the enumerator","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Due to the creation of the coseismic pipeline, there are some shared datasets. It is important to use ","type":"text"},{"text":"NOT \"Coseismic\"","type":"text","marks":[{"type":"code"}]},{"text":" in the query box to ensure coseismic datasets are ignored. More specific pipelines must ignore the ","type":"text"},{"text":"machine tag","type":"text","marks":[{"type":"code"}]},{"text":" called ","type":"text"},{"text":"s1-gunw-coseismic","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Greylisting","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This likely will require its own page at some point. Greylisting is important for the following reason: when we deliver products over an AOI, we only do so if ","type":"text"},{"text":"all possible GUNWs for a date pair","type":"text","marks":[{"type":"underline"},{"type":"em"}]},{"text":" ","type":"text","marks":[{"type":"em"}]},{"text":"are completed. That means if a GUNW can be completed as completed as outlined by a given acq-list/ifg-cfg, then we cannot deliver all the other GUNWs that have been generated. There are two types of errors that we want to group together that will not be able to finish with topsApp when they crop up for an ifg-cfg that should not hamper our delivery of products: ","type":"text"},{"type":"hardBreak"},{"text":"a. ","type":"text"},{"text":"Exception: Could not determine a suitable burst offset","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"b. ","type":"text"},{"text":"No swaths contain any burst overlaps ... cannot continue for interferometry applications","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"These occur when we have a GUNW that is trying to be produced over water or over an area with extremely low coherence e.g. ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJxdWVyeV9zdHJpbmciOnsicXVlcnkiOiJcIlMxLUdVTlctYWNxbGlzdC1STS1NMVMxLVROMDA0LTIwMTgwMjIyVDIzMDU0NC0yMDE3MDIxNVQyMzA1MzUtcG9lb3JiLTVlMDFcIiIsImRlZmF1bHRfb3BlcmF0b3IiOiJPUiJ9fSwic29ydCI6W3siX3RpbWVzdGFtcCI6eyJvcmRlciI6ImRlc2MifX1dLCJmaWVsZHMiOlsiX3RpbWVzdGFtcCIsIl9zb3VyY2UiXX0="}}]},{"text":". ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Caution!","type":"text","marks":[{"type":"em"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The two relevant trigger rules are: ","type":"text"},{"text":"standard-product-greylist-failed-gunw-burst-overlap","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"standard-product-greylist-failed-gunw-swaths-overlap","type":"text","marks":[{"type":"code"}]},{"text":" (these correspond to the errors above)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The trigger rules for grey listing are in figaro (since they facet on jobs)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The trigger rules (those cited above) require specification of a container - if a topsapp job is being modified be sure to update these trigger rules reference the correct container to ensure the trigger rules are correctly invoked - Charlie M. accidentally changed the name of the job-spec. exchanging a ","type":"text"},{"text":"-","type":"text","marks":[{"type":"code"}]},{"text":" with a ","type":"text"},{"text":"_","type":"text","marks":[{"type":"code"}]},{"text":" and trigger rules were no longer valid. ","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Facet","type":"text","marks":[{"type":"underline"}]},{"text":": On the jobs in figaro using the error codes above.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Standard Product S1-GUNW - Greylist S1-GUNW from topsapp job [python3]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result:","type":"text","marks":[{"type":"underline"}]},{"text":" You will get a greylist id associated with the ifg-cfg which can be identified using the hash-id.","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Notes on Errors:","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"There are some error types that are worth mentioning as they can arise even if the pipeline has been run correctly. ","type":"text"},{"text":"Make sure the errors match exactly to those examples found below as ISCE errors are very, very hard to catch","type":"text","marks":[{"type":"em"}]},{"text":" ","type":"text"},{"text":"and a slight difference in the error output can mean be the result of totally different sources","type":"text","marks":[{"type":"em"}]},{"text":" (note both error examples below mention “burst”):","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Burst overlap errors like ","type":"text"},{"text":"this job","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-mozart.aria.hysds.io/figaro/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiam9iLmpvYl9pbmZvLmpvYl9wYXlsb2FkLnBheWxvYWRfdGFza19pZCI6ImQxNjIwNTQ0LTU5OWItNDg4MS04MDVhLWZiMTBhZjE2OGZhMyJ9fV19fSwic2l6ZSI6MjUsInNvcnQiOlt7IkB0aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIkB0aW1lc3RhbXAiXSwicGFydGlhbF9maWVsZHMiOnsiX3NvdXJjZSI6eyJleGNsdWRlIjpbImpvYi5qb2JfaW5mby5tZXRyaWNzIiwiY29udGV4dC5jb250ZXh0IiwiY29udGV4dC5sb2NhbGl6ZV91cmxzIiwiam9iLmNvbnRleHQuY29udGV4dCIsImpvYi5jb250ZXh0LmpvYl9zcGVjaWZpY2F0aW9uLnBhcmFtcyIsImpvYi5jb250ZXh0LmxvY2FsaXplX3VybHMiLCJqb2IubG9jYWxpemVfdXJscyIsImpvYi5wYXJhbXMiXX19fQ=="}}]},{"text":" - the SLCs (on two different dates) do not have an overlap. This occurs when the metadata used to enumerate the job and create the IFG-CFG was slightly off from what is on the ground and/or the overlap is just not sufficient for ISCE2 to do it’s processing. This means that the IFG-CFG is malformed and should be ignored.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"DEM download errors like ","type":"text"},{"text":"this job","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-mozart.aria.hysds.io/figaro/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsicmVzb3VyY2UiOiJqb2IifX0seyJ0ZXJtIjp7ImpvYi5qb2JfaW5mby5qb2JfcGF5bG9hZC5wYXlsb2FkX3Rhc2tfaWQiOiJmODlmZDkzYS1mYjdmLTQzYTktYTAxZi0zNmZmYTBlOTZhYzMifX0seyJxdWVyeV9zdHJpbmciOnsicXVlcnkiOiJcInRvcHNBcHBcIiBBTkQgXCJkZW1cIiBBTkQgXCJqb2Itc3RhbmRhcmRfcHJvZHVjdC1zMWd1bnctdG9wc2FwcFwiIiwiZGVmYXVsdF9vcGVyYXRvciI6Ik9SIn19XX19LCJzaXplIjoyNSwic29ydCI6W3siQHRpbWVzdGFtcCI6eyJvcmRlciI6ImRlc2MifX1dLCJmaWVsZHMiOlsiQHRpbWVzdGFtcCJdLCJwYXJ0aWFsX2ZpZWxkcyI6eyJfc291cmNlIjp7ImV4Y2x1ZGUiOlsiam9iLmpvYl9pbmZvLm1ldHJpY3MiLCJjb250ZXh0LmNvbnRleHQiLCJjb250ZXh0LmxvY2FsaXplX3VybHMiLCJqb2IuY29udGV4dC5jb250ZXh0Iiwiam9iLmNvbnRleHQuam9iX3NwZWNpZmljYXRpb24ucGFyYW1zIiwiam9iLmNvbnRleHQubG9jYWxpemVfdXJscyIsImpvYi5sb2NhbGl6ZV91cmxzIiwiam9iLnBhcmFtcyJdfX19"}}]},{"text":" - this is likely a transient error and will go away on a re-run. Simply, the DEM was not downloaded successfully from our S3 bucket during processing. If problems persist, please reach out to Nicholas Arenas.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Clobber errors like ","type":"text"},{"text":"this job","type":"text","marks":[{"type":"link","attrs":{"href":"https://100.67.33.56/figaro/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiam9iLmpvYl9pbmZvLmpvYl9wYXlsb2FkLnBheWxvYWRfdGFza19pZCI6ImI2MWRlMmZkLTFhYTQtNDc4Mi1iZTYzLWI0ODcwN2FjYTA4MSJ9fV19fSwic2l6ZSI6MjUsInNvcnQiOlt7IkB0aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIkB0aW1lc3RhbXAiXSwicGFydGlhbF9maWVsZHMiOnsiX3NvdXJjZSI6eyJleGNsdWRlIjpbImpvYi5qb2JfaW5mby5tZXRyaWNzIiwiY29udGV4dC5jb250ZXh0IiwiY29udGV4dC5sb2NhbGl6ZV91cmxzIiwiam9iLmNvbnRleHQuY29udGV4dCIsImpvYi5jb250ZXh0LmpvYl9zcGVjaWZpY2F0aW9uLnBhcmFtcyIsImpvYi5jb250ZXh0LmxvY2FsaXplX3VybHMiLCJqb2IubG9jYWxpemVfdXJscyIsImpvYi5wYXJhbXMiXX19fQ=="}}]},{"text":" - although there are “short circuits” within the topsApp PGE exist, the PGE checks the completed GUNW database. Therefore, if two identical topsApp jobs were called on the same ifg-cfg before either could complete, then we will get these clobber errors. Note the clobber errors will generally not all be identical because it depends what file is uploaded first. However, an easy way to determine if such an error was due to duplication in the operator faceting, facet on a single input ifg-cfg and check the related topsApp jobs. Here is an ","type":"text"},{"text":"example","type":"text","marks":[{"type":"link","attrs":{"href":"https://100.67.33.56/figaro/?base64=eyJxdWVyeSI6eyJxdWVyeV9zdHJpbmciOnsicXVlcnkiOiJcIlMxLUdVTlctaWZnLWNmZy1STS1NMVMzLVROMDQ4LTIwMTcwNDA3VDIzMjgzMi0yMDE3MDMyNlQyMzI3MzktcG9lb3JiLWFiZDhcIiIsImRlZmF1bHRfb3BlcmF0b3IiOiJPUiJ9fSwic2l6ZSI6MjUsInNvcnQiOlt7IkB0aW1lc3RhbXAiOnsib3JkZXIiOiJkZXNjIn19XSwiZmllbGRzIjpbIkB0aW1lc3RhbXAiXSwicGFydGlhbF9maWVsZHMiOnsiX3NvdXJjZSI6eyJleGNsdWRlIjpbImpvYi5qb2JfaW5mby5tZXRyaWNzIiwiY29udGV4dC5jb250ZXh0IiwiY29udGV4dC5sb2NhbGl6ZV91cmxzIiwiam9iLmNvbnRleHQuY29udGV4dCIsImpvYi5jb250ZXh0LmpvYl9zcGVjaWZpY2F0aW9uLnBhcmFtcyIsImpvYi5jb250ZXh0LmxvY2FsaXplX3VybHMiLCJqb2IubG9jYWxpemVfdXJscyIsImpvYi5wYXJhbXMiXX19fQ=="}}]},{"text":" of such faceting in figaro.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If the errors are beyond the scope of those listed above, the relevant logs will be saved on Tosca using ","type":"text"},{"text":"triaging","type":"text","marks":[{"type":"link","attrs":{"href":"https://hysds-core.atlassian.net/wiki/spaces/HYS/pages/199885846/Preprocessor+and+Postprocessor+Functions#triage"}}]},{"text":" HySDS functionality which is currently running for the topsApp PGE; here is an example of ","type":"text"},{"text":"triaged job","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJ0cmlhZ2VkX2pvYiJ9fSx7InF1ZXJ5X3N0cmluZyI6eyJxdWVyeSI6Ilwiam9iLXN0YW5kYXJkLXByb2R1Y3QtczFndW53LXRvcHNhcHBcIiIsImRlZmF1bHRfb3BlcmF0b3IiOiJPUiJ9fV19fSwic29ydCI6W3siX3RpbWVzdGFtcCI6eyJvcmRlciI6ImRlc2MifX1dLCJmaWVsZHMiOlsiX3RpbWVzdGFtcCIsIl9zb3VyY2UiXX0="}}]},{"text":" datasets. Facet on one of the failing ifg-cfg’s and send to current topsApp maintainer (as of March 2021, this is ","type":"text"},{"text":"charlie.z.marshak@jpl.nasa.gov","type":"text","marks":[{"type":"link","attrs":{"href":"mailto:charlie.z.marshak@jpl.nasa.gov"}}]},{"text":").","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Trigger Rule:","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Generally, you want to set a trigger rule related to topsApp prior to running the enumerator.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Trigger rules that are so narrowly faceted can be hard to create if no existing dataset exists. Frequently, we have some existing trigger rule and simply edit that in the menu. For reference, here is a template (what would be in the final query window of the trigger rule):","type":"text"}]}]}]}]}]}]}]},{"type":"codeBlock","content":[{"text":"{\n \"filtered\": {\n \"query\": {\n \"bool\": {\n \"must\": [\n {\n \"term\": {\n \"dataset.raw\": \"S1-GUNW-ifg-cfg\"\n }\n },\n {\n \"query_string\": {\n \"query\": \"metadata.track_number:\",\n \"default_operator\": \"OR\"\n }\n }\n ]\n }\n },\n \"filter\": {\n \"geo_shape\": {\n \"location\": {\n \"shape\": {\n \"type\": \"polygon\",\n \"coordinates\": []\n ]\n }\n }\n }\n }\n }\n}","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Generate AOI-Tracks product","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AOI tracks are often too large to be covered by 1 S1-GUNW for a given date-pair","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"once all the ifgs for a specific date-pair are generated an ","type":"text"},{"text":"S1-GUNW-AOI_TRACK","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLUFPSV9UUkFDSyJ9fV19fSwic29ydCI6W3siX3RpbWVzdGFtcCI6eyJvcmRlciI6ImRlc2MifX1dLCJmaWVsZHMiOlsiX3RpbWVzdGFtcCIsIl9zb3VyY2UiXX0="}}]},{"text":" product is produced","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"to generate these products, facet on the ","type":"text"},{"text":"S1-GUNW","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XIn19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":" products and submit the following job:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Standard Product S1-GUNW - S1-GUNW Completeness Evaluator [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotum-job_worker-standard_product-completeness_evaluator","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": this job will look at an S1-GUNW and check if it “completes” the track for a given date-pair. If so ","type":"text"},{"text":"S1-GUNW-AOI_TRACK","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLUFPSV9UUkFDSyJ9fV19fSwic29ydCI6W3siX3RpbWVzdGFtcCI6eyJvcmRlciI6ImRlc2MifX1dLCJmaWVsZHMiOlsiX3RpbWVzdGFtcCIsIl9zb3VyY2UiXX0="}}]},{"text":" product is generated. Otherwise, the evaluator silently completes without producing anything.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"note that these jobs are automatically submitted by the trigger rule ","type":"text"},{"text":"s1gunw-aws-s1gunw-completeness-evaluator","type":"text","marks":[{"type":"code"}]},{"text":" and you should not normally need to submit them on-demand","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"once an ","type":"text"},{"text":"S1-GUNW-AOI_TRACK","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XLUFPSV9UUkFDSyJ9fV19fSwic29ydCI6W3siX3RpbWVzdGFtcCI6eyJvcmRlciI6ImRlc2MifX1dLCJmaWVsZHMiOlsiX3RpbWVzdGFtcCIsIl9zb3VyY2UiXX0="}}]},{"text":" product is produced, the ","type":"text"},{"text":"S1-GUNW","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba-grq.aria.hysds.io/search/?base64=eyJxdWVyeSI6eyJib29sIjp7Im11c3QiOlt7InRlcm0iOnsiZGF0YXNldC5yYXciOiJTMS1HVU5XIn19XX19LCJzb3J0IjpbeyJfdGltZXN0YW1wIjp7Im9yZGVyIjoiZGVzYyJ9fV0sImZpZWxkcyI6WyJfdGltZXN0YW1wIiwiX3NvdXJjZSJdfQ=="}}]},{"text":" products will be published to ASF and ARIA-products via the following pipeline: <<","type":"text"},{"text":"TODO: delivery pipeline","type":"text","marks":[{"type":"strong"}]},{"text":" - there are a bunch of PGEs here and for most intents and purposes they work using trigger rules that have been set up>>","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Checking the Delivery to ASF","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"In addition to using the various ops reports, you can go directly to ASF: ","type":"text"},{"text":"https://search.asf.alaska.edu/","type":"text","marks":[{"type":"link","attrs":{"href":"https://search.asf.alaska.edu/"}}]},{"text":" and us their search by “list” feature. Copying the GUNW ids into this feature can illustrate the delivery publically! This is generally a good method of “delivering” the final AOI to science customers. Here is an ","type":"text"},{"text":"example","type":"text","marks":[{"type":"link","attrs":{"href":"https://search.asf.alaska.edu/#/?searchType=List%20Search&listSearchType=Product&searchList=S1-GUNW-A-R-091-tops-20190217_20190205-233502-34820S_36938S-PP-db61-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20190205-233437-36283S_38413S-PP-db98-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20180129-233405-37742S_39884S-PP-5bca-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20180129-233455-34819S_36938S-PP-6a9e-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20151229-233506-36282S_38412S-PP-cfe2-v2_0_4,S1-GUNW-A-R-091-tops-20190112_20181231-233438-36282S_38412S-PP-6ec1-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20170116-233455-34819S_36938S-PP-fdf6-v2_0_4,S1-GUNW-A-R-091-tops-20190313_20190205-233502-34820S_36939S-PP-9837-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20180117-233431-36282S_38412S-PP-c800-v2_0_4,S1-GUNW-A-R-091-tops-20150115_20141222-233510-35310S_37430S-PP-314f-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20170116-233455-34819S_36938S-PP-08f1-v2_0_4,S1-GUNW-A-R-091-tops-20150316_20150115-233510-35300S_37440S-PP-76a8-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20180306-233456-34820S_36939S-PP-61d2-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20150115-233521-35307S_36938S-PP-703b-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20180129-233405-37742S_39884S-PP-e846-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190217-233418-37743S_39885S-PP-8b33-v2_0_4,S1-GUNW-A-R-091-tops-20200319_20200307-233418-37744S_39885S-PP-073b-v2_0_4,S1-GUNW-A-R-091-tops-20200319_20200313-233429-37744S_38740S-PP-99dd-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20170116-233405-37744S_39885S-PP-7741-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20170116-233456-34820S_36939S-PP-82fd-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20170116-233431-36283S_38413S-PP-4e83-v2_0_4,S1-GUNW-A-R-091-tops-20190313_20190217-233502-34820S_36939S-PP-26e7-v2_0_4,S1-GUNW-A-R-091-tops-20190313_20190217-233437-36283S_38413S-PP-1082-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20180210-233405-37743S_39885S-PP-cdde-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20180210-233430-36282S_38412S-PP-69ea-v2_0_4,S1-GUNW-A-R-091-tops-20200313_20190430-233544-35316S_37099S-PP-0981-v2_0_4,S1-GUNW-A-R-091-tops-20200313_20190430-233522-36291S_38737S-PP-0041-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20171212-233431-36282S_38412S-PP-abf0-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20190217-233502-34820S_36939S-PP-e2d9-v2_0_4,S1-GUNW-A-R-091-tops-20190124_20190112-233502-34819S_36938S-PP-4f79-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20180129-233502-34819S_36938S-PP-2a2b-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20180318-233405-37744S_39885S-PP-d780-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190430-233418-37743S_39885S-PP-c32e-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20180117-233405-37742S_39884S-PP-4294-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20170116-233456-34820S_36938S-PP-2249-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20170116-233431-36283S_38412S-PP-b75a-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20170116-233430-36282S_38412S-PP-82e1-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20151229-233500-36282S_38412S-PP-6fc2-v2_0_4,S1-GUNW-A-R-091-tops-20190313_20190205-233437-36283S_38413S-PP-0e5f-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20160122-233500-36282S_38412S-PP-2248-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20161223-233531-34818S_36937S-PP-8998-v2_0_4,S1-GUNW-A-R-091-tops-20190124_20181231-233412-37743S_39884S-PP-53d9-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20180222-233405-37743S_39885S-PP-0fe7-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20160122-233435-37743S_39885S-PP-8965-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20180129-233430-36282S_38412S-PP-8b02-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20150208-233500-36282S_38412S-PP-b7d2-v2_0_4,S1-GUNW-A-R-091-tops-20200313_20200307-233544-35316S_37099S-PP-0628-v2_0_4,S1-GUNW-A-R-091-tops-20200325_20200313-233544-35316S_37100S-PP-a618-v2_0_4,S1-GUNW-A-R-091-tops-20150316_20150208-233445-36762S_38914S-PP-e169-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20151229-233441-37742S_39884S-PP-d6c5-v2_0_4,S1-GUNW-A-R-091-tops-20190112_20181219-233438-36282S_38412S-PP-507e-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20180306-233431-36283S_38413S-PP-0915-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20200325-233444-36283S_38413S-PP-edaf-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20190325-233508-34820S_36939S-PP-1df1-v2_0_4,S1-GUNW-A-R-091-tops-20200325_20200319-233544-35316S_37099S-PP-7ec4-v2_0_4,S1-GUNW-A-R-091-tops-20150316_20150115-233445-36762S_38914S-PP-5e49-v2_0_4,S1-GUNW-A-R-091-tops-20150115_20141222-233445-36772S_38905S-PP-30d1-v2_0_4,S1-GUNW-A-R-091-tops-20150115_20141128-233445-36772S_38905S-PP-3e46-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190430-233508-34820S_36939S-PP-e718-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20151229-233500-36283S_38412S-PP-4ee1-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20171212-233405-37743S_39885S-PP-0a64-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20171212-233455-34819S_36938S-PP-1c4d-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20180306-233430-36283S_38413S-PP-133e-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20161129-233506-36282S_38412S-PP-de9d-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20171130-233431-36283S_38412S-PP-e0a7-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20180129-233412-37743S_39885S-PP-834b-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20180222-233455-34820S_36939S-PP-bff4-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20180222-233430-36283S_38413S-PP-3317-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20180318-233412-37743S_39885S-PP-e82e-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20180318-233502-34820S_36939S-PP-3c19-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20151205-233500-36283S_38412S-PP-5e5e-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20180210-233455-34819S_36938S-PP-72cf-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20160122-233525-34820S_36938S-PP-098e-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190406-233418-37743S_39885S-PP-ce5e-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190406-233508-34820S_36939S-PP-86ac-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190406-233443-36283S_38413S-PP-e019-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20170116-233405-37743S_39885S-PP-3ed5-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20170116-233455-34820S_36939S-PP-ed0f-v2_0_4,S1-GUNW-A-R-091-tops-20150208_20150115-233445-36773S_38905S-PP-10c3-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20150115-233443-37743S_38903S-PP-e308-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20151229-233531-34818S_36937S-PP-29b5-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20160215-233525-34820S_36938S-PP-eb15-v2_0_4,S1-GUNW-A-R-091-tops-20200313_20200307-233522-36291S_38737S-PP-6b1a-v2_0_4,S1-GUNW-A-R-091-tops-20190124_20181231-233438-36282S_38412S-PP-885e-v2_0_4,S1-GUNW-A-R-091-tops-20190124_20181231-233502-34819S_36938S-PP-67db-v2_0_4,S1-GUNW-A-R-091-tops-20200319_20200313-233443-36283S_38413S-PP-a4a2-v2_0_4,S1-GUNW-A-R-091-tops-20200319_20200313-233503-35308S_36939S-PP-28d5-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20200319-233418-37743S_39885S-PP-9dc7-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20200319-233444-36283S_38413S-PP-cbdf-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20200325-233429-37743S_38740S-PP-69c4-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20200325-233503-35308S_36939S-PP-ce3b-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20150115-233500-36283S_38412S-PP-298a-v2_0_4,S1-GUNW-A-R-091-tops-20200319_20200307-233443-36283S_38413S-PP-7d7e-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20180117-233455-34819S_36938S-PP-0ea7-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20170116-233455-34819S_36938S-PP-d831-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20190313-233412-37743S_39885S-PP-502f-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20190313-233437-36283S_38413S-PP-0348-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20171212-233406-37743S_39885S-PP-5c14-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20171212-233431-36283S_38412S-PP-e040-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20180210-233437-36283S_38413S-PP-ec77-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20180210-233412-37743S_39885S-PP-50a4-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20190217-233412-37743S_39885S-PP-58cb-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20190217-233437-36283S_38413S-PP-70f7-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20180129-233431-36282S_38412S-PP-3f78-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190430-233443-36283S_38413S-PP-9690-v2_0_4,S1-GUNW-A-R-091-tops-20190112_20181219-233503-34819S_36938S-PP-55b7-v2_0_4,S1-GUNW-A-R-091-tops-20150115_20141128-233510-35310S_37430S-PP-f79f-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20170116-233405-37743S_39885S-PP-5488-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20151229-233435-37743S_39884S-PP-385f-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20190205-233412-37743S_39885S-PP-5428-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20190325-233418-37743S_39885S-PP-5792-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20190124-233502-34819S_36938S-PP-fcc4-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20151229-233525-34819S_36938S-PP-c46c-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20180222-233430-36282S_38412S-PP-9f22-v2_0_4,S1-GUNW-A-R-091-tops-20190124_20190112-233438-36282S_38412S-PP-2438-v2_0_4,S1-GUNW-A-R-091-tops-20150208_20141222-233509-35310S_37431S-PP-15cb-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20180129-233437-36282S_38412S-PP-61be-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20190124-233502-34820S_36938S-PP-5305-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20180117-233431-36282S_38412S-PP-5a92-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20150208-233520-35308S_36938S-PP-a70e-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20170116-233405-37742S_39884S-PP-0ded-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20170116-233431-36282S_38412S-PP-4192-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20160215-233435-37743S_39885S-PP-4c50-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20160215-233500-36283S_38412S-PP-6716-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20170116-233430-36283S_38413S-PP-b4d5-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20160122-233525-34819S_36938S-PP-b6c7-v2_0_4,S1-GUNW-A-R-091-tops-20150316_20150208-233510-35300S_37440S-PP-5ba5-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20190112-233502-34819S_36938S-PP-8160-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190217-233508-34820S_36939S-PP-c579-v2_0_4,S1-GUNW-A-R-091-tops-20200307_20190217-233443-36283S_38413S-PP-89bc-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20170116-233405-37743S_39885S-PP-3a0d-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20170116-233455-34819S_36938S-PP-35ff-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20170116-233405-37742S_39884S-PP-4d42-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20170116-233430-36282S_38412S-PP-7595-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20160122-233500-36283S_38412S-PP-df80-v2_0_4,S1-GUNW-A-R-091-tops-20190112_20181231-233503-34819S_36938S-PP-de5a-v2_0_4,S1-GUNW-A-R-091-tops-20200325_20200319-233522-36292S_38737S-PP-c935-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20170116-233406-37743S_39885S-PP-dd7c-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20180306-233405-37744S_39885S-PP-3346-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20161223-233441-37742S_39884S-PP-30a4-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20180318-233456-34820S_36939S-PP-caaf-v2_0_4,S1-GUNW-A-R-091-tops-20180330_20180318-233431-36283S_38413S-PP-f691-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20190124-233437-36282S_38412S-PP-2066-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20180210-233455-34819S_36938S-PP-fedc-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20171212-233456-34820S_36938S-PP-cb97-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20180129-233455-34819S_36938S-PP-5c79-v2_0_4,S1-GUNW-A-R-091-tops-20150208_20141222-233445-36773S_38905S-PP-c52a-v2_0_4,S1-GUNW-A-R-091-tops-20190124_20190112-233412-37743S_39884S-PP-4948-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20180306-233405-37743S_39885S-PP-069c-v2_0_4,S1-GUNW-A-R-091-tops-20190112_20181219-233413-37743S_39884S-PP-6ab1-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20161129-233441-37742S_39884S-PP-0c76-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20171130-233406-37743S_39885S-PP-f4a0-v2_0_4,S1-GUNW-A-R-091-tops-20180210_20180117-233455-34819S_36938S-PP-f5ab-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20150208-233443-37743S_38903S-PP-5c0e-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20180222-233405-37743S_39885S-PP-7689-v2_0_4,S1-GUNW-A-R-091-tops-20180306_20180222-233455-34819S_36938S-PP-bd01-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20161129-233531-34818S_36937S-PP-673a-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20190313-233502-34820S_36939S-PP-c8ce-v2_0_4,S1-GUNW-A-R-091-tops-20180117_20171130-233456-34820S_36938S-PP-1445-v2_0_4,S1-GUNW-A-R-091-tops-20180318_20180306-233455-34820S_36939S-PP-df14-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20180210-233502-34820S_36938S-PP-9b00-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20151229-233435-37743S_39884S-PP-95a6-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20190124-233412-37743S_39885S-PP-bfc1-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20190112-233412-37743S_39885S-PP-5dfe-v2_0_4,S1-GUNW-A-R-091-tops-20190205_20190112-233437-36282S_38412S-PP-50ee-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20151205-233435-37743S_39884S-PP-0c21-v2_0_4,S1-GUNW-A-R-091-tops-20160122_20151205-233525-34819S_36938S-PP-29f3-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20180210-233405-37742S_39884S-PP-2b95-v2_0_4,S1-GUNW-A-R-091-tops-20180222_20180210-233430-36282S_38412S-PP-fea1-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20150208-233443-37743S_38904S-PP-ade7-v2_0_4,S1-GUNW-A-R-091-tops-20160310_20150208-233500-36283S_38412S-PP-601f-v2_0_4,S1-GUNW-A-R-091-tops-20170116_20161223-233506-36282S_38412S-PP-1af8-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20151229-233525-34819S_36938S-PP-eb9f-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20150208-233520-35307S_36938S-PP-1931-v2_0_4,S1-GUNW-A-R-091-tops-20200325_20200313-233522-36292S_38737S-PP-60bc-v2_0_4,S1-GUNW-A-R-091-tops-20160215_20160122-233435-37743S_39884S-PP-b09b-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20170116-233431-36282S_38412S-PP-f2b6-v2_0_4,S1-GUNW-A-R-091-tops-20190325_20180318-233437-36283S_38413S-PP-aef4-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20190325-233444-36283S_38413S-PP-ad89-v2_0_4,S1-GUNW-A-R-091-tops-20150208_20150115-233509-35310S_37431S-PP-9ee1-v2_0_4,S1-GUNW-A-R-091-tops-20200319_20200307-233508-34820S_36939S-PP-08e2-v2_0_4,S1-GUNW-A-R-091-tops-20200331_20200319-233508-34820S_36939S-PP-92b3-v2_0_4,S1-GUNW-A-R-091-tops-20190112_20181231-233413-37743S_39884S-PP-c0c5-v2_0_4,S1-GUNW-A-R-091-tops-20190313_20190205-233412-37743S_39885S-PP-2398-v2_0_4,S1-GUNW-A-R-091-tops-20180129_20180117-233405-37743S_39885S-PP-7e19-v2_0_4,S1-GUNW-A-R-091-tops-20190313_20190217-233412-37743S_39885S-PP-5361-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20190124-233412-37743S_39885S-PP-9be1-v2_0_4,S1-GUNW-A-R-091-tops-20190217_20190124-233437-36283S_38413S-PP-3737-v2_0_4&resultsLoaded=true&granule=S1-GUNW-A-R-091-tops-20200331_20190325-233508-34820S_36939S-PP-1df1-v2_0_4&zoom=5.926666666666667¢er=-72.211914,-41.874257"}}]},{"text":".","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Delivery Failures (flavor 1)","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"An AOITrack Exists but the GUNW wasn’t delivered. This means that the completeness evaluator succeed but some PGE downstream didn’t. Below is taken verbatim from ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"__confluenceMetadata":{"isRenamedTitle":true,"linkType":"page","contentTitle":"Data Accountability And Delivery Verification","versionAtSave":"2"},"href":"/wiki/spaces/ARIA/pages/950796309"}}]},{"text":". This is part of the “delivery pipeline” which is not documented here, but hopefully will be at some point.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Facet","type":"text","marks":[{"type":"underline"}]},{"text":": AOITrack datasets that are not delivering","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Product Delivery of S1-GUNW-AOI_TRACK [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Queue","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"factotum-job_worker-{small,large}","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"pub_sns_arn","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"arn:aws:sns:us-east-1:406893895021:ingest-prod-jobs","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"callback_sns_arn","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"arn:aws:sns:us-west-2:151169893255:aria-torresal-011-daac-cnm-response","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result","type":"text","marks":[{"type":"underline"}]},{"text":": submits individual product delivery jobs of each ifg in the track","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Notes on Other Errors:","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If any GUNW from an entire date pair is absent, then ","type":"text"},{"text":"none of the GUNWs","type":"text","marks":[{"type":"em"}]},{"text":" will deliver. So even if you facet on missing GUNWs, this process will complete without error, but not create an AOITrack dataset and therefore not deliver your desired ASF.","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Generate AOI-Tracks product using Greylist IDs (Completeness via greylist)","type":"text"}]},{"type":"paragraph","content":[{"text":"Did you have some greylist IDs that were generated on demand? We did! The below action needs to be tested, but we are pretty confident it works","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Facet","type":"text","marks":[{"type":"underline"}]},{"text":": Greylist Ids.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Action","type":"text","marks":[{"type":"underline"}]},{"text":": ","type":"text"},{"text":"Standard Product S1-GUNW - S1-GUNW Completeness Evaluator By GreyList [develop]","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Result:","type":"text","marks":[{"type":"underline"}]},{"text":" You will run the completeness evaluator on the new GUNWs.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Clean up","type":"text","marks":[{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Purge localized SLCs as done ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"__confluenceMetadata":{"isRenamedTitle":true,"linkType":"page","contentTitle":"Purge SLCs from a Given Track","versionAtSave":"6"},"href":"/wiki/spaces/ARIA/pages/1007616001"}}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Delete trigger rules associated with TopsApp - clutters trigger rules - or turn them off!","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Check stray instances. ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"In certain cases, RabbitMQ does not accurately capture all the instances that are running from a given queue in ASG. You can check the queues used for the standard product pipeline in EC2 > Auto Scaling Groups (sidebar) and check the queues there. Then, set the “desired capacity” to 0 in the topmost menu when you click on a given queue. Alternatively, to view all the running instances, AWS console > EC2 > Instances (sidebar) and check the instances that are “running”. ","type":"text"}]}]}]}]}]},{"type":"rule"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Notes","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Faceting in Tosca and Figaro","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"It is important to “debug” and view the various progress of all the datasets in the pipeline. There are some very crucial tools that will make life much, much easier.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Spatial queries","type":"text","marks":[{"type":"underline"}]},{"text":" - you can use any dataset to query a region via ","type":"text"},{"text":"Query Region","type":"text","marks":[{"type":"code"}]},{"text":". As mentioned in the section on creating an AOI above, this can help find the related datasets in Tosca.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This type of query can be done with any dataset. Here are a few examples:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":" all the staged SLCs associated with an ifg-cfg, use the extent of the ifg-cfg","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"all the possible SLCs from an acquisition list, use the extent of the acquisition list","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"See the GUNWs that use a given SLC, use the extent of the SLC.","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Timestamp queries","type":"text","marks":[{"type":"underline"}]},{"text":" - generally if you have just created some number of datasets, its helpful to use a timestamp to query based on when they were generated, not by location. Note these are ","type":"text"},{"text":"all in UTC!","type":"text","marks":[{"type":"underline"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Figaro example","type":"text","marks":[{"type":"em"}]},{"text":":","type":"text"},{"text":" ","type":"text","marks":[{"type":"em"}]},{"text":"@timestamp: {2021-04-02T00:00:00 TO *}","type":"text","marks":[{"type":"code"}]},{"text":" (in UTC)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Tosca example","type":"text","marks":[{"type":"em"}]},{"text":":","type":"text"},{"text":" ","type":"text","marks":[{"type":"em"}]},{"text":"creation_timestamp: {2021-04-05T00:00:00 TO *}","type":"text","marks":[{"type":"code"}]},{"text":" (in UTC)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If your query is not working with the said examples, the general formula is ","type":"text"},{"text":": { TO }","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Examples use-cases:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"You want to facet on only ifg-cfgs for topsApp jobs that you created in the last hour to avoid calling topsApp jobs for-ifg-cfgs that have already been created","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"You want to determine any given jobs of a particular job type that were called/triggered in a certain time frame to determine what type of failures occured.","type":"text"}]}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Metadata Queries","type":"text","marks":[{"type":"underline"}]},{"text":" - sometimes its helpful to select a particular value in Tosca. One that comes up a lot is selecting a given a track number for datasets to ensure that a trigger rule is sufficiently constrained. The general format is ","type":"text"},{"text":": ","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Examples:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"metadata.track_number:","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"metadata.trackNumber:","type":"text","marks":[{"type":"code"}]},{"text":" (depending on the ES dataset). This is useful for creating a trigger rule for TopsApp because an AOI, which is created to align with a track, will have this extra constraint and this ensures that no other ifg-cfgs created in the system will generate GUNWs.","type":"text"}]}]}]}]}]}]}]},{"type":"paragraph","content":[{"text":"General Notes on Cost Management and Processing","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"The two most important sources of cost within the pipeline are:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"s3 storage of the staged SLCs","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"topsApp jobs","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Importantly, the cost from s3 storage is dependent on ","type":"text"},{"text":"time. ","type":"text","marks":[{"type":"em"}]},{"text":"The longer staged SLCs stay on the system, the more they cost. The operator has to manage these costs in the following way:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Ensuring that SLCs for an AOI are downloaded en masse. That is every acquisition list has all its SLCs. Of course, getting all the SLCs on the system is never attainable in practice. However, the more SLCs from an AOI that are downloaded, the more directly and thus faster the topsApp processing can be done. Also the purging can be done more quickly.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"speed of processing staged SLCs (post enumeration) into GUNWs using topsapp - in other words, ensuring the topsApp jobs are run quickly once the SLCs have been staged so that you are not waiting","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"this is most efficiently done with trigger rules on ifg-cfgs (see the topsapp section above).","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Purging SLCs that are no longer needed","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Removing the datasets also purges the SLCs from S3","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"While it is beneficial to purge SLCs that are no longer needed, note figuring which are needed and which are not is complicated and is why its best to get as many of the required SLCs downloaded at once","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If you have a small number of GUNWs that are missing it’s best to purge the existing SLCs and repeat the pipeline on the related acquisition lists/ifg-cfgs as those required to produce the GUNWs.","type":"text"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Removing Jobs after bad Facets","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":"It is inevitable there will be times","type":"text"}]},{"type":"paragraph","content":[{"text":"TopsApp Bug Documented (related to intermediate datasets)","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph"}],"version":1}