Page Comparison

...

• once the AOI has been created, facet on the AOI in Tosca and run the following jobs:

  • acq scraper jobs

    • Action: AOI based submission of acq scraper jobs [develop]

    • Queue: factotoum-job_worker-small

    • Result: this job will submit individual acq scraper jobs over the AOI

  • IPF scraper jobs

    • Action: AOI based submission of IPF scraper jobs [develop]

    • Queue: factotoum-job_worker-small

    • Result: this job will submit individual IPF scraper jobs over the AOI

• these update the acquisition-S1-IW_SLC dataset

• Notes:

  • An “overloaded” term is AOI and track. Here are the definitions:

    • An AOI is the area which we want to cover with GUNWs. The SLCs to be downloaded and paired will be according to the enumeration strategy determined by the enumerator submitter below.

    • A track is the path in which the satellite follows and repeats during its orbits around Earth. Here is an image for Sentinel 6’s tracks. These track numbers are called path numbers in ASF Search (see the Filters menu; the example in the linked search shows Track 48 in January)

    • How are they related? We purposefully create AOIs that align with a particular track so that all the SLCs come from a given track. This is ensured in the enumeration because we have a minimum coverage threshold for SLCs and only those within the track will satisfy that threshold. Note that every SLC is collected on a given track so every SLC has a track number.

    • We can facet on tracks within Tosca using:

      • metadata.track_number:<track_number> or metadata.trackNumber:<track_number> (depending on the ES dataset)

  • After creating AOIs, the spatial extent of the AOI dataset in tosca will be the single most important way to query the subsequent datasets related to an AOI. Here is the rough process to do so:

    • Select the recently created AOI in Tosca

    • Click Query Region within said dataset

    • Facet on other dataset or query strings for further filtering.

Clean up Upstream Datasets (optional)

Context: Important datasets for monitoring are:

Acquisition Lists (created by the enumerator) : Ifg-cfgs (created by acq-list evaluator) : GUNWs (created by topsApp)

If the upstream datasets, namely acq-lists and ifg-cfgs, have been generated previous to the enumeration job (next step), then it will be difficult to track. Ultimately, the GUNWs are what we deliver (thus do NOT delete this) and the acquisition lists and ifg-cfgs are internal. Thus, it is safe and recommended to delete these upstream datasets prior to running the enumeration. For the ops report discussed in this link, we start at the upstream dataset (acquisition lists) and then go to datasets downstream to track where processing is failing (basically, you launch jobs to ensure the one-to-one correspondence above). Therefore, if you facet on:

• AOI extent

• Track number

• Upstream dataset: acq-lists or ifg-cfg

And you find acquisition lists and ifg-cfgs, then it will be helpful to purge these datasets. Such datasets could be due to previous AOI processing or migration from previous AWS clusters.

• Facet on (a) AOI extent, (b) Track number, and (c) acq-list or ifg-cfg

  • Action: Purge Dataset [develop-es]

  • Queue: systems-jobs-queue

Run AOI based enumeration job

...

(Alternative enumeration strategy to above where we want SLCs to be within month range)To Be Reviewed by Jack McNelis.

• once the scraper jobs have completed, facet on the starttime of S1-AUX_POEORB. Here is a sample query to get January 1 to April 1: (starttime: {2014-01-01T00:00:00 TO 2014-04-01T00:00:00}) OR (starttime: {2015-01-01T00:00:00 TO 2015-04-01T00:00:00}) OR … OR (starttime: {2020-01-01T00:00:00 TO 2020-04-01T00:00:00}). You have to fill in those …! Here is an example.

  • Action: Standard Product S1-GUNW - aoi_track_acquisition_enumerator [develop]

  • Queue: TBD

  • track_number: the track number of the AOI you are processing (provided by scientist/customer)

  • AOI Name: determined via customer

  • enumeration_job_version: develop

  • enumerator_queue: aria-standard_product-enumerator

    • note the default queue is stale

  • min_match: number of nearest neighbors (provided by scientist/customer)

  • acquisition_version: v2.0

  • skipDays: number of days to skip before pairing (provided by scientist/customer)

  • Result: this job will iterate over the S1-AUX_POEORBs that we faceted over AND cover the AOI. The individual enumeration jobs produce products in the following datasets:

    • S1-GUNW-acq-list: I think of these as a shopping cart that carry the IDs of the SLCs needed to produce an S1-GUNW

      • each of these correspond to a unique ifg-cfg and an S1-GUNW

    • S1-GUNW-acqlist-audit_trail: these are evaluation assessments of each viable pair of SLCs evaluated by the enumerator

      • they are later used by data accountability tools

...