Page Comparison

...

• once all acq-lists have been generated, facet on such acq-lists in Tosca

  • you can query by the AOI id and then facet on the S1-GUNW-acq-list dataset

• then submit the localizer jobs:

  • Action: Standard Product S1-GUNW slc_localizer [develop]

  • Queue: factotum queue is factotum-job_worker-standard_product-slc_localizer and the ASG queue is aria-standard_product-localizer (Perhaps create a dedicated queue on factotum like we did for the evaluator.)

  • asf_ngap_download_queue: factotum-job_worker-slc_sling-asf

    • Note the other queue slc-sling-extract-asf is an ASG

  • esa_download_queue: factotum-job_worker-slc_sling-scihub

    • Note the other queue slc-sling-extract-scihub is an ASG

  • spyddder_sling_extract_version: develop

  • Result: this job will iterate over the SLCs listed in the acq-list and submit a data sling job

    • these sling jobs take acquisition-S1-IW_SLCs as an input and will download the corresponding SLC from ASF (relatively old acquisition) or Scihub (acquisition is less than 2 weeks old) to s3 and register the SLC in the S1-IW_SLC dataset in GRQ

  • Notes:

    • Acquisition lists are in one-to-one correspondence with ifg-cfgs

    • SLCs can be shared among acquisition lists and ifg-cfg's within an AOI. Therefore, #SLCs < # acq-lists = #ifg-cfgs within your AOI. As an example, within an AOI, there were ~700 SLCs for 2300 ifg-cfgs.

    • Say you run the localizer and you see that you have a bunch ifg-cfgs haven’t been created even though most of the sling jobs have been completed successfully. You may have only a few SLCs to download (or much less than the ifg-cfgs that are missing). Check the unique SLCs in the ops report.

    • If you have the proper trigger rules set up and activated, every time a new SLC is slinged and put into the system, then an ifg-cfg is created. This is a helpful trigger rule to have. Currently it is called acqlist_evaluator.

• if you ever need to download a particular SLC, facet on the corresponding acquisition-S1-IW_SLC (the SLC is a substring of the acquisition id) and submit the following job

  • Action: Data Sling and Extract for {asf, Scihub} [develop]

  • Queue: factotum-job_worker-{large,small}

• sling jobs have a tendency to fail since certain products are archived in the DAACs

  • retrying/resubmitting the failed jobs a little later will usually complete

...

• facet on the SLCs in Tosca and submit the following job:

  • Action: Standard Product S1-GUNW - acqlist_evaluator_ifgcfg [develop]

  • Queue: factotum-job_worker-standard_product-slcacqlist_evaluator_ifgcfg

  • acqlist_version: v2.0.0

  • acquisition_version: v2.0

  • slc_version: v1.1

  • Result: this job will look at an SLC and check the acqlists to see if this SLC “completes” it (ie. was the last SLC needed to be ingested in order to proceed). If all SLCs have been downloaded, a S1-GUNW-ifg-cfg product will be produced. These produced are the input/configuration parameters for the topsapp PGE.

  • Notes:

    • This is extremely fast.

    • Sometimes it’s helpful to facet on the AOI and run the SLCs that you have staged to see what new ifg-cfgs are ready to be processed.

• note that these jobs are automatically submitted by the trigger rule acqlist_evaluator and you should not normally need to submit them on-demand

• if you do need to facet on the SLCs over an AOI to submit jobs on-demand, do the following:

  • query by the AOI region

  • refine your facet by adding metadata.trackNumber: <track number> to the query box

    • note other data products have the field named metadata.track_number instead

...

• AOI tracks are often too large to be covered by 1 S1-GUNW for a given date-pair

• once all the ifgs for a specific date-pair are generated an S1-GUNW-AOI_TRACK product is produced

• to generate these products, facet on the S1-GUNW products and submit the following job:

  • Action: Standard Product S1-GUNW - S1-GUNW Completeness Evaluator [develop]

  • Queue: factotum-job_worker-large-standard_product-completeness_evaluator

  • Result: this job will look at an S1-GUNW and check if it “completes” the track for a given date-pair. If so S1-GUNW-AOI_TRACK product is generated. Otherwise, the evaluator silently completes without producing anything.

• note that these jobs are automatically submitted by the trigger rule s1gunw-aws-s1gunw-completeness-evaluator and you should not normally need to submit them on-demand

• once an S1-GUNW-AOI_TRACK product is produced, the S1-GUNW products will be published to ASF and ARIA-products via the following pipeline: TODO: delivery pipeline

• Checking the Delivery to ASF

  • In addition to using the various ops reports, you can go directly to ASF: https://search.asf.alaska.edu/ and us their search by “list” feature. Copying the GUNW ids into this feature can illustrate the delivery publically! This is generally a good method of “delivering” the final AOI to science customers. Here is an example.

• Notes on Errors:

  • This is not an error, but a confusing behavior. If any GUNW from an entire date pair is abset, then none of the GUNWs will deliver. So even if you facet on missing GUNWs, this process will complete without error, but not deliver your desired ASF.

...