E2E On-Demand Pipeline Walkthrough

This is a step by step procedure for the bul`k processing of a new request

create a Request-S1GUNW and generate an estimate

• walk though the following procedure: Submit an On-Demand Estimate Request

• the API call will create a Request-S1GUNW object and submit hysds-io-request-s1gunw-acq_enumerator-location:develop enumeration jobs to the aria-standard_product-enumerator queue

• these enumeration jobs will generate the runconfig-acq-lists and runconfig-acqlist-audit_trails

• make sure all enumeration jobs complete successfully

• the number of runconfig-acq-lists generated (you can query by the Request-S1GUNW id in Tosca to see the count) are the expected number of S1-GUNWs to be generated

  • verify with the customer/scientist that the count looks okay before proceeding with the submission step of processing

call request-submit to trigger the rest of processing

• walk through the following procedure: Submit an On-Demand Submit Reuqest

• the API call will start by submitting a hysds-io-create_aoi:develop job in the factotum-create_aoi-queue for every item in the enumeration_list field in the corresponding Request-S1GUNW object

• the API will then submit individual data sling jobs for the SLCs referenced in the runconfig-acq-lists

  • job-spyddder-sling-extract-local-asf-singularity:ARIA-446_singularity in spyddder-sling-extract-local-asf or job-spyddder-sling-extract-local-scihub-singularity:ARIA-446_singularity in spyddder-sling-extract-local-scihub

  • note that these jobs stage the SLCs to the pleiades file system rather than in s3

  • the downloaded SLCs are registered in the S1-IW_SLC-local dataset

• as the SLCs are ingested, they will kick off the acqlist evaluator jobs via the trigger rule slc-local-request-s1gunw-acqlist_evaluator-develop

  • these jobs will generate a runconfig-topsapp object once all the SLCs in a given runconfig-acq-list are ingested

• the runconfig-topsapp objects will invoke one of the following trigger rules:

  

  • note that there are three distinct queues, each of which corresponds to the group account responsible for the request

  • if you ever need to submit a topsapp job manually, make sure the job parameters match those specified in the trigger rules (can click Edit in the trigger rules to view the parameterization of a particular trigger rule)

generate AOI-Tracks product

• AOI tracks are often too large to be covered by 1 S1-GUNW for a given date-pair

• once all the ifgs for a specific date-pair are generated an S1-GUNW-AOI_TRACK product is produced

• to generate these products, facet on the S1-GUNW products and submit the following job:

  • Action: Standard Product S1-GUNW - S1-GUNW Completeness Evaluator [ARIA-561]

  • Queue: factotum-job_worker-large

  • Result: this job will look at an S1-GUNW and check if it “completes” the track for a given date-pair. If so S1-GUNW-AOI_TRACK product is generated. Otherwise, the evaluator silently completes without producing anything.

• note that these jobs are automatically submitted by the trigger rule s1gunw-on_demand-acqlist-completeness-evaluator and you should not normally need to submit them on-demand

• once an S1-GUNW-AOI_TRACK product is produced, the S1-GUNW products will be published to ASF and ARIA-products via the following pipeline: TODO: delivery pipeline