E2E On-Demand Pipeline Walkthrough

This is a step by step procedure for the bul`k processing of a new request

create a Request-S1GUNW and generate an estimate

• walk though the following procedure: Submit an On-Demand Estimate Request

• the API call will create a Request-S1GUNW object and submit hysds-io-request-s1gunw-acq_enumerator-location:develop enumeration jobs to the aria-standard_product-enumerator queue

• these enumeration jobs will generate the runconfig-acq-lists and runconfig-acqlist-audit_trails

• make sure all enumeration jobs complete successfully

• the number of runconfig-acq-lists generated (you can query by the Request-S1GUNW id in Tosca to see the count) are the expected number of S1-GUNWs to be generated

  • verify with the customer/scientist that the count looks okay before proceeding with the submission step of processing

call request-submit to trigger the rest of processing

• walk through the following procedure: Submit an On-Demand Submit Reuqest

• the API call will start by submitting a hysds-io-create_aoi:develop job in the factotum-create_aoi-queue for every item in the enumeration_list field in the corresponding Request-S1GUNW object

• the API will then submit individual data sling jobs for the SLCs referenced in the runconfig-acq-lists

  • job-spyddder-sling-extract-local-asf-singularity:ARIA-446_singularity in spyddder-sling-extract-local-asf or job-spyddder-sling-extract-local-scihub-singularity:ARIA-446_singularity in spyddder-sling-extract-local-scihub

  • note that these jobs stage the SLCs to the pleiades file system rather than in s3

  • the downloaded SLCs are registered in the S1-IW_SLC-local dataset

• as the SLCs are ingested, they will kick off the acqlist evaluator jobs via the trigger rule slc-local-request-s1gunw-acqlist_evaluator-develop

  • these jobs will generate a runconfig-topsapp object once all the SLCs in a given runconfig-acq-list are ingested

• the runconfig-topsapp objects will invoke one of the following trigger rules:

  

  • note that there are three distinct queues, each of which corresponds to the group account responsible for the request

  • if you ever need to submit a topsapp job manually, make sure the job parameters match those specified in the trigger rules (can click Edit in the trigger rules to view the parameterization of a particular trigger rule)