Upstream Ancillary Ingest is the first subsystem in the processing pipeline. It manages keeping the SDS archive upto date with:

...

• Our pipeline is triggered by the ancillaries

• SDS archive should always be in sync with SciHub

• Failure to do so results in failure in processing pipeline

• Advantage - At time of urgent response, we need not depend on uptime of external service as we have our own local metadata archive (GRQ acquisition index).

I presented the necessity of the Ancillary Ingest at ESIP Summer Meeting 2019. Here are the slides:

...

Related Github Repos and tickets

Repos

https://github.com/aria-jpl/scihub_acquisition_scraper

https://github.com/aria-jpl/s1_qc_ingest

Tickets

https://jira.jpl.nasa.gov/browse/ARIA-10

...

https://aria.atlassian.net/browse/ARIA-29

Job Runtime

Objective

• Upstream ancillary ingestion of SAR observations for rapid and large-scale analysis

How to set up the inputs

Job Inputs:

CI Integration (Jenkins)

HySDS-io and Jobspec-io

hysds-io.json.<pge>

...

job-spec.json.<pge>

...

Job Outputs

Main file that gets executed is <script>

• <steps>

Output directory structure

<screen shot>

Output structure of merged/

...