/
Bulk Processing and Keep-Up Ops Overview

Bulk Processing and Keep-Up Ops Overview

Owned by Marjorie Lucas
Last updated: Jan 24, 2020
3 min read

Scientists who submit AOI’s will generally want GUNW products to be generated over the complete temporal span of Sentinel acquisitions and continue to generate products as new acquisitions are ingested. This page outlines the steps to perform the bulk processing and set up the trigger rules for future product generation.

Ingest AOI

First thing to do is to create/ingest the AOI.

  1. Select the Create Area of Interest PGE and fill in the PGE parameters.

    • type: leave as monitoring

    • starttime: the time when the earliest acquisition for that area of interest was acquired (generally around 10/2014)

    • endttime: the time until which keep-up/automated product generation will continue

 

 

2. Submit job then confirm the AOI was ingested.

Create Trigger Rules

Create the trigger rules that will continue processing products as new acquisitions are ingested.

  1. Click on Trigger Rules on the Tosca front page.

  2. Select the Standard Product S1-GUNW - aoi_track_acquisition_enumerator PGE and fill in the parameters as needed. Here are the more common processing requests. Be sure to double check what type of processing is requested.

    • Annual n-1 processing:

      • minMatch: 1

      • skipDays: 365

    • n-3 processing:

      • minMatch: 3

      • skipDays: 0

    • n-2 processing:

      • minMatch: 2

      • skipDays: 0

One trigger rule will have to be submitted for each of the processing types.

 

 

Check Acquisition & IPF Status

Before bulk processing is triggered, make sure acquisitions and IPF’s are caught up.

  1. Facet on the AOI, select the AOI based submission of IPF scraper jobs PGE, then submit the job.

    1. Alternative: IPF ASF Scraper for a given acquisition or IPF ASF Scraper for a given acquisition

  2. Facet on the AOI, select the AOI based submission of acquisition scraper jobs PGE, then submit the job.

    Both of these jobs will submit individual jobs to ingest any missing acquisitions and IPF’s. Once it looks like all of these jobs have completed, check to make sure no acquisitions and IPF’s are missing.

  3. Log into C-factotum

  4. Navigate to /home/ops/verdi/ops/ops_scripts/check_acquisitions.

  5. Run the acquisition completeness script.
    ex. ./check_acquisition_completeness.py --aoi AOI_monitoring_houston_D143 --track 143
    The script will output the ID’s to any missing acquisitions.

  6. Run the IPF completeness script.
    ex. ./check_ipf_completeness.py --aoi AOI_monitoring_houston_D143 --track 143
    The script will output the ID’s to any missing IPF’s.

    If there are any missing acquisitions or IPF’s, rerun steps 1 & 2.

Submit Bulk Processing Jobs

Now that everything is set-up, submit the bulk processing jobs. This is the same process outlined under Create Trigger Rules except these jobs will be submitted as On-Demand jobs instead of Trigger Rules.

  1. Click the On-Demand button on the Tosca front page.

  2. Follow step 2 under Create Trigger Rules.

Generate AOI Ops Report

Once it looks like most of the bulk processing is complete, generate the AOI Ops report to see if there are any missing products.

  1. Facet on the AOI and select the Standard Product S1-GUNW AOI Ops Report PGE.

  2. Submit the job. When the job completes, open the excel sheet and check to see what GUNW’s are missing and try to work backwards to generate them.

Reporting Processing Status

Create a new page under Monitoring AOIs and title it after the AOI ID. There should be a template called AOI-Monitoring that you can use to create the page.

Related content