...
Mozart Trigger Rules for handling common failures for S1-GUNW processing pipeline
...
Mozart trigger rule: retry-s1gunw evaluator could not find record with orbit file
Condition: evaluator failed due to “Couldn't find record with orbit file"
...
Mozart trigger rule: retry-s1gunw-evaluator-Failed to resolve SLC IDs for all acquisition IDs
Condition: evaluator failed due to “Failed to resolve SLC IDs for all acquisition IDs”
Mozart trigger rule: retry-s1gunw-blacklisting
...
Condition: all failed topsApp jobs that has retry count of 3
...
Rule Name | Condition | Action |
|---|---|---|
s1gunw topsapp blacklist failed | all failed job-standard_product-s1gunw-topsapp job types whose retry count has reached 5 times | hysds-io-blacklist_s1-gunw_from_job:release-20190610 |
s1gunw topsapp greylist failed no suitable burst offset | hysds-io-greylist_s1-gunw_from_job:release-20190610 | |
s1gunw topsapp greylist failed no swath with burst overlap |
...
If set required_retry_count to 0, then will always run.
Since we constrain on retry_count=3 now, it will only get triggered for failed jobs of retry count 3.
...
hysds-io-greylist_s1-gunw_from_job:release-20190610 | |
s1gunw evaluator retry failed to find precise orbit |
...
hysds-io-lw-mozart-retry:v0.0.5 |
Mozart trigger
...
Condition: all failed enumerator with orbit file not found.
...
Example:
RuntimeError: Failed because orbit file : None NOT FOUND
...
rules for standard product pipeline in AWS that matches on job type: job-standard_product-s1gunw-topsapp:develop
standard product s1gunw topsapp blacklist failed
standard product s1gunw topsapp greylist failed no suitable burst offset
standard product s1gunw topsapp greylist failed no swath with burst overlap
standard product s1gunw evaluator retry failed to find precise orbit
Mozart trigger rules for request on-demand on Pleiades that matches on job type: job-request-s1gunw-topsapp-local-singularity:ARIA-446_singularity
request ondemand s1gunw topsapp blacklist failed
request ondemand s1gunw topsapp greylist failed no suitable burst offset
request ondemand s1gunw topsapp greylist failed no swath with burst overlap
request ondemand s1gunw evaluator retry failed to find precise orbit