...
...
(WARNING: PGE is only compatible with python3, please make sure your docker container has it)
Github Repo and branch
Current Github repo: https://github.com/aria-jpl/ariamh
Current branch: s1-gunw-merged-aoi_track
...
To run the merged track stitcher you need to facet on S1-GUNW-MERGED-AOI_TRACK data products
(WARNING: The PGE is iterative so make sure you are faceting on a handful of products, not 1000+)
...
| Code Block | ||
|---|---|---|
| ||
{
"label" : "AOI Merged Track Stitcher",
"submission_type": "individual",
"recommended-queues" : [ "grfn-job_worker-small" ],
"allowed_accounts": [ "ops" ],
"params" : [
{
"name": "extra_products",
"from": "submitter",
"type": "text",
"default": "los.rdr.geo",
"placeholder": "space separated list of products to process, e.g. los.rdr.geo"
},
{
"name": "localize_products",
"from": "dataset_jpath:_source.metadata.s1-gunw_urls", <-- JSONPath to the localize URLS
"type": "text"
}
]
} |
...
Main Stitch Function
The inputs stored in json file ifg_stitch.json & read by the MergedTrackStitcher.stitch() method:
filenamesis ordered a array of rawGUNWfilesdirectionis hardcoded toalongoutnameis hardcoded tofilt_topophase.unw.geo
| Code Block | ||
|---|---|---|
| ||
{
"extra_products": [
"los.rdr.geo"
],
"filenames": [
[
"S1-GUNW-MERGED_RM_M1S2_TN014_20190320T152600-20190224T152533_s123-poeorb-11b7/merged/filt_topophase.unw.geo"
],
[
"S1-GUNW-MERGED_RM_M1S3_TN014_20190320T152625-20190224T152533_s123-poeorb-9dd9/merged/filt_topophase.unw.geo"
],
[
"S1-GUNW-MERGED_RM_M1S3_TN014_20190320T152650-20190224T152558_s123-poeorb-8c3e/merged/filt_topophase.unw.geo"
],
[
"S1-GUNW-MERGED_RM_M1S2_TN014_20190320T152714-20190224T152623_s123-poeorb-1ac7/merged/filt_topophase.unw.geo"
]
],
"direction": "along",
"outname": "filt_topophase.unw.geo"
} |
PGE Output:
| Code Block | ||
|---|---|---|
| ||
##########################################
Running AOI merged track stitcher sciflo: Thu May 23 22:50:14 UTC 2019
master_timestamp: 20190224T152533
slave_timestamp: 20190320T152714
curl -n -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/dem/usgs_mirror/SRTMSWBD.003/2000.02.11/N15E039.SRTMSWBD.raw.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 49138 100 49138 0 0 28548 0 0:00:01 0:00:01 --:--:-- 28568
.
.
.
curl -n -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/dem/usgs_mirror/SRTMSWBD.003/2000.02.11/N08E043.SRTMSWBD.raw.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12767 100 12767 0 0 57122 0 --:--:-- --:--:-- --:--:-- 56995
API open (R): /tmp/tmp731qgsuv
API close: /tmp/tmp731qgsuv
Writing geotrans to VRT for /tmp/tmp731qgsuv
stitch_sequence
stitch_pair
stitch_pair
stitch_pair
Writing geotrans to VRT for filt_topophase.unw.geo
Writing geotrans to VRT for filt_topophase.unw.conncomp.geo
Writing geotrans to VRT for phsig.cor.geo
Writing geotrans to VRT for los.rdr.geo
Finished running aoi merged track stitcher sciflo: Thu May 23 22:51:06 UTC 2019
status: 1 |
Files
...
the PGE writes out
| Code Block |
|---|
# Files in dataset directory size date 4729 Jun 5 14:28 file 853 May 24 20:19 los.rdr.geo.vrt 4625 May 24 20:19 los.rdr.geo.xml 535 May 24 20:19 phsig.cor.geo.vrt 4211 May 24 20:19 phsig.cor.geo.xmlS1-GUNW-MERGED_TN041_20180402T115755-20180414T120208-poeorb-3e9c.dataset.json 1879 Jun 5 14:28 S1-GUNW-MERGED_TN041_20180402T115755-20180414T120208-poeorb-3e9c.met.json 1495754 Jun 5 14:28 filt_topophase.unw.geo.browse.png 210444 Jun 5 14:28 filt_topophase.unw.geo.browse_small.png 480 Jun 5 14:32 merged/ # Files in merged/ directory 143286120 Jun 5 14:28 filt_topophase.unw.conncomp.geo 548 549Jun May 245 2014:1928 filt_topophase.unw.conncomp.geo.vrt 4387 4388Jun May 245 2014:1928 filt_topophase.unw.conncomp.geo.xml 1146288960 Jun 5 14:29 filt_topophase.unw.geo 875 874 Jun May 245 2014:1929 filt_topophase.unw.geo.vrt 4380 4381Jun May 245 2014:1929 filt_topophase.unw.geo.xml 268910712 May 24 20:19 1421 Jun 5 14:29 inputs.json 1146288960 Jun 5 14:31 los.rdr.geo 852 Jun 5 14:31 los.rdr.geo 134455356 May 24 20:19 phsig.cor.geo 268910712 May 24 20:19 filt_topophase.unw.geo **MAIN OUTPUT FILE** 33613839 May 24 20:19 filt_topophase.unw.conncomp.geo 8013600 May 24 20:19 dummy.out |
TODO:
write out the output files to create a dataset file
create
_dataset.jsonon completion (Reference code)create
dataset.met.jsonfile (Reference code)integrate the naming convention in the dataset
S1-GUNW-MERGED_TN<track>_<master end time>-<slave start time>-poeorb-XXXXAlready integrated code to get the
slave start timeandmaster end time, given the array oflocalize_urlsWrite all output files to dataset ID directory
S1-GUNW-MERGED_TN<track>_<master end time>-<slave start time>-poeorb-XXXX
Expected Output (In Progress)
...
.vrt
4624 Jun 5 14:31 los.rdr.geo.xml
573144480 Jun 5 14:32 phsig.cor.geo
534 Jun 5 14:32 phsig.cor.geo.vrt
4210 Jun 5 14:32 phsig.cor.geo.xml |
PNG file created
...
Publishing dataset
add entry into
./verdi/etc/datasets.json
| Code Block | ||
|---|---|---|
| ||
{
"ipath": "ariamh::data/S1-GUNW-MERGED",
"match_pattern": "/(?P<id>S1-GUNW-MERGED_.*?_TN.*?_.*?-(?P<year>\\d{4})(?P<month>\\d{2})(?P<day>\\d{2})(?P<rest>.+))$",
"alt_match_pattern": null,
"extractor": null,
"level": "l2",
"type": "interferogram",
"publish": {
"s3-profile-name": "default",
"location": "s3://s3-us-west-2.amazonaws.com:80/aria-ops-dataset-bucket/datasets/{type}/{version}/{year}/{month}/{day}/{id}",
"urls": [
"http://aria-ops-dataset-bucket.s3-website-us-west-2.amazonaws.com/datasets/{type}/{version}/{year}/{month}/{day}/{id}",
"s3://s3-us-west-2.amazonaws.com:80/aria-ops-dataset-bucket/datasets/{type}/{version}/{year}/{month}/{day}/{id}"
]
},
"browse": {
"location": "davs://pucops:puc_0ps@aria-dav.jpl.nasa.gov/browse/{type}/{version}/{year}/{month}/{day}/{id}",
"urls": [
"https://aria-dav.jpl.nasa.gov/browse/{type}/{version}/{year}/{month}/{day}/{id}"
]
}
} |