...
...
...
...
...
...
...
...
...
| Table of Contents |
|---|
Related Github Repos and tickets
https://github.com/aria-jpl/topsstack
Versions
Tickets:
Jira Legacy server System JIRA serverId 88de5227-42b1-365b-8364-d731c8efaf35 key ARIA-47
Job Runtime
Depends on how many SLCs are being processed
1+ hours for 8 SLCs
GNU Parallel
...
c5d.9xlarge
(36 vCPU, 72 GiB)
...
c5.24xlarge
(96 vCPU, 192 GiB)
...
1 year (~30 SLCS, 4 bursts)
...
7 hrs, 24 mins, 46 secs
...
4 hrs, 38 mins, 33 secs
...
2 year (~60 SLCS, 4 bursts)
...
...
8 hrs, 16 min, 46 secs
Multiprocessing
...
c5d.9xlarge
(36 vCPU, 72 GiB)
...
c5.24xlarge
(96 vCPU, 192 GiB)
...
1 year (~30 SLCS, 4 bursts)
...
6 hrs, 43 mins, 27 secs
...
4 hrs, 19 mins, 30 secs
...
2 year (~60 SLCS, 4 bursts)
...
12 hrs, 58 mins, 32 secs
...
8 hrs, 30 mins, 43 secs
...
3 year (~90 SLCS, 4 bursts)
...
18 hrs, 14 mins, 44 secs
...
10 hrs, 56 mins, 5 secs
Objective
Creating a stack of SLCs
...
Code Block raise Exception('Could not determine a suitable burst offset')There must only be one track in your SLC inputs
correct facet SLC inputs incorrect facet SLC inputs
...
Job Inputs:
Bbox (*required)
min_latmax_latmin_lonmax_lon
...
Link: http://b-ci.grfn.hysds.io:8080/job/ops-bcluster_container-builder_aria-jpl_topsstack_master/
WARNING: If rebuilding on the same branch (master), make sure to remove docker image so that it reloads when restarting the job:
docker rmi <topsStack docker image id>
HySDS-
...
IO and Jobspec-
...
IO
hysds-io.json.topsstack
| Code Block |
|---|
{
"label": "topsStack Processor",
"submission_type": "individual",
"allowed_accounts": [ "ops" ],
"action-type": "both",
"params": [
{
"name": "min_lat",
"from": "submitter",
"type": "number",
"optional": false
},
{
"name": "max_lat",
"from": "submitter",
"type": "number",
"optional": false
},
{
"name": "min_lon",
"from": "submitter",
"type": "number",
"optional": false
},
{
"name": "max_lon",
"from": "submitter",
"type": "number",
"optional": false
},
{
"name":"localize_products",
"from":"dataset_jpath:",
"type":"text",
"lambda" : "lambda met: get_partial_products(met['_id'],get_best_url(met['_source']['urls']),[met['_id']+'.zip'])"
}
]
}
|
...
| Code Block |
|---|
{
"ipath": "ariamh::data/STACK",
"match_pattern": "/(?P<id>coregistered_slcs-(?P<year>\\d{4})(?P<month>\\d{2})(?P<day>\\d{2})(?P<time>\\d{6}).+)$",
"alt_match_pattern": null,
"extractor": null,
"level": "NA",
"type": "stack",
"publish": {
"s3-profile-name": "default",
"location": "s3://s3-us-west-2.amazonaws.com:80/##BUCKET##/datasets/{type}/{version}/{year}/{month}/{day}/{id}",
"urls": [
"http://##WEBDAV_URL##/datasets/{type}/{version}/{year}/{month}/{day}/{id}",
"s3://##S3_URL##:80/##BUCKET##/datasets/{type}/{version}/{year}/{month}/{day}/{id}"
]
},
"browse": {
"location": "davs://##WEBDAV_USER##@##WEBDAV USER##/browse/{type}/{version}/{year}/{month}/{day}/{id}",
"urls": [
"https://##WEBDAV##/browse/{type}/{version}/{year}/{month}/{day}/{id}"
],
}
} |
...
ASG (Auto Scaling Group)
...
Configurations
EC2 Instance Type | c5d.9xlarge |
...
Not enough CPU
Takes 11.5 Hrs to run 30 scenes
...
May need to upgrade to c5d.18xlarge or i-instances
...
STILL TODO:
...
c5.24xlarge | ||
Block Devices | /dev/sdc
/dev/sda1
/dev/sdb
| /dev/sdc
/dev/sda1
/dev/sdb
|
Spot Price | 1.728 | 2.57 |
...
Job Runtime
Depends on how many SLCs are being processed
1+ hours for 8 SLCs
GNU Parallel
c5d.9xlarge (36 vCPU, 72 GiB) | c5.24xlarge (96 vCPU, 192 GiB) | |
1 year (~30 SLCS, 4 bursts) | 7 hrs, 24 mins, 46 secs | 4 hrs, 38 mins, 33 secs |
2 year (~60 SLCS, 4 bursts) | 13 hrs, 37 mins, 39 secs | 8 hrs, 16 min, 46 secs |
Multiprocessing
c5d.9xlarge (36 vCPU, 72 GiB) | c5.24xlarge (96 vCPU, 192 GiB) | |
1 year (~30 SLCS, 4 bursts) | 6 hrs, 43 mins, 27 secs | 4 hrs, 19 mins, 30 secs |
2 year (~60 SLCS, 4 bursts) | 12 hrs, 58 mins, 32 secs | 8 hrs, 30 mins, 43 secs |
3 year (~90 SLCS, 4 bursts) | 18 hrs, 14 mins, 44 secs | 10 hrs, 56 mins, 5 secs |