How to Update Aria Packages in ARIA/HySDS

Table of Content

1 Overview
2 Update ARIA Packages with Fabric Command
3 Adding ARIA Packages to update_aria_packages

Overview

"Update ARIA Packages" is the adaptation part of a HySDS install and consists of all the adaptation-specific code. This guide will teach you how the ARIA/HySDS system updates ARIA packages for PCM components (like Verdi and Factotum) via the Fabric command and show how new ARIA packages can be added to ARIA package update process.

Update ARIA Packages with Fabric Command

Figure 1: ARIA Package Update Flow Chart

Figure 1 shows a simplified workflow of how ARIA packages are updated from Mozart to workers such as Factotum and Verdi. The flow chart starts with Mozart’s ARIA packages which are git repos cloned into ~/mozart/ops/ . Next, the we run fab command -f ~/.sds/cluster.py –R <role> update_aria_packages on Mozart which uses the fabfile cluster.py. The fab command runs an update_aria_packages function in cluster.py (Figure 2) which uses roles assigned with the -R flag to determine which PCM components (i.e. Mozart, Factotum, and Verdi) will be updated with the ARIA packages from Mozart. The update_aria_packages function will remove the selected packages from the PCM components ops directory and then copy the selected packages from Mozart’s ops directory (i.e. ~/mozart/ops/) to the PCM components ops directory (i.e. ~/verdi/ops/).

from sdscli.adapters.hysds.fabfile import *


#####################################
# add custom fabric functions below
#####################################

def test():
    """Test fabric function."""

    run('whoami')


def update_aria_packages():
    """Update verdi and factotum with ARIA packages."""

    role, hysds_dir, hostname = resolve_role()

    if role == 'mozart':
        hysds_dir = 'verdi'

    # update common packages between verdi and factotum
    if role in ('mozart', 'verdi', 'factotum'):
        rm_rf('%s/ops/ariamh' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/ariamh',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/tropmap' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/tropmap',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/apihub' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/apihub',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/scihub' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/scihub',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/asf' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/asf',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/unavco' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/unavco',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/scihub_acquisition_scraper' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/scihub_acquisition_scraper',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")

    # update packages for factotum
    if role == 'factotum':
        rm_rf('%s/ops/qquery' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/qquery',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('~/dataset-AOI')
        rsync_project('~/', '~/mozart/ops/dataset-AOI',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('~/aoi-converter')
        rsync_project('~/', '~/mozart/ops/aoi-converter',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/scihub_acquisition_scraper' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/scihub_acquisition_scraper',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/bos_sarcat_scraper' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/bos_sarcat_scraper',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/mkarim' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/mkarim',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/usgs_neic' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/usgs_neic',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/update_aoi' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/update_aoi',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        #run("~/aoi-converter/mcrinstaller.py ~/mcr")
        rm_rf('%s/ops/s1_qc_ingest' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/s1_qc_ingest',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        #rm_rf('%s/ops/aria-pdl-clone' % hysds_dir) #don't delete data, storage and log directories
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/aria-pdl-clone',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
        rm_rf('%s/ops/aoi-converter' % hysds_dir)
        rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/aoi-converter',
                      ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")
    # update ariamh settings
    rm_rf('%s/etc/settings.conf' % hysds_dir)
    send_template('ariamh_settings.conf', '%s/etc/settings.conf' % hysds_dir)

    # update spyddder-man settings
    rm_rf('%s/etc/settings.json' % hysds_dir)
    cp_rp('%s/ops/spyddder-man/settings.json' % hysds_dir, '%s/etc/settings.json' % hysds_dir)                                                                                                                89,1          Bot

Figure 2: Cluster.py with update_aria_packages function (line 14).

Adding ARIA Packages to update_aria_packages

To add new packages to the update_aria _packages function, we first need to git clone the Github repos into Mozart’s ops directory:

cd ~/mozart/ops/
git clone <http_to_github_repo>

Next, we need to append the cluster.py file in ~/.sds/ with the new ARIA packages. In cluster.py, find the update_aria_packages command (Figure 2, Line 14). Then depending on the PCM component we want to add the package to, we append the block of code under the line if role == <role> with the following commands:

rm_rf('%s/ops/<__NEW_ARIA_PACKAGE__>' % hysds_dir)
rsync_project('%s/ops/' % hysds_dir, '~/mozart/ops/<__NEW_ARIA_PACKAGE__>',
              ssh_opts="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no")

The commands rm_rf and rsync_project come from fabfile.py, which are found in Mozart’s ~/mozart/ops/sdscli/sdscli/adapters/hysds/fabfile.py directory. The rm_rf command will remove the __NEW_ARIA_PACKAGE__ from the PCM component if it exists in the ops directory. The rsync_project command will copy the __NEW_ARIA_PACKAGE__ from Mozart’s ops directory (i.e. ~/mozart/ops/) to the PCM components ops directory (i.e. ~/verdi/ops/).

Once the code has been added to cluster.py, we can finally run the update process using the fab command based on the <role> specified: