Packaging & Installing A Third-Party App On Arista Devices Using EOS Extensions

For a project I am working on, I have a requirement to install a third-party application (aka. an “agent”) on an Arista EOS device. The details of the inner-workings of this agent application are beyond the scope of this blog post, but at a high level, the agent is a Golang binary that runs a series of performance measurements across a fleet of Arista devices and reports the results back to an analytics platform.

Rather than launch the application in a Docker container running in EOS shell, the preference is to run the application as a native binary executable in EOS shell. In addition, the application should read its runtime configuration from a properties file. Finally, the app’s status of being installed & running should survive a reboot. So, where to begin?

As it turns out, Arista provides a very convenient feature called EOS Extensions for handling this type of use case. "EOS" stands for "Extensible Operating System", and much of the extensibility that EOS is built on is facilitated through the use of EOS Extensions (e.g. Arista deploys security patches using this mechanism).

In a nutshell, an EOS Extension is pre-packaged application code that is in an RPM (RPM Package Manager) or SWIX (Software Image Extension) format. With EOS Extensions, we simply have to load our application's RPM package into a specific partition on the Arista device and install it using a single CLI command.

This blog post summarizes my notes in packaging a pre-compiled Golang binary (the "agent") in RPM format, and installing it on an Arista device using EOS Extensions.

Generating A Tarball For The App

The first thing we need to do is create a tarball for the application. For my particular use case, I just have to bundle two artifacts into the tarball. Both of these artifacts are in a directory called agent-1.0.0:

1. A Golang binary file for the agent application (agent-binary in the output below)
2. An external properties file used to seed some runtime parameter in the Golang app (agent-properties.conf in the output below)

$ tree agent-1.0.0/
agent-1.0.0/
├── agent-binary
└── agent-properties.conf

To create the tarball, issue the following command from outside the agent directory, which will create a tarball named agent-1.0.0.tar.gz:

$ tar -czvf agent-1.0.0.tar.gz agent-1.0.0/

Packaging The Third-Party App As An RPM

Now that we have bundled our application artifacts into a tarball, we are ready to create an RPM package for it. One convenient way to do this is with the rpmbuild utility. If you’re on an RPM-based system (e.g. RHEL, Centos), you can install it using the following:

$ sudo yum install rpm-build

On a Debian-based system (e.g. Ubuntu), you can install it by using the following:

$ sudo apt install rpm

Next, we have to create some top-level default directories in our workspace which rpmbuild will use during the build process. The 6 main directories that we need are:

1. BUILD: the location where the application is unpacked and built
2. BUILDROOT: a staging area that looks like the final installation directory
3. RPMS: where the newly created binary package files are written
4. SOURCES: where the original source tarball is placed
5. SPECS: where the .spec file for the RPM package is placed
6. SRPMS: where the newly created source package files are written

To create these top-level directories, issue the following command:

$ mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}

This will create the following directory structure in your home directory:

$ tree
.
└── rpmbuild
    ├── BUILD
    ├── BUILDROOT
    ├── RPMS
    ├── SOURCES
    ├── SPECS
    └── SRPMS
$

With the above directory created, there a couple of preliminary things we need to do before we can build the RPM package:

1. Copy the application tarball (agent-1.0.0.tar.gz) to the SOURCES directory
2. Create a .spec file (agent-1.0.0.spec) and save it in the SPECS directory

First, let’s copy the application tarball over to the SOURCES directory:

$ cp agent-1.0.0.tar.gz rpmbuild/SOURCES/

Next, let's create our agent-1.0.0.spec file in the SPECS directory. The SPEC file tells rpmbuild how to build and package the application, and specifies any steps to perform during and after the installation. The SPEC file written for this use case is shown below (comments have been added where appropriate to explain the contents of the file):

Name: agentVersion: 1.0.0
Release: 1%{?dist}
Summary: Agent App
Group: EOS/Extension
License: Selector Software License
SOURCE0 : %{name}-%{version}.tar.gz
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root

%description
%{summary}

# Commands necessary to prepare for the build
%prep
# Extract the files from the tarball
%setup -q

%build

%install
# Commands to install the application
# In this step, we copy all the files from the tarball to their appropriate location
# (1) 'agent-binary' gets copied to '/usr/local/bin'
# (2) 'agent-properties.conf' gets copied to '/etc'
mkdir -p $RPM_BUILD_ROOT/usr/local/bin
mkdir -p $RPM_BUILD_ROOT/etc
cp agent-binary $RPM_BUILD_ROOT/usr/local/bin
cp agent-properties.conf $RPM_BUILD_ROOT/etc/

%clean

%files
# Set the default attributes for the files
# Set permissions mode to 755 and use 'root' for the default user id and group id
%defattr(755,root,root,-)
# Add the list of files to be installed 
/usr/local/bin/agent-binary
/etc/agent-properties.conf

%post
# We could run the agent binary here, as commented out below, but we are going
# to run the agent using the Arista CLI 'daemon' config command instead.
# /usr/local/bin/agent-binary -c /etc/agent-properties.conf

At this point, our directory structure looks like the following, where we have our tarball (agent-1.0.0.tar.gz) in the SOURCES directory, and the our SPEC file (agent-1.0.0.spec) in the SPECS directory:

$ tree rpmbuild
rpmbuild
├── BUILD
├── BUILDROOT
├── RPMS
├── SOURCES
│   └── agent-1.0.0.tar.gz
├── SPECS
│   └── agent-1.0.0.spec
└── SRPMS

6 directories, 2 files
$

We are now ready to to use the above SPEC file to run rpmbuild and create the RPM package for our application.  To do this, run the following command (be sure to point to your SPEC file):

$ rpmbuild -bb SPECS/agent-1.0.0.spec

Executing(%prep): /bin/sh -e /home/jag/rpmbuild/tmp/rpm-tmp.atswAE
+ umask 022
+ cd /home/jag/rpmbuild/BUILD
+ cd /home/jag/rpmbuild/BUILD
+ rm -rf agent-1.0.0
+ /bin/gzip -dc /home/jag/rpmbuild/SOURCES/agent-1.0.0.tar.gz
+ /bin/tar -xof -
+ STATUS=0
+ [ 0 -ne 0 ]
+ cd agent-1.0.0
+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ exit 0
Executing(%build): /bin/sh -e /home/jag/rpmbuild/tmp/rpm-tmp.8NaqSC
+ umask 022
+ cd /home/jag/rpmbuild/BUILD
+ cd agent-1.0.0
+ exit 0
Executing(%install): /bin/sh -e /home/jag/rpmbuild/tmp/rpm-tmp.2JrcbB
+ umask 022
+ cd /home/jag/rpmbuild/BUILD
+ cd agent-1.0.0
+ mkdir -p /home/jag/rpmbuild/BUILDROOT/agent-1.0.0-1.x86_64/usr/local/bin
+ mkdir -p /home/jag/rpmbuild/BUILDROOT/agent-1.0.0-1.x86_64/etc
+ cp agent /home/jag/rpmbuild/BUILDROOT/agent-1.0.0-1.x86_64/usr/local/bin
+ cp agent-comcast-dell.conf /home/jag/rpmbuild/BUILDROOT/agent-1.0.0-1.x86_64/etc/
+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/brp-strip /usr/bin/strip
+ /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip
+ /usr/lib/rpm/brp-strip-comment-note /usr/bin/strip /usr/bin/objdump
Processing files: agent-1.0.0-1.x86_64
warning: Missing build-id in /home/jag/rpmbuild/BUILDROOT/agent-1.0.0-1.x86_64/usr/local/bin/agent
Provides: agent = 1.0.0-1 agent(x86-64) = 1.0.0-1
Requires(interp): /bin/sh
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires(post): /bin/sh
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/jag/rpmbuild/BUILDROOT/agent-1.0.0-1.x86_64
Wrote: /home/jag/rpmbuild/RPMS/x86_64/agent-1.0.0-1.x86_64.rpm
Executing(%clean): /bin/sh -e /home/jag/rpmbuild/tmp/rpm-tmp.4yfypH
+ umask 022
+ cd /home/jag/rpmbuild/BUILD
+ cd agent-1.0.0
+ exit 0

And that’s it!  After running the above command, our directory structure looks like the output shown below, where the following new artifacts were generated:

1. The newly generated RPM binary package (agent-1.0.0-1.x8664.rpm) in directory RPMS/x86_64
2. The contents of the application tarball unpackaged under directory BUILD
3. The staging of what the target installation will look like under directory BUILDROOT

$ tree rpmbuild
rpmbuild
├── BUILD
│   └── agent-1.0.0
│       ├── agent-binary
│       └── agent-properties.conf
├── BUILDROOT
│   └── agent-1.0.0-1.x86_64
│       ├── etc
│       │   └── agent-properties.conf
│       └── usr
│           └── local
│               └── bin
│                   └── agent-binary
├── RPMS
│   └── x86_64
│       └── agent-1.0.0-1.x86_64.rpm
├── SOURCES
│   └── agent-1.0.0.tar.gz
├── SPECS
│   └── agent-1.0.0.spec
├── SRPMS
└── tmp
    └── rpm-tmp.4yfypH

14 directories, 8 files
$

Installing The App On Arista As An EOS Extension

With the RPM package built, we are now ready to install our agent application on the Arista device as an EOS Extension.  The first thing we need to do is transfer our RPM package to the device.  It is important to note that the RPM must be copied over to the device partition named extension: before it can be installed as an EOS Extension.

To install an EOS Extension on an Arista device, be sure to first transfer your application RPM to the extension: partition on the device.

Transferring the RPM to the extension: partition is a two-part task:

1. SCP the RPM over to /mnt/flash/ on the Arista device
2. From EOS, copy the RPM from flash: to extension:

For the first task, in the following command, we use SCP to copy over the application RPM to partition /mnt/flash/:

$ scp RPMS/x86_64/agent-1.0.0-1.x86_64.rpm admin@<ARISTA-IP-ADDRESS>:/mnt/flash/
Password: 
agent-1.0.0-1.x86_64.rpm                      100% 4515KB   3.0MB/s   00:01    
$

Logging onto the Arista device, we can verify the successful transfer by looking at the contents of the /mnt/flash/ drive using the following EOS command.  We see, as highlighted below, that our RPM has been successfully transferred to the Arista device.

$ ssh admin@<ARISTA-IP-ADDRESS>
Password: 
Last login: Thu Jul 2 14:37:35 2020 from 10.162.0.2
ARISTA1#dir flash:
Directory of flash:/
   ...
   -rw- 4623286 Jul 2 15:02 agent-1.0.0-1.x86_64.rpm
   ...
4093313024 bytes total (3179036672 bytes free)
ARISTA1#

For the second task, from the EOS CLI, we copy the RPM package over from the flash: partition to the extension: partition.

ARISTA1#copy flash:agent-1.0.0-1.x86_64.rpm extension:
Copy completed successfully.
ARISTA1#

We can subsequently verify the transfer by using the following command from EOS to view the contents of the extension: partition:

ARISTA1#dir extension:
Directory of extension:/$ 

  -rw- 4623286 Jul 2 15:16 agent-1.0.0-1.x86_64.rpm

4093313024 bytes total (3174412288 bytes free)
ARISTA1#

If we issue the show extensions command in EOS, we see the following output below.  Here, we see the RPM listed as an available extension under the “Name” column.  But, note the “A, NI” flags under the “Status” column, which indicate that the extension is “available” but “not installed”.

ARISTA1#show extensions
Name                            Version/Release      Status      Extension
------------------------------- -------------------- ----------- ---------
agent-1.0.0-1.x86_64.rpm        1.0.0/1              A, NI       1 


A: available | NA: not available | I: installed | NI: not installed | F: forced
S: valid signature | NS: invalid signature
ARISTA1#

To install the EOS Extension, we simply issue the following command (extension agent-1.0.0-1.x86_64), as shown below.  When we re-issue the show extensions command, we now see under the “Status” column that the “NI” flag has changed to “I”, which means that the EOS Extension is now installed.

ARISTA1#extension agent-1.0.0-1.x86_64.rpm
ARISTA1#
ARISTA1#show extensions
Name                            Version/Release      Status      Extension
------------------------------- -------------------- ----------- ---------
agent-1.0.0-1.x86_64.rpm        1.0.0/1              A, I        1 


A: available | NA: not available | I: installed | NI: not installed | F: forced
S: valid signature | NS: invalid signature
ARISTA1#

NOTE: The EOS Extension installed above will not persist (survive) a device reboot. If you try to reboot, you will see the “I” flag change to “NI” in the “Status” column of the show extensions command. To persist the extension installation across reboots, be sure to copy it into the boot-extensions partition.

To copy the EOS Extension to the boot-extensions partition, run the following command:

ARISTA1#copy installed-extensions boot-extensions 
Copy completed successfully.
ARISTA1#

We can subsequently verify that the EOS Extension has been enable for boot persistence using the following command:

ARISTA1#show boot-extensions 
s2agent-1.0.0-1.x86_64.rpm
ARISTA1#

Running The EOS Extension As A Daemon

Arista provides a very powerful capability whereby you can daemonize a third-party user script or EOS Extension using the daemon configuration stanza.  This allows the application to run as a background process independently of the shell.  Furthermore, if the application exits for some reason, EOS will automatically restart it.  Finally, this behaviour will persist across reboots.

The Arista EOS CLI configuration snippet below shows how we can run our agent EOS Extension as a daemon:

ARISTA1#show running-config | begin daemon
daemon s2agent
    exec /usr/local/bin/agent-binary -c /etc/agent-properties.conf
    no shutdown
!

Let’s confirm that the process is indeed running by running the shell command shown below.  Here, we can see that the agent process (PID 4935) is indeed running.

ARISTA1#bash ps -ef | grep s2agent | grep -v grep
root 4935 1913 1 20:08 ? 00:00:26 /usr/local/bin/agent-binary -c /etc/agent-properties.conf
ARISTA1#

NOTE: If you make any change to the daemon config, the changes won’t take effect until you issue a shutdown, followed by a no shutdown on the daemon config.

Now, as a test of agent restartability using the daemon config, let’s terminate the agent by killing the process with PID 4935.  As shown below, we see that EOS automatically relaunches the agent again (new PID 5370).

ARISTA1#bash sudo kill 4935
ARISTA1#bash ps -ef | grep s2agent | grep -v grep
root 5370 1913 2 20:36 ? 00:00:00 /usr/local/bin/agent-binary -c /etc/agent-properties.conf
ARISTA1#

Finally, to test agent restartability after a reboot, let’s reload the Arista device.  As shown below, we see that the agent process is automatically restarted (new PID 2863).

ARISTA1#reload
! Image does not support next image compatibility checks. Running only checks from the current image.
Proceed with reload? [confirm]

Broadcast message from root@ARISTA1 (Mon Jul 6 20:45:45 2020):

The system is going down for reboot NOW!
Connection to <ARISTA-IP-ADDRESS> closed by remote host.
Connection to <ARISTA-IP-ADDRESS> closed.


        ...[ SSH BACK INTO DEVICE AFTER IT REBOOTS]...


jag@jag-dev:~$ ssh admin@<ARISTA-IP-ADDRESS>
Password: 
ARISTA1#bash ps -ef | grep s2agent | grep -v grep
root 2863 1916 1 20:50 ? 00:00:27 /usr/local/bin/agent-binary -c /etc/agent-properties.conf
ARISTA1# 

And there you have it: in this blog post we accomplished the following:

• Packaged the tarball for our application as an RPM
• Installed the RPM as an EOS Extension on an Arista device
• Launched our application as an automatically-restarting daemon using the Arista EOS CLI

-- Jag --