1. NetSpyGlass Agent

Contents

• NetSpyGlass Agent
  • Hardware Requirements
  • How to download the agent
  • Docker
  • Configuration
    • Configuration file agent.conf
      • Basic rules of the config file format
      • Configuration parameters
  • Running the Agent
  • fping
  • AutoUpgrades
  • Advanced Parameters
    • Agent Name
    • Region
    • IP Binding

1.1. Hardware Requirements

NetSpyGlass Agent will run on Linux. Any reasonably modern Linux distribution is suitable, as long as you can install Java 8 on it. We recommend Ubuntu 16.04LTS or CentOS. Prerequisite package on Ubuntu is openjdk-8-jre. On CentOS you’ll need java-1.8.0-openjdk-headless.

We recommend a server or a VM with at least 4G of RAM. Actual amount of RAM depends on the number of devices the agent is going to poll. Our preliminary estimate is about 2Mb of RAM per device on average, but this really depends on the number of metrics collected from the device.

NetSpyGlass Agent can utilize multiple CPU cores, so running on a machine with 4 or more cores improves performance.

The agent does not store lots of data on the local disk, mostly logs. The agent writes logs locally and rotates them to conserve disk space. The size of the log files depend on the number of devices served by the agent. Logs go to the /opt/nsg-agent/var/logs, we recommend that this directory be located on the file system with at least 10G of space.

NetSpyGlass agent process runs under an unprivileged account “nw2”; it does not need to run as root. The user nw2 is created by the rpm and deb packages automatically.

The agent communicates with NSG cluster in the cloud via Kafka and needs Kafka certificate and password. Both are bundled with the agent package so you do not need to copy any files to the machine where the agent runs. Because the package includes certificate and password which are unique for each customer, we generate unique agent packages for each customer.

Important

The agent requires sensitive information (like a certificate and credentials) to talk to the service. The downloaded package contains such secret information and should be treated as a secret.

1.2. How to download the agent

To download agent package, log in to your NSG service UI, then click “Help” at the bottom of the navigation bar on the left, then click “Getting Started”:

There are two ways to download and install the agent using the panel that appears when you click Getting Started:

Clicking button with downward arrow downloads agent package of corresponding type. Once download completes, just copy the file to the VM where you intend to run the agent and install it there. Since the package includes certificate and credentials used to access the service, and communication channels are configured on the server side, this is it, you do not need to edit any configuration files on the agent side. Just start the agent how you would normally do with the package type you use.

You can also click button “Install cmds” to get commands you can copy and paste on the shell prompt on the machine where you want to run the agent. These commands will download the package and install it for you.

1.3. Docker

If you prefer to run NSG agent on docker, you can use Docker image that can be downloaded using the same agent download page in NSG UI. To make this more convenient, simplify image management, and provide password protection for the image download, NSG servers in the cloud act as Docker registry. Click “install cmds” icon in the UI page to get commands to pull and run nsg agent docker image. You should be able to just copy these commands from the UI panel and paste them into shell prompt. These commands look like this:

echo <access_token> | sudo docker login cluster-registry.netspyglass.com --password-stdin
sudo docker run --restart always -d cluster-registry.netspyglass.com/nsg-agent:latest

Commands given by the download page will include actual unique access token to authenticate to the registry. Note that we generate unique token for each NSG cluster so one customer can not use theirs to download agent generated for another. Your NSG cluster url will appear in place of cluster-registry.netspyglass.com used in this example.

1.4. Configuration

The agent takes its configuration parameters on the command line and from the configuration file agent.conf stored with the NSG servers in the cloud. You can find this file in directory conf in the git repository where you keep server configuration files and Python apps. To change something in the agent configuration, such as add or update channel definition, just edit file conf/agent.conf, commit and push. Servers will pick up the change and push it to all agents automatically.

Note

Agent used to read its configuration from files on the machine where it runs. This is not the case anymore, nsg-agent v3.x pulls its configuration from servers in the cloud. You do not need to edit any files on the machine where the agent runs.

1.4.1. Configuration file agent.conf

Most of the configuration used by the agent comes from this file. The file is in “HOCON” format described here: https://github.com/typesafehub/config

1.4.1.1. Basic rules of the config file format

• Comments, with # or //
• Allow omitting the {} around a root object
• Allow = as a synonym for :
• Allow omitting the = or : before a { so you can use foo { a : 42 }
• Allow omitting commas as long as there’s a newline
• Allow trailing commas after last element in objects and arrays
• Allow unquoted strings for keys and values
• Unquoted keys can use dot-notation for nested objects, foo.bar=42 means foo { bar : 42 }
• Duplicate keys are allowed; later values override earlier, except for object-valued keys where the two objects are merged recursively
• include feature merges root object in another file into current object, so foo { include "bar.json" } merges keys in bar.json into the object foo
• include with no file extension includes any of .conf, .json, .properties
• you can include files, URLs, or classpath resources; use include url("http://example.com") or file() or classpath() syntax to force the type, or use just include "whatever" to have the library do what you probably mean (Note: url()/file()/classpath() syntax is not supported in Play/Akka 2.0.)
• substitutions foo : ${a.b} sets key foo to the same value as the b field in the a object
• substitutions concatenate into unquoted strings, foo : the quick ${colors.fox} jumped
• substitutions fall back to environment variables if they don’t resolve in the config itself, so ${HOME} would work as you expect. Also, most configs have system properties merged in so you could use ${user.home}.
• substitutions normally cause an error if unresolved, but there is a syntax ${?a.b} to permit them to be missing.
• += syntax to append elements to arrays, path += "/bin"
• multi-line strings with triple quotes as in Python or Scala

1.4.1.2. Configuration parameters

Agent configuration file agent.conf mostly holds definition of communication “channels”. Communication channels describe SNMP configurations used by the devices on your network. Each channel defines one SNMP v1, v2 or v3 configuration; there can be any number of channels of each type if you use different SNMP configurations on your network.

Each entry in the section channels is a dictionary with the following items:

• protocol: must be “snmp”
• version: SNMP version (1, 2, 3). If this parameter is missing, the version is assumed to be 2.
• community: used with versions 1 and 2. If the value of this parameter is a blank string (e.g. community = “”), devices using this SNMP configuration are not going to be discovered or polled but will appear in maps if L3 link to it could be constructed using information collected from another device
• user: used with SNMP v3
• password: corresponding password
• secLevel: This is a string that defines security level for SNMPv3. The following levels are recognized: noAuthNoPriv, authNoPriv, authPriv
• auth: authentication. MD5 and SHA are supported.
• privacy: privacy setting, DES, 3DES, AES128, AES192, AES256 are supported.

Here is an example of agent.conf configuration file that defines several SNMP channels of different types:

agent {
    configuration {

        network {
            channels = {

                v1public : {
                    protocol : snmp,
                    version : 1,
                    community : public
                },

                v2public : {
                    protocol : snmp,
                    version : 2,
                    community : public
                },

                v3aes128: {
                    protocol: snmp,
                    version: 3,
                    secLevel: authPriv,
                    auth: SHA,
                    privacy: AES128,
                    user: snmpuser,
                    password: v3SecretRedacted
                },
            }
        }
    }
}

You can add as many channels as you need, just make sure to use unique names.

Note

Limitations in SNMPv3 support in the current version of NetSpyGlass:

• only USM security model is supported
• in case security model authPriv is used (both authentication and privacy are turned on), the same password is used for both

1.5. Running the Agent

Once you have created and updated files agent.conf and SSL access configuration files, you can start the agent. To do this, use the usual command:

service nsg-agent start

Command:

service nsg-agent status

can be used to check of the agent is running and command:

service nsg-agent stop

can be used to stop it.

The following command can be used to update agent jar file:

service nsg-agent update

this command downloads new jar file but does not restart the agent if it is already running. Use this command to upgrade the agent manually when you run it with disabled automatic upgrades.

The agent writes its logs to the directory /opt/nsg-agent/var/logs. Watch logs agent.log, stdout and stderr in this directory when you start the agent, especially when you do it for the first time. All errors and warnings will be recorded in the logs.

Agent reads few configuration parameters from the file /etc/default/nsg-agent (see below).

1.6. fping

The agent uses external utility fping to ping devices to collect ICMP round trip time and packet loss statistics. Provided rpm and deb packages list fping as a dependency to make sure it is installed on the machine. We pre-install fping in the Docker image we ship.

1.7. AutoUpgrades

The rpm/deb package you use to install the agent does not really contain Agent code at all. The package installs a stub program that is based on getdown library ( https://github.com/threerings/getdown/wiki/Design ). When you start the agent, this program activates, connects to the download web site (it uses encrypted connection to do this), checks for the last available version of the agent, downloads and activates it. All files that it downloads are stored in the directory /opt/nsg-agent/var

Agent can upgrade itself if new version is available. There are two mechanisms to trigger the upgrade:

• you can simply restart the agent using command service nsg-agent restart. This stops the running agent, re-launches the stub program getdown, which repeats the process of checking if the upgrade is available, downloads it and starts it. Another way to do this is to use command service nsg-agent update, this command downloads new jar file but does not restart the agent if it is already running
• another way to upgrade the agent is by the command from the NSG server. We use this to upgrade agents remotely during our scheduled maintenance window whenever new agent has been released. NSG UI panel “System” has button “restart/update” that will issue this command to a selected agent.

You can disable ability to upgrade the agent remotely when you install the agent. To do this, add line to the file /etc/default/nsg-agent:

AUTO_UPDATE=0

This line will be added automatically if you set environment variable AUTO_UPDATE when you install the agent:

AUTO_UPDATE=0 apt-get install ./package.deb

The same method works for rpm package as well.

Use -e AUTO_UPDATE=0 if you run on docker:

sudo docker run --restart always -d -e NSG_NAME="unique-agent-name" -e AUTO_UPDATE=0 cluster-registry.netspyglass.com/nsg-agent:latest

1.8. Advanced Parameters

1.8.1. Agent Name

Each agent must have unique name, this is especially important in large clusters that run with many agents. Agent name appears in NSG UI panel “System”:

When you install the agent from rpm or deb package, the agent assumes the name of the host it is running on by default. If you want to change it, enter it in the file /etc/default/nsg-agent as follows:

$ cat /etc/default/nsg-agent
NSG_NAME="my-agent-name"
$

Agent running on docker will also use container host name as the agent name, which means it will change every time the container restarts. This can be inconvenient if you need to send some commands to the agent to execute. To set the name of the agent running on docker, pass environment variable NSG_NAME:

sudo docker run --restart always -d -e NSG_NAME="unique-agent-name" cluster-registry.netspyglass.com/nsg-agent:latest

1.8.2. Region

Large NSG clusters divide the network onto several regions to make sure nsg agents run “close” to the devices and to improve control over sets of devices monitored by each agent (see Regions). By default the agent belongs to the region “world”.

Add variable INITIAL_REGION to the file /etc/default/nsg-agent to define the region the agent belongs to:

$ cat /etc/default/nsg-agent
NSG_NAME="my-agent-name"
INITIAL_REGION="sjc"
$

When the agent runs on docker, it takes region name from the environment variable with the same name:

sudo docker run --restart always -d -e NSG_NAME="unique-agent-name" -e INITIAL_REGION=sjc \
    cluster-registry.netspyglass.com/nsg-agent:latest

1.8.3. IP Binding

NSG Agent sends SNMP and ICMP requests to monitored devices and by default, for SNMP and ICMP communication, agent binds to all network interfaces and lets the OS decide what source IP address to use with all packets it sends. Sometimes it is preferable to use specific source address. This can be done using BIND_ADDRESS environment variable.

Depending on NSG Agent deployment, there are two ways to define the BIND_ADDRESS environment variable:

• when agent was installed from Linux distribution package (.deb or .rpm), variable BIND_ADDRESS can be specified in the file /etc/default/nsg-agent:

  $ cat /etc/default/nsg-agent
  BIND_ADDRESS=198.51.100.1
  

• when agent was deployed as docker container, variable BIND_ADDRESS can be provided using docker run arguments:

  $ sudo docker run --restart always -d -e BIND_ADDRESS=198.51.100.1 -e NSG_NAME="unique-agent-name" -e INITIAL_REGION=sjc \
      cluster-registry.netspyglass.com/nsg-agent:latest