17. Integration with Kentik

NetSpyGlass v2.1 can be integrated with netflow collection and analytics service by Kentik Technologies ( https://www.kentik.com/ ). Current implementation of the integration allows NetSpyGlass to run queries against Kentik to get netflow-related data for the devices and interfaces it monitors. This data appears in the pop-up inspector window that opens when user clicks an intefrace in the network map or Graphing Workbench.

_images/kentik_integration_screenshot_1.png

Since devices are added to NetSpyGlass and Kentik separately and it is not guaranteed that these two systems should have the same set of devices or that the device that exist in both have the same name, NetSpyGlass uses device’s management IP address to match them.

To configure the integration, add the following segment to your main nw2.conf configuration file:

# The following section is used to describe integrations with third-party
# systems and services that are not used as alert notification streams or
# the time series databases.
#
# In most cases, server restart is not needed when you make changes to
# the parameters in this section.
#
integrations {

    kentikEval {

        type = kentik

        # set this to true to enable Kentik integration
        enable = false

        # access can be 'proxy' or 'direct'. At this time only 'proxy' is supported
        access = proxy

        # authentication data for the API calls.
        auth {
            email = "user@domain.suffix"
            token = "your_api_token"
        }
    }
}

Note

At this time, integration only with one Kentik account is allowed. We expect to lift this restriction in the future.

Put your Kentik authentication credentials into the section auth and set the value of the parameter enable to true. NetSpyGlass server restart is required when you add Kentik integration for the first time, however subsequent changes to the configuration parameters do not require server restart. This includes changes to the value of the parameter enable that can be used to enable and disable the integration without server restart.

Once integration is configured and turned on, watch server log for errors. If everything works correctly, you should see log records that look similar to these:

2017-07-18 18:16:12,546 INFO  pool-59-thread-1                   [n.h.n.i.k.KentikConnector]: Kentik device:  "13365" "latch" "10.0.15.42"
2017-07-18 18:16:12,555 INFO  pool-59-thread-1                   [n.h.n.i.k.KentikConnector]: Kentik device:  "13342" "ex2200" "10.0.15.20"
2017-07-18 18:16:12,569 INFO  ForkJoinPool.commonPool-worker-30  [n.h.n.i.k.KentikConnector]: All Kentik devices saved or updated: 2 devices
2017-07-18 18:16:12,559 INFO  pool-59-thread-1                   [n.h.n.i.k.KentikConnector]: Reading Kentik query template '/opt/netspyglass/home/etc/kentik/queries/proto_by_avg.json'

this means NetSpyGlass was able to find two devices it monitors in Kentik. Integration with Kentik will be activated after the next network discovery.

Now if you click an interface of a device that exists both in NetSpyGlass and Kentik in a network map, the popup inspector window that appears should have tab “Kentik”.

Possible reasons why the tab “Kentik” may be missing in the inspector:

  • this device was not added to Kentik or we could not find it there using its management IP address
  • the device was found but tags have not been assigned yet. Run network discovery in NetSpyGlass or wait till the next scheduled discovery.

Tab “Kentik” in the inspector window can be used to run several preconfigured queries against Kentik service. These queries are configured to match the device and the interface for which the inspector has been opened.

Preconfigured Kentik queries are stored in files in the directory /opt/netspyglass/home/etc/kentik/queries/ on the server where NetSpyGlass primary server runs. The package installs several queries “out of the box” but you can add your own.

Here is an example of one query:

{
  "queries": [
    {
      "query": {
        "query_title": "Top Flows by Avg Bits/s",
        "dimension_column_name": "topflowsip",
        "dimension": [
          "TopFlowsIP"
        ],
        "cidr": 32,
        "cidr6": 128,
        "metric": "in_bytes",
        "topx": $show_top,
        "depth": 75,
        "fastData": "Auto",
        "outsort": "avg bit/sec",
        "lookback_seconds": $lookback,
        "hostname_lookup": false,
        "starting_time": "",
        "ending_time": "",
        "device_name": "$KentikDeviceName",
        "all_selected": false,
        "filters_obj": {
          "connector": "All",
          "filterGroups": [
            {
              "connector": "All",
              "filters": [
                {
                  "filterField": "$direction",
                  "operator": "=",
                  "filterValue": "$ifIndex"
                }
              ],
              "not": false
            }
          ]
        },
        "descriptor": "",
        "aggregates": [
          {
            "name": "avg bit/sec",
            "column": "f_sum_in_bytes",
            "fn": "average",
            "leftOperand": "",
            "raw": false
          }
        ]
      },
      "bucket": "Table",
      "bucketIndex": 0
    }
  ]
}

This is a slightly modifed copy of the query you can copy from Kentik Data Explorer with the following changes:

  • key query.query_title has been added
  • key query.dimension_column_name has been added
  • some fields have been replaced with macros that have format $name.

The value of the key query.query_title appears in the drop-down list Query in the NetSpyGlass UI; this is how you choose the query when you use it.

The value of the key dimension_column_name is used to find the column with the query key values in the response received from Kentik.

Here is the list of recognized macros:

  • $KentikDeviceName the name of the device in the form recognized by Kentik
  • $ifIndex interface ifIndex value
  • $show_top the value is taken from the UI control “Show top”
  • $lookback the value is taken from the UI control “Lookback time”
  • $direction direction of the traffic flow, the value comes from the UI control “Direction of traffic”

Whenever you make a change in the tab “Kentik”, NetSpyGlass server reads query template from the file, replaces macros with corresponding values and makes Kentik API call to get the data. Once the call returns, the data appears in NetSpyGlass UI.

To add your own query, download it in JSON format from Kentik Data Explorer or API Tester, add keys query.query_title and query.dimension_column_name , replace values with corresponding macros per the list above and finally save the file in the directory /opt/netspyglass/home/etc/kentik/queries/ on the server. The server should notice that something has changed there and try to reload its configuration and read Kentik query templates from the files in this directory. It will log errors if it discovers that some query template files do not parse. Log records similar to these indicate normal operation:

2017-07-18 18:16:12,587 INFO  pool-59-thread-1 [n.h.n.i.k.KentikConnector]: Reading Kentik query template '/opt/netspyglass/home/etc/kentik/queries/port_src_by_avg.json'
2017-07-18 18:16:12,587 INFO  pool-59-thread-1 [      n.h.n.f.FileWatcher]: Watching file /opt/netspyglass/home/etc/kentik/queries/port_src_by_avg.json for changes; event description: kentik
2017-07-18 18:16:12,588 INFO  pool-59-thread-1 [n.h.n.i.k.KentikConnector]: Reading Kentik query template '/opt/netspyglass/home/etc/kentik/queries/top_flows_by_avg.json'
2017-07-18 18:16:12,591 INFO  pool-59-thread-1 [      n.h.n.f.FileWatcher]: Watching file /opt/netspyglass/home/etc/kentik/queries/top_flows_by_avg.json for changes; event description: kentik