9.3. Configuration

Syslog Server configuration is part of the NSG Agent configuration and can be found at syslogServer section in the file agent.conf stored with the NSG servers in the cloud. This file is located in directory conf in the git repository where all server configuration files and Python apps are kept. To change something in the Syslog Server configuration, such as server port or read timeout, just edit file conf/agent.conf, commit and push. Servers will pick up the change and push it to all agents automatically.

Syslog Server configuration contains two kinds of parameters, one defines network settings like interface and port server binds to and receives syslog messages from. Another one defines Grok parser settings. Grok parser is the process inside NSG Agent responsible for real-time conversion of incoming syslog messages into structured data (Json documents) that eventually be published to ElasticSearch by NetSpyGlass server.

Below is an example of syslogServer configuration

syslogServer {
    # Transport layer
    host = "0.0.0.0"
    port = 5514
    timeoutReadMillis = 20000
    timeoutConnectMillis = 20000
    messageMaxLength = 65535

    # Grok parser
    include "grok.conf"
    timeAuthority = agent

    # Internal
    # Changes to parameters below affect the robustness of the Syslog Server
    # and expected to be done by NetSpyGlass personal only
    # The content of processor parameter is omitted
    processors = []
    messageFieldNameToKafkaKey = "logSource"
}

Configuration begins with definition of networking transport layer parameters

  • host: network interface server binds to, if 0.0.0.0 is specified, server listens all interfaces
  • port: TCP/UDP port server listens to
  • timeoutReadMillis: server read timeout in milliseconds
  • timeoutConnectMillis: server connection timeout in milliseconds
  • messageMaxLength: maximum length of syslog message allowed, server drops messages longer than messageMaxLength

and ends with Grok parser configuration.

  • include "grok.conf": instruction to load custom (user-defined) Grok patterns and expressions from configuration file grok.conf
  • timeAuthority: source of timestamp for ES document containing the syslog message. There are two options available, agent - the system time of the machine running the agent and syslog - time extracted from the syslog message header
  • messageFieldNameToKafkaKey: (internal) ES document field name that contains identification of the device issued the syslog message
  • processors: (internal) Grok parser configuration

Important

The processors section of Grok parser configuration must be considered as unchangeable and may be updated only along with the newer versions of NSG Agent or NetSpyGlass server.

Grok parser is configured with built-in patterns defined in processors section and custom patterns that can be modified or extended by users. Custom or user-defined patterns are stored in grok.conf configuration file that can be found next to agent.conf in directory conf in the git repository where all server configuration files and Python apps are kept.

Below is the initial content of grok.conf

grok.regex = [
    {
        name = "INTERFACENAME"
        expression = "[a-zA-Z][a-zA-Z0-9\\-./]+"
    }
    {
        name = "INTERFACESTATUS"
        expression = "(up|down)"
    }
]

grok.patterns = [
    "Native VLAN mismatch discovered on %{INTERFACENAME:logIfLocalName} \\(%{NONNEGINT}\\), with %{HOSTNAME} %{INTERFACENAME:logIfRemoteName} \\(%{NONNEGINT}\\)"
    "Line protocol on Interface %{INTERFACENAME:logIfLocalName}, changed state to %{INTERFACESTATUS:logIfLocalOperStatus}"
    "Interface %{INTERFACENAME:logIfLocalName}, changed state to (%{WORD} )?%{INTERFACESTATUS:logIfLocalOperStatus}"
]

There are only two sections in this file. One is grok.regex, this is a collection of custom Grok expressions that can be defined in addition to the built-in Grok expressions. Every object of this collection must have expression’s name name and the regular expression in the field expression. Those custom expressions along with built-in can be used in user-defined Grok patterns that are collected in grok.patterns list.