9.4. Syslog message

NSG Agent receives Syslog messages from TCP/UDP connections and with the help of Grok parser converts them into key-value data structure. Messages received this way are prepended with a text header as stated in the specifications RFC3164 and RFC5424. Depending on the specification and where message was originated headers contain different set of fields in different formats, but always include timestamp, priority, severity and facility.

Example of Syslog message sent over TCP/UDP

<13>May 18 11:22:43 172.16.1.10 Line protocol on Interface GigabitEthernet0/10, changed state to down

Grok configuration defined in syslogServer.processors describes how parser should read such headers that covers most of well known formats. This part of Syslog Server configuration must be considered as unchangeable and may be updated only along with the newer versions of NSG Agent or NetSpyGlass server. At the same time syslogServer includes configuration file grok.conf where user-defined Grok expressions and patterns live.

Example of grok.conf

grok.regex = [
    {
        name = "INTERFACENAME"
        expression = "[a-zA-Z][a-zA-Z0-9\\-./]+"
    }
    {
        name = "INTERFACESTATUS"
        expression = "(up|down)"
    }
]

grok.patterns = [
    "Native VLAN mismatch discovered on %{INTERFACENAME:logIfLocalName} \\(%{NONNEGINT}\\), with %{HOSTNAME} %{INTERFACENAME:logIfRemoteName} \\(%{NONNEGINT}\\)"
    "Line protocol on Interface %{INTERFACENAME:logIfLocalName}, changed state to %{INTERFACESTATUS:logIfLocalOperStatus}"
    "Interface %{INTERFACENAME:logIfLocalName}, changed state to (%{WORD} )?%{INTERFACESTATUS:logIfLocalOperStatus}"
]

First, Grok parser tries to parse message header and if no syntax errors found goes to the message body. There are many patterns of Syslog messages from different vendors already collected in the syslogServer.processors, so Grok parser applies them first and if no pattern matches the message it applies patterns from the grok.conf file.

The following code block shows fields Grok parser extracts from the message <13>May 18 11:22:43 172.16.1.10 Line protocol on Interface GigabitEthernet0/10, changed state to down if user-defined grok patterns are configured like in the example above.

{
    "logTimestamp": "2021-05-18T11:22:43.000Z",
    "logSource": "172.16.1.10",
    "logSyslogPriority": 13,
    "logSyslogFacilityCode": 1,
    "logSyslogFacilityName": "user",
    "logSyslogSeverityCode": 5,
    "logSyslogSeverityName": "notice",
    "logSyslogText": "<13>May 18 11:22:43 172.16.1.10 Line protocol on Interface GigabitEthernet0/10, changed state to down",
    "logText": "Line protocol on Interface GigabitEthernet0/10, changed state to down",
    "logIfLocalName": "GigabitEthernet0/10",
    "logIfLocalOperStatus": "down"
}

As you can see, there are fields that belong to the message header

  • logTimestamp - timestamp from the device that generated this message
  • logSource - this message was forwarded by rsyslog that’s why source device address can be found in the message, otherwise Syslog Server populates this field with remote address taken from the connection
  • logSyslogPriority - message priority
  • logSyslogFacilityCode - message facility code
  • logSyslogFacilityName - message facility code translated to text
  • logSyslogSeverityCode - message severity code
  • logSyslogSeverityName - message severity code translated to text
  • logSyslogText - original message

The following fields were extracted from the message body using pattern from grok.conf

  • logText - message body
  • logIfLocalName - field generated by Grok expression INTERFACENAME
  • logIfLocalOperStatus - field generated by Grok expression INTERFACESTATUS

Important

logSource field must present in every document produced by Grok parser. This field is used by NetSpyGlass server to identify monitored device that produced the message and append device information to the ES document that contains the message.

In addition to the fields found in the incoming Syslog messages, NSG Agent adds some extra fields.

  • agentName - text, name of the NSG Agent that received the message
  • agentRemotePort - integer, TCP port of the remote peer that sent the message
  • agentRemoteAddress - text, IP address of the remote peer that sent the message
  • agentLocalAddress - text, IP address on the NSG Agent where message was received
  • agentVersion - text, NSG Agent version
  • agentUuid - text, NSG Agent globally unique identifier
  • kafkaTopic - text, Kafka topic used to deliver message to NSG Server
  • kafkaKey - text, Kafka message key, must be equal to logSource
  • kafkaOffset - integer, offset in the Kafka topic used to deliver message to NSG Server
  • kafkaPartition - integer, Kafka partition ID
  • logSequence - integer, added to the message by log forwarder like rsyslog
  • logEventTag - mark message as recognized network event, there are the following events available ifDown - network operation interface change status to down and ifUp - network operation interface change status to up

Important

In order to distinguish fields extracted from Syslog message from those added by NSG Agent or NSG Server, all extracted fields must be prepended with prefix “log”.

Default user-defined fields declared in the grok.conf

  • logIfLocalIndex - integer, index of the local network interface
  • logIfLocalName - text, name of the local network interface
  • logIfRemoteName - text, name of the remote network interface
  • logIfLocalOperStatus - text, status of the operation local interface
  • logIfLocalAdminStatus - text, status of the administrative local interface

When NSG Server receives message from the agent it tries to associate the message with device that issued the message. NSG Server uses field logSource to search for the device by name, system name, or IP address. If device could be found, the following fields are added to the message.

  • deviceId - integer
  • physicalDevice - boolean
  • pingOnly - boolean
  • discoverySnmpStatus - boolean
  • discoveryPingStatus - boolean
  • isisOn - boolean
  • ospfAdminStat - integer
  • generation - integer
  • device - text
  • address - text
  • sysDescr - text
  • productName - text
  • chassisId - text
  • boxDescr - text
  • sysName - text
  • sysContact - text
  • sysLocation - text
  • shortName - text
  • isisSysId - text
  • ospfRouterId - text
  • discoveryStatus - text
  • signature - text
  • sysObjectID - text
  • lastDiscoveryAttemptTime - text
  • lastSuccessfulDiscoveryTime - text
  • channels - text

In case Grok parser has extracted field logIfLocalName from the message, NSG Server also adds interface-specific fields.

  • ifAddress - text
  • ifCompIndex - integer
  • ifRfc1213Index - integer
  • ifAlternateIndex - integer
  • ifAdminStatus - text
  • ifOperStatus - integer
  • ifHighSpeed - integer
  • ifSpeed - text
  • ifType - integer
  • ifName - text
  • ifSnmpName - text
  • ifDescription - text
  • ifMacAddress - text