.. _syslog_message: Syslog message ============== :ref:`NSG Agent` receives Syslog messages from TCP/UDP connections and with the help of :ref:`Grok parser` converts them into key-value data structure. Messages received this way are prepended with a text header as stated in the specifications RFC3164 and RFC5424. Depending on the specification and where message was originated headers contain different set of fields in different formats, but always include timestamp, priority, severity and facility. .. _syslog_intf_down: Example of Syslog message sent over TCP/UDP .. code-block:: none <13>May 18 11:22:43 172.16.1.10 Line protocol on Interface GigabitEthernet0/10, changed state to down Grok configuration defined in ``syslogServer.processors`` describes how parser should read such headers that covers most of well known formats. This part of :ref:`Syslog Server configuration` must be considered as unchangeable and may be updated only along with the newer versions of :ref:`NSG Agent` or NetSpyGlass server. At the same time ``syslogServer`` includes configuration file ``grok.conf`` where user-defined Grok expressions and patterns live. Example of ``grok.conf`` .. code-block:: none grok.regex = [ { name = "INTERFACENAME" expression = "[a-zA-Z][a-zA-Z0-9\\-./]+" } { name = "INTERFACESTATUS" expression = "(up|down)" } ] grok.patterns = [ "Native VLAN mismatch discovered on %{INTERFACENAME:logIfLocalName} \\(%{NONNEGINT}\\), with %{HOSTNAME} %{INTERFACENAME:logIfRemoteName} \\(%{NONNEGINT}\\)" "Line protocol on Interface %{INTERFACENAME:logIfLocalName}, changed state to %{INTERFACESTATUS:logIfLocalOperStatus}" "Interface %{INTERFACENAME:logIfLocalName}, changed state to (%{WORD} )?%{INTERFACESTATUS:logIfLocalOperStatus}" ] First, Grok parser tries to parse message header and if no syntax errors found goes to the message body. There are many patterns of Syslog messages from different vendors already collected in the ``syslogServer.processors``, so Grok parser applies them first and if no pattern matches the message it applies patterns from the ``grok.conf`` file. The following code block shows fields Grok parser extracts from the message ``<13>May 18 11:22:43 172.16.1.10 Line protocol on Interface GigabitEthernet0/10, changed state to down`` if user-defined grok patterns are configured like in the example above. .. code-block:: none { "logTimestamp": "2021-05-18T11:22:43.000Z", "logSource": "172.16.1.10", "logSyslogPriority": 13, "logSyslogFacilityCode": 1, "logSyslogFacilityName": "user", "logSyslogSeverityCode": 5, "logSyslogSeverityName": "notice", "logSyslogText": "<13>May 18 11:22:43 172.16.1.10 Line protocol on Interface GigabitEthernet0/10, changed state to down", "logText": "Line protocol on Interface GigabitEthernet0/10, changed state to down", "logIfLocalName": "GigabitEthernet0/10", "logIfLocalOperStatus": "down" } As you can see, there are fields that belong to the message header - ``logTimestamp`` - timestamp from the device that generated this message - ``logSource`` - this message was forwarded by rsyslog that's why source device address can be found in the message, otherwise Syslog Server populates this field with remote address taken from the connection - ``logSyslogPriority`` - message priority - ``logSyslogFacilityCode`` - message facility code - ``logSyslogFacilityName`` - message facility code translated to text - ``logSyslogSeverityCode`` - message severity code - ``logSyslogSeverityName`` - message severity code translated to text - ``logSyslogText`` - original message The following fields were extracted from the message body using pattern from ``grok.conf`` - ``logText`` - message body - ``logIfLocalName`` - field generated by Grok expression ``INTERFACENAME`` - ``logIfLocalOperStatus`` - field generated by Grok expression ``INTERFACESTATUS`` .. important:: **logSource** field must present in every document produced by Grok parser. This field is used by NetSpyGlass server to identify monitored device that produced the message and append device information to the ES document that contains the message. In addition to the fields found in the incoming Syslog messages, :ref:`NSG Agent` adds some extra fields. - ``agentName`` - text, name of the NSG Agent that received the message - ``agentRemotePort`` - integer, TCP port of the remote peer that sent the message - ``agentRemoteAddress`` - text, IP address of the remote peer that sent the message - ``agentLocalAddress`` - text, IP address on the NSG Agent where message was received - ``agentVersion`` - text, NSG Agent version - ``agentUuid`` - text, NSG Agent globally unique identifier - ``kafkaTopic`` - text, Kafka topic used to deliver message to NSG Server - ``kafkaKey`` - text, Kafka message key, must be equal to ``logSource`` - ``kafkaOffset`` - integer, offset in the Kafka topic used to deliver message to NSG Server - ``kafkaPartition`` - integer, Kafka partition ID - ``logSequence`` - integer, added to the message by log forwarder like ``rsyslog`` - ``logEventTag`` - mark message as recognized network event, there are the following events available ``ifDown`` - network operation interface change status to **down** and ``ifUp`` - network operation interface change status to **up** .. important:: In order to distinguish fields extracted from Syslog message from those added by NSG Agent or NSG Server, all extracted fields must be prepended with prefix "log". Default user-defined fields declared in the ``grok.conf`` - ``logIfLocalIndex`` - integer, index of the local network interface - ``logIfLocalName`` - text, name of the local network interface - ``logIfRemoteName`` - text, name of the remote network interface - ``logIfLocalOperStatus`` - text, status of the operation local interface - ``logIfLocalAdminStatus`` - text, status of the administrative local interface When NSG Server receives message from the agent it tries to associate the message with device that issued the message. NSG Server uses field ``logSource`` to search for the device by **name**, **system name**, or **IP address**. If device could be found, the following fields are added to the message. - ``deviceId`` - integer - ``physicalDevice`` - boolean - ``pingOnly`` - boolean - ``discoverySnmpStatus`` - boolean - ``discoveryPingStatus`` - boolean - ``isisOn`` - boolean - ``ospfAdminStat`` - integer - ``generation`` - integer - ``device`` - text - ``address`` - text - ``sysDescr`` - text - ``productName`` - text - ``chassisId`` - text - ``boxDescr`` - text - ``sysName`` - text - ``sysContact`` - text - ``sysLocation`` - text - ``shortName`` - text - ``isisSysId`` - text - ``ospfRouterId`` - text - ``discoveryStatus`` - text - ``signature`` - text - ``sysObjectID`` - text - ``lastDiscoveryAttemptTime`` - text - ``lastSuccessfulDiscoveryTime`` - text - ``channels`` - text In case Grok parser has extracted field ``logIfLocalName`` from the message, NSG Server also adds interface-specific fields. - ``ifAddress`` - text - ``ifCompIndex`` - integer - ``ifRfc1213Index`` - integer - ``ifAlternateIndex`` - integer - ``ifAdminStatus`` - text - ``ifOperStatus`` - integer - ``ifHighSpeed`` - integer - ``ifSpeed`` - text - ``ifType`` - integer - ``ifName`` - text - ``ifSnmpName`` - text - ``ifDescription`` - text - ``ifMacAddress`` - text