.. _syslog_analysis: Syslog analysis =============== After Syslog messages are collected by :ref:`NSG Agent`, they get converted into :ref:`key-value structures` by :ref:`Grok parser`. In addition to the fields extracted from inbound message :ref:`NSG Agent` appends field ``logSource``. It is set to the address of the device issued the message if the message was received directly from the device so that the source IP address of the corresponding TCP/UDP packet could be used. If the message arrived via syslog forwarder (e.g. `rsyslogd`), device host name or address is extracted by a specialized built-in Grok parser and inserted into the same field ``logSource``. Once message conversion is complete, :ref:`NSG Agent` serializes it into Json document and sends to NetSpyGlass server for further processing. For every incoming message server tries to find the device that sent it using field ``logSource`` for identification. If device is found, than essential device fields are added to the inbound Json document. Similarly, if original message mentions network interface name and :ref:`Grok parser` extracted this information into the field ``logIfLocalName``, the server tries to find that interface and adds its fields to the Json document. In the end, server sends resulting Json to the ElasticSearch for indexing. There are two time-based ElasticSearch indexes where Syslog messages can be found ``${network.name}-syslog-long`` and ``${network.name}-syslog-short`` ( ``${network.name}`` is the name of the cluster). Examples shown below assume ``network.name = labdcdev``. NetSpyGlass server sends all messages to index ``${network.name}-syslog-short`` and duplicates some of them to ``${network.name}-syslog-long`` if syslog facility value is **auth**, **authpriv** or **security**. The difference between these indexes is in retention policy. ``${network.name}-syslog-long`` rolls over index after 18 months or if it exceeds 120Gb of space limit. ``${network.name}-syslog-short`` rolls over index after 45 days or if it exceeds 120Gb of space limit. Several tools are available for analysis of syslog messages stored in the ElasticSearch: Kibana, :ref:`NSGQL` or ES API. Kibana ------ Kibana is the best tool for visually exploring the ingested messages, checking their structure and searching for the particular device, interface or message. NetSpyGlass comes with pre-built syslog indexes and index pattern ``${network.name}-syslog`` which can be found in the "Discover" view in Kibana. .. image:: ../images/syslog-kibana.png NsgQL ----- Another powerful tool that can be used to search events and build :ref:`alerts` based on Syslog messages is :ref:`NSGQL`. NsgQL queries access data only in the index ``${network.name}-syslog-short``. To build NsgQL queries that access syslog messages use table ``syslog``: .. code-block:: bash $ nsgql "select device from syslog limit 10" The good starting point for exploring the syslog data is to get familiar with fields available in the table ``syslog``. .. code-block:: bash $ nsgql describe syslog ----------------------------+------------+------------ columnName | columnType | description ----------------------------+------------+------------ timestamp | STRING | logSource | STRING | logSyslogFacilityCode | INTEGER | logSyslogFacilityName | STRING | logSyslogPriority | INTEGER | logSyslogSeverityCode | INTEGER | logSyslogSeverityName | STRING | logSyslogText | STRING | logText | STRING | ... List of predefined fields is available at :ref:`Syslog Message` section. Example below demonstrates how to obtain most recent 10 events for the last 5 minutes when message was generated by "auth" facility. .. code-block:: bash $ nsgql "select device, logText from syslog where logSyslogFacilityName='auth' and timestamp between 'now-5m' and now order by timestamp desc limit 10 " --------+----------------------------------------------------------------------------------------------------------- device | logText --------+----------------------------------------------------------------------------------------------------------- ex2200 | SNMPD_AUTH_FAILURE: nsa_log_community: unauthorized SNMP community from 10.7.25.171 to 10.7.25.11 (public) ex2200 | last message repeated 125 times gw-wrdo | SSHD_LOGIN_FAILED: Login failed for user 'root' from host '10.0.1.18' gw-wrdo | LIBJNX_LOGIN_ACCOUNT_LOCKED: Account for user 'root' has been locked out from logins gw-wrdo | Failed password for root from 127.0.0.1 port 38426 ssh2 ex2200 | last message repeated 125 times ex2200 | last message repeated 125 times carrier | Failed password for root from 123.456.22.11 port 36812 ssh2 ex2200 | SNMPD_AUTH_FAILURE: nsa_log_community: unauthorized SNMP community from 10.7.25.171 to 10.3.25.13 (public) ex2200 | SNMPD_AUTH_FAILURE: nsa_log_community: unauthorized SNMP community from 10.7.25.171 to 10.3.25.13 (public) --------+----------------------------------------------------------------------------------------------------------- Count: 10, served by: labdcdev-nsg-api-1, processing time: 0.918 sec; query id: 1637686633408 API Call -------- If there is a need of another kind of automation, ElasticSearch API is also available. .. code-block:: bash curl -s "${ELASTICSEARCH}/labdcdev-syslog-short/_doc/_search?size=1&q=logText:*mismatch" | jq . .. code-block:: json { "took": 1006, "timed_out": false, "_shards": { "total": 6, "successful": 6, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 10000, "relation": "gte" }, "max_score": 1, "hits": [ { "_index": "labdcdev-syslog-short-000001", "_type": "_doc", "_id": "9361691678227663", "_score": 1, "_source": { "logSequence": "8746039", "relayTimestamp": "Nov 25 12:32:58", "ciscoTag": "CDP-4-NATIVE_VLAN_MISMATCH", "logSyslogSeverityCode": 4, "deviceId": 5, "productName": "catalyst3560G48TS", "sysDescr": "Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version 12.2(35)SE1, RELEASE SOFTWARE (fc1)\r\nCopyright (c) 1986-2006 by Cisco Systems, Inc.\r\nCompiled Tue 19-Dec-06 10:54 by antonino", "chassisId": "FOC1034Y09Z", "SoftwareRev": [ "12.2(35)SE1" ], "logSyslogFacilityName": "local7", "sysName": "c3560g-1", "OSPFArea": [ "0" ], "ciscoSeverityLevel": "4", "VlanId": [ "0", "22", "1", "100", "18" ], "syslog5424Pri": "188", "kafkaKey": "10.0.15.228", "generation": 830, "lastDiscoveryAttemptTime": "2021-11-24T20:02:50.554Z", "agentRemotePort": 45694, "ospfAdminStat": 1, "@metadata": { "pipeline": "filebeat-7.10.0-labdcdev-agent-syslog-pipeline" }, "logSyslogFacilityCode": 23, "index": "labdcdev-syslog-short", "agentName": "carrier", "agentRemoteAddress": "127.0.0.1", "ciscoFacility": "CDP", "discoveryPingStatus": true, "Role": [ "Switch", "Router", "iBgpPeer" ], "channels": [ "v2public" ], "SerialNumber": [ "FOC1034Y09Z" ], "device": "c3560g-1", "physicalDevice": true, "discoverySnmpStatus": true, "kafkaTopic": "nsg.labdcdev.syslog", "signature": "FOC1034Y09Z", "pingOnly": false, "logIfRemoteName": "GigabitEthernet0/10", "discoveryStatus": "ds_complete", "boxDescr": "WS-C3560G-48TS", "logText": "Native VLAN mismatch discovered on GigabitEthernet0/1 (1), with c3560g-1 GigabitEthernet0/10 (100).", "logTimestamp": "2021-11-25T20:32:58.770Z", "VlanName": [ "VLAN0018", "default", "VLAN0022", "VLAN0100" ], "kafkaPartition": 19, "ciscoMnemonic": "NATIVE_VLAN_MISMATCH", "Protocol": [ "BGP4", "OSPF" ], "timestamp": "Apr 27 17:47:38", "isisOn": false, "address": "10.0.15.228", "logSource": "10.0.15.228", "ospfRouterId": "10.0.22.228", "sysObjectID": "1.3.6.1.4.1.9.1.617", "Vendor": [ "Cisco" ], "logSyslogPriority": 188, "BGP4LocalAS": [ "AS65001" ], "logSyslogSeverityName": "warning", "lastSuccessfulDiscoveryTime": "2021-11-24T20:02:50.554Z", "logSyslogText": "<188>Nov 25 12:32:58 10.0.15.228 8746039: *Apr 27 17:47:38: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet0/1 (1), with c3560g-1 GigabitEthernet0/10 (100).", "logIfLocalName": "GigabitEthernet0/1", "Model": [ "catalyst3560G48TS" ], "agentVersion": "4.1.0-1-b997-feature-NET-6004-feature/NET-6004-b9971", "agentLocalAddress": "172.19.0.1", "kafkaOffset": 563762, "agentUuid": "32c53926-06f6-11ec-ad0a-67e36d14b268" } } ] } }