9.5. Syslog analysis¶
After Syslog messages are collected by NSG Agent, they get converted
into key-value structures by Grok parser.
In addition to the fields extracted from inbound message NSG Agent appends field logSource.
It is set to the address of the device issued the message if the message was received directly from the device
so that the source IP address of the corresponding TCP/UDP packet could be used. If the message arrived via
syslog forwarder (e.g. rsyslogd), device host name or address is extracted by a specialized built-in Grok
parser and inserted into the same field logSource.
Once message conversion is complete, NSG Agent serializes it into Json document and sends to NetSpyGlass server for further processing.
For every incoming message server tries to find the device that sent it using field logSource for identification.
If device is found, than essential device fields are added to the inbound Json document.
Similarly, if original message mentions network interface name and Grok parser
extracted this information into the field logIfLocalName, the server tries to find that interface and adds
its fields to the Json document. In the end, server sends resulting Json to the ElasticSearch for indexing.
There are two time-based ElasticSearch indexes where Syslog messages can be found ${network.name}-syslog-long and
${network.name}-syslog-short ( ${network.name} is the name of the cluster). Examples shown below assume
network.name = labdcdev. NetSpyGlass server sends all messages to index ${network.name}-syslog-short and
duplicates some of them to ${network.name}-syslog-long if syslog facility value is auth, authpriv or security.
The difference between these indexes is in retention policy. ${network.name}-syslog-long rolls over index after 18
months or if it exceeds 120Gb of space limit. ${network.name}-syslog-short rolls over index after 45 days or if it
exceeds 120Gb of space limit.
Several tools are available for analysis of syslog messages stored in the ElasticSearch: Kibana, NSGQL or ES API.
9.5.1. Kibana¶
Kibana is the best tool for visually exploring the ingested messages, checking their structure and searching for the particular device, interface or message.
NetSpyGlass comes with pre-built syslog indexes and index pattern ${network.name}-syslog which can be found in
the “Discover” view in Kibana.
9.5.2. NsgQL¶
Another powerful tool that can be used to search events and build alerts based on Syslog messages is NSGQL.
NsgQL queries access data only in the index ${network.name}-syslog-short.
To build NsgQL queries that access syslog messages use table syslog:
$ nsgql "select device from syslog limit 10"
The good starting point for exploring the syslog data is to get familiar with fields available in the table syslog.
$ nsgql describe syslog
----------------------------+------------+------------
columnName | columnType | description
----------------------------+------------+------------
timestamp | STRING |
logSource | STRING |
logSyslogFacilityCode | INTEGER |
logSyslogFacilityName | STRING |
logSyslogPriority | INTEGER |
logSyslogSeverityCode | INTEGER |
logSyslogSeverityName | STRING |
logSyslogText | STRING |
logText | STRING |
...
List of predefined fields is available at Syslog Message section. Example below demonstrates how to obtain most recent 10 events for the last 5 minutes when message was generated by “auth” facility.
$ nsgql "select device, logText from syslog where logSyslogFacilityName='auth' and timestamp between 'now-5m' and now order by timestamp desc limit 10 "
--------+-----------------------------------------------------------------------------------------------------------
device | logText
--------+-----------------------------------------------------------------------------------------------------------
ex2200 | SNMPD_AUTH_FAILURE: nsa_log_community: unauthorized SNMP community from 10.7.25.171 to 10.7.25.11 (public)
ex2200 | last message repeated 125 times
gw-wrdo | SSHD_LOGIN_FAILED: Login failed for user 'root' from host '10.0.1.18'
gw-wrdo | LIBJNX_LOGIN_ACCOUNT_LOCKED: Account for user 'root' has been locked out from logins
gw-wrdo | Failed password for root from 127.0.0.1 port 38426 ssh2
ex2200 | last message repeated 125 times
ex2200 | last message repeated 125 times
carrier | Failed password for root from 123.456.22.11 port 36812 ssh2
ex2200 | SNMPD_AUTH_FAILURE: nsa_log_community: unauthorized SNMP community from 10.7.25.171 to 10.3.25.13 (public)
ex2200 | SNMPD_AUTH_FAILURE: nsa_log_community: unauthorized SNMP community from 10.7.25.171 to 10.3.25.13 (public)
--------+-----------------------------------------------------------------------------------------------------------
Count: 10, served by: labdcdev-nsg-api-1, processing time: 0.918 sec; query id: 1637686633408
9.5.3. API Call¶
If there is a need of another kind of automation, ElasticSearch API is also available.
curl -s "${ELASTICSEARCH}/labdcdev-syslog-short/_doc/_search?size=1&q=logText:*mismatch" | jq .
{
"took": 1006,
"timed_out": false,
"_shards": {
"total": 6,
"successful": 6,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 10000,
"relation": "gte"
},
"max_score": 1,
"hits": [
{
"_index": "labdcdev-syslog-short-000001",
"_type": "_doc",
"_id": "9361691678227663",
"_score": 1,
"_source": {
"logSequence": "8746039",
"relayTimestamp": "Nov 25 12:32:58",
"ciscoTag": "CDP-4-NATIVE_VLAN_MISMATCH",
"logSyslogSeverityCode": 4,
"deviceId": 5,
"productName": "catalyst3560G48TS",
"sysDescr": "Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version 12.2(35)SE1, RELEASE SOFTWARE (fc1)\r\nCopyright (c) 1986-2006 by Cisco Systems, Inc.\r\nCompiled Tue 19-Dec-06 10:54 by antonino",
"chassisId": "FOC1034Y09Z",
"SoftwareRev": [
"12.2(35)SE1"
],
"logSyslogFacilityName": "local7",
"sysName": "c3560g-1",
"OSPFArea": [
"0"
],
"ciscoSeverityLevel": "4",
"VlanId": [
"0",
"22",
"1",
"100",
"18"
],
"syslog5424Pri": "188",
"kafkaKey": "10.0.15.228",
"generation": 830,
"lastDiscoveryAttemptTime": "2021-11-24T20:02:50.554Z",
"agentRemotePort": 45694,
"ospfAdminStat": 1,
"@metadata": {
"pipeline": "filebeat-7.10.0-labdcdev-agent-syslog-pipeline"
},
"logSyslogFacilityCode": 23,
"index": "labdcdev-syslog-short",
"agentName": "carrier",
"agentRemoteAddress": "127.0.0.1",
"ciscoFacility": "CDP",
"discoveryPingStatus": true,
"Role": [
"Switch",
"Router",
"iBgpPeer"
],
"channels": [
"v2public"
],
"SerialNumber": [
"FOC1034Y09Z"
],
"device": "c3560g-1",
"physicalDevice": true,
"discoverySnmpStatus": true,
"kafkaTopic": "nsg.labdcdev.syslog",
"signature": "FOC1034Y09Z",
"pingOnly": false,
"logIfRemoteName": "GigabitEthernet0/10",
"discoveryStatus": "ds_complete",
"boxDescr": "WS-C3560G-48TS",
"logText": "Native VLAN mismatch discovered on GigabitEthernet0/1 (1), with c3560g-1 GigabitEthernet0/10 (100).",
"logTimestamp": "2021-11-25T20:32:58.770Z",
"VlanName": [
"VLAN0018",
"default",
"VLAN0022",
"VLAN0100"
],
"kafkaPartition": 19,
"ciscoMnemonic": "NATIVE_VLAN_MISMATCH",
"Protocol": [
"BGP4",
"OSPF"
],
"timestamp": "Apr 27 17:47:38",
"isisOn": false,
"address": "10.0.15.228",
"logSource": "10.0.15.228",
"ospfRouterId": "10.0.22.228",
"sysObjectID": "1.3.6.1.4.1.9.1.617",
"Vendor": [
"Cisco"
],
"logSyslogPriority": 188,
"BGP4LocalAS": [
"AS65001"
],
"logSyslogSeverityName": "warning",
"lastSuccessfulDiscoveryTime": "2021-11-24T20:02:50.554Z",
"logSyslogText": "<188>Nov 25 12:32:58 10.0.15.228 8746039: *Apr 27 17:47:38: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet0/1 (1), with c3560g-1 GigabitEthernet0/10 (100).",
"logIfLocalName": "GigabitEthernet0/1",
"Model": [
"catalyst3560G48TS"
],
"agentVersion": "4.1.0-1-b997-feature-NET-6004-feature/NET-6004-b9971",
"agentLocalAddress": "172.19.0.1",
"kafkaOffset": 563762,
"agentUuid": "32c53926-06f6-11ec-ad0a-67e36d14b268"
}
}
]
}
}