.. _tags: Tags **** Tags in NetSpyGlass are organized in "facets". Facets represent certain characteristics or aspects of devices, interfaces or hardware components. For example, devices can be classified by their role, such as "Switch", "Router" or "Load Balancer". Tags that belong to facet "Role" describe this characterisation. Other tag facets can be used to classify devices and components in many different ways. It is possible to add many tags that belong to the same facet. NetSpyGlass groups facets internally to separate tags that belong to a device from tags that belong to its interfaces. This way, the system can display only essential tags in different parts of the UI to avoid clutter. NetSpyGlass server defines set of tag facets it uses to attach information about devices, network interfaces, hardware components, protocols and firewall counters to objects created during network discovery. Some of these facets are explained below. You can add your own tags using tag selector Python hook script (see :ref:`user_defined_tags`). Standard Tag Facets =================== Tags that describe device ------------------------- **Vendor** : vendor name, 'Cisco', 'Juniper' etc **Role** : device role, one of the following: - Unknown - SimulatedNode - SimulatedBridge - Router - Switch - Server - LoadBalancer - WirelessClient - WirelessAP - Firewall - eBgpPeer - iBgpPeer **Protocol** : protocols this device runs. Possible values are: - Unknown - Stp_Off - UnknownStp - Stp - Sstp ( 802.1q ) - Rstp ( 802.1W ) - PVST - PvstPlus - RapidPvst - Mistp - MistpPvstPlus - Mst ( IEEE 802.1s Multiple Spanning Tree (MST) - VPLS - OSPF - BGP4 **OSPFArea** : OSPF areas this device is a member of, as numbers **BGP4LocalAS** : Local AS **BGP4Peer** : this facet holds two kinds of tags: AS numbers (strings such as "AS174") and BGP peer name (e.g. "myrouter1") Tags that describe interfaces ----------------------------- **ifRole** : interface role. Possible values: - Unknown - RouterInterface - UntaggedSwtichPort - TaggedSwtichPort - OutOfBandManagement - VlanInterface - VirtualInterface - BroadcastTypeInterface - LoopbackInterface - SimulatedInterface - WLan - Internal (e.g. unit 32767 on Juniper. We ignore these while building L2 topology) - Tunnel - PeeringInterface - OspfInterface **ifSpeed** : interface speed as a string, e.g. "1G", "10G" **ifType** : the value of ifType from RFC1213 MIB **ifAdminStatus** : administrative status (Up, Down, Testing) **ifOperStatus** : operational status (Up, Down, Testing) **ifDescription** : this facet holds tags generated from interface descriptions **Link** : tags in this facet describe the opposite end of the network link terminated on the interface **ifBGP4Peer** : tags in this facet describe BGP4 peer connected to the inteface. Tag words are in the form "ASxxxx" (where 'xxxx' is the AS number) or the host name of the iBGP peer. **OSPFState** : tags in this facet describe OSPF state of the interface. Values can be as follows: - unknown - down - loopback - waiting - pointToPoint - designatedRouter - backupDesignatedRouter - otherDesignatedRouter **ifVlan** : vlan number the interfave is a member of. Tagged ports have multiple tags in this facet **Aggregator** : tags in this facet are assigned to 802.3ad aggregation ports and describe aggregator port they belong to. For example, if 802.3ad aggregator port Port-Channel1 consists of aggregation ports Gi0/10, Gi0/11, then interface Port-channel1 will gets tag "ifRole.Aggregator" and interfaces Gi0/10 and Gi0/11 get tags "ifRole.AggregationPort" and "Aggregator.Po1" How Tags are Assigned ===================== Many tags are assigned by the server at the end of the discovery run to attach information about devices and their components to internal objects that represent them in NetSpyGlass. At this point NetSpyGlass have already learned everything it can about devices: vendor, model, inventory of interfaces and hardware components and network topology. This informaion is used to assing tags to objects that represent devices, interfaces, hardware components, protocols, firewall counters etc. We call tags assigned this way "implicit", thet are immutable and are reassigned again are the end of the next discovery run. The next step after implicit tags are assigned is to call Python hook script described by the configuration parameter ``tagSelector`` in the configuration file `nw2.conf`. If this parameter is absent, the system uses default tag selector script that ships as an internal Java resource. We provide a copy of this script in the file `tag_selector.py` in the directory `doc` in the distribution tar archive. This copy is not actually used by the program, it is provided so you can inspect it and use it as a reference. The default `tag_selector` script does not assign any tags but defines the class and its functions and provides additional documentation in the doc strings. As with other Python hooks, configuration parameter `tagSelector` describes Python module and class name:: network { tagSelector = "tags.UserTagSelector" In this example, NetSpyGlass will try to load class `UserTagSelector` from `tags.py`. Class :class:`UserTagSelector` must be derived from the class :class:`TagSelector` described in the next section :ref:`tag_selector_class`. Once tags are assigned, they can be used in many ways. First, we can use tags to analyse devices and components and decide if they should be monitored or not. See :ref:`selection_of_components_to_monitor` for an explanation and example of how to use tags in monitoring variable builder hook script. There you will see an example of the Python function that decides which interfaces should be monitored by checkig tags of the :class:`PyNetworkInterface` object passed to it as a parameter. Once monitoring variables are created, they carry copy of the set of tags taken from the device and the component the variable corresponds to. You can use these tags to find, filter and group monitoring variables in your data processing rules script (see :ref:`rules`). This high level diagram illustrates how tags are assigned and then propagate through the system: .. image:: images/tags_assignment_flow.png .. _tag_selector_class: Tag Selector Class ================== .. automodule:: tag_selector :members: :undoc-members: :show-inheritance: .. _user_defined_tags: User Defined Device Tags ======================== User can create their own tags using Python script `tags.py` located in the directory `scripts` in the git repository co-located with the cluster. All clusters ship with this script as a prototype, the copy we include with new cluster simply defines required class and methods but does not do anything. NetSpyGlass watches this script for changes and reloads its whenever the script is modified. There is no need to restart the system, jsut edit, commit and push, then watch System/Logs/Python panel for errors and warnings. A class should be defined in this script, using `tag_selector.TagSelector` as its base class. The class should have the following methods: * assign_device_tags() : this method is used to add tags to the device object * assign_interface_tags() : this method is used to add tags to interface objects A copy of the script included with new cluster already has these methods and provides some usage examples in comments. Example:: import tag_selector class UserTagSelector(tag_selector.TagSelector): def __init__(self, log, graph): super(UserTagSelector, self).__init__(log, graph) def assign_device_tags(self, device): if device.name in ['router1', 'router2']: device.tags.add('Explicit.core') device.tags.add('Explicit.test1') In this simplified example, we add tags `Explicit.core` and `Explicit.test` to devices "router1" and "router2" simply by matching their name. You can use regular expressions via Python module `re` to implement complex pattern matches or module `ipaddr` to match device addresses. In the end, to add a tag just add a string in the format "TagFacet.word" to the set `device.tags`. .. _tags_examples: User Defined Interface Tags: Parsing Interface Descriptions ----------------------------------------------------------- The same python hook script can be used to add interface tags. To do this, add function `assign_interface_tags` to the class as follows:: import re from net.happygears.nw2.py.py_wrappers import PyDevice from net.happygears.nw2.py.py_wrappers import PyNetworkGraph import tag_selector class UserTagSelector(tag_selector.TagSelector): def __init__(self, log, graph): super(UserTagSelector, self).__init__(log, graph) def assign_interface_tags(self, device, interfaces): for intf in interfaces: m = re.search('T=peerAS(\d+)', intf.description) if m: asn = m.group(1) intf.tags.add('ifDescription.AS' + str(asn)) In this example we use regular expression to match pattern "T=peerAS(\d+)" in the interface description and add tag in facet `ifDescription`. The tag looks like `ifDescription.AS174` (i.e. it has peer AS number encoded in the description). You can add any number of tags to the same interface this way, however they all must belong to the facet `ifDescription` to be processed correctly by the rest of the system. The meaning of these tags is completely up to you. Example above demonstrates how we can parse interface description and assign tags to the network interface objects. We can take this one step further and assign tags extracted from interface description to devices on either side of the link. To do this, we'll use object :py:class:`PyNetworkGraph` passed to the constructor of the tag selector class:: import re from net.happygears.nw2.py.py_wrappers import PyDevice from net.happygears.nw2.py.py_wrappers import PyNetworkGraph class UserTagSelector(object): def __init__(self, log, graph): self.log = log self.graph = graph def assign_interface_tags(self, device, interfaces): for intf in interfaces: if 'T=transit' in intf.description: intf.tags.add('ifDescription.transit') opposite = self.graph.get_connected_device(device, intf) if opposite: opposite_dev, opposite_intf = opposite assert isinstance(opposite_dev, PyDevice) self.log.info('Device ' + device.name + ': adding tag "transit" to ' + opposite_dev.name) opposite_dev.tags.add('Explicit.transit') We use simple check to see if string "T=transit" appears somewhere in the interface description and set interface tag 'ifDescription.transit". Then we call :py:func:`get_connected_device()` on the network graph object to get device and interface on the opposite side of the link. It is possible that such device does not exist, if NetSpyGlass could not determine where this interface is connected. If this is the case, this function returns `None`. Otherwise it returns a tuple `(opp_dev, opp_intf)` where `opp_dev` is the device on the opposite side of the link (a :py:class:`PyDevice` object) and `opp_intf` is corresponding network interface (a :py:class:`PyNetworkInterface` object). We can use these two objects to add tags just like we did before, using their `tags` attribute. Tags in facet NsgControl ======================== Tags in this special facet can be used to control various aspects of NetSpyGlass operation related to devices and interfaces. At this time, only the following tag is recognized: - `NsgControl.noLink` Tag `NsgControl.noLink` can be assigned to an interface and makes NetSpyGlass ignore this interface for the purpose of the network topology discovery. For example, a decommissioned link can be marked with this tag to make sure it disappears from network maps even if it is still connected. There is also an interaction of this tag with link retention algorithm: normally, if a link does down, link retention algorithm will keep it in the network map for a few days (depends on the value of the configuration parameter `discovery.linkRetention.generations`). However, if the link was decommissioned, it should disappear right away. You can set tag `NsgControl.noLink` on the decommissioned interface to purge the link from maps immediately. One way to set the tag is to add a code word to the interface description, then use `tags.py` NetSpyGlass Python all to parse it and set the tag and shown in :ref:`tags_examples`. External Tags ============= NetSpyGlass supports special category of device and component tags that we call "External tags". These tags can be updated using API call that writes them to the database. Every time NetSpyGlass updates tags of a device and its components, it does this in three steps: * run built-in algorithms that assign standard tags using information collected about the device during discovery process * apply external tags stored in the database at that moment * run user's Python app to assign user-defined tags .. note:: The Python app has access to all tags created previously, including external ones. The API call used to update external tags only manipulates records in the database but does not update actual tags attached to the "live" device objects. We assume that the user is going to build an automation to use this API call, for example by integrating with Netbox or another external database that acts as a source of meta data about devices, interfaces and components. The automation can run on schedule and can compare tags present in the external source with those stored in NetSpyGlass, then use the API call to synchronize them. It may be convenient to do this in batches to reduce amount of data that needs to be stored and compared at a time. To avoid possible race conditions and necessary internal queueing, the API call used to update external tags only updates database records. Once all external tags have been updated, the automation should make another API call to initiate tag update in NetSpyGlass. API call used to query and update external tags ----------------------------------------------- GET /v2/tags/net:netid/external[?id=:devID|name=:name|address=:address] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This call will return external tags for all devices or a device identified by the query parameters. Only one device match parameter can be provided at a time, but if the query parameter is missing, the call returns external tags for all devices and components. The response format is as follows: .. code-block:: json [ { "category": "device", "device": { "id" : 12345, "name" : "device_name", "address" : "10.1.2.3" }, "tags": [ "Location.SJC", "AcmeRole.BB" ] }, { "category": "interface", "device": { "id" : 12345, "name" : "device_name", "address" : "10.1.2.3" }, "component": { "index": 1010, "name": "ge-0/0/0" }, "tags": [ "NsgControl.noLink" ] }, ] See the next section for the description of the "category". Information about the device includes its id, name and address. This information is provided to make it easier to match the device against external sources, such as Netbox, where NetSpyGlass device ID is unlikely to be present, however the name and address can be used. The address returned by this call is the one NetSpyGlass uses to communicate with the device. We also provide both the index and name for the interface and component. If the value of "category" is "device", the tag is meant to be attached to the device rather than interface or a component and section "component" may be skipped. Note that the call does not support filtering by the component and always returns external tags attached to all components (if any) of the matching device. POST /v2/tags/net:netid/external ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The body should have the following format:: [ { "category": , "device": { }, "component" : { } "tags": [ "Facet.Word" ] } ] The category can be one of the following: * "device" - the tag will be assigned to the device; key "index" is ignored * "interface" - the tag will be assigned to the interface with ifIndex given by key "index" * "component" - the tag will be assigned to the hardware component with index given by the key "index" Device match clause can be one of: * "id" - the value should be numeric device ID * "name" - the value should be device name * "address" - the value should be device IP address only one device match clause is supported at a time. Component match can be one of: * "index" - interface or h/w component is matched by its index * "name" - interface or h/w component is matched by the name Each dictionary inside of the envelope array assigns one tag, but the call can pass any number of these in any combination. If the call tries to update multiple tags for many devices and some of the device or component match clauses do not match, the response will mention them under "errors" but the server will continue to process the rest and return HTTP response 200. The response of this call has the following format: .. code-block:: json { "errors": [ "Device not found: name=invalid-name", "Device not found: address=10.99.32.1" ], "status": "tags updated for 10 devices" } Example ^^^^^^^ The following example demonstrates different combinations of device and component match clause formats. .. code-block:: json [ { "category": "device", "device": { "id" : 12345, }, "tags": [ "Location.SJC" ] }, { "category": "device", "device": { "id" : 98765, }, "tags": [ "Location.IAD" ] }, { "category": "interface", "device": { "id" : 1001, }, "component" : { "index": 1010 } "tags": [ "NsgControl.noLink" ] }, { "category": "interface", "device": { "id" : 1001, }, "component" : { "name": "ge-0/0/0" } "tags": [ "NsgControl.noLink" ] }, { "category": "component", "device": { "name": "jnpr-router" }, "component": { "name" : "Routing Engine 0" }, "tags": [ "NsgControl.SomeTag1", "NsgControl.Another", ] } ] .. note:: The input data format for this API call supports matching of device and interface/component by their numeric id, name or address. Since internal object models in NetSpyGlass rely on the ids, the server uses provided name or address to find the device or interface and records their id when it writes information about external tag to the database. The match of the name or address happens once when the API call is executed. If the name or address changes later, the tag will remain attached to the same device or interface until you make this API call again. Removing External Tags ^^^^^^^^^^^^^^^^^^^^^^ The POST API call described above always replaces all external tags for the matching device and component. If you need to remove all external tags for a device or a component, execute the same API call with body data that does not have "tags" key. The scope of this call is determined by the category, device and component match clause. If the call has category "device", then all external tags attached to the matching device will be removed but any external tags attached to its components will remain intact. The following would remove all external tags for the device with id=56: .. code-block:: json [ { "category": "device", "device": { "id" : 56 } } ] The following would remove all external tags for the component and device identified by name: .. code-block:: json [ { "category": "device", "device": { "name" : "router1" }, "component": { "name" : "Routing Engine 0" } } ]