IT Operations Management (ITOM)
cancel

SNMP Trap Management in NNMi 10.10

SNMP Trap Management in NNMi 10.10

JackYangHpe

The management of SNMP (Simple Network Management Protocol) traps in NNMi (Network Node Manager i) has been evolved significantly since NNMi 9.2x. Although all the new features have been documented in NNMi Online Help and Deployment Reference, customer have yet to see a clear picture about the process workflow in which they can control the behavior at various stages, such as filtering, blocking, forwarding and logging, etc. This blog is intended to server for this purpose.

NNMi works with SNMP traps at the following three levels:

• NNMi nnmtrapreceiver process (running as an operating system’s service)
• NNMi Trap Service (controlled by ovjboss)
• NNMi Event Service (controlled by ovjboss)

The details at each level are explained in the follow sections.

NNMi nnmtrapreceiver Process
The NNMi nnmtrapreceiver process runs as an operating system service. You can see this service by running the command “ps –ef | grep nnmtrapreceiver” on a Linux NNMi server:

f1.png

There are a few things to be aware of about the nnmtrapreceiver process:

• The nnmtrapreceiver process listens to the SNMP trap port (port 162 by default) on the NNMi server and sends SNMP traps to the NNMi Trap Service which will be described in the next section.
• The nnmtrapreceiver process is coupled with nnmtrapreceivermd which is integrated with ovspmd and responds to ovstart/ovstop messages.  During ovstart, the nnmtrapreceivermd starts the nnmtrapreceiver process, but leaves it running during ovstop.
• The nnmtrapreceiver process will queue traps for a certain period of time if NNMi Trap Service is not started. This feature is especially useful in High Availability and Application Failover clusters. It runs on both nodes (active and standby) in the cluster. This means that during failover, no trap is lost.

NNMi Trap Service

The NNMi Trap Service is a service running under ovjboss. You can see this service by running the command “ovstatus –v” on the NNMi server, as shown in the figure below:

f2.png

The diagram below illustrates the process workflow for SNMP traps in NNMi Trap Service.
Note: to keep the discussion in context, the diagram also included the source of SNMP traps and the nnmtrapreceiver process discussed in the previous section.f3.png

Each step in the workflow is explained as follows:

• trapFilter.conf – If the trap’s source IP address, OID, or both are listed in the trapFilter.conf file, the trap is dropped. Otherwise, the trap is stored in the Trap Binary Store and is used in trap analysis (for example, to calculate the trap rate).
Note the following:
    - trapFilter.conf is located in <NnmDataDir>/shared/nnm/conf directory
    - The trap binary store consists of up to five binary files: traplog0 through traplog4. They are located in the
       <NnmDataDir>/shared/nnm/databases/traps directory, as shown in the picture below:
f4.png

• Trap logging – If the trap logging is enabled, traps are logged to the trap.csv, trap.log files, or both in the <NnmDataDir>/log/nnm directory:
f5.png

You can enable/disable trap logging and change the trap logging format using the nnmtrapconfig.ovpl command:
nnmtrapconfig.ovpl –setProp trapLoggingMode < TXT, CSV, BOTH, OFF>

• “Hosted Object Trap Storm” detection – This step is also called “NarrowTrapAnalysis”.
In this step, the NNMi Trap Service compare each trap with the trap storm filter specified in the hosted on-trapstorm.conf file. If a trap meets a trap storm definition, the trap is dropped.
The hosted-on-trapstorm.conf file is located in the <NnmDataDir>/shared/nnm/conf directory.

• Trap storm detection – This step is also called “WideTrapAnalysis” in which if a trap storm based on the overall trap rate and threshold is detected, the traps are dropped. Note: run the command “nnmtrapconfig.ovpl –help” to see how to configure a trap storm definition

• Check trap age - If the time stamp of the trap is older than 10 minutes, the trap is dropped.

• nnmtrapd.conf – If the trap’s source IP address, OID, or both are listed in the nnmtrapd.conf, the trap is dropped.
Note the following:
- nnmtrapd.conf file is located in the <NnmDataDir>/shared/nnm/conf directory.
- The difference between nnmtrapd.conf and trapFilter.conf is that nnmtrapd.conf only stops traps from being created or stored in the event database, but they are still stored in the trap binary store and logged in the trap log file if logging is enabled, and they are also used in the trap analysis, while trapFIlter.conf completely blocks traps from entering the NNMi server, as if these traps were never received.

• Undefined trap check – If the NNMi server is configured to accept undefined traps, the traps will be sent to the “Trap Forwarding” stage, bypassing the check for whether the trap is enabled.
Note: The property com.hp.nnm.events.allowUndefinedTraps in the $NNM_PROPS/nms-jboss.properties file determines whether NNMi accepts undefined traps.

• Trap enablement check – If NNMi is not configured to accept undefined traps, then we will check whether the trap is defined in NNMi SNMP Trap Configuration and whether itis enabled. You can enable/disable traps in NNMi UI Console: “Configuration > Incidents > SNMP Trap Configuration”, then open individual trap configuration, check/uncheck the “Enabled” checkbox:

f6.png
If the trap cannot be found in the trap configuration (i.e., undefined), or it is found but not enabled, it will be dropped.

• Trap forwarding - Traps are forwarded to the specified trap forwarded destinations, if any, and to the NNMi Event Service which is discussed in the next section.
You can configure trap forward destinations in the NNMi console:

f7.png

NNMi Event Service

The NNMi Event Service is also a service running under ovjboss. You can see this service by running the command “ovstatus –v” on the NNMi server, as shown below:

f8.png

 The diagram below illustrates the process workflow for SNMP traps in NNMi Event Service.

f9.png

Each step in the workflow is explained as follows:

1. SNMP Trap Receiver – The Event Service itself has an SNMP Trap receiver which receives SNMP traps forwarded from Trap Service described in the previous section.

2. Global/Regional Incident Receiver – If the NNMi server is configured as a global NNMi server in a global-regional configuration, the Global/Regional Incident Receiver receives traps forwarded from regional NNMi servers.

3. Check source node in topology – If the source node of the trap was not discovered by NNMi, the trap is unresolved. Check the configuration for unresolved traps (Step 4). If the source node is was discovered, check its management mode (Step 5)

4. Check unresolved trap configuration – By default, NNMi discards the traps emitted from the devices that were not discovered by NNMi (unresolved traps). If you changed this configuration, unresolved traps will be sent to the “Trap Suppression Check” (Step 9) for further evaluation.
Note: you can change the unresolved trap configuration on the NNMi UI console, “Configuration > Incidents > Incident Configuration”, and uncheck the box “Discard Unresolved SNMP Traps and Syslog Messages”:

f10.png

5. Check source node management mode – If the source node is not managed, the trap is always discarded.

6. Check trap definition – If the trap is defined, it is sent to the “Trap Suppression Check” (Step 9) for further evaluation. Otherwise, check if undefined traps are accepted.

7. Undefined trap check – If the NNMi server is configured to accept undefined traps, the trap will be sent to “Trap Suppression Check” (Step 9) for further evaluation. Otherwise it is discarded.
Note: The property com.hp.nnm.events.allowUndefinedTraps in the $NNM_PROPS/nms-jboss.properties file determines whether NNMi accepts undefined traps.

8. Log traps to the incident.csv file – In this step, traps are logged in the incident.csv file which is located in the <NnmDataDir>/log/nnm directory.

9. Check whether the trap is suppressed – Trap suppression is configured on each individual trap. You can configure the suppression from the NNMi UI console, “Configuration > Incidents > SNMP Trap Configuration”, then open the trap of the interest, and select the “Suppression” tab:

f11.png

10. Trap customization – You can customize each individual trap in this step, such as Enrichment, Dampening, Deduplication, Action, etc. You can also apply various interface settings and node settings for the trap.

11. Check whether the number of traps in storage exceeds the limit - NNMi allows 100,000 traps to be stored in the NNMi database. After this limit is reached, no additional traps are stored in the NNMi database.

12. Save the trap in NNMi database and forward it to NNMi Northbound Interface – Finally, traps are persisted in NNMi database, and forwarded to NNMi Northbound Interface, which is beyond the scope of this blog.

Summary

This blog discussed three services on an NNMi server that process SNMP traps, i.e., nnmtrapreceiver process, NNMi Trap Service and NNMi Event Service.  The process workflow and configuration details were explained. Let me know your thoughts in the comments below!

  • infrastructure management
About the Author

JackYangHpe

Comments
Outstanding Contributor.

Nice Information about NNMi 10.10 and well explained