IT Operations Management (ITOM)
cancel

How to tame the HP Operations Manager message storms – rises again!

How to tame the HP Operations Manager message storms – rises again!

GirishMatti

Co-written by Tobias Mauch, a very senior and much respected engineer on the HP OM team.

 

 Here comes the third and the final part of this series on managing message storms in the HP Operations Manager (HP OM) infrastructure monitoring environment.

 

In the first two posts, we saw how the message or event storm can be tamed on the HP OM server either using ECS or with the HP OMU 9.20 Event Storm Filter (ESF). Both of these mechanisms worked on the HP OM server side.

 

If you missed them, then have a look at the previous posts here:

 

After reading these blog posts which outline how to easily and simply beat the message storms, you may be wondering if there is a way to tame the storms right where they began—on the Operations agent servers. Keep reading to find out how to accomplish this task.  

 

 Operations Agent Message Storm Suppression

 

In this mechanism, message storm detection is performed on the agent node and is configured via configuration settings which can be deployed as nodeinfo policy centrally from the HP OM server to one or more agent servers.

To enable this all you have to do is set the config setting OPC_MSG_STORM_DETECTION to TRUE in the namespace eaagt.msgstorm.

 

This setting can be manually set on the agent using the command ‘ovconfchg –ns eaagt.msgstorm -set OPC_MSG_STORM_DETECTION TRUE’ or if you want to use this setting on several managed nodes, you could use a nodeinfo policy to configure it centrally from the HP OM server and then deploy these settings to one or more agents.

 

This feature is available with Operations Agent version 11.12 and later and since this is not an ECS policy has better performance.

Along with the setting OPC_MSG_STORM_DETECTION, you need to define a category for which to filter (OPC_MSG_STORM_DETECTION_CATEGORY). Possible values are POLICY, MSGGROUP, APPLICATION, OBJECT, and SEVERITY.

 

Threshold, time and reset are configured via OPC_MSG_STORM_RATE.

Once enabled, for example to set a threshold of 100 messages in an interval of 60 seconds and a reset value of 45:

ovconfchg -ns eaagt.msgstorm -set OPC_MSG_STORM_RATE 100:60:45

 

You can configure if internal messages are sent during a message storm (OPC_SEND_INTERNAL_MSG_ON_MSGSTORM). You can configure if messages are suppressed during a storm (OPC_SUPPRESS_MSG_ON_MSG_STORM, default is TRUE). You can configure if suppressed messages are logged to System.txt (OPC_MSG_STORM_TRACE_SUPPRESSED_MSGS).

 

For more information, see the "Message Storm" section in the HP Operations Agent User Guide for your Operations Agent release, for example OA 11.14.

 

Hasta la vista, baby

 

That’s it, as the Terminator would say, you can say “Hasta la Vista, baby” to message storms in three different ways!

 

For more information on how HP Operations Manager can help you with infrastructure monitoring visit with the product home page here. 

  • operations bridge
About the Author

GirishMatti