Should you be concerned about event duplicates in your Operations Manager (OM) or Operations Manager i (OMi) setup? What harm they could do, you may ask. Well, if you have a large environment with thousands of events coming into OM and OMi on a daily basis, large duplicate counts can have some performance implications (believe me some customers have events with duplicate count running in 3 to 5 digits). But you don't need to worry about it. There are enough configuration options in Operations Management to handle this situation.
Why Event Duplicates?
Duplicate events are event reporting the same or similar information about something
The same event being sent repeatedly so it is shown as one event with increasing message/event count (called duplicate count) in the event browser on the server
Sometimes (or more often?) poor monitoring and event management practices cause loads of duplicates
Duplicates in HP Operations Manager (OM Unix/Linux/Windows)
In the HP OM(U/L/W) world, two events are considered duplicates when either they have the same “key” attribute OR in case of messages with no keys, a set of event attributes are exactly the same. In such a case, the duplicate count of the original event is incremented while the second event itself may be dropped or stored in the database as per the configuration.
In OMW, you can configure many aspects related to duplicates -
Some important configuration settings define:
Which attributes of the event should match for them to be duplicates?
Should duplicate messages be stored in the data base? If yes, how many? (You can also set it to unlimited.)
Should the severity and message text of the original messages be shown or that of the latest duplicate message?
Should annotations be added for duplicate messages and if so, how many?
In flexible management environment, what all event changes to forward and what all to accept to/from other management servers
For more details about message suppression configuration, refer to OMW online help, section - “configure duplicate message suppression”
In OMU/L, some of the similar configuration options for duplicates are:
Attributes used to identify duplicates are fixed and cannot be modified
Enable/disable duplicate suppression and addition of duplicate as annotation to the original message using following command on the server:
opcsrvconfig -dms -enable anno
Most of the configuration is done by setting server configuration variables using a command like:
Similar to OM, OMi also provides duplicate suppression with somewhat different configuration options.
One important difference is that OMi does not store the duplicate messages in the database. Instead it drops them completely after updating the original event. Some configuration options are:
Duplicates based on keys (just like in OM) – enabled by default
Duplicates based on ETIs(same CI, same HI and same HI Value) – enabled by default
Duplicates based on attributes (like in OM) – disabled by default
Should the title and severity of the original message be updated from the newer duplicate message?
Should history entries be added for duplicate count changes on the event? (Please note that this infra setting to disable history entries for duplicates is applicable only for events that are de-duplicated by OMi.)
Duplicates in the synchronized OM - OMi Environment
When OM is integrated with OMi to forward all events, new events as well as event updates from OM are forwarded into OMi. The event updates include change in duplicate count, message title, message severity, lifecycle state, custom message attributes and annotations. These synchronized updates have certain implications on OMi which are important to understand:
Every event update coming from OM into OMi, adds a history entry to the event. So, if there are hundreds of updates on an event, then there will be those many history entries under the history tab of the event details window in the OMi event browser.
When OM performs de-duplication, it can result in the addition of multiple history lines (for duplicate count, for annotations, for title and severity change , if applicable and depending on the settings in OM) to the corresponding OMi event. Please do not forget that configuration setting in OMi to “not generate history lines for duplicate messages” applies only when OMi de-duplicates the events and not in this case.
As a result of above, for a single event in event database table, there could of hundreds of records in the tables for storing history lines and event changes. Such events with large number of history lines can cause significant performance impact on the OMi event browser and event archival, as has been seen with many large customers. There could be other factors impacting the performance, this is just one of the factors.
So, what should be done to avoid these performance issues?
Firstly, consider the reasons for so many duplicates being generated in the monitoring setup. If possible, modify your monitoring to reduce/avoid the duplicates.
If you cannot change the monitoring and do not care about the duplicate counts in OMi, then
If events are coming into OMi from non OM sources and OMi is doing de-duplication, then by default , history lines will not be added as per the infrastructure setting mentioned in earlier section; make sure this setting is set to false
If events are coming from OM into OMi and OM is doing the de-duplication, configure OM to not send the ‘change in duplicate count’ update to OMi by setting “operations to be forwarded” configuration in OMW; (in OMU/L, you can use OPC_SEND_* variables to enable and disable the similar settings) ;
Even when sending of the duplicate count change is disabled, if a duplicate suppression adds annotation entries in OM and/or changes event title, severity, those updates will still be sent to OMi and add history entries unless you have configured otherwise. It will be good to disable adding of annotation entries for duplicates in OM using relevant configuration settings.
You also have the option to disable the duplicate suppression in OM and instead do it in OMi. This allows OMi to make sure no history entries are added for duplicate count changes. But this is not a recommended practice, especially in large environments with huge number of incoming events. In the end-to-end event flow, data should be suppressed as early as possible to avoid unnecessary network traffic and processing on higher level servers.
These are just some simple and quick ways to configure event duplicates to avoid potential performance issues caused by large number of history lines and event changes in OMi.
Stay tuned for more detailed document on this topic but in the meanwhile feel free to share your comments/feedback/questions….