Yes, you read it right. Though it sounds funny at first, but if we give it a thought, the title makes complete sense. Most of the enterprises today, depend on Big Data infrastructure to whip out meaningful and profitable insights from its overload of data. The enterprises are now looking for a Big Brother to overlook the availability and performance of the Big Data System, and that’s exactly what the new HP Omi Management Pack for Hadoop is. A Big Brother for your Hadoop based big data solution.
Figure 1. Big Data
Why do you need a Big Brother for your Hadoop Cluster?
1. Complexity – The Hadoop Infrastructure is inherently complex. With ever increasing volume, velocity and variety of data, this infrastructure continues to grow in size and complexity, this will impact its manageability and maintenance. Imagine a cluster with thousands of nodes, how would one ensure the availability of such a big cluster without an intelligent monitoring solution for assistance?
2. “Always On” – We are in the age of real time analytics where businesses are taking decisions on the fly to compete better. For example; Retailers can now gain fast insight into buying patterns, demographics, change selling methods, sales plays, launch loyalty programs and multiple simultaneous marketing campaigns to suit. More examples exist in practically all industries. This reliance of business on the big data systems like Hadoop necessitates these systems to be “Always On”. Achieving this would be difficult without a monitoring solution.
3. “Maintaining Throughput” – Hadoop systems are designed to perform complex analysis of huge amount of data in very short time. How does one ensure that the throughput of these jobs is not degraded over time? It is very difficult to analyze and detect signals of the performance degradation unless one uses a monitoring solution to do it.
How can HP help?
As IT environments become more dynamic and complex the operations teams need a unified view for management and monitoring. HP’s new evolved “Management Cockpit” is a solution that helps you achieve this by enabling automatic detection, and automated monitoring setup and configuration of your IT Infrastructure
“HP Operations Manager i” forms the foundation of this solution and the “Omi Management Packs” extend its capability to a given domain. HP’s “OMi Management Pack for Hadoop” is automated, scalable monitoring solution that allows you to visualize and monitor your entire Hadoop Cluster in a single pane of glass, and directly correlate its health with the business services and applications that depend on it. It is one of the offerings from the suite of OMi Management Packs available from HP. If the OMi Monitoring Automation and OMi Management Pack are new to you, then take a look at the blog articles below that introduce these new offerings from HP:
HP’s “OMi Management Pack for Hadoop” is an answer to all of the challenges listed earlier. It helps the user to overcome each of the above challenges with its inherent features.
1. Complexity – “OMi Management Pack for Hadoop” is highly scalable monitoring solution which helps you to visualize the entire Hadoop Infrastructure in a single pane of glass. The Management Pack also contains dashboards, like the one shown in the figure below, which provides panoramic view of the health of the entire Hadoop Cluster in a single window. The dashboards are made available as part of the “OMi Content Pack for Hadoop” which is a pre-requisite for the “OMi Management Pack for Hadoop”. The automated RTSM based deployment allows for the automated monitoring of the dynamic Hadoop Cluster.
Figure 2 NOC Dashboard of OMi Management Pack for Hadoop
2. “Always On” – “OMi Management Pack for Hadoop” comes with out of the box Aspects like “Hadoop Service Availability” to monitor the availability of the Hadoop Cluster. It monitors the critical components of the Hadoop Cluster like NameNode, DataNode, JobTracker, TaskTracker and SecondaryNameNode. It also generates events and proactive alerts for the users to take appropriate actions to ensure the availability of the Hadoop Cluster.
Figure 3 Management Template and Aspects in OMi Management Pack for Hadoop
3. “Maintaining Throughput” – “OMi Management Pack for Hadoop” monitors the performance of the “Hadoop Cluster” and generates events to notify any degradation of the performance. This is achieved via the out of the box Aspects like “NameNode File Operations”, “Job status and map-reduce statistics”, etc.
HP’s Big Data platform HAVEn is creating waves in the Big Data world; the “Management Cockpit for Big Data” will complement this platform by adding monitoring capabilities. There is also an opportunity to apply the capabilities of the HP Operational Analytics on the operational data generated by the “Omi Management Pack for Hadoop” to pinpoint and predict issues in your big data environment. This solution combines the benefits of big data analytics provided by “HP Operation Analytics” and monitoring capabilities tailored for Hadoop by “Omi Management Pack for Hadoop”; in effect providing a faster, safer and cost effective alternative to ensure that your Big Data platforms run optimally.
With all these possibilities and its features HP Omi Management Pack for Hadoop positions itself as the Big Brother for your Hadoop infrastructure. To learn more about this solution and other Management Packs visit OMi Management Pack page.
Manoj Mohanan is a Software Engineer working in the HP OMi Management Packs team. Apart being a developer he also dons the role of an enabler, working with HP Software pre-sales and support teams providing technical assistance with OMi Management Packs. He has experience of more than 8 years in this product line.