Given all the press and media coverage, there’s an increasing adoption of BigData Platforms out there. I wanted to take a closer look at this trend, and I recently had the chance to gather some public information about a large USA retailer and their adoption of Hadoop.
The results compared to older RDBMS systems are quite staggering. Marketing campaigns that once took 30 days, now take two or three. With the older systems a campaign could only be applied all their customer base. With the power of Hadoop, they can apply multiple campaigns to multiple groups, and their analysis can now look at years of history. In fact, now they can look at all of the data they have, rather than the much smaller window their archives allowed previously.
They quote costs savings to nearly a third of the legacy systems.
So Big Data platforms are cutting edge, and clearly their performance and availability have rapidly become mission critical. With this kind of performance (faster than lightning) departments are measured on their ability to offer analytics and launch new initiatives. They are depend on these analytics and initiatives being "always on".
As for all IT systems that are mission critical, their deployment should be associated with IT monitoring and management—so IT has the measurements it needs to manage the availability and performance of these platforms. Although there is of some level of resilience with these Big Data platforms, (depending on how they have been architected and built using clusters there are, however, key components in the cluster will need to be correctly monitored. In the case of large and multiple implementations, a central solution that also manages the applications being served by the BigData Platforms, is essential for IT Management to handle the situation.
These Big Data concerns are the top priority of the new OMi Management Pack Software options—an HP BSM solution software portfolio. These are brand new and work with OMi Monitoring Automation. This means that new instances or nodes that arrive on the IT landscape, are automatically discovered and specific value add management templates and configurations defined by these Management Pack software modules are deployed to the target environments
For Hadoop these typically include:
Monitoring Hadoop Services
Node Infrastructure and Cluster monitoring
Correlation by OMi with cross domain/components JVM, RPC, Network, Database like Vertica, MongoDB and other databases.
According to some subject matter experts, Vertica best practices define monitoring of:
Low Disk Space
Read-Only File System
Loss of K Safety
Current Fault Tolerance at Critical Level
Too Many ROS Containers
WOS Over Flow
Node State Change
Recovery Lock Error
Recovery Projection Retrieval Error
Refresh Lock Error
Tuple Mover Error
Timer Service Task Error
HP OMi Monitoring Automation Management Packs are now available for ORACLE DataBase, Infrastructure, Hadoop and Vertica.
Each solution results in the provision for dashboards like these which greatly assist operators and administrators to understand the overall health of the whole platform, deal with any issues concerning them, and govern all configurations centrally