IT Operations Management (ITOM)

Why are fire trucks red? Also, why are all my VMs red in Cloud Optimizer's treemap?

Why are fire trucks red? Also, why are all my VMs red in Cloud Optimizer's treemap?

Ramkumar Devana

I remember a joke I heard in my childhood about why fire trucks are red. It goes like this:

The fire truck has 8 wheels and 4 fire men totaling 12; 12 inches make a foot; a ruler (scale) is one foot long; Queen Elizabeth was a ruler; Queen Elizabeth is the name of a ship, too; ships sail on the sea; seas have fishes; fishes have fins; the people of Finland are called Finns; the Finns fought the Russians; Russians are red (sorry, this joke is pretty old) and so the fire truck, always Russian (i.e. rushing), is red.


Setting aside the political incorrectness of this joke, I take away the more important meaning of how all things in the world are connected in one way or another. Which brings us to less important things, like why all your HyperV VMs may show up red in HPE's Cloud Optimizer tree-map view, here's some background.

As you would know, the hypervisor (ESXi, HyperV, KVM, etc) controls the virtual machines and among other things, also the memory allocation to guest OS.

With ESXi, if one allocates some memory to a virtual machine (say, 2GB) one can find out exactly how much of that memory is used by the guest OS (say, 780MB). This usage information is used in over-committing memory resources, so you can run VMs together with more memory allocated than available. While it is debatable how much to over-commit, the key thing here is that the exact amount of memory used by the guest OS is known in ESXi world.

This is not exactly the case in the KVM, Xen and HyperV worlds. Memory utilization done by the guest OS is not known to the hypervisor. If someone allocated 2GB to a Xen guest, they can consider that memory as completely off limits. There's no over-commit possibility.

Until Dynamic memory was introduced as a feature in HyperV with Windows Server 2008 R2 SP1, HyperV suffered from this problem too. Even with Windows 2012 HyperV, if a VM is setup with static memory allocation, the memory usage by guest OS against allocation will show as 100%. Set the VM properties to use dynamic memory and sure enough, you'll see better details of the memory consumed by the guest OS. An excellent KMine doc here that explains in more detail.


KVM and Xen resource measurement is usually done via libvirt APIs. Libvirt's dommemstat as well as dominfo commands return the value of memory used “actual” as equal to the complete allocated memory. So again, if 2GB is allocated to a Xen domU, or a KVM guest, for all practical purposes the memory is “lost” to the guest. If you are familiar with the libvirt tool “virt-manager,” the tool does not even show memory utilization for the guest in the main dashboard—for the reason that memory utilization is always a static 100% for the guests.

But here's where Cloud Optimizer does the clever bit: it uses cAdvisor, which is an open-source tool to get resource utilization information for containers and virtual machines running on a Linux host. This is installed at the time a KVM host is added to the cloud optimizer list of targets. So if VMs running on KVM host (or the Helion compute nodes) show up all red, check if cAdvisor installed correctly. You can install it manually, too.

For Xen hosts, there's no solution yet. We would love to hear input on this, however.

Is it important to have right amount of memory usage from the VMs? I would consider this important—if the intent is to get more out of the available capacity. That's the aim of virtualization: to make better use of hardware. If a VM always shows 100% memory utilization, actual memory used is never known and there's no scope for “governed” sharing of memory.

I hope this post helped you to understand why the fire truck is red, and also why the treemap shows all HyperV/KVM VMs as always having 100% memory utilization. Now you can take steps to ensure that you get precise memory utilization.

Download and try out HPE Cloud Optimizer - free eval for 60 days. There's also a free community edition for use with upto 25 instances (VMs + hosts).



HPE Software Rocks!
  • infrastructure management
About the Author

Ramkumar Devana

Ramkumar Devanathan (twitter: @rdevanathan) is Product Manager for HPE Cloud Optimizer (formerly vPV). He was previously a member of the IOM-Customer Assist Team (CAT) providing technical assistance to HP Software pre-sales and support teams with Operations Management products including vPV, SHO, VISPI. He has experience of more than 14 years in this product line, working in various roles ranging from developer to product architect.