Virtual Host Metrics Daemon (vhostmd)
=====================================

vhostmd provides a "metrics communication channel" between a host and
its hosted virtual machines, allowing limited introspection of host
resource usage from within virtual machines.  This functionality may be
useful in hosting environments, giving virtual machine administrators
a limited view of host resource consumption - potentially explaining a
performance degradation within the virtual machine.

vhostmd will periodically write metrics to a disk.  The metrics to
write, how often, and where to write them are all adjustable via the
/etc/vhostmd/vhostmd.conf configuration file.  The disk can then be
surfaced read-only to virtual machines using tools provided by the host's
virtualization platform.


Contact
-------

All comments / suggestions / patches should be directed to the
virt-tools-list mailing list:

  http://www.redhat.com/mailman/listinfo/virt-tools-list


Usage
-----

vhostmd [options]

  Options:
   -v | --verbose         Verbose messages
   -d | --no-daemonize    Process will not daemonize
   -f | --config <file>   Configuration file (/etc/vhostmd/vhostmd.conf default)
   -p | --pid-file <file> PID file (/var/run/vhostmd.pid default)


Configuration File
------------------
The default configuration file (example listed below) defines a 256Kbyte
metrics disk in /dev/shm/vhostmd0, updated every 5 seconds.  It also
includes a few examples of user-defined metrics, which provide a
(currently simplistic) mechanism for extending metrics gathered by vhostmd.

  <vhostmd>
    <globals>
      <disk>
        <name>host-metrics-disk</name>
        <path>/dev/shm/vhostmd0</path>
        <size unit="k">256</size>
      </disk>
      <virtio>
        <max_channels>1024</max_channels>
        <expiration_time>15</expiration_time>
      </virtio>
      <update_period>5</update_period>
      <path>/usr/bin:/usr/sbin:/usr/share/vhostmd/scripts</path>
      <transport>vbd</transport>
      <transport>virtio</transport>
    </globals>
    <metrics>
      <metric type="string" context="host">
        <name>HostName</name>
        <action>virsh hostname | tr -d '[:space:]'</action>
      </metric>
      <metric type="string" context="host">
        <name>VirtualizationVendor</name>
        <action>/bin/rpm -q --info xen | grep Vendor: | \
                 awk '{print substr($0, index($0,$5)) }'</action>
      </metric>
      <metric type="uint32" context="host">
        <name>TotalPhyCPUs</name>
        <action>xm info | gawk '/^nr_cpus/ {print $3}'</action>
      </metric>
      <metric type="uint32" context="host">
        <name>NumCPUs</name>
        <action>xm info | gawk '/^nr_cpus/ {print $3}'</action>
      </metric>
      <metric type="uint64" context="host">
        <name>TotalPhyMem</name>
        <action>xm info | gawk '/^total_memory/ {print $3}'</action>
      </metric>
      <metric type="uint64" context="host">
        <name>UsedMem</name>
        <action>xentop -b -i 1 | gawk '/Domain-0/ {print $5}'</action>
      </metric>
      <metric type="uint64" context="host">
        <name>FreeMem</name>
        <action>xm info | gawk '/^max_free_memory/ {print $3}'</action>
      </metric>
      <metric type="uint64" context="host">
        <name>PagedInMemory</name>
        <action>vmstat -s | gawk '/pages paged in/ {print $1}'</action>
      </metric>
      <metric type="uint64" context="host">
        <name>PagedOutMemory</name>
        <action>vmstat -s | gawk '/pages paged out/ {print $1}'</action>
      </metric>
      <metric type="group" context="host">
        <name>PageRates</name>
        <action>pagerate.pl</action>
        <variable name="PageInRate" type="uint64"/>
        <variable name="PageFaultRate" type="uint64"/>
      </metric>
      <metric type="real64" context="host">
        <name>TotalCPUTime</name>
        <action>virsh dominfo 0 | sed 's/: */:/' | \
                 gawk -F: '/CPU time/ {print $2;}'</action>
      </metric>
      <metric type="real64" context="vm">
        <name>TotalCPUTime</name>
        <action>virsh dominfo NAME | sed 's/: */:/' | \
                 gawk -F: '/CPU time/ {print $2;}'</action>
      </metric>
      <metric type="xml" context="vm">
        <name>my-metric</name>
        <action>xml-metrics-test.sh</action>
      </metric>
    </metrics>
  </vhostmd>

A valid configuration file must contain the root element <vhostmd>.

The <globals> element contains configuration global to vhostmd, such as the
metrics refresh interval and the metrics transport mechanism. The <transport>
element defines how the metrics are transported between the host and VMs. The
vbd transport uses a virtual disk, described in the <disk> element, to share
metrics data between host and VM. The virtio transport, described by the
<virtio> element, uses a virtio-serial connection to share the metrics data.

The <metrics> element is a container for all of the <metric> elements.
A metric element is used to define a metric, giving it a name and an action
that produces the metric value.

The supplied vhostmd configuration file provides a useful set of default
metrics to be collected.  This can be extended or modified by editing
/etc/vhostmd/vhostmd.conf and changing existing metric definitions or
adding new metric definitions under the metrics container.

Defined metrics begin with the <metric> element, which contains two
attributes: type and context.  The type attribute is used to describe the
metric's value type.  Supported types are int32, uint32, int64, uint64,
real32, real64, string, group, and xml. group is used when an action returns
more than one metric value. xml is the most flexible type and specifies that
the metric's action returns valid metric XML.  The context attribute is used
to indicate whether this is a host or vm metric.  Supported contexts are
host and vm.

Currently, the metric element contains 3 elements: name, action, and variable.
The name element defines the metric's name.  The action element describes a
command or pipeline of commands used to gather the metric.  For metrics of
vm context, the tokens NAME, ID, and UUID may be used where these attributes
of a VM are normally provided in a command.  When the metric is sampled, these
tokens will be substituted with the actual name, ID, or UUID of the vm currently
being sampled by vhostmd.  If the metric type is xml, action is expected to
retrun valid metric XML as defined below in "XML Format of Content".


Metrics Disk Format
-------------------

Currently, the disk format is quite simple: a raw, file-backed disk
containing a header, immediately followed by metric content.

The header contains the following, all in network-byte order

 - 4 byte signature, 'mvbd'
 - 4 byte busy flag
 - 4 byte content checksum
 - 4 byte content length

The busy flag permits simple reader/writer synchronization.  The busy
flag can be checked for clear, content read into a buffer, and the
busy flag checked again for clear to ensure stable content.


XML Format of Content
---------------------

The content is an XML document containing default and user-defined
metrics.  The format is quite similar to the metrics definitions
found in the vhostmd configuration file. A notable addition, as
illustrated below, is the <value> element containing the metric's
current value.

    <metrics>
      <metric type='real64' context='host'>
        <name>TotalCPUTime</name>
        <value>846.600000</value>
      </metric>
      <metric type='uint64' context='host'>
        <name>PageInRate</name>
        <value>0.000000</value>
      </metric>
      <metric type='uint64' context='host'>
        <name>PageFaultRate</name>
        <value>0.000000</value>
      </metric>
      <metric type='uint64' context='host'>
        <name>PagedOutMemory</name>
        <value>6885044</value>
      </metric>
      <metric type='uint64' context='host'>
        <name>PagedInMemory</name>
        <value>2367980</value>
      </metric>
      <metric type='uint64' context='host'>
        <name>FreeMem</name>
        <value>829</value>
      </metric>
      <metric type='uint64' context='host'>
        <name>UsedMem</name>
        <value>1369088</value>
      </metric>
      <metric type='uint64' context='host'>
        <name>TotalPhyMem</name>
        <value>1919</value>
      </metric>
      <metric type='uint32' context='host'>
        <name>NumCPUs</name>
        <value>2</value>
      </metric>
      <metric type='uint32' context='host'>
        <name>TotalPhyCPUs</name>
        <value>2</value>
      </metric>
      <metric type='string' context='host'>
        <name>VirtualizationVendor</name>
        <value>SUSE LINUX Products GmbH</value>
      </metric>
      <metric type='string' context='host'>
        <name>HostName</name>
        <value>laptop</value>
      </metric>
      <metric type='real64' context='vm' id='0' uuid='00000000-0000-0000-0000-000000000000'>
        <name>TotalCPUTime</name>
        <value>847.700000</value>
      </metric>
      <metric type='real64' context='vm' id='2' uuid='6be3fdb8-bef5-6fec-b1b7-e61bbceab708'>
        <name>TotalCPUTime</name>
        <value>69.400000</value>
      </metric>
    </metrics>

Default Metrics
----------------

Metrics meaning is derived from the context and the name.
Current list of default metrics supported:

Host Context:
  NumCPUsUtilized:  Number of Physical CPUs Utilized
  TotalCPUTime:     Total CPU Time
  PagedOutMemory:   Paged Out Memory
  PageInRate:       Page In Rate
  PageFaultRate:    Page Fault Rate
  FreePhyMemFree:   Physical Memory
VM Context:
  NumCPUsUtilized:  Number of Physical CPUs Utilized
  TotalCPUTime:     Total CPU Time:


Surfacing Metrics Disk to VM
----------------------------

The virtualization platform's native block device tools should be
used to surface the metrics disk to guest virtual machines.  For
example, if using Xen as the virtualization technology

  xm block-attach <vm> tap:aio:/path/to/metrics/disk /dev/xvdd r

If using a Linux-based virtualization technology (Xen, KVM, qemu, ...),
libvirt would be the preferred tool, e.g.

  virsh attach-disk <vm> /path/to/metrics/disk xvdd --driver tap
    --subdriver aio --type disk --mode readonly

You can also edit the configuration of a domain using
'virsh edit guestname' and add the following clause into the <devices>
section:

  <disk type='block' device='disk'>
    <source dev='/dev/shm/vhostmd0'/>
    <target dev='hdb' bus='ide'/>
    <readonly/>
  </disk>

(Note: Change target dev and bus as appropriate for the domain)



Notes on Virtio Transport
-------------------------

The virtio transport uses a virtio serial device to transport metrics data
between the host and VMs. Basically for a virtio serial device, QEMU creates
- a unix domain socket on the Host
- a serial port on the VM
- 'connects' both to a 'communication channel'

It can be configured in the virtio section of the vhostmd configuration file.
<max_channels> defines the maximum number of virtio channels/VMs supported
by the vhostmd instance with a default value of 1024.
<expiration_time> is the time after which the virtio serial channel of a VM
is detached when vhostmd did not receive updates. It's default value is
'3 * update_period' of the <globals> section and the configured value is
checked to be at least '3 * update_period'.

  <vhostmd>
    <globals>
      <virtio>
        <max_channels>1024</max_channels>
        <expiration_time>15</expiration_time>
      </virtio>

Sample VM config with virtio serial:

  <channel type='unix'>
    <source mode='bind'/>
    <target type='virtio' name='org.github.vhostmd.1'/>
    <address type='virtio-serial' controller='0' bus='0' port='1'/>
  </channel>

The target name of a channel in a QEMU VM configuration must be
  'org.github.vhostmd.1'


Vhostmd accepts metric requests 'GET /metrics/XML\n\n' and responds with

  <metrics>
    <metric type='real64' context='host'>
      <name>TotalCPUTime</name>
      <value>179645.910000</value>
    </metric>
  ...
    <metric type='uint64' context='vm' id='9' uuid='a70605c8-7d69-8c44-7e1a-5ecd092cb1e1'>
      <name>UsedMemory</name>
      <value>524288</value>
    </metric>
  </metrics>


Guest Tool/Library for Accessing Metrics Data
---------------------------------------------

Tool: vm_dump_metrics
 Stand alone static utility will read all the metrics and write them
 to stdout or optionally an argumented file.
 Usage:
   vm_dump_metrics -b|-i|-x [-d dest_file]

Library: libmetrics.so.0
 Dynamic library that supports individual metrics gathering


Build
------
 sh autogen.sh
 ./configure --prefix=/usr --libdir=/usr/lib64 --disable-shared
 make


Install
-------
 make install
   /etc/init.d/vhostmd
   /usr/sbin/rcvhostmd
   /etc/vhostmd/vhostmd.conf
   /etc/vhostmd/vhostmd.dtd
   /usr/share/doc/vhostmd/vhostmd.xml
   /usr/share/doc/vhostmd/vhostmd.dtd
   /usr/share/doc/vhostmd/mdisk.xml
 
 vm_dump_metrics
   Install on all interested VMs in any accessable directory

-- End