XAPI RRDs
xapi stores persistent performance data in 'round robin databases' (RRDs). Each of these is a fixed size structure containing data at multiple resolutions. 'Data sources' are sampled every few seconds and points are added to the highest resolution RRD. Periodically each high-frequency RRD is 'consolidated' (e.g. averaged) to produce a data point for a lower-frequency RRD.
Jon Ludlam writes:
RRDs are maintained for individual VMs (including dom0) and the host. Internally, the RRD is updated and maintained by a module similar to (but not actually) rrdtool (http://oss.oetiker.ch/rrdtool/). RRDs are resident on the host on which the VM is running, or the pool master when the VM is not running. For this reason, to obtain the data requires knowledge of where the VM is running.
The RRDs can be extracted via a http handler registered at /host_rrd or /vm_rrd. Both of these require authentication to retrieve the data, either by use of http auth, or by a XenAPI session. The session authentication works by passing the 'OpaqueRef:...' as the query parameter 'session_id'. The vm_rrd handler requires the uuid of the VM, passed by the query parameter 'uuid'. As an example, connecting to the URI 'http://<server>/host_rrd' using a web browser will prompt for a username and password, then get an XML dump of the RRD. This XML is documented on the rrdtool webpage, and can be imported into rrdtool itself for further analysis. Example URIs:
http://localhost/host_rrd?session_id=OpaqueRef:abcd.... http://localhost/vm_rrd?uuid=abcd...&session_id=OpaqueRef:cdef...
It's my expectation that this won't be used as often as the following updates URI.
Recent updates to the RRDs can also be queried so that the entire RRD does not need to be downloaded when you already have most of the data. This is via another http handler: /rrd_updates. Once again this needs to be authenticated either using a session or http auth. The parameter 'start' needs to be provided, and is a Unix-type 'number of seconds since Jan 1 1970'. This will provide data in an rrdtool 'xport' style xml format, for every VM resident on the particular host that is being queried. In order to differentiate which column in the export is associated with which VM, the 'legend' field is prefixed with the VM's uuid. Also the type of archive from which it came is also prefixed, e.g. AVERAGE or MIN, etc. To obtain host updates too, use the query parameter 'host=true'. Example URI:
http://localhost/rrd_updates?session_id=...&cf=AVERAGE&start=1000000000&host=true
Example host rrd (note that a VM rrd is identically structured, but with different data sources):
<?xml version="1.0"?>
<rrd>
<version>0003</version>
<step>5</step>
<lastupdate>1213616574</lastupdate>
<ds>
<name>memory_total_kib</name>
<type>GAUGE</type>
<minimal_heartbeat>300.0000</minimal_heartbeat>
<min>0.0</min>
<max>Infinity</max>
<last_ds>2070172</last_ds>
<value>9631315.6300</value>
<unknown_sec>0</unknown_sec>
</ds>
<ds>
<!-- other dss - the order of the data sources is important
and defines the ordering of the columns in the archives below -->
</ds>
<rra>
<cf>AVERAGE</cf>
<pdp_per_row>1</pdp_per_row>
<params>
<xff>0.5000</xff>
</params>
<cdp_prep> <!-- This is for internal use -->
<ds>
<primary_value>0.0</primary_value>
<secondary_value>0.0</secondary_value>
<value>0.0</value>
<unknown_datapoints>0</unknown_datapoints>
</ds>
...other dss - internal use only...
</cdp_prep>
<database>
<row>
<v>2070172.0000</v> <!-- columns correspond to the DSs defined above -->
<v>1756408.0000</v>
<v>0.0</v>
<v>0.0</v>
<v>732.2130</v>
<v>0.0</v>
<v>782.9186</v>
<v>0.0</v>
<v>647.0431</v>
<v>0.0</v>
<v>0.0001</v>
<v>0.0268</v>
<v>0.0100</v>
<v>0.0</v>
<v>615.1072</v>
</row>
...
</rra>
... other archives ...
</rrd>
Example rrd_updates - only 1 VM present, no host updates:
<xport>
<meta>
<start>1213578000</start>
<step>3600</step>
<end>1213617600</end>
<rows>12</rows>
<columns>12</columns>
<legend>
<entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry> <!-- nb - each data source might have multiple entries for different consolidation functions -->
<entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
<entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
<entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
<entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
<entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
<entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
<entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
<entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
</legend>
</meta>
<data>
<row>
<t>1213617600</t>
<v>0.0</v> <!-- once again, the order or the columns is defined by the legend above -->
<v>0.0282</v>
<v>209715200.0000</v>
<v>0.0</v>
<v>0.0201</v>
<v>209715200.0000</v>
<v>0.0</v>
<v>0.0445</v>
<v>209715200.0000</v>
<v>0.0</v>
<v>0.0243</v>
<v>209715200.0000</v>
</row>
...
</data>
</xport>
Here is some sample python code to read and parse the rrd_updates - it uses the local domain socket to log in to xapi, and hence the code must be executed on the server. To modify it to work off-box, simply change the bit that gets the session and the URI it's requesting:
#!/usr/bin/python
#
# Example code for reading RRDs
# Contact: Jon Ludlam (jonathan.ludlam@eu.citrix.com)
#
# Mostly this script is taken from perfmon, by Alex Zeffert
#
import XenAPI
import urllib
from xml.dom import minidom
from xml.parsers.expat import ExpatError
import time
# Per VM dictionary (used by RRDUpdates to look up column numbers by variable names)
class VMReport(dict):
"""Used internally by RRDUpdates"""
def __init__(self, uuid):
self.uuid = uuid
# Per Host dictionary (used by RRDUpdates to look up column numbers by variable names)
class HostReport(dict):
"""Used internally by RRDUpdates"""
def __init__(self, uuid):
self.uuid = uuid
class RRDUpdates:
""" Object used to get and parse the output the http://localhost/rrd_udpates?...
"""
def __init__(self):
# params are what get passed to the CGI executable in the URL
self.params = dict()
self.params['start'] = int(time.time()) - 1000 # For demo purposes!
self.params['host'] = 'true' # include data for host (as well as for VMs)
self.params['cf'] = 'AVERAGE' # consolidation function, each sample averages 12 from the 5 second RRD
self.params['interval'] = '60'
def get_nrows(self):
return self.rows
def get_vm_list(self):
return self.vm_reports.keys()
def get_vm_param_list(self, uuid):
report = self.vm_reports[uuid]
if not report:
return []
return report.keys()
def get_vm_data(self, uuid, param, row):
report = self.vm_reports[uuid]
col = report[param]
return self.__lookup_data(col, row)
def get_host_uuid(self):
report = self.host_report
if not report:
return None
return report.uuid
def get_host_param_list(self):
report = self.host_report
if not report:
return []
return report.keys()
def get_host_data(self, param, row):
report = self.host_report
col = report[param]
return self.__lookup_data(col, row)
def get_row_time(self,row):
return self.__lookup_timestamp(row)
# extract float from value (<v>) node by col,row
def __lookup_data(self, col, row):
# Note: the <rows> nodes are in reverse chronological order, and comprise
# a timestamp <t> node, followed by self.columns data <v> nodes
node = self.data_node.childNodes[self.rows - 1 - row].childNodes[col+1]
return float(node.firstChild.toxml()) # node.firstChild should have nodeType TEXT_NODE
# extract int from value (<t>) node by row
def __lookup_timestamp(self, row):
# Note: the <rows> nodes are in reverse chronological order, and comprise
# a timestamp <t> node, followed by self.columns data <v> nodes
node = self.data_node.childNodes[self.rows - 1 - row].childNodes[0]
return int(node.firstChild.toxml()) # node.firstChild should have nodeType TEXT_NODE
def refresh(self, session, override_params = {}):
params = override_params
params['session_id'] = session
params.update(self.params)
paramstr = "&".join(["%s=%s" % (k,params[k]) for k in params])
# this is better than urllib.urlopen() as it raises an Exception on http 401 'Unauthorised' error
# rather than drop into interactive mode
sock = urllib.URLopener().open("http://localhost/rrd_updates?%s" % paramstr)
xmlsource = sock.read()
sock.close()
xmldoc = minidom.parseString(xmlsource)
self.__parse_xmldoc(xmldoc)
# Update the time used on the next run
self.params['start'] = self.end_time + 1 # avoid retrieving same data twice
def __parse_xmldoc(self, xmldoc):
# The 1st node contains meta data (description of the data)
# The 2nd node contains the data
self.meta_node = xmldoc.firstChild.childNodes[0]
self.data_node = xmldoc.firstChild.childNodes[1]
def lookup_metadata_bytag(name):
return int (self.meta_node.getElementsByTagName(name)[0].firstChild.toxml())
# rows = number of samples per variable
# columns = number of variables
self.rows = lookup_metadata_bytag('rows')
self.columns = lookup_metadata_bytag('columns')
# These indicate the period covered by the data
self.start_time = lookup_metadata_bytag('start')
self.step_time = lookup_metadata_bytag('step')
self.end_time = lookup_metadata_bytag('end')
# the <legend> Node describes the variables
self.legend = self.meta_node.getElementsByTagName('legend')[0]
# vm_reports matches uuid to per VM report
self.vm_reports = {}
# There is just one host_report and its uuid should not change!
self.host_report = None
# Handle each column. (I.e. each variable)
for col in range(self.columns):
self.__handle_col(col)
def __handle_col(self, col):
# work out how to interpret col from the legend
col_meta_data = self.legend.childNodes[col].firstChild.toxml()
# vm_or_host will be 'vm' or 'host'. Note that the Control domain counts as a VM!
(cf, vm_or_host, uuid, param) = col_meta_data.split(':')
if vm_or_host == 'vm':
# Create a report for this VM if it doesn't exist
if not self.vm_reports.has_key(uuid):
self.vm_reports[uuid] = VMReport(uuid)
# Update the VMReport with the col data and meta data
vm_report = self.vm_reports[uuid]
vm_report[param] = col
elif vm_or_host == 'host':
# Create a report for the host if it doesn't exist
if not self.host_report:
self.host_report = HostReport(uuid)
elif self.host_report.uuid != uuid:
raise PerfMonException, "Host UUID changed: (was %s, is %s)" % (self.host_report.uuid, uuid)
# Update the HostReport with the col data and meta data
self.host_report[param] = col
else:
raise PerfMonException, "Invalid string in <legend>: %s" % col_meta_data
def main():
xapi = XenAPI.xapi_local();
xapi.login_with_password("","")
session=xapi._session
rrd_updates = RRDUpdates()
rrd_updates.refresh(session,{})
for uuid in rrd_updates.get_vm_list():
print "Got values for VM: "+uuid
for param in rrd_updates.get_vm_param_list(uuid):
print "param: "+param
data=""
for row in range(rrd_updates.get_nrows()):
data=data+"(%d,%f) " % (rrd_updates.get_row_time(row),
rrd_updates.get_vm_data(uuid,param,row))
print data
main()
Definitions of counters
Host
| memory_total_kib |
| memory_free_kib |
| xapi_memory_usage_kib |
| xapi_free_memory_kib |
| xapi_live_memory_kib |
| xapi_allocation_kib |
| cpuN |
| loadavg |
| pif_IF_rx |
| pif_IF_tx |
| pif_IF_rx_errors |
| pif_IF_tx_errors |
| sr_SR_cache_size |
| sr_SR_cache_hits |
| sr_SR_cache_misses |
VM
| cpuN | |
| memory | Memory currently allocated to VM(Bytes) |
| memory_target | |
| memory_internal_free | Free memory size(KB) reported by the guest agent |
| vif_IF_tx | |
| vif_IF_rx | |
| vif_IF_tx_errors | |
| vif_IF_rx_errors | |
| vbd_DEV_write | |
| vbd_DEV_read | |
| vbd_DEV_write_latency | |
| vbd_DEV_read_latency | |
| runstate_fullrun | |
| runstate_full_contention | |
| runstate_concurrency_hazard | |
| runstate_blocked | |
| runstate_partial_run | |
| runstate_partial_contention |
