IOPS reservations in SIOC 6 , what’s the deal?

Storage I/O control has been available for a long time now since vSphere 4.1, if you don’t know what SIOC is you can read about it in many blogs out there, my personal favorite for anything storage is Cormac Hogan‘s blog, here is also a link to Cormac’s post about SIOC.

Some of you might have read about the new SIOC feature in vSphere 6 called IOPS reservations.

In case you didn’t let’s quickly review it, In version 5.5 VMware introduced a new scheduler call mClock, this I/O scheduler is more efficient but also it has the capability to set I/O reservations on VMDK’s. In vSphere 6 VMware added the ability to set those reservations on the VMDK level, not through the web client but by setting the “reservation” property on the VMDK, see this post by William Lam that has a nice PowerCLI script to do this for you.

How does SIOC know what the storage device is capable of delivering in terms of IOPS? This is done using the I/O injector that now supports testing not only the latency capability of the device (which is used to set the dynamic threshold)  but also it’s IOPS capability. It does that by injecting random reads to the storage device, this works at the same schedule as it always did which is once every 24 hours and only when the storage device is idle, same ol’ injector enhanced.

The thing I would like to note here is that an IOPS reservation does not behave like the reservation of RAM you know from DRS, it is not a guarantee of a specific amount of IOPS, the term reservations might be confusing, for example the system will not pre-check IOPS capabilities and prevent an sVmotion from happening if it thinks IOPS capabilities will be violated, nor the creation of new VMDKs or an increase in the reservations of VMDK’s will be stopped for violation. It is more a guarantee for a slice of the IOPS capabilities of the storage device when it is over -reserved.
Allowing for over-reservation does make sense, IOPS capabilities is not a pool of resources like RAM and is more fluid and subject to change even after you already set reservations for teh VMDK’s that reside on it, a storage device’s IOPS capabilities can go up or down based on configuration changes of the physical storage device or other factors like cache utilization.

What happens if I over-reserve the datastore(e.g. I sampled it for 2000 IOPS and reserved 3000)?  At that point the IOPS will be scheduled to the VMDK’s based on the proportional allocation of reservations instead of the proportional allocation of shares, practically this means that SIOC switches from managing latency to managing IOPS. You can find here  post that explains this change very nicely with graphics in the VMware blogs.

Also, note that since sVmotion is a VMDK attribute it might take longer to complete when SIOC kicks in because it is subject to the IOPS allocation policy of SIOC.

If you have ideas about teh direction SIOC should take in the future, the VMware storage BU are looking for feedback from the field on how SIOC is being utilized today and which changes are required to boost its adoption, if you want to influence this very important feature please submit your thoughts to the survey here.

I hope this clears things up and helps better understand SIOC IOPS reservations, I recently delivered a session about SIOC as part of vBrownbag VCP6-DCV series, I will add a link to it in this blog post as soon as it will be published.

As always, comments are welcome,

Cheers,

Niran