Utilizing Cloud Storage Like Never Before

We use the cloud every day: Google drive, Microsoft Azure, OneDrive, Box. It is what allows us to access our information from anywhere, at any time. Even if your phone finally decides to break for good, you can still access those priceless photos.

Despite the cloud being ubiquitous, however, storage on cloud platforms is seriously underutilized, according to new research from Electrical and Computer Engineering professor Jian Huang - who is also an Illinois Computer Science affiliate faculty member - and his students, in collaboration with Microsoft Research. “We found that actually only ~30% of storage is utilized. And so that’s the problem we’re trying to solve, how can we improve storage utilization,” said Huang. 

Illinois CS PhD students Benjamin Reidys and Jinghan Sun recently presented their work on enabling storage harvesting to improve storage utilization in cloud platforms at the 16^th Annual USENIX Symposium on Operating Systems Design and Implementation in San Diego.

“The core idea of storage harvesting,” Huang explained, “is the way we enable the cloud storage infrastructure to harvest unused storage capacity and bandwidth from other virtual machines (VMs).”

Sun expanded, saying “it is a more elastic usage of idle resources in the cloud.” Furthermore, “we also want to make storage more cost-efficient. We don’t want to waste cloud resources when they’re not being used, therefore, we can reduce the operating cost for cloud platform operators."

Cloud storage is a core service in data centers today, so why is it so underutilized?

There are two main reasons for this underutilization: allocated and unallocated storage.

Allocated storage: Storage allocation is conducted in a static manner, meaning there is no flexibility in the amount of storage allocated. However, the storage usage of each end user varies and may change over time. For example, each student and staff member on our campus will have a default storage amount in Google Drive, but many of us will not use the entirety of that allocated storage.

Unallocated (unsold) storage: Cloud storage providers will also over-provision VMs to meet the flexibility requirement from their customers. Most people during the day aren’t on their social media profiles, but in the evening at home, they relax and log in for an evening of scrolling their feeds and using online services to interact with their friends. Cloud providers must reserve sufficient resources to handle this situation. Since each VM that is unsold requires a certain amount of resources (memory, storage, and processor cores) the result will be storage resources being underutilized.

In a hypothetical analysis of storage resources, Reidys and Sun report that both allocated and unallocated storage have sufficient storage capacity and bandwidth to harvest and both are available for a long enough duration to facilitate harvesting.

To improve resource utilization, a different VM technique called a harvest VM shows promise. Harvest VMs are lower priority than regular VMs, meaning they can be kicked out if resources are needed by a regularly allocated VM. They can grow to utilize available resources but they can also shrink to avoid eviction by the regularly allocated VMs.

Storage harvesting comes with its own challenges, however.

“Current system virtualization support is hard. If you want to harvest storage, you need to know what resources are available, where those resources are, how much and for how long they’re available,” Reidys explained.

BlockFlex is a learning-based storage harvesting framework that aims to improve the utilization of cloud storage by harvesting available storage resources (both allocated and unallocated). Sun said that “BlockFlex utilizes a lightweight machine learning technique to help us predict how many resources are available for harvesting, and for how long.”

For the harvest VMs, BlockFlex will make a prediction of the necessary storage capacity and bandwidth as well as for how long the demand will last, rather than relying on users to specify their storage resources.

Storage harvesting presents another concern: data security. Since resources are being harvested, it is important that those resources don’t contain stored sensitive information. To ensure data security, BlockFlex erases data after harvesting. Reidys reassured that “there’s nothing that can be learned about other user data in the cloud and sensitive data can’t be leaked” from using BlockFlex.

Cloud storage utilization has been improved by implementing BlockFlex: Reidys and Sun report that BlockFlex improves storage utilization by 1.3x on average and improves harvest VM performance by up to 60%. While storage utilization will never get to 100%, it can certainly be improved and BlockFlex is a step in that direction.

Read the original story from the Coordinated Science Laboratory.