The cloud has opened up opportunities for virtually every industry. The ability to quickly acquire all the compute resources you need has made projects that once seemed infeasible completely practical. Do you have an unexpected rendering job and no available capacity on premises? Use the cloud to spin up the extra nodes. Do you want to use machine learning algorithms to analyze a large pool of gene expression data? Create a virtual supercomputer in the cloud and get years of compute in a few hours.
But life in the cloud isn’t perfect. One of the biggest gotchas for many companies is file storage. While object storage is a mature technology and actively advocated by cloud providers, file storage in the cloud comes with many limitations and pain points. Lack of protocol support, performance limitations, problematic retrofitting of legacy solutions for the cloud, and data mobility are four common problems.
Cloud file storage gotchas
In terms of protocol support, customers who rely on SMB file services often find that it’s not supported. You’ll find SMB file storage on Azure, but elsewhere in the cloud file storage is much more likely to support NFS.
Performance for cloud-based file storage is tricky. Many products either have limited scalability or performance is explicitly tied to capacity. The latter is true of the AWS Elastic File System (EFS). Customers who don’t have enough data stored but want better performance sometimes store dummy files to get more bandwidth. In other words, they resort to creating false capacity to get the performance they need.
Shoehorning legacy, on-prem storage approaches onto the cloud comes with many difficulties. For instance, setting up a fleet of Windows Distributed File System (DFS) instances in AWS allows the use of the popular SMB protocol, but management of the required licensing server component becomes a single point of failure that makes access to file-based data unreliable. (DFS is a Windows Server feature that allows you to create a single namespace to provide a replicated sharing infrastructure across the network.)
Finally, most customers need data mobility. They want to move data back and forth between their on-prem storage and their cloud-based storage. It’s a rare user who stores everything in the cloud.
When faced with so many hurdles, some companies feel that they have no choice but to rewrite their applications to use object storage rather than file storage. Obviously, this is an expensive, time-consuming approach and not always possible. For many companies, all their applications are file-based and use file service protocols to communicate with each other. The need to rewrite these applications to use object storage is preventing many organizations from moving their file-based workloads to the cloud.
Introducing Qumulo File Fabric
Qumulo File Fabric (QF2) is a modern, file storage system that can scale to billions of files and that runs in the data center and in AWS. In QF2, cloud instances or computing nodes with standard hardware work together to form a cluster that has scalable performance and a single, unified file system. All QF2 clusters form a globally distributed but highly connected storage fabric tied together with continuous replication. In the rest of this article, we’ll concentrate on how QF2’s capabilities solve the problems people have encountered when looking for file storage in the cloud.
QF2 protocol support
QF2 supports SMB, NFS, and FTP. This means that you can use QF2 for AWS as a standard file server and easily allocate SMB shares or NFS exports to your users.
QF2 performance
QF2 meets the high performance requirements for heavy compute workloads on premises and in the cloud. A standard four-node cluster that runs on m4.16xlarge AWS EC2 instances typically has a read throughput of 1.8 GB/s and a write throughput of 1.3 GB/s. FuseFX, a visual special effects studio, used a four-node QF2 cluster to support a rendering farm of 1000 compute EC2 instances (32,000 cores) that achieved a peak read throughput of 3.87 GB/s and peak IOPS of 40K.
Performance and capacity can scale linearly with QF2. Adding another EC2 instance with the Qumulo AMI installed adds both. You can also scale performance and capacity independently of each other, depending on the number and sizes of EC2 instances you choose.
QF2 data mobility
QF2 uses continuous replication to move data where it’s needed, when it’s needed. Continuous replication creates a copy of the data in a directory on your primary cluster and transfers it to a directory on a second, target cluster. Continuous replication is always running (unless you configure it to not run during certain hours of the day or days of the week). QF2 takes your latest changes and replicates them without your needing to worry about it.
Continuous replication means you can easily transfer data from your on-prem QF2 cluster to your QF2 cluster in AWS, perform your computations, and then transfer the results back to the on-prem storage.
The Qumulo REST API
Cloud architects and devops engineers expect to be able to access a cloud service programmatically and to automate its configuration and management. With QF2, you can inspect and control the file system with a UI, command-line tools, and a REST API.
The QF2 UI gives you a complete picture of what’s happening in the file system in real time. Here is the main dashboard.
QumuloWhereas the dashboard gives you an overview of the cluster, you can also drill down to the file level using the different menu items to get more specific information. All the information available from the dashboard is also available programmatically through the Qumulo REST API.
Design of the Qumulo API
The Qumulo API covers both the control plane (system configuration and statistics) and the data plane (files and metadata stored in the file system). Its response content, unlike file system protocols such as SMB or NFS, returns all the information that is available.
Endpoints are classified as one of these resource archetypes:
- Document: A database record, with both fields and links to related resources.
- Collection: A directory of resources (usually document resources).
- Controller: An executable function (for example, send-mail).
- Store: A client-managed resource repository (generally not used in the API).
The URIs are structured so that it’s simple to write scripts against the endpoints or to use tools such as Curl. For example, here’s how to read a directory:
/v1/files/%2F/entries/?limit=1000
where:
/v1: This is the version of the endpoint, which always comes first./files/: This is the document, collection, or controller to access. In this case, /files/ is a collection.%2F: This is the ID of a document in the files collection. In this case, this is the root directory of the file system./entries/: This is the action to take on the specified file or folder.?limit=1000: This is an optional query parameter.
The only HTTP header required is authorization for OAuth2-style bearer tokens:
curl -k -X GET -H “Authorization: Bearer <token>” https://server:8000/v1/file-system
To learn more about the design of the Qumulo API, such as how it prevents accidental overwrites and how it handles long-running actions, read “Engineering at Qumulo: REST API.”
Using the Qumulo API interactively
You can experiment with the API interactively by clicking “API & Tools” from the dashboard. Here is an example of what you’ll see.
QumuloIf you scroll down, under Files, you’ll see the method for listing the entries in a directory.
QumuloIf you click on the method, it expands to show the call and the information you need to enter. For this method, you need to enter the directory you want to read and the number of files you want to list.
QumuloIn this example, the directory is CERT_ROOT and the maximum number of files returned is 1000.
After you click “Try It,” you get the response header, which shows if the call succeeded.
QumuloIf the call succeeded, the rest of the response shows the information you requested. Here is an example.
QumuloUsing the Python library for the Qumulo API
You can also access the API with Python. The instructions for installing and using the Python library are here. For example, here is how to create a file.
# create file
curl -X POST \
-H "Content-Type: application/json" \
-H"Authorization: Bearer $API_TOKEN" \
-k https://$API_HOSTNAME:8000/v1/files/%2F/entries/ \
-d "{
\"action\": \"CREATE_FILE\",
\"name\": \"test-file1.txt\"
}"
You can see more examples of how to use Python here.
Deploying QF2 for AWS with Terraform
You can use popular automation platforms to automatically deploy QF2 for AWS. As an example, you can use the Terraform template, qumulo_clustered.tf, to deploy QF2 clusters. Set the number of nodes in the cluster_config variable to either 1 or to a number greater than or equal to 4. You can also use a .tfvars file to automatically populate the variables.
Try QF2 for yourself
You can create a standalone instance of QF2 on AWS with 5TB of storage for free. You pay only for the underlying AWS infrastructure. After adding the QF2 AMI to your AWS account, work through the tutorial called “Installing a single-node cluster on AWS” or watch the accompanying video.
If you would like to try out a multi-node cluster, call us at +1 (206) 260-3588 or chat with us online and learn how you can qualify for 10,000 free QF2 consumption units (Qnits). With 10,000 Qnits, you’ll have more than enough time to evaluate large-scale production workloads on a four-node QF2 cluster.
—
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.