IABM Member Blog – Accelerated Ingest to Cloud Services
Signiant Flight: Cloud-Native Upload Technology for AWS & Microsoft Azure
Written by Megan Cater, Senior Manager Digital Content at Signiant
Find out more about Signiant Flight here
INTRODUCTION
We are in an era of data inundation. According to a report by IDC, by 2020 there will be as many digital bits in existence on the Internet and in storage devices as there are known stars in the universe.
Much of that data has a neglected life, created only to be forgotten moments later. But a subset is highly valued, from personal account information to business assets and intellectual property. Those are the files that we want to protect, analyze, edit, share, backup, and store.
However, as we move into a future where data will only continue to grow in amount and importance, we will need more storage for the valuable bits we want to keep as well as much improved digital mechanisms to move them around.
[bctt tweet=”As we move into a future where data will only continue to grow in amount and importance, we will need more storage for the valuable bits we want to keep as well as much improved digital mechanisms to move them around – Megan Cater, Signiant”]
Many businesses want to utilize cloud object storage services such as Amazon S3 and Microsoft Azure to solve the storage issue, and to gain access to business specific cloud applications. But moving large data to and from the cloud, especially when multiple locations or distance is involved, remains an obstacle to effectively incorporating cloud services.
This article discusses the technical reasons behind this challenge, explores the pros and cons of data transfer options provided by cloud vendors, and introduces Signiant Flight, the fastest and most flexible way to move data to and from cloud services.
CLOUD SERVICES AND THE NETWORK BOTTLENECK DILEMMA
Cloud services such as object storage fit the needs of many companies, giving them scalability and global access to their business-critical files. For many applications, it’s important to get files into and out of the cloud very quickly — and that’s where cloud services fall short, because the standard mechanisms for cloud I/O (input/output) have significant limitations, especially when dealing with latency and congestion.
[bctt tweet=”Cloud services such as object storage fit the needs of many companies, giving them scalability and global access to their business-critical files – Megan Cater, Signiant”]
Data is typically moved over the Internet, including in and out of cloud services, through HTTP (hypertext transfer protocol) and the underlying TCP (transmission control protocol). While adequate for most Internet traffic, HTTP with TCP is optimized for loading web resources in a browser, not sending large blocks of data to a destination, and don’t perform well for the transfer of large files over long distances when latency and packet loss come into play. Especially when you get high bandwidth pipes and some distance involved, HTTP with TCP starts to slow down quite dramatically.
It’s important to note that bandwidth and latency are independent variables, so simply getting a higher-bandwidth connection doesn’t solve the problem. Here’s the difference: bandwidth is the theoretical maximum rate data can be transferred over a given network or network segment (usually measured in bits per second). Latency is the amount of time it takes for data to travel across the network or network segment (usually measured in thousandths of a second, or milliseconds).
If you think of a network as a garden hose, the bandwidth is the width of the hose and the latency is the time it takes for water to travel the full length of the hose. Latency is not reduced by increased bandwidth; no matter how big your hose is, the data still has to travel the distance. In theory, data travels at the speed of light. But, in practice, switching and other overhead in the network increase latency and slow it down.
The speed of light is pretty fast: it takes light approximately 1/10th of a second (or 100ms) to travel around the world; but that’s without considering any network switching overhead — and there are typically more switches involved for longer distances. The latency when sending data halfway around the world and back can be more than 1/3 of a second.
There are two ways to look at how far the data packet has to travel: physical distance, and network distance. Even when sending data between two locations in the same building, data can travel the equivalent of across the continent if it’s on different service provider’s networks; so, it’s the network distance, rather than the physical distance that’s important. Network distance is what causes throughput bottlenecks in most systems, slowing down content transmission even if you’ve got a high-bandwidth pipe. These network bottlenecks can disrupt a successful business model if workflows depend on moving data into and out of cloud object storage or other cloud services in a fast, reliable manner.
GETTING DATA TO THE CLOUD
Public cloud platforms like as Amazon Web Services and Microsoft Azure offer a few solutions for ingesting data including free upload utilities, shipping physical media and dedicated network connections. Some of these address latency better than others, and each has pros and cons depending on the use case.
[bctt tweet=”Public cloud platforms like as Amazon Web Services and Microsoft Azure offer a few solutions for ingesting data including free upload utilities, shipping physical media and dedicated network connections – Megan Cater, Signiant”]
Upload Utilities
Cloud provider upload utilities are a useful and cost-effective way to ingest data. However, without acceleration built in, they can be slow and unreliable when transferring large amounts of data over a long distance. Some upload utilities attempt to address the latency issues with HTTP by breaking objects into chunks and uploading the chunks via multiple parallel HTTP streams. While these “multipart upload utilities” (such as Microsoft AzCopy) do not include acceleration, they can be used with Signiant Flight to optimize each stream for the most efficient data movement possible. We’ll discuss how this works on the following page.
Physical Media
For storing large amounts of data, one of the most basic approaches is to stick your content on data tapes or disk drives and ship them to a provider who then loads it into their cloud object storage. Offerings such as AWS Snowball provide a storage appliance bundled with shipping and ingest services. While shipping physical media works and you can’t beat the bandwidth of a truck, when you have to do this over and over again, it starts to have some serious logistical challenges. And it’s not very easy to automate. So, for ongoing workflows, most of us prefer an electronic, network-based approach.
Dedicated Connectivity
Dedicated network connections such as AWS Direct Connect and Azure ExpressRoute provide ways to connect directly into cloud infrastructure rather than going through the public Internet. A direct and fixed connection from your data center to your cloud provider’s data center will generally minimize latency.
[bctt tweet=”Dedicated network connections such as AWS Direct Connect and Azure ExpressRoute provide ways to connect directly into cloud infrastructure rather than going through the public Internet – Megan Cater, Signiant”]
But in order to take advantage of a dedicated connection, first you have to go to your friendly neighborhood Telco and order a fixed circuit that runs from your premise to the cloud provider’s POP. You can then buy connectivity from there to the cloud, but it isn’t cheap and isn’t available in every location.
If it is available, this approach can be a good choice if you have a predictable amount of data being transported every day, and it’s always coming from the same location. But it doesn’t work well if data is being moved from multiple locations into the cloud.
And, because you’re deploying fixed network infrastructure to address the problem, it is especially inefficient if you have peaks and valleys in the amount of data you are sending. A dedicated network is not agile or elastic and cannot adapt to changing demands — you’re paying for that connection even when you’re not using it.
In summary, a dedicated network connection is cost inefficient if your direct connection isn’t always fully utilized, it isn’t viable if data is coming from multiple locations, and you simply may not be able to get a direct connection from your location to your cloud provider of choice.
INTRODUCING FLIGHT: HIGH-PERFORMANCE CLOUD INGEST
Flight is a cloud-native SaaS offering that employs Signiant’s patented acceleration technology and scale out architecture while ensuring that all available bandwidth can be fully utilized to achieve unprecedented multi-Gbps speeds. Flight scales automatically to adapt to high data loads and is highly secure — Flight is the highest performance, most flexible way to transfer data to and from the cloud.
[bctt tweet=”Flight is a cloud-native SaaS offering that employs Signiant’s patented acceleration technology and scale out architecture while ensuring that all available bandwidth can be fully utilized to achieve unprecedented multi-Gbps speeds – Megan Cater, Signiant”]
While off-the-shelf offerings from cloud providers work in many situations, there are a growing number of mission-critical cloud use cases that demand higher-performance ingest capabilities. Signiant developed Flight, a cloud utility for accelerated ingest, to meet the needs of businesses that require more flexibility along with better speed, reliability and security in transferring data between on-premises locations and cloud object storage and other cloud services. Flight is particularly useful in distributed ingest applications, where data originates in many locations around the world.
Flight is simple to set up and connects to your on-premises storage through any client software including Signiant, AWS CLI or Microsoft AzCopy, optimizing client transfer capabilities (such as multi-part upload) and creating an accelerated and streamlined pathway directly to your cloud services.
How Flight Achieves Extremely High Transfer Rates
Flight provides end-to-end accelerated transfers to or from cloud services. Utilizing Signiant’s core file acceleration technology and Flight’s patented scale out architecture, Flight achieves extremely high transfer rates.
- Core Acceleration: Like all Signiant products, Flight leverages our core acceleration technology, which improves on standard Internet transmission speeds up to 200 fold and is increasingly impactful with longer distances, higher bandwidth and more congested networks.
- Scale Out Architecture: Flight’s patented scale out architecture delivers multi-Gbps speed by enabling the use of multiple cloud compute instances for a single transfer. An enormous improvement over first-generation accelerated cloud data transfer utilities, Flight eliminates bottlenecks associated with “scale up” transfers that transmit each file via a single cloud compute instance. In addition to the raw speed benefit, this architecture greatly simplifies the entire data management workflow. There’s no longer a need to re-organize data for optimal transfer speed. Simply initiate the transfer of any data set, and Flight will ensure the fastest possible delivery.
MORE THAN JUST SPEED: THE ADDITIONAL ADVANTAGES OF FLIGHT
Cloud-Native Implementation
Flight is a managed Software-as-a-Service offering. Signiant runs the cloud servers and software for you, so there’s no need to procure and turn up VMs, install software, and manage it in the cloud. In contrast with accelerated file transport solutions that are not truly cloud-native, with Flight you don’t have ongoing costs associated with managing your own cloud infrastructure, like applying software updates and monitoring your servers for availability.
Automatic Scalability
Flight scales automatically and can handle your largest bursts of files or a wave of new projects with auto-scaling, redundancy, and load balancing built-in. That’s all included in your SaaS subscription and you never have to worry about spinning up new servers or deploying more software.
Security
Security of transfers to and from the cloud is a top area of concern for many businesses, and Flight incorporates a number of features designed to ensure a very high level of security. The Signiant transfer protocol incorporates Transport Layer Security (TLS), and all transfers are tracked and time-stamped for traceability. Encryption in flight and end-to-end encryption are supported, along with various capabilities for maintaining the secrecy of cloud services keys and controlling access.
[bctt tweet=”Security of transfers to and from the cloud is a top area of concern for many businesses, and Flight incorporates a number of features designed to ensure a very high level of security – Megan Cater, Signiant”]
Client-Side Software Flexibility
With Flight, customers have the flexibility to use any client-side software.
Easily accelerate transfers via familiar interfaces such as web browsers or
CLIs and SDKs from cloud vendors. Flight improves the performance of any cloud workflow where AWS or Microsoft Azure client software is already embedded.
Global Access to Any Cloud Service
Flight is the only solution that makes it easy to move files to multiple cloud platforms from the same system, allowing access to any cloud service, including streams. This gives you the freedom to choose the best platform and services for your business, so you can avoid cloud vendor lock in or use multiple cloud platforms like AWS and Microsoft Azure. With the same Flight subscription, you can put some assets in AWS and some in Azure— or switch providers if your business needs change.
4 STEPS TO GET STARTED
Fast, easy to use, easy to get started, scalable to your requirements, and compatible with multiple clouds, Flight requires only four simple steps to get started:
1. Install the Flight proxy software on your premises,
2. Configure your client application to use Flight as a proxy,
3. Configure your destination cloud storage/service in the Flight control console,
4. Start transferring to and from the cloud. Within minutes, you will be moving content to your cloud services such as object storage in an accelerated, high-performance, and secure manner.
SUMMARY
Slow access to cloud storage and other services has been a barrier to cloud adoption for many businesses. Flight’s advanced acceleration technology and scale out architecture addresses throughput bottlenecks and latency, providing unprecedented multi-Gbps transfer rates with enhanced reliability and security.
Flight is a true SaaS solution — Signiant runs the servers and software in the cloud so you don’t have to, and it scales automatically as data loads change.
Increasingly, cloud data storage and other cloud services are becoming a critical aspect of software-defined companies. And, when time is of the essence, the power to efficiently interact with cloud services provides a definite competitive edge.
If your business needs a high-speed bridge to the cloud, talk to a specialist to discuss how Flight and Signiant can help you achieve your cloud strategy goals.
[bctt tweet=”Increasingly, cloud data storage and other cloud services are becoming a critical aspect of software-defined companies. And, when time is of the essence, the power to efficiently interact with cloud services provides a definite competitive edge – Megan Cater, Signiant”]
ABOUT SIGNIANT
Signiant software enables high-speed movement of large, valuable files in their entirety over public and private networks. With acceleration technology to minimize the impact of latency, enterprise class security, and central control, our software powers most of the world’s Media & Entertainment companies — from small post-production houses to Hollywood studios — and is increasingly being adopted for use with satellite imagery, big data analytics, and genome sequencing. At Signiant, we’ve embraced cloud-native software development, and offer unique SaaS file transfer and cloud storage upload solutions.
