Integrating Cloud Storage and Delivery of Large Files and Video into an Intranet

Ben Cole

Netsight improves usability and server performance by integrating into an intranet cloud-based storage and delivery of videos and large files. This solution hands off uploads, so that content editors no longer need to wait for file uploads to complete. It also provides automated transcoding and resolves performance bottlenecks.

One of Netsight’s customers has a Plone-based intranet, which they use for distributing videos to large numbers of employees. The publication of these corporate-message videos is accompanied by an automatic email alert, notifying all employees that the video is ready to be viewed. The mass notification tends to produce issues when large numbers of users try to view the same video concurrently, using up all the available bandwidth of the server and creating a general slowdown of the site for all users.

In addition to the performance issues, the customer also had ongoing issues when videos supplied by various production companies were in formats unsuitable for playback in a browser. This meant they’d need either to transcode themselves or (more commonly) give videos to us to transcode.

Our intention, then, was to host this large content in cloud storage and instruct the user’s browser to request it from the new location. This would provide a seamless way for the content editor to upload any video and have it play in the browser, with any transcoding being taken care of behind the scenes.

For the cloud solution we chose Amazon Web Services (AWS), as the customer already had an account with them. The Amazon Elastic Transcoder service provides a convenient solution to the customer’s transcoding requirements, and their Simple Storage Service (S3) provides a simple, secure and cost-effective solution for storing arbitrary data in the cloud.

To improve the user experience whilst uploading content to S3, we opted to make use of Celery, a distributed task queue written in Python. Celery allows us to run tasks asynchronously, allowing Plone to finish serving the user’s browser request whilst the lengthy upload job runs in the background. We then updated the download action within Plone, so that when the user clicks the download link of a file we instead generate a short-lived URL to the content in S3 and redirect the request to that. The user then seamlessly downloads the file from S3 rather than Plone. This same method is also used when streaming videos in the browser; with the content being served from S3 rather than Plone, this frees up Plone to serve other requests whilst the user watches the video.

This solution has solved the immediate needs of our customer, by resolving the concurrency and transcoding issues. As we developed this solution, however, we realised there were many opportunities to expand on the functionality and offer a more flexible solution to benefit a wider audience. We are now working to abstract away the reliance upon Amazon AWS and make the solution more modular. This will allow us to write other backends to make use of other cloud services, or indeed just store the files locally and have them served by a separate web server, which will still provide the performance benefit of not serving the files through Plone.

Tweet about this on TwitterShare on FacebookShare on Google+