Using AWS S3 to Power Your Digital World

As a designer, web developer, and techie-geek, I need a versatile and robust data storage solution that I can afford, but also use without learning some new language. So far, I’ve only found one service that can handle the large majority of my needs. This article covers how I use the Amazon Web Services Simple Storage Service (AWS S3) to meet most of my needs.

AWS S3

AWS S3 is Amazon’s cloud storage solution. It’s versatile, reliable, fast, and scalable to fit almost anyone’s needs. Of course with a service that sounds this great you would expect it to be expensive but it’s actually the most affordable storage solution I’ve found on the web, considering the features you get.

AWS S3 is intended for developers, but thanks to some great tools, it’s easy enough for just about anyone to use. Before I get into how I use AWS S3, I want to mention that this storage solution doesn’t use the traditional file structure of folders/files, etc. Instead AWS S3 uses “buckets” in which you store objects. The tools I use make AWS S3 appear to be a normal file system with the exception of “buckets”. Think of a bucket as a separate hard drive where you’ll store your files. You might also want to read the Amazon S3 page on Wikipedia. So let’s get on with how I use AWS S3.

AWS S3 + Jungle Disk

I probably use Jungle Disk the most often because it makes it easy to use and manage my AWS S3 buckets, perform automated backups and centralize my data for access anywhere, at any time. When you use Jungle Disk with your AWS S3 account, you decide which of your individual buckets Jungle Disk can mount as a network drive. Then, you have drag-and-drop access to your AWS S3 files! Jungle Disk also encrypts your files, so they’re safe and secure.

Jungle Disk has plenty of options for bucket management, automatic backups, encryption, bandwidth limiting, and even more. It also has a monitoring tool to view and manage transfers in progress. It typically runs in the background, but it comes in very handy when you would like to take action on something or just watch what’s going on.

If you’re worried about cross-platform compatibility, don’t be! Jungle Disk has versions of their software for 32- and 64-bit Windows, Linux and Mac. They even have a version that you can run from a USB flash drive on all three platforms for quick access to your files from anywhere.

Of course, if you forget your flash drive, they also have web access to your files. If you work with other people who need access to your files, Jungle Disk can do that, too. They have multi-user options to make accessing AWS S3 buckets very easy for several people.

So, we have cross-platform cloud storage that’s drag-and-drop easy and that we can access anywhere with tons of great options. What else do we need?

AWS S3 as a “CDN” or Public File Access

Most of you probably have blogs or websites that you have hosted on a web server you pay for. As we all know, quality web hosting isn’t cheap, especially when it comes to storage space. I don’t want to use my expensive web server storage for images and other file downloads and I especially don’t want to bog down my web server with file requests from visitors when there’s a better way to do it.

S3Fox for Firefox

S3Fox is a Firefox addon that lets you manage your AWS S3 buckets and files. Why do we need S3Fox when we could use Jungle Disk? S3Fox does a few things Jungle Disk wasn’t intended for, such as managing CloudFront distributions which we’ll get into later. I’ve setup a bucket called “files.jremick.com” which I plan on using to host images and files for my blog as well as other websites and other random purposes.

Then I setup a CNAME on my web server directing “files” and “www.files” to “files.jremick.com.s3.amazonaws.com.” which will then allow me to use the subdomain “http://files.jremick.com” to access files I’ve placed in the “files.jremick.com” bucket for public viewing. The other two are used by CloudFront which we’ll get into later.

So now we have an easy way to access files at http://files.jremick.com. We could use it as a sort of “CDN” (even though it wouldn’t be a true CDN) or we could just use it to provide file downloads that won’t bog down our web server. If you’re wondering, yes, you can view and download the panorama image from my S3 account and no, I’m not worried about bandwidth because it’s super cheap! :-) You can find it here: http://files.jremick.com/red-rock-panorama.jpg. Did you notice the “wp-content” directory? Familiar eh? On to using AWS S3 with WordPress!

AWS S3 plugin for WordPress

The AWS S3 plugin for WordPress is one of my favorite plugins for WordPress because it lets me use my AWS S3 account to host media for my blog rather than my expensive web server. Of course I could do this manually if I wanted but the plugin integrates this functionality with WordPress so I can upload files without leaving my WordPress control panel.

You might be wondering why this is beneficial. Well, for starters, images and other media loaded from your AWS S3 account will likely load faster simply because you’re using Amazon’s servers rather than your own (possibly puny) server. Also, your web server won’t be bogged down loading these media files and your regular PHP/HTML files.

Your website will also load faster for most people because in most browsers you are limited to the number of parallel downloads from a single domain. If you’re hosting your images on your AWS S3 account which will be from a secondary domain then browsers will be able to load more files at the same time. See Maximizing Parallel Downloads in the Carpool Lane for more information.

AWS S3 + CloudFront

OK, so I’ve covered how I use AWS S3 for networked storage as well as for my websites and reducing the load on my web server. If you run a high traffic website (which I don’t) or you’re just a nerd (like me) and want things to run as fast as possible then you’ll want to check out Amazon CloudFront as well.

Earlier in the article I put “AWS S3 as a ‘CDN’ or Public File Access” with CDN in quotes. The reason I did that is because AWS S3 is NOT a true CDN. A CDN is a Content Delivery Network that delivers your files from a distribution of servers around the world. Visitors get access to your files from the fastest resource available (usually the closest server). AWS S3 only has a few data centers around the world and your data will most likely be in one location making it far from a CDN.

If you want the best speed for visitors across the globe, you’ll want to use a real CDN like CloudFront. Thankfully Amazon has made it super easy to use these services together. I’ve already signed up for CloudFront and now I just need to configure it using S3Fox.

Simply right click on the bucket you want distributed to Amazon’s CloudFront and click “Manage Distributions”. From here you can configure your CloudFront distribution. You’ll be assigned a unique domain for the distribution; “d1i7xb2p8w9276.cloudfront.net” is what this distribution has been assigned.

I’ve also used “cdn.jremick.com” as the CNAME for this distribution so I can access the files at http://cdn.jremick.com. You’ll see the status as “InProgress” until the distribution has been deployed and the status will change to “Deployed”.

Then I setup the CNAME on my web server.

Now when I request files at http://cdn.jremick.com they will be requested from the CloudFront servers which will pull the files from your AWS S3 account and cache them for all subsequent requests.

There are some disadvantages to CloudFront (and other true CDNs) though. Once a file has been cached on the CloudFront servers, it won’t be requested from your AWS S3 account again. That means you’ll need to version your files (filename_v1.css, filename_v2.css, etc.) so they’ll actually reflect the changes for your users. It’s a great service but it really is intended more for high traffic purposes. In most situations for average people with blogs, AWS S3 will do just fine. I will be using CloudFront to host JavaScript, CSS and other static files though, just because I’m a nerd and I want performance! :-)

AWS S3 + S3Sync = Automated Offsite Server Backups

I’m a worry wart when it comes to losing data. My web server hosts around 20 accounts for other people and it’s very important to make sure all that data is backed up, safe and secure. That’s where S3Sync comes in. I can use it to automatically backup my web server to a specified AWS S3 bucket.

Here, I’ve jumped into Transmit (FTP for Mac with AWS S3 support) and logged into my AWS S3 account. I’m looking at my “servintbackups” bucket which shows the different backup folders. Each night the backups are updated automatically on my AWS S3 account.

If you would like to do this as well check out these tutorials.

• Automating backups with Amazon S3 on Linux
• How I automated my backups to Amazon S3 using S3Sync

Conclusion

Using AWS S3 and a variety of tools I’ve managed to get a lot for a little.

• Centralized file access in the cloud, anywhere, on any platform.
• Automated backups for desktop and server computers.
• Web access to your files.
• Media hosting outside of your web server to reduce load and speed things up.
• Easy to setup “CDN” and/or providing file access for users.
• Easy to setup true CDN with CloudFront.

As I said earlier as well, AWS S3 is built for developers; so if I do need to use it for even more solutions, then the opportunity is there.

As great as AWS S3 is, it may not fit the bill for every problem you have. For instance, AWS S3 servers don’t gzip files and backing up 200GB of data (like an iTunes library) would cost $30 per month vs. $5 or $10 per month on other services. AWS S3 is just one of the tools I use among many.

Do you use AWS S3? Or do you prefer another similar solution? Tell us about it in the comment!

This article was originally posted on the ThemeForest blog. We are currently porting over some of the more popular articles to Nettuts+.