I'm hoping to use Amazon Elastic Compute Cloud (EC2) for an upcoming project that involves multi-hosting a bunch of Drupal websites. I like the "pay for what you use" model, and the ability to ramp up capacity within minutes instead of weeks. Integrating one of Drupal's provisioning solutions with Amazon's "Datacentre On-Demand" is my ultimate goal.

But first, EC2 has a few limitations which makes multi-site hosting a little more challenging:

1. Storage is not persistent across crashes. Reboots are ok, but crashes aren't and they do happen.
2. Dynamic IP addresses only. You can't reserve or request an IP.
3. Single external IP address. One IP per SSL site is the current norm.
4. No choice of kernel. This might be ok if you didn't hit bugs in the supplied kernel.

Limitations #1 & #2 can be addressed by scripting. Frequent backups to S3 will minimise data loss, and a dynamic dns service can be used.

There's a couple potential solutions for #3, the single IP address, neither of which I've tried. Multi-domain SSL certificates would seem to do pretty much the same as wildcard certificates, but for completely different domain suffixes. mod_gnutls is another potential, but it lacks support in some still-important browsers (IE6 at least).

For the lack of kernel choice, the only options are to wait until Amazon releases a newer version (they have for RedHat images) or to workaround any issues that surface. I came across http://bugzilla.kernel.org/show_bug.cgi?id=6873 when removing the LVM snapshots I use for backups. I have yet to find a good workaround...

So, with the usual "at your own risk" warnings, here's my guide to automating Drupal hosting on Amazon's EC2. Expect heavy changes as I learn more about what works best for EC2.

Current features

• Builds on the Centos V5_10 image provided by RightScale.
• Converts /mnt to four LVM filesystems; /data/www, /data/mysql_master, /data/mysql_slave & /usr/local.
• Runs MySQL 5.0.22 master and slave on the same host. InnoDB is used by default
• Apache 2.2.3 installed in /usr/local with document root set to /data/www. TODO: Support for multiple domains is next on my to-do list.
• Incremental (rsync-style) back up of databases, document root and /usr/local to S3 storage every 15 minutes.
• Filesystem snapshots are used to get consistent copies of the data. The MySQL master is backed up online without any locking. Apparently this is fine for InnoDB.
• MySQL slave is shutdown before the snapshot of its filesystem is taken. (In case a lock really was needed!)
• Update DynDNS.
• The installer will restore data if you've built a server with this domain name before.
• On first installs, Drupal HEAD is checked out from cvs.drupal.org.

Next steps

• Replace/enhance Drupal setup.
• Multiple domains/aliases configuration.
• Proper provisioning!
• Drupal module for EC2 image management and monitoring.

Instructions

1. Launch AMI ami-08f41161

   I use the EC2UI, a Firefox extension for this. Optionally include the following parameters in the "User data" box, or wait to be prompted by the installer:
   
DOMAIN=croome.org
DOMAIN_ALIASES=www.croome.org
AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

   NOTE: Your Amazon credentials are needed to use S3 for backups. If you don't supply them then backups will fail (ungracefully atm). They are stored in /root/.hosting.conf, and readable only by root. If you specified them when you launched the AMI, they will also be stored in /var/spool/ec2. The installer will make sure this area is also readable only by root.

2. Login
   
   imac:~ simon$ ssh -i drupal.pem -l root ec2-67-202-41-211.compute-1.amazonaws.com
   The authenticity of host 'ec2-67-202-41-211.compute-1.amazonaws.com (67.202.41.211)' can't be established.
   RSA key fingerprint is 7a:40:a0:da:14:41:16:cb:9a:93:4b:4a:1a:84:0e:61.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added 'ec2-67-202-41-211.compute-1.amazonaws.com,67.202.41.211' (RSA) to the list of known hosts.
   ___ _ __ __ ____ __
   / _ \ (_)___ _ / / / /_ / __/____ ___ _ / /___
   / , _// // _ `// _ \/ __/_\ \ / __// _ `// // -_)
   /_/|_|/_/ \_, //_//_/\__//___/ \__/ \_,_//_/ \__/
   /___/

   Welcome to a public Amazon EC2 image brought to you by RightScale!
   

3. Download automation scripts
   
root@domU-12-31-39-00-55-C3:~] cd /tmp
[root@domU-12-31-39-00-55-C3:/tmp] svn co http://svn.openresort.org/repos/automation
A automation/setup
A automation/setup/stop_slave
A automation/setup/setup_mysql
A automation/setup/stop_apache
A automation/setup/setup_filesystems
A automation/setup/backup_files
A automation/setup/start_slave
A automation/setup/verify_config
A automation/setup/start_apache
A automation/setup/setup_packages
A automation/setup/templates
A automation/setup/templates/mysqld_multi
A automation/setup/templates/php.ini
A automation/setup/templates/CentOS-Base.repo
A automation/setup/templates/httpd.conf
A automation/setup/templates/ddclient.conf
A automation/setup/templates/my.cnf
A automation/setup/stop_master
A automation/setup/remove_snaps
A automation/setup/setup_apache
A automation/setup/load_config
A automation/setup/restore_files
A automation/setup/setup_php
A automation/setup/setup_drupal
A automation/setup/create_snaps
A automation/setup/start_master
A automation/init.d
A automation/bin
A automation/bin/backup_mysql
A automation/build_image.sh
A automation/etc
A automation/etc/hosting.conf
Checked out revision 37.

4. Run installer
   
[root@domU-12-31-39-00-4D-C7:/tmp] cd automation/
[root@domU-12-31-39-00-4D-C7:/tmp/automation] ./build_image.sh
Domain name [croome.org]:
Domain alias(es), separated by space []:
AWS Access Key ID [1DCMY68GD372A75GDA82]:
AWS Secret Access Key [****************************************]:
Updating packages ... Done.
Creating filesystems ... Done.
Configuring MySQL ... Done.
Configuring PHP ... Done.
Configuring Apache ... Done.
Restoring from backup ... Done.

5. Login to Drupal

   For me, this was at http://ec2-67-202-41-211.compute-1.amazonaws.com
   Note: Database user is 'root', with password 'private'.

Note: Look in automation/etc/hosting.conf for other options, including DynDNS setup. I'll streamline this later...

Suggestions welcome, but DON'T USE IN PRODUCTION!!!