When using Apache VirtualHosts you will be serving up sites from different directories, which also means you’ll be dumping logs in to different locations.

Rotating these logs can be a bit clumsy, however if you structure your DocumentRoot‘s in a logical fashion you can setup logrotate to do this with one config.

Usually logrotate config would look like this:

/var/www/domain1.com/logs/*.log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        ...
}
/home/domain-owner/domain2.com/logs/*.log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        ...
}
/var/www/some-other-dir/domain3.com/logs/*.log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        ...
}
...

Yet the path accepts a wildcard for the DIR, so why not set up your VirtualHost directories like this:

/var/www/vhosts
            domain1.com
                   logs
                   httpdocs
            domain2.com
                   logs
                   httpdocs
            domain3.com
                   logs
                   httpdocs

Now you can rotate logs in a one-liner with something like this:

/var/www/vhosts/*/logs/*.log {
        daily
        missingok
        rotate 30
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                /etc/init.d/apache2 reload > /dev/null
        endscript
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

The problem

So you’ve got your nice form, and want to allow users to attach a file. If the user tries to upload a file which exceeds the ‘post_max_size’ value in php.ini then WHAMMY…the entire $_POST variable will be empty.

This is a massive headache in terms of UX, your user has just spent 5 minutes entering information and simply because they tried to attach a 4Tb file then all this is lost. Validating the size of the file they attach is not the issue, the issue is that if the upload is part of a form with other text fields then their input will be lost.

The solution

First off, lets get the current value for ‘post_max_size’ and ‘upload_max_filesize’ from php.ini.

upload_max_filesize

post_max_size

As you can see, my post_max_size is 20Mb and my upload_max_filesize is 10Mb.

Why are they different? Well, anything over 10Mb will upload but will return a decent error you can catch, but if a user tries to upload a file over 20Mb then the internet explodes and PHP simply dumps the whole lot, $_POST will be empty, as will $_FILES, $_REQUEST etc.

How this would look to the user uploading 15mb file

How this would look to the user uploading 15mb file

How this would look to the user uploading 25mb file

How this would look to the user uploading 25mb file

As you can see, if post_max_size is exceeded (the second screenshot), all the $_POST data is lost and the user has to start again.

One solution to this would be to have 2 separate forms, or split up the steps in the process so the file upload is on a separate page.

However there is a better solution: Ajax.

This approach uses two separate forms but we do it on the same page in such a way the UX is seamless.

1. When the user clicks “Save and Continue” we submit form 1 (just the text/select fields) via ajax as a serialized array
2. Validate it on the server and return a JSON array of info (such as: did it validate? if there are errors what are they? etc.)
3. If it does validate, save it in a session
4. Check the response client side, and if ok we submit form 2 (just the files) the usual way

Here’s how it looks.

The HTML for the forms

The JS (I’m using jQuery)

My code on the server validates the data from the text fields, saves it to a session, and returns a JSON object. If the validation fails it returns something like this.

We can use the response to show errors BEFORE the form with the files is submitted (the normal way).

Validation errors, used a modal because i'm hip

If the response in the JSON is ‘success’ then we simply submit the second form as normal. If the file/s still exceed the post_max_size then at least the previous text input is on the session and can be used to populate the fields again.

How this would look to the user uploading 50mb file

Huge upload, $_POST is lost but we have the text fields in the session

I have been playing with Rackspace “cloud” servers for a few weeks now. They’re easy to fire up and shut down, and you only pay for the time the servers are online, so they are great for spinning up a development server for a few hours.

I’ve put together the following as my own selfish reference to setting up a working CentOS LAMP server.

This post assumes you have knowledge of using SSH and have some idea of what you are trying to achieve which in this case is a working LAMP web server.

But it also involves managing your own stuff. Updates, checking logs, making sure your websites are up and running, fixing issues etc. will now be your responsibility.

If the sound of all that makes you dizzy, then you might want to consider a fully managed web hosting service from a reputable hosting company instead who will do all that boring stuff for you.

Still with me?

Good. Lets fire up a server. Remember to give your server a cool name, this will help gain you the respect of more seasoned server admins. If you own a retro Transformers T-shirt, put it on now.

Create a server

When you create a server on rackspace, you will be issued with the important basics. The unique IP address, and your root username and password. So fire up a terminal and get your eyes dirty. For the purpose of this post, I will be using the server “clapton” (cool, huh?) and the IP address is 46.38.165.175, I’m logged in with a clean install of CentOS  5.5.

[root@clapton ~]#

Yum is your friend

You know that cool retro Tansformers T-shirt you’re wearing? Well Yum is even cooler.

First up, lets update any existing packages that have been included in your CentOS install by using the “update” option

yum update

The installed packages are now up to date, now lets move on to the stuff we want. I want to install the following;

• Apache
• MySQL
• PHP
• Open SSL
• Sendmail
• Git
• VIM

I’m not going to bother with FTP (real men use Git) and I want to keep things lightweight and locked down as much as possible. Having FTP running just means faffing with IP lists in an attempt to stop 7 million Chinese teenagers pwning your FTP login.

Open SSL comes included with CentOS, we just need the apache mod_ssl. I will write another post regarding setting up SSL certs, as this was my one stumbling block when I first started. I’d previously been spoiled by admin panels like Plesk when it came to things like that.

Core settings

There isn’t really much house work to take care of, other than checking the servers HOSTS file, located at /etc/hosts;

127.0.0.1    localhost    localhost.localdomain
46.38.165.175    clapton    46.38.165.175.static.cloud-ips.co.uk

In the Rackspace control panel, if you view the DNS for the server you will see a xxxx.cloud.co.uk record, add this (along with the server name if not already there) to the IP in your HOSTS file.

I also add a localhost.localdomain entry, this will make life easy for things like sendmail.

VIM

The rest of the setup may require editing files, be nice to your eyes and use VIM.

yum install vim-enhanced

Apache

Lets get Apache up and running. We want SSL support too so we’ll also install the apache mod now (OpenSSL should be installed by default as mentioned above, we just need the mod itself)

yum install httpd mod_ssl

Fire up VIM and edit the apache conf file, we need to tell apache to “listen” on port 80, its around line 135 of httpd.conf;

vim /etc/httpd/conf/httpd.conf

httpd conf listen port 80

Now Apache is listening to port 80, we now need to open this port as CentOS by default will have every port locked down for obvious reasons.

To open port 80 (http) and 443 (https) we add a rule for each to the iptables config;

iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT
iptables -I INPUT 1 -p tcp --dport 443 -j ACCEPT

and then commit these new rules;

/etc/init.d/iptables save

That’s it, fire up apache;

service httpd start

and go to your IP address in a browser;

apache default page

PHP

Use Yum to install PHP, the MySQL extension, and MySQL itself in one fell swoop;

yum install php php-mysql mysql-server

As of writing this, the latest PHP version that Yum will use is 5.1.x, which lets be fair, is pretty outdated.

5.2.3+ is fine for me, so I’ll go with that. If you want to go big and have 5.3+ (but have never installed manually before) then I would suggest get going with 5.2.x to start with, then upgrade to 5.3 later on once you have a basic web server up and running.

To update to 5.2+ we need to add a .repo file which instructs Yum to do some kung foo for us when finding which packages we want.

We then just update PHP using Yum.

So first, add the .repo file as follows;

touch /etc/yum.repos.d/CentOS-Testing.repo

Then open the empty file;

vim /etc/yum.repos.d/CentOS-Testing.repo

Then add the following config to the repo file you just created;

# CentOS-Testing:
# !!!! CAUTION !!!!
# This repository is a proving grounds for packages on their way to CentOSPlus and CentOS Extras.
# They may or may not replace core CentOS packages, and are not guaranteed to function properly.
# These packages build and install, but are waiting for feedback from testers as to
# functionality and stability. Packages in this repository will come and go during the
# development period, so it should not be left enabled or used on production systems without due
# consideration.
[c5-testing]
name=CentOS-5 Testing
baseurl=http://dev.centos.org/centos/$releasever/testing/$basearch/
enabled=1
gpgcheck=1
gpgkey=http://dev.centos.org/centos/RPM-GPG-KEY-CentOS-testing
includepkgs=php*

Now update PHP as normal with Yum;

yum update php

Restart apache

service httpd restart

MySQL users !IMPORTANT

MySQL is now installed, but comes with a default user….with NO PASSWORD. So we need to start MySQL, drop the users with no passwords, and create a user with a password;

service mysqld start

Log in as the default password-less root user;

mysql -u root

Set a password for your root user;

SET PASSWORD FOR root@localhost = PASSWORD('newpassword');

and

SET PASSWORD FOR root@127.0.0.1 = PASSWORD('newpassword');

Drop the ‘any’ user too

DROP USER ''@localhost;

All done. Exit MySQL;

exit;

Email (Sendmail)

yum install sendmail sendmail-cf

service sendmail start

Done

Services

At this stage we have 3 main services installed; Apache, MySQL and Sendmail.

But these are currently manually started, which means if you had to reboot your server, you would also have to also manually start each service again, which is pants.

To set all 3 to start by default use the chkconfig command;

chkconfig httpd on

chkconfig mysqld on

chkconfig sendmail on

You can also use the –list parameter to see the changes and view any other default services and their run levels.

chkconfig --list

Git

Git is cooler than both Yum and your Transformers T-shirt combined.

To install Git we can employ the EPEL (Extra Packages for Linux) RPM;

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/
epel-release-5-4.noarch.rpm

We can now Yum install;

yum install git

git --version

Should display;

git version 1.5.5.x

Great Success

That’s it. You now have a LAMP stack on Rackspace. A great tool in the Rackspace Panel is “My Server Images”. Now would be a good time to create an image of what you have done so far, then you can roll this out on other servers really quickly in future.

But don’t be a dingbat and loose all your shit, image or no image. Remote file storage is seriously cheap, see this post on Remote Backups to Amazon S3 and save yourself a huge headache one day

Reddit is a big site (in terms of traffic and data storage)  built by some guys in the US, one of their former developers talks openly about some of the issues they have had to overcome.

Lessons Learned while at Reddit from Carsonified on Vimeo.

Well, first post. Straight in at the deep end with something turbo-geeky I guess! I signed up for Amazon S3 last week, my initial intention was to use S3 for a cheap and cheerful place to host non-essential files for some of my current projects, such as avatars, user uploads, blog images etc.

Reading through Amazons S3 REST API got me thinking about my current backup solution (local and off-site, but with some manual admin involved) and how this would be the perfect solution for automated off-site backups. So I hit uncle Google in search for either a shell script or PHP/PEAR class I could run on a cron to take care of this for me….and would you believe it, I found s3cmd.

S3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don’t fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

Perfect! This is a command line tool, so the rest of this post assumes you are on a dedicated or VPS web server, are  familiar with SSH and basic Linux commands, and have root access. If you’re not quite ready to get your eyes dirty on the command line, here is a useful post on FTP, download your web files, keep them safe, and you could then use an S3 client like s3hub to sync your files to Amazon instead.

OK, so for those of you running CentOS this post is for you. I’ll run through each step of installing and setting this up, this is no reflection on s3cmd’s docs (which are great) but its always nice to have a more specific guide focusing on your type of server setup.

Step 1

First we need to install s3cmd. Login to your web server via SSH. You’ll need root privileges.

You'll need root privileges

Remember, you’re logged in  a root now. Bad things can happen, so concentrate. This isn’t a good time to answer the phone, spill coffee on your machine or try and play Angry Birds at the same time. First, lets move in to /etc/yum.repos.d. We can then use wget to download the latest version on s3cmd.

$ cd /etc/yum.repos.d

Use wget to grab the s3cmd repo

You will see the repo download, and you can then install with;

$ yum install s3cmd

s3cmd installing....great success!

Step 2

We can now configure s3cmd to work with our Amazon S3 account, so get your access key and secret key from the “Account” section of your Amazon web services user section, under “Security Credentials”;

Get your access key and secret key

Back to terminal for the next bit, if all has gone correctly so far, you should be able to run the following command, and will be presented with the config screen;

$ s3cmd --configure

Enter your Amazon access key and secret key

I have opted to use HTTPS for obvious security reasons. You don’t want some pesky script kiddy sniffing database config files and other sensitive files.

Step 3

Nearly there. Amazon S3 uses “buckets”, these are basically directories within your account to store stuff. I went ahead and created a new bucket called “backups” (You can do this by logging in to Amazon S3 console, or with a desktop client like s3hub).

Lets check s3cmd is now running correctly on your server. Type the following;

$ s3cmd ls

This will output a list of current buckets on your Amazon account. If this doesn’t work, something bad has happened. Go put your head in the toilet.

If you’re still with me, lets put some files on S3. For this example, I want to put the file /var/backups/myfile.txt in to backups directory (bucket) on S3. Use the following (where my bucket is called “bucketname”);

$ s3cmd put /var/backups/backups/myfile.txt s3://bucketname/backups/

It worked, didn’t it!? POW!

Right, so putting single files up is going to take a  loooonnnggg time but luckily we can put an entire directory. You can also use sync if you only want to upload changed/new files. Lets use sync this time and upload my directory /var/backups/folder…

$ s3cmd sync /var/backups/folder/ s3://bucketname/backups/

Step 4

Ok lets add this command to our cron file. Im going to run this at 3am every night while things are quiet. I will be backing up multiple domains, with different owners so I will add this to my root users crontab file;

$ vi /var/spools/cron/root

Run a late night cron

That’s it. Your web files are now synced with your Amazon S3 bucket every night.