I broke my favourite mug

mug with chip in rim and broken handle, image on mug says "only my computer understands me"

About 18 or so years ago (I can’t believe it was that long ago) a friend (Jules, drop me a line if you read this?) from Canberra gave me this mug. She picked mugs with similar messages for me, Damo and Stevo. Mine said “Only my computer understands me”, I think Damo’s said “Do I look like a fucking help desk?” I can’t remember what Stevo’s said. I do remember thinking at the time that she really chose the mugs well.

I used my mug nearly every day I drank literately thousands of coffees from it. Anyway, it slipped out of my hand while I was washing it the other day.

Oh well, end of an era.

Import historical data from Apache logs into AWStats

One of my clients had a problem where the last 6 months of data was not in Google Analytics. Upon investigating it turned out that for some reason the WordPress Google Analytics plugin was not active. I could not determine why it was not active when I am sure I set it up in the past.

I had all the Apache logs for the period in question so it seemed a simple idea to put the data into something useful that would show charts to the client. AWStats is perfect for that. In fact I used to use it long ago before Google Analytics was available but I had forgotten about it. As with all good open source software, the project is still there and ticking along.

Configuring AWStats turned out to be a but tricky. By default, debian sets AWStats up for a one domain host. My Apache logs are configured in the vhost_combined format which is one access.log file for all the virtual hosts.

The log files are rotated by logrotate and numbered access.log.1 access.log.2 access.log.3 .. access.log.10 etc. This presents another problem as you need to get them into order and normal alphabetical sorting does not work as there are no leading 0s in the file names.

Further, Apache was misconfigured and all the virtual host entries which should have indicated which virtual host was serving that access were in fact showing the ServerName. Luckily the entries do include the actual URL that was requested so with a bit of grep and sed it was easy to reconstruct what the virtual host should have been.

I wrote little bash script that would take a file name, either (eg access.log or access.log.gz) and would output that file after having parsed it to fix up the errors (later I discovered zcat -f will cat a file whether it is gziped or not so invalidating the need for the mycat function). You’ll see in the sed regular expression that I change the : to a space, AWStats does not like having a : between the hostname and the port and I could find no way to making AWStats parse that correctly. The reason there is two regex replacements in the sed command is that I fixed the apache logging of the host name prior to running this script, so needing to take into account both cases of old hostname and new hostname.

I could have made the sed regex taking into account the port number but I’m only interested in port 80 anyway and didn’t see the need to spend time on getting that working.

Log file format:

# Actual
old.host.name:80 199.7.156.141 - - [16/Sep/2012:17:25:51 +1000] "GET /wp-content/themes/grip/style.css HTTP/1.1" 200 7108 "http://correct.host.name/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; BTRS126493; EasyBits GO v1.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; eSobiSubscriber 2.0.4.16; InfoPath.2)"
old.host.name:80 199.7.156.141 - - [16/Sep/2012:17:25:51 +1000] "GET /wp-content/themes/grip/stylesheet/nivo-slider/nivo-slider.css HTTP/1.1" 200 968 "http://correct.host.name/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; BTRS126493; EasyBits GO v1.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; eSobiSubscriber 2.0.4.16; InfoPath.2)"

# Required for importing to AWStats
correct.host.name 80 199.7.156.141 - - [16/Sep/2012:17:25:51 +1000] "GET /wp-content/themes/grip/style.css HTTP/1.1" 200 7108 "http://correct.host.name/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; BTRS126493; EasyBits GO v1.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; eSobiSubscriber 2.0.4.16; InfoPath.2)"
correct.host.name 80 199.7.156.141 - - [16/Sep/2012:17:25:51 +1000] "GET /wp-content/themes/grip/stylesheet/nivo-slider/nivo-slider.css HTTP/1.1" 200 968 "http://correct.host.name/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; BTRS126493; EasyBits GO v1.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; eSobiSubscriber 2.0.4.16; InfoPath.2)" 

catlogs.sh finds relevant lines in given log file and reformats them to be suitable for importing into AWStats and outputs to stdout:

#!/bin/bash

mycat() {
    local f;
    for f; do
        case $f in
            *.gz) gzip -cd "$f" ;;
            *) cat "$f" ;;
        esac;
    done;
}

mygrep() {
    #get all the lines from log file which have accesses to the correct.host.name
    mycat $1 | grep 'http://correct.host.name' | \ 
        sed -e 's/old.host.name:80/correct.host.name 80/ ; s/correct.host.name:80/correct.host.name 80/' # replace incorrect hostnames
}

mygrep $1

Then I needed to loop through all the access.log files in the apache log directory in historical order. To do that I wrote a simple for loop on the command line.

for i in $(ls /var/log/apache2/access.log* | sort -r -n -k 3 -t '.' ) ; do sudo -u www-data /usr/lib/cgi-bin/awstats.pl -showcorrupted -showsteps -LogFile="bash /home/jason/catlogs.sh $i |" -config=/etc/awstats/awstats.correct.host.name.conf ; done;

A nice thing with AWStats is you can pass in a command that outputs to stdout as the log file -LogFile="bash /home/jason/catlogs.sh $i |". I used sort to get the files into numerical order. sort’s -k and -t options let you sort by a “KEY”. The logs need to go from oldest at the top to newest at the bottom, so you have to process the files in reverse number order.

Lastly, to ensure AWStats can read the apache access logs in future, I changed the apache vhost_combined format to:

LogFormat "%V %p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined

and I changed awstats log format to:

LogFormat = "%virtualname %other %host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"

Apache Virtual Host configuration for a Networked WordPress Installation

Direct any URL request that Apache receives to the WordPress installation. You need to do it if you are setting up a WordPress Network multi-site installation that has sites with their own unique domain names. e.g. site1.org, site2.com, someothersite.co.uk etc.

/etc/apache2/sites-enabled$ ls -al
total 8
drwxr-xr-x 2 root root 4096 Jul 25 13:18 .
drwxr-xr-x 7 root root 4096 Jul 24 12:28 ..
lrwxrwxrwx 1 root root 40 Jul 24 12:06 000-wordpress-network-ssl -> ../sites-available/wordpress-network-ssl
lrwxrwxrwx 1 root root 36 Jul 24 12:02 010-wordpress-network -> ../sites-available/wordpress-network

Order of the files is very important. wordpress-network contents below:

<VirtualHost *:80>
UseCanonicalName Off

ServerAlias *.examplehost.com examplehost.com
ServerName examplehost.com
DocumentRoot /var/www

Options All
ServerAdmin myname@examplehost.com

# Store uploads of www.domain.com in /srv/www/wp-uploads/$0
RewriteEngine On
RewriteRule ^/wp-uploads/(.*)$ /var/www/wp-uploads/%{HTTP_HOST}/$1

# try and make server-status return server status
#RewriteRule ^/server-status - [L]
RewriteCond %{REQUEST_URI} !=/server-status
<Location /server-status>
SetHandler server-status

Order Deny,Allow
# Deny from all
# Allow from localhost
Allow from all
</Location>
<Directory />
Options FollowSymLinks
AllowOverride All
</Directory>
CustomLog /var/log/apache2/access.log vhost_combined
ErrorLog /var/log/apache2/error.log

# this is needed when activating multisite, WP needs to to a
# fopen("http://randomname.domain.com") to verify
# that apache is correctly configured
php_admin_flag allow_url_fopen on

</VirtualHost>

Migrating single site WordPress installations to a multisite network

I have seen this question about migrating to a networked multisite WordPress install come up more than once so I thought I’d write down my experience. I used to host about 10 individual WordPress installations. Managing them and keeping them all up to date was a pain, and that meant I always had sites that were out of date. So a few months ago I spent some time migrating them all to a new WordPress network installation. That allows you to host multiple sites all in the one WordPress installation. The only problem is that its quite cumbersome to migrate sites into the new site.

I’ve tried to document the overall process. If anything isn’t clear please let me know and I’ll and elaborate.

  1. Backup all your old sites and databases
  2. Go to each site and export the site
  3. Select a suitable main domain name for your new networked WordPress site. When you create a new blog in the network, the admin of the blog will get an email from your multi site saying it was set up, so choose something that is suitable bearing that in mind.
  4. Set up the multi site with subdomain (not sub folder).
  5. Set up apache virtual hosts to direct ALL host traffic to the wp install.
  6. Set up subdomain sites for each site you wish to import.
  7. Install the WordPress MU Domain Mapping plugin
  8. For each subdomain:
    1. Import the old blog
    2. re-delegate the domain or set your hosts file up to point the domain name to your new WordPress network.
    3. On the Info tab in your network settings for the domain, change the Domain to the actual domain you want,
    4. On settings tab, change siteurl to the blogs correct domain,
    5. Change “home” to the correct domain
    6. Check through the list and fix up any other instances of incorrect domain. Save the changes.
    7. Note down the site id (from the url. something like: http://mynewmultisite/wp-admin/network/site-settings.php?id=4  means site ID is 4)
    8. Go to Settings/Domains and in the New Domain section, enter the site ID and correct domain for the new site
    9. Click save and that’s it – the new domain should be working. Test it out.
    10. Fix up all the old links to images. they will currently be links with the old subdomian in them. You can fix them up with the most excelent Search and Replace plugin.
  9. That’s it, you should be done.

It’s a fairly long winded process but it can be done and is worth the effort. Running a networked site instead of many individual installs saves a lot of time in upgrades and  maintenance.

Moving across the plugins and themes is also a bit of a pain. You just have to manually do it per site. There are plugins that are supposed to help move widget settings across but I had limited success with them. “Widget Data – Setting Import/Export Plugin” seemed to add the settings not replace them so you end up with duplicates of widgets and things. it may have improved since I tried it.

I don’t like to site wide activate plugins but WordPress does not have an option like with themes, to only enable a plugin on a per site basis. There is however a plugin for that: Multisite Plugin Manager. With that you can turn on plugins for each site in the Network Admin/Edit Site area. Then you have to go to the site’s dashboard and activate the plugin in there afterwards.

Another useful plugin for a networked site is called “Network Plugin Auditor” it adds a column to the Network/Admin/Plugins page that shows which sites are using each plugin. It also also shows the converse on the Sites page.

There are tools to assist this process also. One such tool that comes highly recommended, that I have not personally used is Backup Buddy. If you have more than a few sites to migrate it may well be worth investing in something like that to help you.

Ordered list with letters in Twenty Twelve

UPDATE: It turns out I was wrong. type= is deprecated. The correct way to do this is style="list-style-type:lower-alpha". I updated my post to reflect that.

It appears there is a bug in Twenty Twelve as of this writing where specifying the type of an ordered list is not respected. So you can’t get an OL of letters. The second list below should have letters.

  1. first line
  2. second line
  1. first line
  2. second line
<ol>
	<li>first line</li>
	<li>second line</li>
</ol>
 
<ol style="list-style-type:lower-alpha">
	<li>first line</li>
	<li>second line</li>
</ol>

A word on WordPress security

First of all, I am not an expert on security. So please check this info for yourself, and don’t be shy to suggest improvements either.

In order to keep your WordPress site secure, one of the things you can do is slow down a potential brute force attack, so the attacker can try fewer login/password combinations per second.

There is an excellent plugin by convissor over at wordpress.org called Login Security Solution that does just that. I have been running it for a while now and it is very good. It emails you if your site is under attack and informs you about what its doing. It also ensures users have secure passwords. I have mentioned it before.

Recently I have had a few attacks and encountered a minor problem with the plugin. Daniel has been extremely proactive in tracking down the issue. But during the process I found that the attacks were coming from 3 IPs.

That made me think I should ban those IPs. And that led me to fail2ban. This is an excellent tool that monitors for failed login attempts and simply temporarily bans that IP using your firewall.

In debian you install it with aptitude install fail2ban. If you want to enable apache monitoring, you need to add:

[apache]

enabled = true

to /etc/fail2ban/jail.local

restart fail2ban and that’s it. Within minutes of me activating it, it had already banned a few IP addresses due to failed ssh login attempts.