I found collectd causing rrd illegal attempt to update using time errors. I was seeing a whole load of lines like this in my syslog:
Aug 20 16:27:12 mythbox collectd: rrdtool plugin: rrd_update_r (/var/lib/collectd/rrd/mythbox/df-root/df_complex-free.rrd) failed: /var/lib/collectd/rrd/mythbox/df-root/df_complex-free.rrd: illegal attempt to update using time 1345444032 when last update time is 1345444032 (minimum one second step)
It was adding one message like that every second so my logs were completely full of it. Google didn’t reveal much except that this sort of error is either because there are two instances of RRD trying to write the RRD database at the same time, or that my server’s date and time are way out of sync. Neither of these were true in my case.
I asked on #collectd on freenode and a very nice person by the name of tokkee told me that it’s a known issue of sorts. The df plugin for collectd uses /proc/mount to determine which drives to check free space on and if / is in there twice, it tries to update the entry for / twice and causes the problem.
The solution is to add the following to the /etc/collectd/collectd.conf file:
<Plugin df> FSType "rootfs" IgnoreSelected true </Plugin>
Then I restarted collectd and my logs were peaceful again.
I was getting these errors again on one of my VPS hosts. In this instance, / only appeared once in /proc/mounts but /run was in there multiple times:
root@new:/etc/collectd# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,relatime,errors=remount-ro,data=ordered 0 0 devtmpfs /dev devtmpfs rw,relatime,size=1085360k,nr_inodes=271340,mode=755 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=217328k,mode=755 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=460860k 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0 root@new:/etc/collectd#
The solution is to ignore tmpfs instead of rootfs:
<Plugin df> FSType "tmpfs" IgnoreSelected false </Plugin>