Collectd causing rrd illegal attempt to update using time errors

โ€”

by

in , ,

I found collectd causing rrd illegal attempt to update using time errors. I was seeing a whole load of lines like this in my syslog:
Aug 20 16:27:12 mythbox collectd[32167]: rrdtool plugin: rrd_update_r (/var/lib/collectd/rrd/mythbox/df-root/df_complex-free.rrd) failed: /var/lib/collectd/rrd/mythbox/df-root/df_complex-free.rrd: illegal attempt to update using time 1345444032 when last update time is 1345444032 (minimum one second step)

It was adding one message like that every second so my logs were completely full of it. Google didn’t reveal much except that this sort of error is either because there are two instances of RRD trying to write the RRD database at the same time, or that my server’s date and time are way out of sync. Neither of these were true in my case.
I asked on #collectd on freenode and a very nice person by the name of tokkee told me that it’s a known issue of sorts. The df plugin for collectd uses /proc/mount to determine which drives to check free space on and if / is in there twice, it tries to update the entry for / twice and causes the problem.
The solution is to add the following to the /etc/collectd/collectd.conf file:


        FSType "rootfs"
        IgnoreSelected true

Then I restarted collectd and my logs were peaceful again.
Update 2014-04-10:
I was getting these errors again on one of my VPS hosts. In this instance, / only appeared once in /proc/mounts but /run was in there multiple times:

root@new:/etc/collectd# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,relatime,errors=remount-ro,data=ordered 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1085360k,nr_inodes=271340,mode=755 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=217328k,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=460860k 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
root@new:/etc/collectd#

The solution is to ignore tmpfs instead of rootfs:


        FSType "tmpfs"
        IgnoreSelected false


Comments

13 responses to “Collectd causing rrd illegal attempt to update using time errors”

  1.  Avatar
    Anonymous

    add it where please ?

    1. Jason Lewis Avatar
      Jason Lewis

      I see that wasn’t clear from my post. Add it to the /etc/collectd/collectd.conf file. I’ve updated my post to reflect that.

  2. Thomas Avatar
    Thomas

    Your blogpost saved my day ๐Ÿ™‚

    1. Jason Lewis Avatar
      Jason Lewis

      Glad to hear it!

  3. It saved my day too ๐Ÿ™‚ Thanks

  4.  Avatar
    Anonymous

    +1!

  5. Saved my day, thanks a lot!

  6. Dorian Avatar
    Dorian

    thank’s for sharing this tips ! ๐Ÿ™‚

  7. Harm Avatar
    Harm

    Awesome, simple solution to an annoying problem!

  8. kevin Avatar
    kevin

    Awesome! Didn’t think having a /run/shm and /run/lock would cause this problem.. based on the previously known bugs.. THANKS!

  9. solved my problem – thanks!

  10. Hello! It didn’t saved mine ๐Ÿ™ I have added the few lines but what do you mean by “restarted collectd”? The problem remains in my case…

  11. I see this is now the default config but I’m still seeing this issue? Anyone else? But only on one host and it’s an Azure Ubuntu host…

Leave a Reply

Your email address will not be published. Required fields are marked *