Disk crash brought down hack.org & others

While I was on vacation the main hack.org server, totoro, suffered a disk crash. Timeline:

• 2015-07-24 totoro crashed. Investigated where to set up a virtual server before mail started to bounce.
• 2015-07-25 Got hold of someone in the co-lo and had them flip the power switch. totoro came back up with one less disk in the ZFS mirror.
• 2015-08-07 The second disk in the mirror crashed! Three year old Seagate disks in a ZFS mirror. I guess they were from the same batch or something… No hot spare, unfortunately.
• 2015-08-08 VPSes created and configured. Backup restored. Mail starts flowing again. Also properly set up everything in Ansible. Most of our stuff was already in Ansible playbooks, but not everything.

totoro was later brought to my home. If I removed the hard disks and turned one of them up-side down I heard a loud "click"!

We had backup of many things but not of the XMPP server data, so accounts, rosters, and any offline messages not yet delivered were lost.

If you don't see any presence from your hack.org XMPP contacts, add them again.