MC's Journal

Disk crash brought down hack.org & others

While I was on vacation the main hack.org server, totoro, suffered a disk crash. Timeline:

  • 2015-07-24 totoro crashed. Investigated where to set up a virtual server before mail started to bounce.

  • 2015-07-25 Got hold of someone in the co-lo and had them flip the power switch. totoro came back up with one less disk in the ZFS mirror.

  • 2015-08-07 The second disk in the mirror crashed! Three year old Seagate disks in a ZFS mirror. I guess they were from the same batch or something... No hot spare, unfortunately.

  • 2015-08-08 VPSes created and configured. Backup restored. Mail starts flowing again. Also properly set up everything in Ansible. Most of our stuff was already in Ansible playbooks, but not everything.

totoro was later brought to my home. If I removed the hard disks and turned one of them up-side down I heard a loud "click"!

We had backup of many things but not of the XMPP server data, so accounts, rosters, and any offline messages not yet delivered were lost.

If you don't see any presence from your hack.org XMPP contacts, add them again.


Go and xmpp-client

The last two weeks I've been programming mostly in Go. The language, I think, feels a lot like Python would if it had been re-designed by an old C programmer. I got productive rather quickly and it feels very much at home. I'll probably write something about my experience later.

I've been mostly happy with emacs-jabber as my XMPP chat client for the last few years but as more and more of my contacts move to mandatory use of OTR encryption it leaves something to be desired. There's Magnus' jabber-otr which uses a pure-Python OTR implementation under the hood, but it's rather fragile and very much a work in progress.

Linus Nordberg pointed me to Adam "agl" Langley's xmpp-client a couple of months ago. It's a minimalist command line XMPP chat client written in Go with excellent OTR support. I started using it, but missed emacs-jabber a bit.

When I started learning Go two weeks ago I had an idea that I would do an Emacs frontend to xmpp-client and get back some of the emacs-jabber goodness. My idea was that, if the current user interface in xmpp-client was isolated from the functionality, I could write my own user interface code, specifically meant to be easy to parse from Emacs Lisp.

I delved into the code, but found that the terminal user interface is sprinkled all over xmpp-client together with the code that defines its function as a chat client. This means that if I go this way I would either have to reproduce a lot of the chat client functionality in my code or, probably better, do a lot of work to isolate the UI code.

While I was in the xmpp-client code I decided to at least make it nicer for me to use. I added a few commands and a new configuration item. My changes so far:

  • /whois command. Gives information about a user, their current state online/offline away/chatty/xa/dnd and if we have a current OTR session.

  • /who command. Lists online users in a short form. This ties in to the next feature.

  • InitialAwayIsOnline, a new configuration item. If true it means people in our roster which are initially marked as "away" are considered online as far as /who is concerned. If set to false, those people are not listed. This is the default and the same as the old behaviour.

  • /m command. Like /query in most IRC clients. Sets the default target we're chatting with. Also prints some data about our target: are they even online? Do we have an OTR encryption session? Are their identities verified?

Code here:

https://github.com/mchackorg/xmpp-client

All features merged in master branch. Each future is on a similarly named topic branch. InitialAwayIsOnline is on the "who" branch.


Experiments with Raspberry Pis

I'm a bit late to the Raspberry Pi party, but when the local electronics shop had a sale of model B+ I bought two on a whim. The new RPi 2 looks nicer, but B+ seems pretty nice too.

My Squeezebox recently broke down and I figured I might use one of the Raspis as the new livingroom music player. The sound quality might not be as good, but I thought I'd do a test and if I wasn't happy with it I would shell out for a better DAC.

I ended up running Minibian with MPD as the music player. It mounts the music directoy from the FreeBSD file server with NFS 3. It seems to work fine and the audio quality is at least bearable.

I'm mostly controlling MPD with EMMS in Emacs on the laptop or MPDroid on the phone. I really don't miss the Squeezebox remote.

The other Raspi is destined for our country house. I'm probably going to connect a Tellstick Duo and use some wireless thermometers and hygrometers around the house. I'm writing some simple scripts so that the Raspi can call home over 4G and report data now and then.

When I got the raspis I tested some different operating systems: Raspian (Debian-based Linux), Minibian (also Debian-based Linux - but much smaller standard image), RISCOS (minimal OS originally for the Archimedes, the first ARM computer), FreeBSD and Plan 9.

I'm running FreeBSD on most servers and would have gone that way on raspi as way, but FreeBSD on the raspi meant living without binary packages and compiling everything from source was veeeery sloooow. Had a look into cross compiling ports, but it seemed a hazzle. NanoBSD might be an option but I haven't tested it.

Plan 9 was fun, as usual, but not very practical. Haven't done anything on Plan 9 for a while. Tried a modern Plan 9 on my Thinkpad a few years ago and now on the raspi. A raspi seems to be a pretty good (and very cheap) Plan 9 terminal. Interesting to think back and compare the RPi against, say, the MIPS Magnum 3000 and the NeXTStation that Bell Labs once considered as low-end workstations they used as Plan 9 terminals.

The end user experience in Plan 9 hasn't changed much since I had Plan 9 on my workstation at work in the middle 1990s. Some things have changed behind the scenes, though, with a file server these days typically storing blocks of data indexed by their hash. The window system has changed a bit as well: rio, the 8 1/2 replacement, does graphics a bit faster.

Not much seems to happen on Plan 9 at Bell Labs. Most of the Plan 9 hackers seems to have left for Google and seems to work on the Go programming language. Plan 9 lived a long time at Coraid, though, embedded in their storage over Ethernet products, but Coraid seems to have shut down! Some development seems to happen over at 9front, a forked Plan 9.

It would be fun to do something with Plan 9 on the raspi, but I will probably just use some Linux distribution for the summer house raspi as well.


IPv6 tunnel from HE with dynamic endpoint on FreeBSD

Hurricane Electric's tunnelbroker offers IPv6 tunnels that you can use without any extra client software. No need for AICCU, OpenVPN or anything else.

HE's tunnels use built-in support in most most operating systems for the 6in4 tunneling protocol. It even works if you're home router might an odd duck, such as Plan 9. It doesn't work in all the scenarios some of the other tunnel clients support and it doesn't protect from IP spoofing, but it might be preferable in some situations.

My case is an EdgeRouter Lite, a rather nice MIPS-based thing (based on Cavium Octeon) from Ubiquiti Networks. I installed FreeBSD on it, but since this is a rather odd FreeBSD platform I don't get any binary packages and have to rebuild the firmware myself if I want to upgrade. I also don't get any support for the built-in network acceleration hardware.

I tossed the original Linux, since I didn't fancy rewriting all my pf rules nor adding them through a web UI, however nice. I did, however, make a copy of the Linux image and I'm especially interested in what's inside cavium-ip-offload.ko and friends. But that's for another time, if I can find the time.

Typically, you set up a 6in4 tunnel like this on FreeBSD:

ifconfig gif0 create
ifconfig gif0 tunnel 83.249.0.22 216.66.80.90
ifconfig gif0 inet6 2001:470:27:9d2::2 2001:470:27:9d2::1 prefixlen 128
route -n add -inet6 default 2001:470:27:9d2::1
ifconfig gif0 up

where 83.249.0.22 is the current IPv4 tunnel endpoint on my home router. You want some of these in rc.conf to survive reboots:

gif_interfaces="gif0"
ifconfig_gif0_ipv6="inet6 2001:470:27:9d2::2 2001:470:27:9d2::1 prefixlen 128"
ipv6_defaultrouter="2001:470:27:9d2::1"

My router gets its IPv4 from my ISP's DHCP, so the tunnel endpoint on my end might change at any time. Luckily, Hurricane thought of that and provides a web service to change my IPv4 endpoint easily by accessing:

https://${userid}:${updatekey}@ipv4.tunnelbroker.net/nic/update?hostname=${tunnelid}

so I wrote a script, he.sh:

#! /bin/sh

tunnelgw=216.66.80.90
userid=my-id
updatekey=my-update-key
tunnelid=my-tunnel-id

if [ $# != 1 ]
then
    echo Usage he.sh ip-adress
    exit
fi

# $1 should be my own IP
ifconfig gif0 tunnel $1 $tunnelgw

fetch --no-verify-peer -o - "https://${userid}:${updatekey}@ipv4.tunnelbroker.net/nic/update?hostname=${tunnelid}"

ifconfig gif0 inet6 2001:470:27:9d2::2 2001:470:27:9d2::1 prefixlen 128

ifconfig gif0 up

and call it from /etc/dhclient-exit-hooks, a program which, if it exists, runs automatically by FreeBSD dhclient DHCP client.

#! /bin/sh

/root/he.sh $new_ip_address

Ta-da!


Symbolics and the Lisp machines

Some time ago the first domain name in DNS, symbolics.com, turned 30 years old. WHOIS says:

   Domain Name: SYMBOLICS.COM
   Registrar: NETWORK SOLUTIONS, LLC.
   Sponsoring Registrar IANA ID: 2
   Whois Server: whois.networksolutions.com
   Referral URL: http://networksolutions.com
   Name Server: NS1.AMERINOC.COM
   Name Server: NS2.AMERINOC.COM
   Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
   Updated Date: 05-jun-2011
   Creation Date: 15-mar-1985
   Expiration Date: 16-mar-2016

Behind the domain name was once Symbolics, Inc., a legendary manufacturer of Lisp machines, computers with a special architecture suitable to run Lisp programs at high speed.

There's something in the eye of an old hacker when she thinks about Lisp machines. From what I can gather, they seem to have been the ultimate hacker computer: a completely dynamic environment with easy access to source for everything, including the lowest levels of the operating system, easily changed from within the interactive environment with the effects taking place immediately.

There were several LispM makers back before the AI Winter: Symbolics, LMI, Xerox and even Texas Instruments, with their TI Explorer. A brief history of lisp machines can be found here:

http://www.andromeda.com/people/ddyer/lisp/

I'm too young to have experienced the LispM directly myself. When I started at Linköping University the Computer Science department had just thrown out some Xerox LispMs and replaced them with Sun SPARCstation-1s. They still ran some Lisp assignments in an emulator called Medley but since I didn't even had official access to the the Suns, I never experienced it.

I don't know where the Xerox machines ended up. For some reason Lysator, the academic computing society at LiU, didn't get them. Update, Lysator's counterpart in Uppsala, has a few Symbolics machines.

To get a feeling of what it was like to work on a LispM, here's a movie of someone playing with a TI Explorer in Unisys disguise for an hour:

https://www.youtube.com/watch?v=xx6QPgi7RSQ

and here's a collection of screenshots from Symbolics' Genera environment:

http://lispm.de/symbolics-ui-examples/symbolics-ui-examples.html

Remember that many of these screenshots are from the early 1980s, before the first Mac, for instance.

The CPU in most of the LispMs were developed from the architecture of the CONS and CADR machines from MIT's AI Lab. Both Symbolics' and LMI's first machines was based directly on the CADR.

For a couple of years it's been possible to run the CADR machine on an emulator written by Brad Parker. All the source from the original CADR system is also available:

http://www.unlambda.com/cadr/

Björn Victor has collected some information about running emulated LispMs:

http://bjorn.victor.se/lispm.php

Some people still seem to dream about running Lisp on the bare metal. Here are two such projects:

Other than that, the best way of experiencing something like the LispM environment on modern computers is probably to use Emacs as your main environment. This is what I do, although I have sometimes looked longingly at Racket.


Converting a blog from Blosxom to Pelican

My blog has always been just static files on a web server. From 2009 when I started blogging again I used a slightly hacked Blosxom to generate the blog from text files written in Markdown. Blosxom is a bit dated. There are numerous problems with it. One problem in particular is that it relies on the modification time of the files as the publishing date. I thought it was time for a change.

Enter Pelican, a static blog generator written in Python with numerous themes and plugins. I experimented with it a bit and found it quite nice.

Many of the Pelican themes are huge files of Javascript and CSS based on popular frameworks with even more JS and CSS. I made a very minimalist theme instead.

Of course I wanted to convert my old Blosxom blog entries to Pelican. Some of the things I wanted to keep through the conversion:

  1. Titles should be kept from the original. In Blosxom, the first line of the file is the title. Convert that to a "Title: title" in the new file.

  2. Modifiation time of original file should be inserted into a "Date:" timestamp.

  3. Permalinks should not be changed. I want the URL to the original blog entry to be the same after conversion. This is possible with the "url:" and "save_as:" metadata lines.

This is what I came up with:

#! /usr/bin/env python

import os
import time

newprefix = '/tmp/pelicanblog/'

if not os.path.isdir(newprefix):
        os.makedirs(newprefix)

for name in os.listdir('.'):
    if os.path.isfile(os.path.join('.', name)):
        # Get the modification time of file
        mtime = os.path.getmtime(name)

        with open(name, 'r') as f:
            # Now read first line of file as title
            title = f.readline()
            # Write out our collected metadata to name.md instead of
            # name.txt
            newname = name.replace('.txt', '.md')
            nf = open(newprefix + newname, 'w')
            nf.write('Title: {}'.format(title))
            nf.write('Date: {}\n'.format(time.ctime(mtime)))
            nameurl = name.replace('.txt', '.html')
            nf.write('url: {}\n'.format(nameurl))
            nf.write('save_as: {}\n'.format(nameurl))

            # Copy rest of file from original.
            for line in f:
                nf.write(line)

Run this script while standing in the directory where you keep all your Blosxom blog entries. You'll find the result under /tmp/pelicanblog. Weed out unnecessary files and then copy it all to your Pelican's content directory. Done!

The only thing that didn't go smooth in the transition is the syntax highlightning of code snippets. Sometimes the heuristics guess the wrong programming language. I have marked some of the entries manually.

All in all a rather smooth transition to a modern blogging tool.


You've got mail!

I recently switched from Gnus to another mail client, mu4e, which I use with offlineimap. I wrote here about the change a few days ago.

Using offlineimap means I now have all my mail in the local filesystem on my laptop again. Very handy in offline scenarios. Naturally, this also meant I needed a new way of telling me if I've got unread mail.

I duckduckgoed around a bit and found nsbiff, a small Objective C program that shows the number of unread mail in a single Maildir on the OS X status bar. nsbiff is a really simple program, the real code is in BIFFAppDelegate.m and the file is all of 102 lines including empty lines. I thought it quite neat and briefly considered adding stuff like extra Maildirs to check and other small changes...

Then, I accidentelly discovered that the program took up 79 megabytes of resident memory! To count files in a directory and show a number in the status bar!? Really?

Instead, I opted for this in Emacs:

(setq display-time-mail-directory "~/Maildir/hack/INBOX/new")

(setq display-time-string-forms
      '((if display-time-day-and-date
           (format "%s %s %s" dayname monthname day) "")
        (format "%s:%s"
                24-hours minutes)
        (if mail (propertize " " 'display display-time-mail-icon))))

(display-time)

which shows me a small mail icon on the Emacs mode-line if there are any files what so ever in the ~/Maildir/hack/INBOX/new directory.

Since I spend almost all of my tube time in front of Emacs anyway, this will have to do.

I also whipped up this small shell script to list the number of unread mail in some important folders. It's very brute force, since it counts all the files on every run, but on small folders it takes almost no time.

#! /bin/sh

prefix=/Users/mc/Maildir

maildirs="/hack/INBOX /hack/mail.mc-plus and a lot of other folders..."

for maildir in $maildirs
do
    count=`ls "$prefix$maildir/new" | wc -l`
    if [ ${count} != "0" ]
    then
        echo $maildir $count
    fi
done

mu4e — a powerful Emacs mail client

I've been using Gnus as my mail and Usenet News client for over 15 years. For a long time I kept all my mail in local MH folders, updating them with fetchmail. Then I moved to remote IMAP folders with sweet, sweet server-side sorting courtesy of the Sieve sorting language.

Gnus is great a News client but leaves something to be desired when it comes to mail, especially when it comes to search abilities. I've managed so far by SSHing to the server and grepping over the Maildir backend but that, of course, requires SSH access. Naturally I don't have SSH access on $DAYJOB's Exchange server, so grep searching is right out and I had to cope with webmail for searching. Luckily for me, they didn't turn off IMAP on the Exchange server so I can still use Gnus for ordinary mail tasks.

Enter mu4e, a very customisable Emacs-based mail client which works with the very fast mu mail searching tool.

Together with offlineimap and the goodness of the Xapian search engine on which mu is built I now have very quick search available locally.

Running offlineimap on both work and play mail took a while. ~90,000 mail messages covering 1.6 GiB! Actually working with mu4e is quick enough, though. Impressively quick.

Instead of using Emacs' own smtpmail for sending messages, I opted to use the nice msmtp mail sending client and some Emacs Lisp to choose which server to send through depending on the From: line in the message I'm currently composing. See below.

Here's my inital mu4e configuration, subject to change. Some mail addresses has been removed or made slightly more obscure because, you know, spam.

(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e")
(autoload 'mu4e "mu4e" "mu for Emacs." t)
(setq mu4e-date-format-long "%Y-%m-%d %H:%M:%S")
(setq mu4e-headers-date-format "%y%m%d %H:%M:%S")
(setq mu4e-view-show-images t)

;; Silly mu4e only shows names in From: by default. Of course we also
;; want the addresses.
(setq mu4e-view-show-addresses t)

;;; Set defaults.
(setq mu4e-sent-folder (concat "/hack/sent." (format-time-string "%Y-%m" (current-time))))
(setq mu4e-drafts-folder "/hack/draft")
(setq mu4e-trash-folder "/hack/trash")

;; Re-index every 15 minutes.
(setq mu4e-update-interval (* 10 60))

;;; Bookmarks list demands mu4e loaded.
(require 'mu4e)

;;; Create new bookmarks to quickly visit an INBOX and show only
;;; flagged and unread messages.

(add-to-list 'mu4e-bookmarks
             '("maildir:/hack/INBOX flag:flagged OR maildir:/hack/INBOX flag:unread" "Unread or flagged in hack.org INBOX" ?h))

(add-to-list 'mu4e-bookmarks
             '("maildir:/$dayjob/INBOX flag:flagged OR maildir:/$dayjob/INBOX flag:unread" "Unread or flagged in $DAYJOB INBOX" ?s))

(setq mu4e-user-mail-address-list
      '(
        "long list of e-mail addresses deleted for spam harvesting
      reasons."
      ))

;;; Default html2text is no good. Can't use shr since I don't link
;;; Emacs with libxml2. So use w3m instead.
(setq mu4e-html2text-command "w3m -I utf8 -O utf8 -T text/html")

;;; FIXME: Should depend on what folder I'm replying from.
(setq mu4e-compose-signature "MC, http://hack.org/mc/\nIRC: mchack @ Freenode, OFTC\nTwitter: @mchackorg")

(defun mc-set-from-address ()
  "Set the From address based on where the folder of the parent,
if this is a reply. Otherwise, use default."
  (setq user-mail-address
        (if mu4e-compose-parent-message
          (cond
           ((string-match "\/slu.*" (mu4e-message-field mu4e-compose-parent-message :maildir))
            "michael.cardell.widerkrantz@$dayjob.se")
           (t "mc at the domain hack.org"))
          "mc at the domain hack.org")))

(add-hook 'mu4e-compose-pre-hook 'mc-set-from-address)

(defun mc-set-archive-folder ()
  "Set archive and draft folder name based on time and which mail server we use."
  (interactive)
  (let ((from-address (message-field-value "From"))
        (to-address (message-field-value "To")))
    (cond
     ((string-match "mc at the domain hack.org" from-address)
      (setq mu4e-sent-folder
            (concat "/hack/sent." (format-time-string "%Y-%m" (current-time))))
      (setq mu4e-drafts-folder "/hack/draft"))
     ((string-match "michael.cardell.widerkrantz@$dayjob.se" from-address)
            (setq mu4e-sent-folder
                  (concat "/$dayjob/sent." (format-time-string "%Y-%m" (current-time))))
            (setq mu4e-drafts-folder "/$dayjob/draft"))
     )))

(add-hook 'mu4e-compose-mode-hook 'mc-set-archive-folder)

;; Borrowed from http://ionrock.org/emacs-email-and-mu.html
(defun choose-msmtp-account ()
  "Choose account label to feed msmtp -a option based on From
header in Message buffer. This function must be added to
message-send-mail-hook for on-the-fly change of From address
before sending message since message-send-mail-hook is processed
right before sending message."
  (if (message-mail-p)
      (save-excursion
        (let*
            ((from (save-restriction
                     (message-narrow-to-headers)
                     (message-fetch-field "from")))
             (account
              (cond
               ((string-match "mc at the domain hack.org" from) "hack")
               ((string-match "michael.cardell.widerkrantz@$dayjob.se" from) "$dayjob")
               )))
             (setq message-sendmail-extra-arguments (list '"-a" account))))))

(setq message-sendmail-envelope-from 'header)
(add-hook 'message-send-mail-hook 'choose-msmtp-account)

(defun mc-message-mode ()
  "Define shortcuts to sign or encrypt a message."
  (define-key message-mode-map (kbd "\C-cp") 'mml-secure-message-sign-pgpmime)
  (define-key message-mode-map (kbd "\C-ce") 'mml-secure-message-sign-encrypt))

(add-hook 'message-mode-hook 'mc-message-mode)

(setq message-send-mail-function 'message-send-mail-with-sendmail
      sendmail-program "/usr/local/bin/msmtp")

Using DuckDuckGo as a Tor Hidden Service in Firefox search box

I mainly use the DuckDuckGo search engine. DuckDuckGo is also available as a Tor Hidden Service at:

http://3g2upl4pq6kufc4m.onion/

Using this .onion address means you never leave the Tor network and will, of course, require you to have Tor installed to be able to get at it at all.

I've made a simple search plugin for Firefox that uses this hidden service. No fancy packaging yet. You will have to copy it to the right place yourself. Get it at http://hack.org/mc/projects/duckduckgo-tor/.

UPDATE: If you go to http://3g2upl4pq6kufc4m.onion/html/ and go to the search bar menu you will see an extra menu option:

Add "DuckDuckGo (HTML)"

This will do the same thing as installing my plugin, except for the name.


OS X User Notifications from rcirc

Modern OS X has a nice way of alerting a user that something happened called User Notifications. They pop up in a corner of the screen and usually fade away after a while. Typically this is used for instant messages and mail notifications, et cetera.

I found that I would like to have User Notification when my nick was mentioned on an IRC channel or I was sent a private message. Here's how to do it.

First, grab terminal-notifier, a neat little program that generates User Notifications. Install it and test it out with, say:

/Applications/terminal-notifier.app/Contents/MacOS/terminal-notifier -title Title -message Message

If that works, on to the next step. Now grab rcirc-notify.el. Patch it like so:

--- a/rcirc-notify.el
+++ b/rcirc-notify.el
@@ -34,9 +34,9 @@ same person.")
 (defun rcirc-send-notification (title message)
   (cond ((and (or (eq window-system 'mac)
                   (eq window-system 'ns))
-              (executable-find "growlnotify"))
-         (start-process "rcirc-notify" nil "growlnotify"
-                        "-t" title "-m" message "-a" "Emacs.app" (if rcirc-notify-sticky "-s" "")))
+              (executable-find "/Applications/terminal-notifier.app/Contents/MacOS/terminal-notifier"))
+         (start-process "rcirc-notify" nil "/Applications/terminal-notifier.app/Contents/MacOS/terminal-notifier"
+                        "-title" title "-message" message))
         ((and (eq window-system 'x)
               (executable-find "notify-send"))
          (start-process "rcirc-notify" nil

Eval the buffer and turn notify on with M-x turn-on-rcirc-notify RET.

Also add something like this to your .emacs.el:

(autoload 'rcirc-notify "rcirc-notify.el" "Notify me when something happens in IRC" t)

(eval-after-load 'rcirc
  '(require 'rcirc-notify))

Enjoy your User Notifications.