Here's a short status update about my racoon hacking (project pages). I've made some progress and found a few silly mistakes.
First, the IPSECKEY RRs I added to the hack.org zone had a mistake. I accidentally marked the keys as DSA keys (algorithm 1) instead of RSA (algorithm 2). On the other hand, my code didn't even look at the algorithm to see that we actually had an RSA key. When I added this, it was the first time I noticed the mistake.
I've also changed the way keys are loaded. Instead of loading the
public key to the internal list of keys, I just set it to be the
peer's public key (the
struct ph1handler element called
after querying DNS.
I was stuck for a while here because when loading the binary key with
binbuf_pubkey2rsa() I got an even modulus! It took a while to find
out why. The reason was I accidentally allocated too large a buffer
for the key.
Anyway, here's the debug output for node ipsec1 documenting the first successful security association setup between two nodes with transport mode opportunistic encryption using DNS keys:
In case you're worried the line about the CERT validation being disabled is of no concern. It's just racoon's way of saying that we're not trying to compare the name in the CN in an X.509 cert with the peer's ID. It would be quite silly to check that in this scenario when there's no CN and no cert.
I need to polish the code at least a bit before publishing. Stay tuned.