Experiments in making IPsec scale

Time-stamp: <2012-11-12 07:24:51 MET>

Presentation given at FSCONS 2012.

About MC

My name is Michael Cardell Widerkrantz. Most people call me “MC”. I work as a consulting software developer specialising in computer network software development.

I live in Malmö in the south of Sweden.

You can reach me at mc AT hack.org.

My web pages:



I'm here to talk about some experiments in making IPsec scale to many nodes. To better give you an understanding on what the problem is I'm first going to talk a little about what IPsec is and what it does.


I assume most of you know what IPsec is but to avoid any confusion I'll present some basics about IPsec.

IPsec is a security extension to the Internet Protocol. It works on the IP layer and protects the layers above: UDP, TCP, et cetera.

IPsec is typically implemented in the kernel as part of the IP stack.

Between two hosts IPsec provides:

IPsec terminology

To understand IPsec we need to know some concepts:

Typical use of IPsec today

Today IPsec is mostly used in two overlapping ways:

The vision

I have a vision. I want to see all nodes on the entire Internet authenticated and all connections encrypted.

Please note that “authentication” here doesn't necessarily mean that we give up personal anonymity. It means that it's much harder to spoof IP packets from another source adress.

The problem - key distribution

The problem is key distribution.

Key distribution and management

There are several existing key distribution systems.

Internet key exchange protocol (IKE)

The most commong key exchange protocol used for IPsec today is IKE, the Internet key exchange protocol. It's typically implemented as a userland server.

How IKE works

I'm briefly going to describe how IKE works. Warning: This is a bit simplified but hopefully described in enough detail so you will understand what I'm talking about later.

Key distribution using X.509 certificates

If we use X.509 certificates for authentication we send a signed certificate in the CERT payload. The signature on our certificate has been done by a Certificate Authority common to both peers.

Of course, this also means that the public key of the Certificate Authority (or a certificate chain leading to the CA) needs to be distributed to all nodes in advance.

In a corporate environment this is simple: We create our own CA and distribute's its public key to all our nodes.

But how do we scale up? What can we do when there are several administrative domains? Which common CA can we use?

Compare the situation with HTTPS. It's a mess! Many CA keys pre-loaded in each browser. How can we trust this mess?

Key distribution by DNS

A simple solution is to use an existing infrastructure with more or less trusted roots: the DNS!

There is already a DNS Resource Record to store our public keys: IPSECKEY, defined in RFC 4025.

We can use DNSSEC to verify the published keys so we know the right administrative domain published the keys.

Earlier attempts at using DNS

Using DNS for publishing public keys is not a new idea. The IPSECKEY record has been around since at least 2005.

Even before that the old FreeeS/WAN Linux IPstack also supported keys in DNS. However, FreeS/WAN used reverse zones and more or less relied on static IP addresses.

The problems with reverse zones is, of course, that most of us are not authorised to change our reverse zones. In this mobile age a lot of us move around a lot and we use dynamic IP adresses so what reverse zone we use may change from hour to hour.

It's much easier to control forward zones. In fact many operating systems support the DNS Update protocol out of the box.

Besides, as far as I know the code used for the old opportunistic encryption has been removed from current FreeS/WAN forks.

Forward DNS key distribution scenarios

I've figured out three different scenarios in how to use forward DNS zones for key distribution. In the following examples I assume we use DNSSEC to validate all DNS data.

Endpoints known by name

This is pretty straightforward:

  1. Alice and Bob already knows each other's name. Perhaps they have been introduced somewhere else.

    They query DNS for each other's address and public keys. When received they load the public key and sets up a Security Policy for the peer's address.

  2. When/if the Security Policy is triggered an IKE dialogue is started automatically.

This scenario provides strong authentication both ways.

Another example: Say I'm working as an external consultant at a company. At our first meeting their IT guy says that he wants me to connect to their VPN concentrator, vpn.example.com. I tell him my laptop is called brain.hack.org. That's it. That's what we need to know to fully authenticate our VPN connection. The rest is automatic.

IKE server queries DNS

Remember that when first connecting over IKE the peer's send each other their identification. This identification can be a fully qualifed domain name.

In effect, Alice sends “Hi, I'm Alice” to Bob. Bob looks in DNS for a public key matching that name and uses it for authentication. And vice versa.

Note that this scenario is open for Man in the Middle attacks during initial handshake if we don't know the names in advance, but it's still “better than nothing”.

If we know the peer's name in advance and have configured our server accordingly, this is as secure as the previous scenario.

Capturing resolver

Now for the most complex scenario. Suppose a local DNS resolver running on the node Alice captures all attempts to query for A or quad-A records in DNS. When a program running on Alice tries to reach Bob it asks the resolver for Bob's adress. The resolver also queries for the corresponding public key.

When the address and the key arrives the resolver loads the key into the IKE server and sets up a Security Policy for the peer's address. When all this is done it returns Bob's address to the program that want's to connect.

When the program attempts a connection it triggers the Security Policy and an IKE dialogue starts. In the dialogue Alice tells Bob “Hi, I'm Alice”. Bob can now query DNS for Alice's key and authenticate her.

Note that this scenario might be open for a man in the middle attack in one direction.

Let me walk through it:

  1. A program asks a local resolver for Bob's address.

  2. Resolver queries DNS for Bob's address and Bob's public key.

  3. DNS replies with address and the public key, if available. The resolver now loads the key and sets up policy.

  4. The resolver tells the program Bob's address.

  5. Alice says “Hi, I'm Alice” to Bob through IKE. Note: This might be open for an attack.

  6. Bob queries DNS for Alice's key.

  7. Bob gets Alice's key and can authenticate Alice through IKE.

Patches to racoon

About a year ago I started hacking on the old racoon IKEv1 server to support these three scenarios. My patches support:

Helper scripts

For this project I wrote two helper scripts in Perl:

These two scripts can and should be used together with a DNSSEC validating resolver.

Experiences using racoon and DNS key distribution

The most promising scenario is “Endpoints known by name”. Provides strong authentication in both directions.

The IKE server queries DNS scenario might be interesting if we add a list of names that we are allowed to query DNS for.

The weakest point in both scenarios is DNS update authentication. How secure is it?

I have found some problems with large DNS records (IPSECKEY) in the field. Also problems with DNSSEC. TCP sometimes filtered.

There is no gain from using DNS in IKE server queries DNS scenario if names not known in advance. CERT payload instead?

It might be better to use a key fingerprint rather than whole key in DNS. This would get around the problems with large resource records, but on the other hand if we use DNSSEC we get rather large records anyway.

It would be better to use IKE version 2 instead of IKE version 1.

racoon is old and it shows. We need a more modern code base.

That concludes the first experiment in making IPsec scale. Now for the second experiment, Better-than-nothing security or BTNS.

Better-than-nothing security

BTNS is IPsec with anonymous keys. This means that even if we don't know anything about the peer in advance we can still use IPsec.

In BTNS the peer's public key, a raw RSA key, is sent in the IKE dialogue.

The key that the peer sends can be validated with a stored fingerprint but we might have a wildcard configured that allows any key to be used.

When using BTNS we can accept the peer's key without validation and just continue to verify that the peer knows the corresponding private key. Yes, this means we're wide open for an active attack, a Man in the Middle, during the initial handshake.

BTNS is defined in RFC 5386 & RFC 5387, both from November 2008.

When I started looking at it no-one had implemented BTNS!

Why use BTNS?

Why would you want to use BTNS?

You want it because it protects the layers above, UDP, TCP, and what have you.

It provides almost everything that ordinary IPsec provides: Confidentiality, integrity and replay protection.

But there's no authentication! It's open for a Man in the Middle!?

Yes, BTNS is open for an active attack but it still protects against passive surveillance. Instead of strong authentication we get continuity of association — we know we are still speaking to the same party we started the communication with.

It also provides importantant spoof protection for long lived sessions. An example were this is much needed is the BGP routing protocol.

We might also use BTNS as a fallback when other keys are not available. For instance, why not use it in combination with the DNS scenarios I mentioned before?

One last point: It's still Better than nothing! Compare the current mail delivery practice: Almost everyone uses SMTP + STARTTLS with self-signed certificates. Why not do the same on the IP level?

OpenBSD's iked

When I started looking into implementing BTNS this summer I surveyed the existing IKE version 2 servers and had a look at their code. OpenBSD's iked caught my eye.

Patches to iked

I have patched iked to support:

Experiences with BTNS

Current limitations

Current limitations in OpenBSD and/or iked:

Potential future work

More information

The projects:

The author:

The sponsor: