I have written a beginning HOWTO type document on how to use IPSECKEY DNS records with my very early racoon patches. Please see Using Racoon with IPSECKEY Records on FreeBSD .
The text is mostly for testing and development purposes, not for production use.
I have only tested on FreeBSD 8 and 9 but it might work equally well under Linux. Please test and report.