I don't know if I'd call what libcap does "compiler theory" it appears to be a darker art that has extra knowledge about a lot of different things. It's remarkably sophisticated though. I implemented a BFF jit engine for a popular ips and spent a lot of time trying to work out a good way to replace libcap before we just went with it. Optimized libcap output will routinely make you scratch your head and wonder how it can work.
I would read with great interest a blog post about your experiences. I have no practical need, but I thought this tour of BPF was fascinating, and it left me hungry for more.
In addition to his blog post, which will probably be great, my reading list would be:
* First, the McCanne BPF+ paper.
* Then, the pcap source code, particularly pcap/optimize.c, perhaps starting with the peephole optimization stuff.
* Finally, one of the BPF JIT projects (there's one on SourceForge). Amusingly: the Linux BPF JIT was at one point used as an aid to memory corruption, by jit-spraying kernel memory. So also consider reading any of the jit-spray papers.
hey Steve here, the BPF+ work was done mainly by Andy Begel... really good stuff but it never got incorporated into libpcap, and I never otherwise published how the original optimizer worked. I did, however, give a retrospective talk on it a couple years back at sharkfest...
