pf (originally from OpenBSD, but now part of FreeBSD as of 5-RELEASE) is also a great option for building firewalls on FreeBSD. The rules syntax feels more modern, and just makes more sense to me.
Depending on how many (hundreds? thousands?) of simultaneous sessions you're planning to have it control and monitor, dualcore dual opterons may even be overkill. Lots of RAM on a P4 box may surprise you on actual performance. I would also suggest using "hardware-based" network adapters (eg ones made by 3com, Intel) rather than the $5 Realtek cards people often try to throw at this stuff. It makes a big difference in high load situations, and the hardware based cards aren't THAT expensive anyways.
More info here on pf :
http://www.freebsd.org/doc/en_US.ISO...ewalls-pf.html
Alternately, you could take a look at some of the "Cisco-killer" open-source routing solutions available now. Vyatta is the big name in open-source router replacements using x86 hardware. These are not some SOHO "D-link" caliber routing solutions, but rather hardened OSS projects meant to replace Cisco equipment in real environments.
More on Vyatta :
http://www.vyatta.com/