you're not crazy, it'd be nice if it worked that way, but it's a bit more complex.
just a warning though: this isn't perfect. referrers are frequently broken or just plain missing. it's not really wise to only allow certain referers in unless you also allow null referers and some of those all x referers some software generates.
ok, with that said, here goes.
let's assume you just want to allow all referers in except one:
RewriteEngine on
RewriteCond %{HTTP_REFERER}
http://www.baddomain.com/ [NC]
RewriteRule /*
http://www.yahoo.com/ [R,L]
you can use ! in front of the
http:// on the RewriteCond line to negate the conditional. check hotlink protection .htaccess rules for more details, or Apache's excellent guide
http://httpd.apache.org/docs/misc/rewriteguide.html