>Seriously? You're attacking, why on earth would you want to do that before removing the enemy flag?

Why I would want to use the enemy anvil? I dunno, you made it sound like your buildings needed a protection ;)

>Physical access is a very, very shaky concept. It practically means that every castle with a hole belongs to everyone. Both teams would essentially open up everything in order to not get locked out - at which point the castle might as well not exist at all.

So your main doubts about that would be that A) you cannot enter your own castle from the outside and B) while the enemy is inside your castle your gates are useless (assuming he actually opened all of them manually instead of doing whatever he wanted to do in your base).
I think we should try to think of interesting solutions without a building restriction (and some magic radius). That could for example be keys that you use to open doors that you set up with locks (but you lose the key on death), "sensor" mechanisms (think of dwarf fortress pressure plates) that allows you to link certain doors to sensors and whatever more.