The Swords were the undisputed whipping boys