The demolition of a building known as "Britain's biggest man cave" has cost taxpayers around £220,000, a council has said.