But to be honest, at least when I sling around such accusations, a lot of it is hyperbole. It's a way of getting up their noses, of ridiculing their pretensions, rather than a real comparison of them with Mussolini and his Black Shirts.
Well, at least usually it is.
Residents of the planned eco towns in England could face strict monitoring of their travel habits, home insulation and even wasted food, to ensure they are truly living a "green" lifestyle.
Experts advising the government on its plans to build up to 10 eco towns by 2020, yesterday called for ministers to toughen environmental standards for the developments with monitoring to ensure their carbon footprint is three times smaller than the British average.
The recommendation is that there should be detailed scrutiny of the number of trips residents make by car, and the types of waste produced by households and businesses. Thermographic cameras should be used to check which homes lose heat, according to the Commission for Architecture and the Built Environment (Cabe).
The monitoring plans are part of the proposed standards for the government's flagship housing programme, which has been criticised for failing to demand the highest levels of sustainability.
I'm all in favour of the idea that we might tread more lightly upon this Earth even as I criticise the methods offered to help us do so. But do we really need to have block captains monitoring our every move?
That really is getting a little close to fascism, isn't it?