Yes, I think I would have felt safe there, living on the top floor.
It was built after Albigensian Crusade against the Cathars in the 13th century. The land was given to Pierre de Voisins, one the lieutenants of Simon de Montfort, 5th Earl of Leicester. In the town there is an interesting little museum, mainly about the Cathars. It gives some information about the various views and opinions about this group of anti-catholic Christians. One of the views is that the Cathar were not all that important, but that the ‘myth’ was created and is developed and maintained to promote tourism in the area. This endeavour, if that is what it is, was helped by the books of Kate Mosse: Labyrinth, Sepulchre and Citadel. They are set in different periods of history in the Languedoc and deal with the Cathar story.