It is located at the same place occupied, in ancient times, by the Roman Acropolis and, in the Middle Ages, by a mighty fortress. This fortress was demolished in 1446 at the end of the domination of Alessandro Sforza, with the approval of Pope Eugene IV, by the inhabitants enraged for the abuses of several lordlings who used it as a garrison and for the frequent sieges that the city of Fermo suffered just because of this almost impregnable fortress (symbol and instrument of power).

The convent became then State property, was sold at a public auction and was finally bought by Count Francesco Paccaroni, of a noble Fermo family, who took possession in 1811. The new owner modified the building and converted it into a noble residence, without altering its original layout and using mainly materials coming from the demolition of the convent, while the vegetable garden and wood became a garden and a park.

Some decades later, the Villa underwent radical changes following the design of architect Giambattista Carducci (1806 – 1878), who set up, among others, the neoclassical façade (with the typical red brick widely used in the neighbouring area), the large covered entrance on the ground floor and on the first floor a spectacular main hall, with a ceiling adorned with classical decorative paintings by the Fermo artists Nunzi, Morettini and Maranesi. The original plans, in colour, by architect Carducci are still preserved in the Civic Library Romolo Spezioli of Fermo.