John VIII, Count of Nassau-Siegen (1583-1638), commissioned the castle in 1630 to serve as the ancestral seat of the Catholic branch of the House of Nassau. The choice of location was political. The main Nassau line - the Princes of Orange - had become Protestant and was leading the Dutch Revolt against Spain. John VIII's branch had converted back to Catholicism, allying itself with the Habsburg Southern Netherlands, and Ronse sat firmly on the Catholic side of that religious frontier. The design borrowed from the Luxembourg Palace in Paris, which Marie de' Medici had completed only a few years earlier. The result was an unmistakably French Renaissance composition set down in a small Flemish textile town. Anthony van Dyck painted the family portrait in 1634 - it now hangs at Firle Place in Sussex - showing John VIII, his wife Ernestine Yolande de Ligne, and their children in the kind of dynastic pose that the castle was built to embody.

John VIII's heirs spent generations arguing over the barony, and their disputes eventually killed the family's interest in actually living in Ronse. By 1745 the property had been sold to the Counts of Merode-Westerloo, who treated it as a property to visit occasionally rather than a home. The Merodes held it through the second half of the eighteenth century, but they too rarely came. When the French armies crossed the Sambre at the Battle of Fleurus on 26 June 1794, they swept away the legal framework that gave the barony of Ronse its meaning. With the title abolished, the castle suddenly had no owner who needed it, and no obvious purpose. It was put up for sale, and the decay began almost immediately.

Under Napoleon, Ronse became one of the first towns in the Southern Netherlands to industrialize. The castle was repurposed for that new economy in the most literal way possible. In 1803, the Lousbergs brothers installed 180 cotton looms in the castle's cellars, opening the first large weaving mill in the city beneath the vaulted Renaissance ceilings of a Nassau palace. For a few years the building did double duty - decaying upper floors above and industrial production below. The textile equipment moved on, the building was bought in 1821 by Alexandre Louis van Hove, and van Hove offered the castle to Ronse's city council. He wanted it used as a hospital or a secondary school. Mayor Eugene Ferdinand Fostier, who had returned to office in 1820 and had personal reasons to dislike van Hove, refused.

By 1823 the castle was sold for 30,000 francs and demolished. The decision was not really architectural or economic. It was a final act in a long-running feud between two families - the van Hoves, who had represented the lord under the old regime, and the Fostiers, who had risen with the French Revolution. The conservative side and the progressive side could not agree on what to do with the building, and so they agreed instead to lose it. In 1844 the notary Charles Alexander Snoeck divided the twenty-hectare estate into parcels. He built a villa for his son Cesar, the famous musicologist, on the foundations of the castle's old gatehouse. Today you can still see the seventeenth-century barrel vaults in that villa's basement, with their meter-thick whitewashed brick walls and sandstone doorways - the underground anatomy of a building that has otherwise vanished completely. The courtyard where Nassau guests once disembarked from their carriages is now a backyard, hemmed in by the old foundations.