Legend says that the first structure on this spot, a timber tower ringed by a moat fed by the River Mark, was built by a Viking named Gelmel in the ninth century, and the local name for the castle - Gelmelslot, the Castle of Gelmel - has carried his memory ever since. Whether Gelmel ever existed is a separate question. What can be documented is that by the late twelfth century the wooden tower had been rebuilt in stone, and that in the first half of the fifteenth century Jan IV van Cuijk turned it into a serious Gothic fortress with two concentric walls and two concentric moats. The castle sat on the northern frontier of the Margraviate of Antwerp, a forward bastion looking out across what would eventually become the Dutch-Belgian border. Through marriage and inheritance it passed into the hands of the Van Culemborg family, who would hand it, in turn, to its most ambitious owners.

Elisabeth of Culemborg, born 1475, was a dame d'honneur at the court of Archduchess Margaret of Austria - one of the most powerful women in the Habsburg Netherlands, spending most of her life close to the centre of imperial politics. In 1509 she married Antoine of Lalaing, a nobleman whose career was about to climb sharply. In 1518 the emperor Charles V created Antoine the first Count of Hoogstraten. Between 1525 and 1540 the couple poured money into the castle, hiring the architect Rombout II Keldermans - who also worked on Mechelen's town hall and the cathedral at Antwerp - to turn the medieval fortress into a Renaissance palace. Three rings of fortifications. Watchtowers and drawbridges. An armoury, two chapels, colonnades, decorated halls, galleries, gardens, fountains. The Lalaings kept city palaces in Antwerp, Brussels, and Mechelen as well. Hoogstraten was the country seat, the place to show what you were.

Antoine II van Lalaing, the third Count of Hoogstraten, was a friend of William of Orange. In April 1566, with Habsburg religious policy increasingly intolerable to the Protestant nobility, William called a meeting of high Dutch noblemen at Hoogstraten castle to decide what to do. What they decided was to send Philip II a formal petition - the Smeekschrift der Edelen - asking him to stop the Inquisition and grant wider religious freedom. The king did not agree. Two years later the crown confiscated Antoine's estates; part of the castle library was packed up and shipped to the Escorial outside Madrid, where it joined the royal collection. The Eighty Years' War was already beginning. Hoogstraten survived a fire in 1581 and a siege during the 1603 Mutiny of Hoogstraten - when unpaid Spanish soldiers seized the castle and Maurice of Orange marched to relieve them - and was patched up in 1615, but when the great powers declared the area neutral in 1618 the outer defences had to be torn down. Restoration plans were drafted in the seventeenth century. They were never paid for. The inner castle stayed in ruins.

Through the seventeenth and eighteenth centuries the castle passed through marriages to the Salm-Salm princely family, who shuttled between Hoogstraten, an abbey town in the Vosges called Senones, and Anholt Castle in Westphalia, never quite committing themselves to any one residence. When the last countess of Lalaing died in 1709, the inventory of her household showed a single great hall and barely ten habitable rooms - no fine furniture, no precious artworks, not even fruit trees in what had once been a famous garden. Nicholas-Leopold zu Salm inherited at the age of eight, and his executor began rebuilding. In 1740 the emperor Charles VI elevated the county to a duchy and Nicholas-Leopold became the first Duke of Hoogstraten. Fires kept setting the work back: kitchens in 1729, the lower court in 1752, the residential core in 1768. By the time Prince Maximilian died in 1773, the inventory was reduced to the armoury and the wardrobe, although the gardens, remarkably, still produced orange, laurel, myrtle, jasmine, fig - and pineapples, raised under glass in eighteenth-century Brabant, a small piece of horticultural showing-off that says something about how the Salm-Salm princes wanted to be seen.

French revolutionary forces confiscated Hoogstraten in 1796 and stripped it. King William I of the Netherlands lifted the seizure in 1815, but kept the castle itself as Dutch state property - the Salm-Salm family went home to Schloss Anholt in Westphalia, where the family still lives. In 1810 the place was repurposed as a shelter for beggars, originally established by Napoleon in Mechelen and moved here when the Mechelen quarters proved inadequate. Long cell-block wings were thrown up to the northeast and northwest, reportedly modelled on the Hotel des Invalides in Paris. From 1880 the buildings hosted an agricultural penal colony, where convicts worked the surrounding fields. Since 1931 it has been a penitentiary education centre - a working prison with a school inside it, which is still its function today. Behind the moat and the great Renaissance gatehouse that Rombout Keldermans designed five hundred years ago, classes are taught and sentences are served. It is one of the strangest second acts in Belgian architecture: a princely palace where Charles V's noblemen plotted the Reformation has become a place where men finish their education behind bars.