Jacob van Wassenaer built the original Rosenburgh sometime in the 1200s, a kilometer north-northeast of his family's Duivenvoorde Castle. The Wassenaers were everywhere in this part of Holland. By the early 14th century the property had passed through Jacob's granddaughter Jacoba and her husband Simon van Benthem, descended from a younger brother of Dirk VI, Count of Holland. Then, in 1351, the castle stood on the wrong side of the Hook and Cod wars - a brutal aristocratic feud over the regency of young Count William V. The Hook faction held Rosenburgh. From 28 March to 21 May, the Cod forces besieged it, at one point under William V's personal command. They brought gunpowder. The castle was probably reduced to rubble. By 1399, the records refer only to the Hofstede, the terrain - meaning whatever had stood there was gone.

Between 1482 and 1534, somebody put a second castle on the site. It may have incorporated remnants of the first - vaulted basements, a section of an old tower. The 17th-century chronicler Van der Houve saw both and recorded them. Then came the Eighty Years' War. The 1573 Siege of Leiden, fought a few kilometers away, destroyed this second castle too. In 1617 Adriaan van IJlem rebuilt it, mounting an old tower on the older work and adding what was described as an elegant new building. A Latin commemorative stone inside recorded the destruction and the rebirth. The castle changed hands through marriages and sales - Stalperts, IJlems, De Waals, Wijntgis - the kind of long inheritance chain that fills Dutch property registers. By the 1630s it belonged to Pieter de Wit, then to a Leiden professor named Jacob Mastertius.

Jan Gerrit baron van Wassenaer was born in 1672 and made his career at sea. He rose to Lieutenant admiral of Holland, which was as high as a Dutch naval officer could climb, and retired sometime after 1709. He had married into the Van Liere family. He bought Rosenburgh, called himself Heer van Rosenburg, and decided to build something his rank deserved. From 1721 to 1723, on the foundations of the second castle's grounds, he raised a new manor. Designs by the French Huguenot architect Daniel Marot - the same Marot who designed William and Mary's interiors at Hampton Court - turned up in the National Archives in the 1960s, connected to this site. The visiting artist Abraham de Haen called the manor overly luxurious and palace-like. Jan Gerrit died in 1723, the year his palace was finished. His thirteen-year-old son Jacob Jan inherited it. Jacob Jan died in 1728.

The next heir was the Lady van Liere van Katwijk, and she made a decision that still seems remarkable: she had the whole manor demolished. Almost everything Jan Gerrit had built, almost everything Marot had designed for the site, was pulled down. Only an annex was left standing. No one fully understands why. The most likely answer is the simplest - upkeep on a palace built for an admiral's ego, on land that no longer fit anyone's plans, was a kind of curse. Easier to flatten it. De Haen's 1725 drawing of the finished manor survives only as a copy by Andries Schoemaker. The actual building lived less than a decade.

In 1952, the archaeologist Jaap Renaud went looking for the medieval castle and found nothing. He concluded its foundations had been removed. In 2010, earth augers turned up heavy debris northwest of where the manor had stood - which meant the medieval castle had been somewhere else entirely. Then in 2012, when Kinderboerderij Voorschoten - the children's farm and petting zoo now occupying the grounds - wanted to expand, archaeologists were called back. They found four stone foundations beneath the planned new building. They were the corners of Daniel Marot's manor. The agency recommended preserving them. The petting zoo's new main building was constructed on top of them anyway. Goats now graze on what was, briefly and expensively, one of the finest country houses in Holland.