In 2008 archaeologists dug into the grounds and found brick walls 1.3 metres thick, dating to the late fourteenth century. Dendrochronology pushed back the timber to 1300, and re-used tuff stone in the foundations hinted at something even earlier. This was a real defensible castle, square in plan, surrounded by a moat and an inner court. What is striking is that the medieval Dommel did not flow where it flows now - it ran just north of the castle, putting Nieuw-Herlaer on the west bank, the same side as nearby Maurick Castle in Vught. The river has since shifted. Today Nieuw-Herlaer sits on the east bank, and the geography of why it was built where it was no longer quite makes sense without an old map.

From 1579 onward the castle became a piece on the chessboard of the Eighty Years' War. In November of that year Claude de Berlaymont made Nieuw-Herlaer his headquarters with a garrison of 's-Hertogenbosch militia. The usual garrison was 25 men. In 1601 the lord of the castle himself, a man called Warnard van Honselaer with no military experience, demanded command of the troops and got it. When Maurice, Prince of Orange besieged 's-Hertogenbosch for the second time in 1603, a contemporary siege map shows Nieuw-Herlaer falling into his hands. In 1610 the garrison was finally withdrawn. The damage was done: by the time the Dutch Republic took 's-Hertogenbosch in 1629, the Catholic owners were finished as political players, and the castle began to slide.

The decline came partly from a strange ownership tangle. After the death of Gerard Proening van Deventer the castle was split in two - one half going to his daughter Maria, the other to her sister Anna - and the halves stayed legally separate for more than a century. In 1710 the Van Gerwen owners of one side ordered a partial demolition that took down the great hall, sparking a dispute with the other side and forcing them to throw up a new facade. By the time Countess Genoveva van Welderen finally bought both halves in 1789 and married Sigismund van Bonstetten, the medieval castle was beyond saving. They razed it. Only the tower was left standing, swallowed up into a new country house. The Batavian Revolution then promptly ruined the Bonstettens, who auctioned everything off in 1798.

What happened next is the part that makes Nieuw-Herlaer unusual. In 1799 priests moved in, turning the manor into the seminary of 's-Hertogenbosch. In 1840 the bishop converted the buildings into the Instituut voor Doven, the first Catholic school for deaf children in the Netherlands - it started with 40 pupils and expanded with classrooms, a laundry, and a bakery for sacramental bread before moving down the road in 1910. French Benedictine sisters from Jouarre Abbey arrived in 1911, fleeing France's anti-clerical secularisation laws; they renamed the place Nieuw-Herlaer Abbey. In 1921 the building sheltered 50 Hungarian children. After 1955 it became a residential institute for people with severe developmental disabilities, with an observation clinic for child psychiatry added in 1960. The Austrian nuns who had taken over after the Benedictines stayed on as nursing staff.

The clinic moved out in 1970 to a new facility called Herlaarhof in Vught - a name that still carries the castle with it. The buildings then went through a difficult decade until 2006, when Cor Pijnenburg bought the property and restored the main house and the convent wing, demolishing the rest. What stands today is a fifteenth-century tower welded onto a late-eighteenth-century manor, set among the meadows where Halder's medieval fulling mill once worked wool cloth within sight of the castle walls. The mill is gone, the moat is gone, the garrison is gone. The stair tower is still here, holding up the corner of a building that has been almost everything a Dutch building can be.