On 29 August 1383, Joanna, Duchess of Brabant, signed away the Lordship of Loon op Zand to Pauwels van Haastrecht, the schout of 's-Hertogenbosch, who had just paid off the previous owner. Pauwels celebrated by building a brick keep on a low rise in the marshland, a square tower 12.7 by 11.44 meters with walls more than two meters thick. The bricks themselves date the work to those years, between 1383 and 1387. Within a generation, an addition was bolted onto the east side, after Pauwels picked the wrong side in a fight with the Duchess and faced a siege of his own walls. The little keep grew outward, gaining an outer bailey ringed by a moat, then a second bailey beyond that, then a free-standing gatehouse dated 1536. By the late sixteenth century it was a small fortress in the Brabant countryside, doing the unglamorous work of guarding a flank.

On 24 June 1587, Dutch troops under Philip of Hohenlohe-Neuenstein wheeled five guns through the Langstraat and began shelling the keep. The garrison, holding for Spain, surrendered after a week of bombardment. What happened next is the kind of detail that history books prefer to summarize. The captains were killed on the spot. The soldiers were taken prisoner. The garrison's wives, the women who had been living inside the castle, were stripped naked and turned out, and they walked with their children to 's-Hertogenbosch to tell what had been done to them. The Dutch held the castle for five weeks, then set it on fire on 30 July and retreated. Peter Ernst of Mansfeld then reoccupied it for the Spanish side and threw up earthen ramparts on the vulnerable approaches. Much of that 1587 earthwork is still visible today, a green ridge in the grass.

Peace, when it came, transformed the building from a weapon back into a home. In the 1660s, Thomas of Immerseel turned the keep upside down. He demolished the north wall above the basement and rebuilt it, shaved brick off the inner walls to gain living space, added a floor, and put on a new roof. The old covered bridge over the moat became a wide, dignified staircase building flanking the keep. The defensible tower had quietly become an apartment complex, with the upper levels rented out as guest suites. The artist Causé etched the result, and for a long time historians used his print as the best surviving picture of the place. The 2009 research finally proved that Causé got it right almost to the brick.

In 1776, Prince Louis Charles Otto of Salm-Salm decided his inheritance needed a fashionable upgrade. He lowered the keep by a floor, lifted off the steep roof and replaced it with a shallower one, thinned the inner walls, and punched out a tidy grid of regular windows. He moved the entrance to the dead center of the north facade. The outer bailey, with all its medieval clutter, was bulldozed and replaced with an open courtyard between two new outbuildings: a carriage house on the east, the Neerhuis on the west. The transformation was so complete that nobody believed any of the medieval fabric survived. Underneath the powdered-wig facade, though, the original keep was still standing, just shorter.

By the early 2000s the castle had run through hands ranging from Dutch nobility to an antiques dealer who hosted art shows on the lawn. Then a wealthy buyer named de Pundert threatened to let the place rot if the municipality did not approve his radical plans. The locals occupied the castle in protest. In the unusual climbdown that followed, de Pundert agreed to hand the entire estate over for free if a charitable foundation were created to take it. Stichting Het Witte Kasteel, the White Castle Foundation, was born. With 100,000 euros from the municipality, 230,000 in donations, and the work of sixty volunteers, the buildings and gardens came back to life. Today the four-hectare estate, with its English garden, French garden, lawn, and forest garden, hosts weddings, a restaurant, and a bed and breakfast in the very rooms where Pauwels van Haastrecht's stonemasons set their bricks in 1383.