Philip van Geldrop and his son Jan built the first castle on this site around 1350, holding the lordship of Geldrop as a loan from the Duke of Guelders. Jan fought on the Brabant side at the Battle of Baesweiler in 1371 - a battle Brabant lost. The family held the castle through five generations: Philip, Jan, another Philip, Rogier, another Philip again, ruling steadily until the line ran into a daughter. Jutta van Geldrop married Aert Daniels van Goor, who became Philip's heir. On 29 June 1462 the couple sold the heerlijkheid - the lordship - to the Teutonic Order for 3,000 guilders. The Order, by an act of the same date, immediately loaned the lordship onward to Philip van Horne. The castle had changed dynasties before the ink was dry.

For the next two hundred years the Van Hornes owned the castle but rarely came to it - Geldrop was a minor possession in a large portfolio. In 1580, Willem van Horne was beheaded for treason in Le Quesnoy; in 1585 the governor of the Netherlands, Alexander Farnese, transferred the lordship to Willem's sister Maria. Their half-brother Amand I van Horne managed the estate, was legitimized in 1609, and moved in. In 1616 Amand renovated the outer bailey into a proper house - the high central building you see today, with its stepped gables and anchor plates spelling out the year of his work. He fought on the royalist side in the Eighty Years' War. His son Amand II rose high in the Spanish Netherlands army. When 's-Hertogenbosch fell in 1629, the Catholic bishop fled here. After Amand II came his son Amand III, who became a priest; then Martin Ignatius, who used the outer bailey as a cereal storage and hid a clandestine Catholic church on the grounds.

The lordship sold again in 1716 to the Irish-named John O'Donnoghue, and again in 1768 to Adriaan van Sprangh of Leiden - the first Protestant lord, who got just four years of ownership before he died. The French Revolution cancelled all feudal rights in 1795, and the castle's income collapsed. The inventory was sold in 1806. The grounds were rented to local cloth-makers, who used them to span freshly fulled cloth on wooden frames. The keep, by then roofless, was offered for demolition in 1838 and torn down in 1840. The castle was saved - and remade - by Hubertus Paulus Hoevenaar, who had made a fortune in Dutch East Indies sugar. Around 1878 he demolished the drawbridge and gatehouse, added the wing that now serves as the wedding room, removed a tower to create the terrace, and standardised every window. He may have saved the building. He certainly stopped it from being medieval.

Around 1866 Hoevenaar laid out an 11-hectare English landscape garden around the castle. He was a tree man. The park still contains a 38-metre giant sequoia he planted, an 18-piece Platanaceae collection - the only one in the Netherlands - and exotic Oriental planes. A famous European beech of the pendula variety lived in the park for over a century until the Kleine Dommel flooded in 2016 and killed it. That seemed somehow apt for the castle. A monumental orangery from 1870 and two greenhouses survive behind the garden wall. So do the houses of the rentmeester (the land agent) and the small generator house that once powered the estate. The current castle, owned since 30 December 1996 by the Stichting Landgoed Kasteel Geldrop foundation, hosts weddings, concerts, and a municipal Oudheidkamer of local antiquities. Tawny owls, kingfishers, sparrowhawks, and woodpeckers nest in the park. Foxes and roe deer drift in from the connected Collse Zeggen wetland. The bishop's hiding place is now where couples sign their marriage register.