On a seventeenth-century engraving of the site, a motte - the artificial earthen mound that supports an early medieval keep - is drawn next to the castle, which suggests the very first version of Baarland Castle was a motte-and-bailey. That earlier structure has vanished into the surrounding fields. The main water castle that occupied the same terrain was probably built in the fourteenth century, with construction unfolding in phases that explain the irregular shape of its bailey. The most useful surviving image is a drawing by Cornelis Pronk, a Dutch topographical artist known for his accuracy, who captured the castle in the eighteenth century alongside the recognizable parish church. Floor plans drawn by Isaac Hildernisse in 1694 confirm the layout but - in a habit Hildernisse repeated elsewhere - give every wall the same thickness regardless of what was really there.

The recorded history opens in 1295 with the Battle of Baarland, when a pro-Holland Zeeland army defeated a Flemish force that had landed nearby. The Flemish were said to have burned everything in their path, houses and churches alike, and any castle near Baarland may not have survived their visit. The following year, in 1296, a Hugo van Baarland was implicated in the assassination of Floris V, Count of Holland - one of the most consequential political murders in medieval Dutch history. Hugo was captured and executed in Dordrecht. His estate, by an old assumption that lacks documentary proof, passed through Aleid van Henegouwen and her 1312 marriage to Wolfert II van Borselen, bringing the castle into the hands of the powerful Van Borselen family who would dominate Zeeland for the next two centuries.

Anna van Borselen, eldest daughter of Wolfert VI, married Philip of Burgundy-Beveren around 1485, and the castle passed to their son Adolf of Burgundy, Admiral of the Netherlands. Adolf left Veere - and his castles - heavily in debt when he died in 1540. His bastard son Philippe de Bourgogne served as bailiff of Veere for nearly thirty years. The castle passed through a tangled succession of marriages to the Smit, Wachtendonck, Van Baarland, Groesbeek, and de Licques families - lords, bailiffs, governors, and Catholics holding out through the wars of religion. In 1734 the Counts of Groesbeek finally sold the castle to Johan Cornelis Lampsins, who restored it in the mid-eighteenth century and actually lived there. Twenty years later it sold again, this time to the Lord of Kerchem.

In September 1840 the demolition was already in progress, the contents advertised in the Middelburgsche courant: marble fireplaces, windows, leather wall-paper, floor tiles, all for sale. Cornelis Adrianus van Bol'es and Jacobus de Backer of Rotterdam, the owners, sold the bare site in January 1855 to Antonie van Hoboken for 65,835 guilders, who turned the remaining bailey into a hunting lodge. The fourteenth-century main building, with its exceptionally thick west and south walls of soft local brick, was gone. The towers were stumps. Only the annex with the two clock gables on the northeast corner survived - the building that today serves as the main house, restored to its seventeenth- and eighteenth-century condition.

When the Feenstras started work in 1965, they drained the moat, leveled the old wall foundations, and built timber forms to pour new concrete - the part that went wrong at first. Mrs. Feenstra did the leveling. They re-used the original soft local brick to face the new walls, so the rebuilt castle has the right surface even where the structure underneath is twentieth-century. Around 1990 a new building was added to the complex at a 90-degree angle. The current Slot Baarland is a working country house, much smaller than its medieval predecessor, surrounded by the same moat that once held off the army of Flanders. The towers on the wall are reconstructions at lower height than the originals, which is honest of the rebuilders - they put back what they could prove and stopped short of what they could not.