Ter Hooge means on the height, which is the most useful clue to what stood here first. In 1291 a Symon van der Hoghe pledged fealty to the Count of Holland, and historians read the date as a likely build year for the original castle. The mound itself, polygonal in plan and roughly forty-two meters across east to west, was almost certainly artificial. A 1695 floor plan by Isaac Hildernisse survives in copy, drawn at the Rijnland foot of 0.314 meters, and it lines up convincingly with a drawing by the meticulous Cornelis Pronk. The castle had buildings on all sides except the east, where a gallery was later added. The northwest corner held a tower house and the slightly taller stair tower that still stands inside the modern central building.

The castle changed hands the way castles did. In 1448 it was bought by Adriaen Jacobsz of Middelburg, whose family promptly began calling itself Van der Hooghe after the place. The privileges were medieval and specific: in 1501, Philip the Fair granted the right to have a moat and bridge; the owner could brew beer untaxed, hunt and fish in his own grounds, and refuse the entry of law officers onto the outer bailey. During the Siege of Middelburg from 1572 to 1574, the castle was occupied by Sea Beggars under Willem Bloys van Treslong and badly damaged. The family split along the great religious fault line of the age: one branch stayed Catholic and was effectively banned from office in Protestant Zeeland; the other turned Protestant and rose. Mayors of Middelburg, members of the Council of State, special envoys to England, favorites of stadtholders. The Protestant branch also developed an interest in claiming nobility they did not have.

Jan van Borssele van der Hooghe was the most ambitious of them. In 1750 he married the very rich Anna Margaretha Elisabeth Coninck, lady of Ritthem and assorted other small lordships. He bought the lordship of Borsele, with its high justice, in 1750. He bought Ter Hooge Castle in 1751. From 1754 to 1757 he tore down most of what was there and built a new manor in the French style, following examples from Jacques-Francois Blondel, with the final design perhaps by Pieter de Swart. The old stair tower became one half of a symmetrical composition; the east side got a brand-new tower built to match it, and two new wings created the U-shaped plan that visitors see today. The drawing room got a ceiling of Rococo stucco, possibly by Carlo Laghi. The front moat was filled in. The back moat was widened into a pond. The whole project announced: this family is ancient and noble and you should treat us accordingly.

Jan also pursued the paperwork. In 1761 he sponsored the publication of a book by Willem and Jona Willem te Water that asserted the Van der Hooghes descended from the prominent medieval Van Borselen family. The argument leaned on a manuscript by Jacob van Grijpskerke, who had died in 1683. A century later, Robert Fruin took a hard look at the documents. He proved that the seventeenth-century Jacob van der Hooghe had never been styled Jonkheer (the Dutch courtesy title for nobility) and had always signed himself plainly Van der Hooghe. He looked at a 1624 list of Zeeland nobility on which Van Borssele van der Hooghe was not mentioned, and found that the family's signature had been inserted, in different ink and a different period, after the fact. The whole edifice came down: the descent from the Van Borselen was a fabrication, and several supporting documents were proven forgeries. The twin tower remained, as it remains today, an architectural lie that has outlived its purpose.

The castle survived its owners. After Jan died in 1764 it passed through the Van Citters family, then in 1846 to Daniel Jacques de Superville (45,000 guilders), then in 1856 to Henri Dignus van Brucken Fock at auction. His son, the composer Gerard von Brucken Fock, was born here in 1859. In 1871 a Count of Lynden bought the property and renovated the interior to neo-Renaissance style; his descendants own it through the Stichting Lynden ter Hooge to this day. The hardest blow came not from owners but from war. In October 1944, Allied bombers breached the dikes of Walcheren to flood the German defenders out of their positions ahead of the Battle of Walcheren Causeway. The water came in. The park around Ter Hooge was destroyed. A new forest was planted in 1947 by the Nederlandsche Heidemaatschappij. The castle itself stands, now divided into four apartments and quietly lived in, the old tower and the false one still flanking the central building like two brothers, only one of them real.