In medieval Dutch, the word huis meant castle, not house, which is why locals still call this place Hoge Huis or Huis te Sabbinge - the High House, or the House at Sabbinge. The walls at the main floor are eighty centimeters thick, three bricks deep. The basement is vaulted. Above that sits the old hall where the lords lived, with a fireplace set into the northern wall. Until 1832 a moat ran around the entire building, and reports from the nineteenth century describe foundations elsewhere on the site, suggesting that this tower may once have been part of something larger - perhaps with stables, outbuildings, an enclosing wall. Whatever surrounded it is gone. What remains is the keep at the center, the part that mattered most when the bars came down.

The Van Sabbinge family appears in records from the 13th century, and the castle was probably founded at that time, perhaps starting as a motte-and-bailey before being rebuilt in brick. Their cousins the Van Schenge - named for the strip of water that once separated this island from Zuid-Beveland - lived here around 1300. A charter from 1321 records the Count of Holland ordering a reconciliation between three branches of the family after, the document delicately puts it, bloodshed and damage in Sabbinge. The count had previously ordered a man named Looper, son of Jan, to break the house of Claes van Schenge, and things had not improved from there. In 1334 most of Wolphaartsdijk island flooded; the polder around the castle probably stayed dry, but the building was damaged.

In 1457 an heiress of Sabbinge married Wolfert van Cats, lord of the nearby village then called Oostkerke. The marriage united two neighboring lordships, and the new lord had no reason to keep his retainers spread across two castles. The Van Cats moved out. Sabbinge Castle, the seat of a family that had once held knights' charters and feuded with the count's permission, was converted to a farmhouse. On the early nineteenth-century cadastre map, the building is drawn with a large barn attached - the kind of practical extension that made medieval defensive towers into useful farm buildings across much of Europe. The tower kept its thick walls, its vaulted basement, and the shafts beside the windows. The bars themselves had long since rotted or been chopped up for firewood.

By 1959 the farmhouse-castle was in ruins and the Lenshoek family planned to demolish it. Instead, they sold it to the municipality of Wolphaartsdijk for one guilder. In 1962 the mayor, Barend ter Haar Romeny, arranged for a restoration. The Rijksdienst voor de Monumentenzorg commissioned a renovation plan from the Middelburg architect de Lussanet de la Sablonière; the detailed working drawings came from architect Heringa of Zierikzee. During the renovation, workers discovered foundations of an octagonal stair tower along the western wall - a later addition that had compromised the original defensive design and was probably built in a second construction phase, when peace had made the windows worth widening and the bars no longer worth keeping ready. The restored tower today is private property, lived in once again, and the wooden shafts are still visible in the walls.