The first written reference, from 1268, calls the place Woninghe ter Nieuwer Schie, the house on the new Schie. In medieval Dutch, "house" was not a euphemism for cottage. It was a category of castle, and the name Huis te Riviere meant house on the river. The river in question was the Merwede, then the main bed of the Rhine after the Old Rhine had been diverted upstream near Wijk bij Duurstede in 1122. Traffic running from Delft to Leiden and on toward Haarlem passed this point, because the branches of the Schie that now feed Rotterdam and Delfshaven did not yet exist. Whoever held this confluence held the toll, the trade, and the leverage over half a province.

Around 1265 someone began to build a castle scaled to that leverage. The plan called for an enceinte forty-five by fifty meters with square corner towers, a footprint enormous for medieval Holland. In the northwest corner stood a separate twelve-meter tower house with walls more than three meters thick, ringed by its own inner moat. The basement was vaulted, lit by slit windows that flared from twelve centimeters outside to ninety-five centimeters inside. Then the money or the will ran out. Construction sputtered between 1300 and 1304 and stopped. For almost forty years the half-built castle sat on its island, the freestanding tower marooned inside its unfinished shell, an arrangement so rare that only Oud Haerlem Castle elsewhere in Holland came close.

In 1339 the castle passed to Dirk of Matenesse, a knight who attached his family name to the place. Dirk seems to have understood that he could not garrison the whole grand enceinte. He concentrated his efforts on the tower house, adding a north wing and turning the keep into something habitable. By 1350 he and his people were living there, with the unfinished outer walls standing around them like the bones of a project no one in the family had ever quite been equipped to complete. The Matenesse family would hold the castle, in one form or another, for centuries, helped along by an inheritance arrangement that kept it in their hands through siblings and children alike.

When the Hook and Cod Wars flared in February 1351, the Cod Alliance moved fast. By 25 March, Te Riviere had been taken. Some chronicles say there was no siege at all; a fifteenth-century account claims Daniel of Matenesse surrendered after one. Either way, the explanation for the speed lies in the castle itself. It was too big for a minor lord to defend, and it stood between the Cod cities of Delft and Schiedam and their trade. The citizens of Schiedam did not wait for politics to swing back. They demolished most of the huge outer enceinte themselves, leaving Dirk with the keep, the north wing, and a vast heap of rubble where the great castle had been.

The castle limped on through the 1400s, was plundered in 1426, and survived in a reduced form until the Eighty Years' War caught up with it. In 1574 Republican soldiers garrisoned in the castle set it on fire twice. The citizens of Schiedam finished what the fires started by pulling the walls down into the moat, where archaeologists would find them exactly four hundred years later. A painting from around 1574 captured the last moment before destruction, and in 1947 and 1948 Dutch archaeologist Jaap Renaud led the excavations that reconciled the painting with the foundations. Renaud thought he was looking at a small castle with an outer bailey. Only in 1962, when archaeologists matched the southern wall of the supposed bailey to the northern one, did the full original plan become clear. The forty-five by fifty meter castle that Holland had almost gained, then forgotten, was finally back in the historical record.