The first version of Dussen was a single square tower, built in or before 1330. In 1387 the lord Arent van der Dussen received a remarkable charter from Albert I, Duke of Bavaria, authorising him to "make and build our house on our manor at Dussen as big and strong as he likes." The Van Dussens used the permission. Over the next six years the castle gained the full footprint of a proper water castle: ringed by water, fortified at the corners, the old keep absorbed into the southeast corner of the new square plan. By the time of the flood it was a fully developed late-medieval fortress, occupying a strategic spot at the disputed edge between the County of Holland and the Duchy of Brabant. Arent II van Dussen had served as bailiff of southern Holland three times and had been ransomed at a high price after losing the Battle of Baesweiler in 1371. The family had political weight.

Arent III had only been lord of Dussen for four years when the water came. The Grote Hollandse Waard, a vast reclaimed polder between Dordrecht and the Maas, had grown careless about its dikes; one breach was enough. The flood waters spilled across the Land van Altena and reached Dussen Castle. The walls above the basement level were destroyed; what masonry survived was probably looted in the difficult years afterwards, when usable brick was a precious commodity in a flooded landscape. From 1422 until 1467 the dikes around the Land van Altena were slowly restored. Only in 1473-1474, half a century after the disaster, did Jan V of Dussen finally rebuild the castle - low wings around an open courtyard, with the two old entrance towers left standing where the water had spared them.

In 1607 Walraven van Gendt, a baron with a long career in the Dutch and Brandenburg armies, bought the castle. He had fought at the Battle of Nieuwpoort in 1600. He rebuilt Dussen almost from the ground up - heightening the 1470 wings by a storey, and adding an elegant Tuscan colonnade around the inner courtyard. A commemorative plaque dates that work to 1609. From the outside Dussen now looks like a Renaissance country house with two improbable medieval towers stuck onto the front. From inside the courtyard, with the columns and round arches running around all sides, it looks like an Italian palazzo that has been picked up and dropped onto a Brabant marsh. The northwestern wing stayed open as late as 1672; it was closed and the western tower restored between 1672 and 1714.

In 1901 the Carmelite sisters arrived. They had been expelled from France by the new French anti-clerical laws and they made Dussen their convent until 1920, adding a chapel along the way. After they left, the castle drifted. It was offered at auction in April 1924 for 10,400 guilders, with a clause forbidding demolition - inserted at the request of the provincial archaeological commission. The first sale fell through. At the second auction the clause was dropped, and the local municipality tried to intervene with a bylaw against demolishing monuments. Eventually Lucas Daniel Suringar from Koudekerken bought the castle for 20,000 guilders. The plans for restoration took years to firm up; the work was finally ordered in July 1940 - while the Netherlands were under German occupation - and started a month later. In the winter of 1944-1945, with the front line nearby, the castle was badly damaged. A second full restoration ran from 1950 to 1953.

Dussen Castle reopened as the town hall of Dussen in June 1954, and stayed in that role until the municipality was absorbed into Werkendam in 1997. Today the castle belongs to Landgoed Brabant B.V., a subsidiary of Monumentenbeheer Brabant, and is used for weddings and cultural events. The volunteers of the Stichting Vrienden Kasteel Dussen organise much of the programming. Stand in the courtyard and look up at the gallery and the two towers and you are looking at three layers of decisions: the medieval tower from before 1421 that the flood could not pull down, the 1470s low rebuild after the dikes were finally fixed, and Walraven van Gendt's 1609 Renaissance heightening with its Tuscan arches. The castle is famously asymmetric. It is asymmetric because, six hundred years ago, the sea came up the Maas and rearranged the building.