Dirk Van Herlaer, the builder, died in 1354. The castle passed within the family for thirty years - to his son Gerhard, then to Gerhard's brother Arent, then to Arent's son Arent Hoeman. Hoeman was one of the closest friends of Duke William of Gelre and Julich, the man who controlled Gelderland in the late 14th century. So when a land war broke out in 1386 between Gelderland and the Duchy of Brabant, the Duke would have expected his friend Arent Hoeman to fight on his side. Instead, Hoeman sided with the Duchess of Brabant. Duke William ended the friendship overnight. Days later he invaded Ammerzoden, captured Hoeman, and seized the castle. The Van Herlaers never got it back. From 1496 it would belong to the Van Arkels by marriage, for five generations and over a century.

Ammersoyen kept getting attacked. The Habsburgs occupied it briefly in 1513 during the Guelderian Wars; Burgundian troops under Henrick van Nassau held it in the mid-16th century; in 1572 Spanish forces took it for a short time before William of Orange drove them out. The castle was patched up after each round - the Van Arkels could afford it - until 1590, when fire came close to finishing the building. The lord at the time, Joris van Arkel, died of his injuries. Because the fire fell in the middle of the Eighty Years' War the family had no money to rebuild, and the castle waited. The big restoration came after the war's end in the mid-17th century, and then in 1672 King Louis XIV's army marched through during the Franco-Dutch War, destroying castles as it went. Thomas van Arkel kept Ammersoyen standing by paying the French commander 7,000 guilders in protection money - which kept the building but ruined the family's finances for good.

In 1873 the last private owner, Arthur de Woelmont, sold what was left of his possessions to the Roman Catholic Church in Ammerzoden. By 1876 the Poor Clares - a contemplative Franciscan order - had moved in, making Ammersoyen only the second Clarisse convent in the Netherlands. The nuns added a chapel that is still standing. They also filled in the moat. The convent ended with the Second World War: the area was bombed heavily during the Allied advance, the castle gave shelter to townspeople, and by the time the fighting stopped much of the building was a ruin. The strangest chapter came next. After the war a local businessman turned what remained of the medieval fortress into a factory that produced washing machines. For a few years one of the great fixed-plan castles of the Netherlands made domestic appliances.

The Friends of Gelderland Castles Foundation bought the castle in 1957 and started over. The plan was to undo everything - to peel back the centuries of renovations and return Ammersoyen to its original medieval form. It was less hard than it sounds. Many of the alterations had simply been plaster and bricks layered over the older walls, and the medieval shape was waiting underneath. Rooms were returned to their original height and footprint. Chimneys were rebuilt. Staircases were moved back to where they had started. The biggest single discovery was in the moat. When the foundation dug it out, the soil produced fragments of pottery, stoneware, glass, silver, bronze and tin - centuries of household debris dropped into the water. The artifacts are now displayed in the castle's attic. The work was finished in 1976.

From 1976 to 1999 Ammersoyen had a quiet double life: a museum open to visitors, and the actual meeting hall of the municipality of Ammerzoden, which sat in the medieval rooms to do its civic business. In 1999 Ammerzoden was merged into the larger municipality of Maasdriel and the council meetings moved elsewhere. What is left is a four-towered medieval water castle, almost uniquely faithful to its 14th-century original, with a moat that has been put back the way it was, surrounded by a small village of about 4,000 people whose ancestors built it up, knocked parts of it down, took shelter inside it, and finally decided to give it back its old shape.