The choice of site was deliberate. Lake Asveja is the longest lake in Lithuania, and the wooded island Vytautas selected sat well inside the water - a defensive position that needed no man-made moat. From here a garrison could threaten the supply routes that Livonian Order raiders used when they pushed south toward Vilnius. The first masonry castle went up in 1412-1413, just as Vytautas was consolidating his power after the Polish-Lithuanian victory at Grunwald two years earlier. Nothing of his original architecture survives in the records. We know it existed; we don't know what it looked like. By the time it next appears in clear documentation, the Radziwiłłs already owned it.

Jerzy Radziwiłł acquired Dubingiai before 1508 and his son built a Renaissance palace on the foundations in the first half of the 16th century. Mikołaj 'the Red' Radziwiłł inherited it next, and under his rule the small town beside the castle became one of the most important centers of the Reformation in Lithuania. This was Calvinist country now. The Radziwiłłs were among the wealthiest noble families in Europe and they used Dubingiai as their main seat for the Biržai-Dubingiai branch of the family until the second half of the 17th century. Contemporary accounts called the palace one of the finest residences in the Grand Duchy, second only to the Royal Palace of the Lithuanian rulers. Around 1620, Janusz Radziwiłł built the Calvinist Church of the Holy Spirit beside the castle - intended explicitly as the family mausoleum.

In 1547, Sigismund Augustus - heir to the Polish-Lithuanian throne - secretly married Barbara Radziwiłł, a noblewoman from this very family. The marriage was a scandal of European scale: the king's mother Bona Sforza opposed it, the Polish nobility despised it, and Barbara was crowned Queen of Poland only after years of political fighting. She lived just two more years after her coronation. But before all of that, immediately after the secret wedding in 1547, Barbara spent five months at Dubingiai Castle. It was here, on this lake island, that one of the most famous queens-by-love in Polish-Lithuanian history quietly waited out the political storm her marriage had caused. She would die in 1551, possibly of cancer, possibly poisoned. Sigismund grieved publicly for years.

The Polish-Lithuanian-Swedish wars of the mid-17th century were brutal on the great noble residences. Bogusław Radziwiłł, the family's most controversial figure, sided with the Swedes during the Deluge and was branded a traitor; Dubingiai was confiscated by armies loyal to the King of Poland and Grand Duke of Lithuania. It was pillaged. It came back to the family in the second half of the 17th century, but the Biržai-Dubingiai branch had moved its seat to Biržai Castle, and the lake palace gradually fell into disrepair. The mausoleum church beside it deteriorated alongside the residence. In 1808, Michał Tyszkiewicz bought the property. Today only foundations and a few cellars survive, and these are still being investigated by Lithuanian archaeologists.

In 2004, archaeologists working at the ruined church found the actual remains of the Radziwiłł family. Mikołaj 'the Black' (died 1565), his wife Elżbieta Szydłowiecka (died 1562), Mikołaj 'the Red' (died 1584), and Janusz Radziwiłł (died 1620) - the most powerful members of one of Europe's great Protestant dynasties - had been lost beneath the ruined floor of their own mausoleum for centuries. On September 5, 2009, they were reburied at Dubingiai with full ceremony. In 2024 a fund was established to rebuild the castle itself. Among the founders is Duke Motiejus Radvila (Maciej Radziwill), a contemporary representative of the family, working alongside Professor Albinas Kuncevičius and Dr. Vydas Dolinskas, who oversaw the restoration of the Palace of the Grand Dukes in Vilnius. Whether the rebuilt castle ever rises again is an open question. The lake, at least, is exactly where Vytautas left it.