Czerwony Dwór, the Polish name, means Red Manor. So does the Lithuanian Raudondvaris. The name comes from the brick. Barbara Radziwiłł's red-brick residence gave its color to the surrounding village, and the village name attached permanently to the building even as ownership changed hands repeatedly. After Barbara died in 1551, the manor fell into disuse and passed to the Gintowt-Dziewałtowski family, who quickly sold it back to the powerful Radziwiłł clan. Between 1653 and 1664, Prince Janusz Radziwiłł commissioned a substantial reconstruction that gave the building its essential current shape. The round tower, still visible on the south corner, is thought to be a piece of the original Teutonic keep, surviving every demolition the Radziwiłłs and their successors authorized.

Ownership passed through the Worłowski family, then the Zabiełło family, and in the 1820s landed with Benedykt Tyszkiewicz, member of one of the great noble houses of the Polish-Lithuanian Commonwealth. The Russian army devastated the castle after the failed November Uprising of 1831 against Russian rule, but the Tyszkiewicz family rebuilt, and the 1832-to-1855 renovation gave the building its current Gothic Revival appearance. They surrounded it with English-style gardens, a large orangery for lemon trees, an ice house, stables, and offices. The chapel, replaced in 1835, was designed by an Italian named Wawrzyniec Cezary Anichini, who died in the manor and was buried beside the chapel he had drawn. The Tyszkiewicz held the castle through the Russian period and into the early twentieth century, and during that century their art collection reportedly held works by Leonardo da Vinci, Rubens, Caravaggio, and the Polish history painter Jan Matejko.

World War I broke the Tyszkiewicz hold. After the war the new Lithuanian state confiscated the property, divided most of the estate into individual plots for land reform, and put the manor itself to public use. It became a school, then an orphanage. World War II badly damaged the building. The Soviet authorities who arrived after 1944 oversaw a slow rebuilding from 1962 to 1975, repurposing the manor as the Lithuanian Institute of Melioration, the science of land drainage and agricultural water management, an unusually prosaic destiny for a building that had once held a Caravaggio. The institute remains there today, technically known as the Lithuanian Institute of Agricultural Engineering. The art that the Tyszkiewicz family owned is scattered, much of it lost in the war, some in museums elsewhere in Europe.

Inside the castle there are now two small museums. One is dedicated to the Tyszkiewicz family, telling the story of the noble dynasty that lived here for nearly a century and shaped the building's current appearance. The other commemorates Juozas Naujalis, the Lithuanian composer, conductor, and organist born in the nearby village of Raudondvaris in 1869. Naujalis is sometimes called the father of Lithuanian professional music; he founded the first Lithuanian song festival, composed sacred works that still appear in Catholic liturgy, and trained a generation of Lithuanian musicians. That his birthplace happens to sit beside the manor where local nobility once kept European masters on their walls is one of the small accidents of geography that the castle's museum quietly catalogs.

Stand at the base of the round tower today, the one historians believe was part of the original Teutonic Koenigsburg, and you are touching brick that has been here since the late fourteenth century. Six centuries. Barbara Radziwiłł's lifetime, the Battle of Grunwald, Napoleon's army passing through, the Russian Empire's century-long grip on Lithuania, two world wars, fifty years of Soviet occupation, and the restoration of independence in 1990. The tower has watched the Nemunas valley change use repeatedly. The composer's birthplace next door, the orphanage that lived in these rooms, the institute studying drainage in chambers once hung with European masters: all of it accreted here, and none of it erased what came before.