Sigismund II Augustus gave Raudonė as a royal manor to Kirschenstein in the late sixteenth century, an unusual transaction that placed Lithuanian crown land in the hands of a German-speaking Prussian merchant. Kirschenstein built a Renaissance manor with the tall cylindrical tower that still defines the building's silhouette. By the eighteenth century the estate had passed to the Olędzki family, members of the szlachta, the Polish-Lithuanian nobility. They commissioned Laurynas Stuoka-Gucevičius, the most prominent Lithuanian architect of the era, to renovate the manor. Gucevičius is best known for designing Vilnius Cathedral's neoclassical façade; here at Raudonė he was working on a smaller scale, but the same hand was on the drawings.

In the early nineteenth century, the estate was acquired by Prince Platon Zubov, a Russian aristocrat who had been the last favorite of Catherine the Great. The Zubov family transformed the building yet again, this time in the Neo-Gothic taste sweeping European country houses. Their architect was Cesare Anichini (whose relative had worked at Raudondvaris Castle a generation earlier). The current Neo-Gothic complex, including a building added in 1877 that served as a warehouse and servants' quarters, is largely the Zubov-era confection: pointed arches, decorative crenellations, the romantic medievalism that wealthy Europeans pasted onto older structures. In 1923 the 1877 building was converted into a mill. The pastoral economy continued around the romantic façade.

From 1898 to 1937, the castle belonged to Sophia Waxell, granddaughter of Sophia von Pirch-Kaiserov who had been Platon Zubov's niece. Sophia Waxell married José Carlos de Faria e Castro, a wine merchant from Madeira, an alliance that brought a Portuguese-Atlantic family connection to a manor in the Lithuanian river valleys. After Sophia's early death the property passed to her husband, then to their son Joseph and his wife Olga Kordashevski and their three boys Nikolai, Vladimir, and Alexander. The family lived through the upheavals of interwar Lithuania, the loss of fortunes, the closing horizons. In 1937 the castle finally became the property of the National Bank of Lithuania, ending three centuries of private noble ownership.

An old park surrounds the castle today, planted with rare specimens: silver fir, Swiss pine, grey walnut, a linden with nine trunks, and the Gediminas Oak. Local legend says the great fourteenth-century Grand Duke Gediminas once had lunch under this oak, on his way somewhere along the Nemunas. The oak stopped putting out leaves some years ago, but the trunk still stands, marked for what it was. The cylindrical tower built by Kirschenstein in the late sixteenth century is open to the public as an observation tower; climb to the top and the Nemunas valley opens below, the same view that has caught Renaissance merchants, romantic Russian princes, and Madeira wine importers in turn. After serving for many years as a public school, the building now operates more like a small art gallery and museum.

Travelers in this part of Lithuania often confuse Raudonė Castle with Raudondvaris Castle further upstream near Kaunas. The names sound nearly identical (raudonas means red in Lithuanian), but the buildings are different and so are the families. Raudondvaris belonged to the Tyszkiewicz dynasty for a century. Raudonė traces its modern shape to the Olędzki and Zubov families. Raudondvaris is Gothic Revival, mostly red brick. Raudonė is Neo-Gothic with the older Renaissance cylindrical tower at its heart. Both face the Nemunas, both survived Soviet occupation, both are open to visitors. The naming overlap is one of the small charms of touring Lithuania's river castles, and the easiest way to keep them straight is to learn which family lived in which one.