Long before Augusta arrived, Koluvere was the kind of place that Baltic history is made of. A castle has stood on the strategic spot since the 13th century, and the high square tower at the heart of the complex probably dates from that earliest period, making it one of the oldest castle towers in Estonia. In 1439 the building came into possession of the bishop of Saare-Lääne and served as one of the bishop's main residences for the next century, an episcopal seat managing the spiritual and secular affairs of western Estonia from a fortified island. During the Livonian War in 1573, an outnumbered Swedish army defeated a Russian army nearby in what became known as the Battle of Lode. Thirteen years earlier, in 1560, Estonian peasants in revolt had reportedly tried to storm the walls.

Between 1646 and 1771 the castle belonged to the von Löwen family, Baltic German nobles who had no further use for military fortifications and turned the building into an aristocratic residence. The cannon-tower added in the 16th century became a quiet curiosity rather than a defense. A sculpted stone above the entrance, carved with the coat of arms of Reinhold von Buxhoeveden, bishop of Saare-Lääne from 1532 to 1541, dated from the same transitional moment when bishops were giving way to private landowners. Around 1770 the interior was extensively redecorated in Rococo style, all curving plaster and pastel pigment, the kind of room a duchess from a German court might have recognized. In 1771 the castle passed to Grigory Orlov, the Russian count and Catherine's former lover, and shortly afterward became Catherine the Great's own property.

Augusta of Brunswick-Wolfenbüttel had been married to Prince Frederick of Württemberg in 1780 when she was 16. He was, by every account, brutal. She wrote desperate letters to her sister Charlotte, the wife of Tsar Paul, who would later become emperor of Russia. The case for asylum reached Catherine, and the empress, herself a German princess by birth, intervened. Augusta was brought to Russia under imperial protection in 1786. Catherine couldn't return her to her husband and couldn't keep her openly at the Russian court without diplomatic disaster, so she sent her to Koluvere, far enough from the political centers to be discreet. Augusta arrived young, traumatized, pregnant, and surrounded by a castle that had been redecorated for an aristocrat's comfort but had no one in it she knew. She lived there for two years.

Augusta died at Koluvere in 1788, only 23 years old. The cause was never properly established. Some accounts suggest complications from childbirth, others suggest poisoning, others suggest her health simply collapsed under the strain of exile and isolation. Her grave is in the Kullamaa parish church a few kilometers away, and her story has fed local folklore for two centuries. People in the region still talk about the lonely young duchess. In 1797 Emperor Paul I, the husband of Augusta's protector Charlotte, presented the estate as a gift to General Friedrich Wilhelm von Buxhoeveden, the Russian commander who shared a name with the long-ago bishop whose coat of arms still hung above the gate. The castle stayed in the Buxhoeveden family until 1919, when Estonian independence ended the era of Baltic German manor ownership.

Between 1924 and 2001 the castle was used by various welfare institutions, the kind of pragmatic 20th-century repurposing that swept through Baltic manor houses after the German nobility lost their estates. The 19th century had given Koluvere a neo-Gothic interior on top of the Rococo, plus stables, granaries, an arched bridge with obelisks, a picturesque park with ponds, and even a small neo-Gothic power station. Several fires damaged the building, in 1840, 1905 and 1963, but the medieval tower and walls survived. Today Koluvere stands as one of the finest castle complexes in Estonia, the layered evidence of bishops, Baltic German barons, an unfortunate young duchess, Russian generals, Soviet welfare workers, and now post-independence preservation, all visible in the same building if you know how to read the stones.