Trakai Island Castle was built in three distinct phases. In the second half of the 14th century, Grand Duke Kestutis ordered construction on the largest of the three Lake Galve islands, moving his main residence and treasury here from the older Trakai Peninsula Castle nearby. The Teutonic Knights raided and damaged it in 1377. After Kestutis was assassinated in 1382, his son Vytautas and his cousin Jogaila fought a power struggle over the title of Grand Duke that included repeated sieges of the castle by both sides. After their reconciliation, Vytautas began the second phase that ran until 1409. During the truce with the Teutonic Order, the construction work was apparently supervised by the Order's own stonemason Radike - four years before the Battle of Grunwald, where Vytautas and Jogaila would crush the Teutonic Knights together.

The second phase added two wings and a six-story keep, 35 meters high, on the southern side. The keep had movable gates separating it from the rest of the castle and contained a chapel and the Ducal living quarters. It connected to a multi-story Ducal Palace built around an inner yard with wooden galleries running along the inside walls - so household staff could reach support facilities without crossing the great hall. The Ducal Hall in the southern wing was about 10 by 21 meters. Only the Upper Castle in the Vilnius Castle Complex was bigger. The principal material was the so-called red Gothic brick that defines the castle's appearance, with stone reserved for foundations and upper sections. Glazed roof tiles, stained glass windows, and decorative brickwork gave it a Gothic profile with Romanesque echoes.

After the Battle of Grunwald in 1410 broke the Teutonic Order's power, the castle's military function began to wane. Vytautas converted it into a residence and added new frescoes, fragments of which have survived. Foreign emissaries were received in the Ducal Palace; the Flemish traveler Guillebert de Lannoy described it admiringly in 1414. In 1430, Vytautas the Great died inside the castle without ever having been crowned King of Lithuania - the crown sent by Holy Roman Emperor Sigismund had been intercepted by Polish nobles who feared an independent Lithuanian monarchy. Jogaila is recorded as having visited the castle thirteen times between 1413 and 1430. Sigismund Augustus later redecorated it in Renaissance style and used it briefly as a royal summer residence. The Lithuanian Metrica - the official archive of the Grand Duchy - was kept here until 1511. After that, the castle became a prison, and during the 17th-century wars with Muscovy it was damaged badly and never rebuilt. It fell, slowly, into ruin.

By the 19th century, the castle was a romantic ruin - painted by Józef Marszewski, photographed in the 1870s as a half-collapsed shell on its quiet island. The Imperial Archaeological Commission documented the remains in 1888; partial restoration began in 1905. German specialists tried during World War I, and Lithuanian and Polish preservationists worked between 1935 and 1941 strengthening the Ducal Palace walls and rebuilding the southeastern tower before the war stopped them. The major reconstruction project began in 1946, with active construction from 1951 to 1952, finishing in 1961. The work proceeded in the face of objections from Soviet authorities in Moscow, who initially considered restoring a feudal castle a glorification of bourgeois nationalism. Lithuanian preservationists pushed back, and the project went forward as a kind of quiet nationalist statement - rebuilding a 15th-century symbol of Lithuanian statehood in an era when Lithuanian statehood did not officially exist. The reconstruction has been criticized in the decades since for over-restoration and for using interpretive license where evidence was thin.

Trakai Island Castle reopened as the Trakai History Museum after the reconstruction completed, and it has been a major Lithuanian tourist attraction ever since. On a clear day, the red-brick towers reflected in Lake Galve are one of the most photographed images in the Baltic. The castle also functions as a national symbol in a country that spent most of the 20th century occupied - first by Tsarist Russia, then briefly independent, then by the Soviet Union, then by Nazi Germany, then by the Soviet Union again. The reconstruction's controversies are real, but so is the role the castle plays in Lithuanian self-conception: an image of the Grand Duchy at its medieval peak, when Lithuania ran from the Baltic almost to the Black Sea, when Vytautas the Great did not need a crown to be a king.