Lida did not stand alone. It was a link in the defensive ring Gediminas drew around his capital at Vilnius, a chain that included Trakai and Senieji Trakai to the west, Kreva and Medininkai to the north, Punia and Kernavė deep inside Lithuanian country. Each fort was meant to slow the Teutonic Knights long enough for the next one to muster, and most of them did exactly that, more than once. Lida itself sits at 141 meters above sea level, on naturally drained ground that the engineers fortified further. Two angle towers rose at the corners of a roughly square plan; the upper stories of both were lived in. A church stood inside the walls until 1533, when it was moved out into the town that had grown up beneath the castle's protection. The town is still called Lida. The original fort gave it its name.

Despite the walls, Lida fell repeatedly. The Teutonic Knights took it in 1384 and again in 1392. In the early fifteenth century, Grand Duke Vytautas handed the castle to an unexpected guest: the deposed Mongol ruler Khan Tokhtamysh, who had been driven out of the Golden Horde by Tamerlane and arrived in Lithuania looking for sanctuary. The chronicles record, almost casually, that Tokhtamysh and his retinue settled in a yurt near the castle. A felt-walled tent of the steppe pitched beside a Baltic brick fortress is the kind of detail that suggests how strange and porous medieval Lithuania actually was. In 1406, the family of Yury of Smolensk was held inside the walls as hostages; Yury's attempt to break in and free them failed. By 1433, Lida had become a flashpoint in the dynastic war between Švitrigaila and his cousin Sigismund Kęstutaitis, two Gediminids fighting over the throne their grandfather had built.

The next centuries were a long inventory of damage. Crimean Tatars sacked Lida in 1506. The Russians stormed it in 1659 during the Russo-Polish War. The Swedes took it twice during the Great Northern War in the early 1700s and blew up both towers as they left. In 1794, the castle grounds saw fighting between Russian troops and the followers of Tadeusz Kościuszko during the doomed insurrection that tried to save what was left of the Polish-Lithuanian Commonwealth. After a major fire swept through the town of Lida in 1891, locals pulled down the southwest tower and parts of the western wall to scavenge stone for rebuilding their houses. Archaeologists from St. Petersburg arrived to stop the dismantling. Most of the damage was already done.

For much of the twentieth century the courtyard inside the surviving walls served as something less serious than fortress: a traveling zoo or a circus would set up there in summer, and every December the town raised a Christmas tree inside the walls. Real restoration finally began in 1982 and continued through 2010, with new red brick deliberately marked off from the original to keep the additions honest. Walls now rise as much as twelve meters again. Each summer, Lida Castle hosts a medieval-style tournament where reenactors in chain mail clatter across ground that once held Tokhtamysh's tent and was burned and stormed by Tatars, Swedes, Russians, and a half-dozen claimants to the Lithuanian crown. A museum is being assembled, slowly, in the rebuilt towers.