In Homer's time, Mytilene was not one place but two. A channel 700 meters long and 30 meters wide separated the main city from a small island, and triremes 34 meters long passed through it under marble bridges that arched across the curve. The Greek word euripus referred to this kind of strait, a working watercourse running through the heart of an organized city since 1054 BC. Over the centuries, the channel silted and filled. By the 16th century at the latest it was simply gone. One of the original marble bridges still survives, buried under a modern building, like an old riverbed under pavement. The castle hill sat between the two harbors, and what is now the Ermou shopping street follows roughly the line where ships once passed.

Lesbos chose the wrong side in the Mithridatic Wars and paid for it. In 88 BC, after the island sided with Mithridates against Rome, the Romans destroyed Mytilene and absorbed what may have been a Temple of Apollo into the foundations of whatever came next. Pompey, a few decades later, restored the city's autonomy. The reason had a name: Theophanes of Mytilene, a 1st-century historian and intellectual who befriended Pompey and, according to Plutarch, won freedoms for his city by personal influence alone. Bronze coins commemorating Theophanes have been excavated both inside the castle and elsewhere in town, his face passed hand to hand for generations as small change. Vespasian revoked the autonomy in 70 AD. Hadrian later restored it. The castle hill kept watching.

In 1373, the island came into Genoese hands when Francesco I Gattilusio, a Genoese adventurer who had married into the Byzantine imperial family, became its lord. He rebuilt the upper castle as a private residence and stamped it with the symbols of his power. A square stone tower rose at the top, decorated with the family arms of the Gattilusi and the Palaiologoi, the imperial dynasty he had married into. It is still called the Queen's Tower today. For nearly a century the Gattilusi held Lesbos, until in 1462 the Ottoman fleet bombarded the castle and took it. Sultan Bayezid II repaired the damage in 1501, after the war with Venice, and added two large round artillery towers and a new line of walls. The castle was no longer a private palace. It was a coastal cannon platform pointed at the sea.

Under the Ottomans, Mytilene castle stopped being a single building and became a small walled city. Bekir Pasha modernized the defenses in 1643 to 1644, perhaps preparing for the Cretan War, perhaps repairing earthquake damage. He built a fresh wall in front of the medieval one and dug a deep wide moat in front of that. The interior filled with Ottoman institutions: the Kule Mosque, an Ottoman seminary, a tekke monastery, the madrasa, an imaret kitchen, a Turkish bath, a fountain, the gunpowder store, and the great cistern. Most of these structures still stand. The Orta Kapu, the Ottoman gate, became the main entrance. People lived inside the castle walls as ordinary residents, not soldiers, and continued to do so until shortly after the Second World War.

Greece took Lesbos from the Ottoman Empire on November 8, 1912, during the First Balkan War. Independence came at a cost most stones could not survive. After 1912, the castle was used as a source of building material to construct refugee housing for Greeks displaced from Asia Minor, and the long slow ruin of the place accelerated. Walls were pulled down. Carved blocks vanished into modest houses around the city. The intramural community held on for another generation, families living between Byzantine ramparts and Ottoman seminaries until shortly after the Second World War. Excavations by the Canadian Archaeological Institute have since uncovered Archaic and Classical buildings inside the walls, layers below the layers. Restoration began on the Orta Kapi gate and the cistern in 2000, and summer evenings in the castle now belong to cultural festivals, a thousand years of fortification reduced to a stage for music.