The castle was built for a Norse king. In 1265 - the year the last Norse king of Mann died at the castle - construction began on a great limestone keep; the Treaty of Perth, which transferred the Hebrides and Mann from Norway to Scotland, followed in 1266. The castle was fortified and added to by successive rulers across the next three centuries. Robert the Bruce besieged and captured Castle Rushen three times during the Scottish wars. The Stanley earls of Derby, who ruled Mann as Lords from 1405 to 1736, lived here. So did the Dukes of Atholl, who succeeded them. The castle has been a fortress, a residence for the Kings and Lords of Mann, a mint, and a prison - past prisoners include a Bishop of Sodor and Man and two newspaper editors. Today it is run by Manx National Heritage and opens to visitors between Easter and October. The keep is one of the best-preserved medieval castles in the British Isles, partly because Mann's relative quiet through the wars of religion and the Commonwealth meant no one ever bothered to slight it.

Tynwald, the Manx parliament, traces itself to the Viking era and claims to be the oldest continuous parliamentary body in the world. Its lower house, the House of Keys, met in Castletown for centuries - first inside Castle Rushen, then in the Old House of Keys on Parliament Square, where the twenty-four Manx members debated under a green dome from 1821 onwards. In 1874 the House of Keys moved north to Douglas, following population and commerce. The Old House of Keys building stood unused until the National Westminster Bank gave it to Castletown's town commissioners in 1973 to serve as a town hall. After the commissioners moved into a purpose-built civic centre in 1989, Manx National Heritage took the building over and reopened it as a museum in 2000, restored as it would have looked on its last working day. Visitors today can sit in the members' seats and watch a recreated debate.

George Quayle (1751-1835) was a Castletown inventor, politician, banker - and, by the persistent local tradition, a smuggler. The Manx position between England and Ireland, with its own customs regime, made smuggling a substantial part of the island's economy in the eighteenth century, and Quayle's harbourside house on Bridge Street was built accordingly. It contains secret passages, hidden doorways, and a replica captain's cabin tucked into its upper floors. Next door, at Bridge House, he ran his own bank - George Quayle & Co., known locally as Quayle's Bank. Opening the bank's safe involved dropping a series of cannonballs through a mechanism Quayle invented himself. In a cellar at the back of the house, bricked up and forgotten for decades, his eighteenth-century yacht the Peggy was rediscovered by workmen in 1935. The Peggy is now in Douglas being conserved, but the house remains as the Nautical Museum, perhaps the most charming small museum in the British Isles.

On the road east out of town, toward Derbyhaven, sits a low grass-covered mound called Hango Hill. The name comes from the Old Norse hanga-haugr, meaning 'gallows hill', and the mound may itself be a Bronze Age burial site - a bronze flat axe was found here. What Hango Hill is best known for is the execution of Illiam Dhone - 'Brown William' - on 2 January 1663. Born William Christian in 1608, Illiam Dhone had been a major Manx figure during the English Civil War, charged with the island's defence by the Stanley lords. When Parliamentary forces approached in 1651, he overthrew the Royalists and surrendered Mann bloodlessly to the Commonwealth - an act some called nationalism, some called treason. After the Restoration, the Stanleys had him tried for high treason and shot. The execution was botched; Dhone died of his injuries. He remains a contested figure in Manx memory - to some a traitor, to others a patriot who saved his island from a worse fate. He is commemorated annually on the anniversary of his death.

Castletown is built almost entirely of local grey limestone, quarried from Scarlett Point at the south-western tip of Castletown Bay. The colour of the town in winter light is unmistakable - a soft, slightly silver grey that holds the rain. The medieval street plan still organises everything. Houses cluster around Market Square, the harbour, and the old military parade ground. The crofts - small enclosed garden plots once attached to each town house - give their name to a residential district. Population peaked at around 3,100 in the early 2000s and has held roughly steady since. Castletown was always smaller than Peel and Ramsey, much smaller than Douglas; it was the capital because of strategy, not size. With the capital function gone, the town settled into a comfortable third or fourth place on the island, dominated by its castle, its harbour, and the daily aircraft from Liverpool descending toward Ronaldsway two kilometres up the road.