Matheson's relationship with the island he had bought was complicated. He spent freely on improvements that genuinely helped: roads, harbour works, drainage of moorland, a chemical plant in Stornoway, employment relief during the potato famine of the late 1840s. He also presided over a period in which significant numbers of islanders were assisted in emigrating to Canada, in arrangements that historians have argued about ever since. Were they voluntary departures aided by a generous laird, or were they pressured clearances dressed up in the language of charity? Some of both, almost certainly. What is not in dispute is that the castle was built with money squeezed out of a drug trade that devastated millions of Chinese lives, and that the social conditions on Lewis during the building years were close to subsistence for many of the people whose horizon now included these new battlements. The garden grounds that surround the castle were planted with non-native trees, an attempted forest on a treeless island, supported by imported soil hauled up the hill by the cartload.

The Matheson family held the castle and the island for more than seventy years. In 1918 they sold the Lews Estate to William Hesketh Lever, Lord Leverhulme, the Lancashire soap magnate whose Sunlight Soap fortune had made him the kind of industrialist who could buy an entire island and consider it a manageable project. Leverhulme had vast plans for Lewis. He was going to industrialise fisheries, scale up the Harris Tweed industry, build new harbours, and even, on his more grandiose days, draft timetables for railways. His vision collided almost immediately with the post-war demand for land resettlement by returning servicemen, and through the early 1920s the conflict between Leverhulme and the land raiders consumed both sides. By 1923 he had had enough of Lewis. He gave the castle, with its grounds, to the people of Stornoway parish. He turned his attention to Harris. In 1925 he died, and his Hebridean ventures collapsed.

During the Second World War, the Royal Navy requisitioned the castle. 700 Naval Air Squadron operated a detachment of six Supermarine Walrus amphibious aircraft from a slipway at Cuddy Point in the grounds. Aircrews slept in rooms once occupied by Matheson's dinner guests; the base was called HMS Mentor. After the war the building served as student accommodation for Lews Castle College through the 1950s. When that closed, the castle was left empty. For several decades it sat above Stornoway with boarded windows and unheated rooms, gradually decaying, its baronial silhouette still recognisable but the interior slowly forgotten. The grounds remained a public park, popular with dog-walkers and joggers, even as the castle itself slumped.

On 22 November 2011 the Heritage Lottery Fund awarded 4.6 million pounds to convert Lews Castle into a bilingual museum and cultural centre. The work took years. In 2016 the ground floor reopened to the public: a restored ballroom, a cafe, and the new home of Museum nan Eilean, the Western Isles museum. Six of the surviving Lewis chessmen, the twelfth-century walrus-ivory figures found in a sandbank near Uig in 1831, are on permanent display there, a homecoming of sorts after more than a century in London and Edinburgh. In 2017 the luxury holiday operator Natural Retreats opened apartments in the upper floors. The building is a Category A listed building and owned by Comhairle nan Eilean Siar, the Western Isles council. The castle still stands above the harbour. Its history is what it is. The island it overlooks chose, in the end, to keep it and tell that history in its own languages.