Long before it was royal, the castle was the work of George Sinclair, 4th Earl of Caithness, who built it between 1566 and 1572 on land that had belonged to the Bishops of Caithness. It began as a Z-plan tower house of three storeys, with a projecting wing at the southeast and a square tower at the northwest - a defensive design built for a hard country and a harder family. The Sinclairs were one of Scotland's most fractious clans, and Caithness was their stronghold. Through the 17th and 18th centuries the castle was extended several times. In 1821 the architect William Burn refashioned it in Tudor Gothic style, and somewhere in that period the name changed too: it became Barrogill Castle, the Sinclair name set aside. Captain Frederic Bouhier Imbert-Terry bought it in 1929, and during the Second World War it served as an officers' rest home. By 1950, when the estate farms were sold off, only the tower itself was still habitable.

The Queen Mother bought Barrogill in 1952, the year King George VI died, and immediately set about restoring it. She stripped away the 19th-century additions she didn't like, reinstated the original name - Castle of Mey - and brought in, for the first time in the building's history, electricity and running water. Work in 1953-1954 made the castle weathertight and habitable. The interior was refurbished over the following years, and the west wing restoration was not completed until 1960. The result was a working country house, not a museum: walled gardens against the salt wind, herds of Aberdeen Angus and flocks of Cheviot sheep on the home farm, and a family retreat where the Queen Mother could be, if not quite a private citizen, then as close to one as she ever got.

Knowing the castle would outlive her, the Queen Mother established the Castle of Mey Trust by a Deed of Trust on 11 June 1996. The Trust's then-president was the Prince of Wales, now King Charles III, and its mandate was sweepingly practical: secure the building, advance historical and architectural education, develop the native breeds of Aberdeen Angus and Cheviot sheep, and benefit the local community. After 2002, when the Queen Mother died, the Trust opened the castle for five months each summer to generate the revenue that would keep it going. Nearly 30,000 visitors came in 2018. King Charles III and Queen Camilla still stay here for ten days at the turn of July and August - the castle closes to the public during their visit, a small annual privacy in a life that has very little.

In May 2019, Charles - then still Prince of Wales - opened a new building on the grounds called Granary Lodge: a ten-bedroom bed and breakfast carved out of the old stables and granary. The project used local materials, local craftsmen, and what the planners called eco-heating, and its profits go back into maintaining the castle. The tearoom too has found a public role: in January 2023 it opened on Tuesdays as a warming centre for locals during the cold months, an initiative of the King's Foundation. Mey is also a piece of pop culture now - the Queen Mother's purchase featured in the first season of Netflix's The Crown, episode eight, "Pride & Joy" - but the castle itself, set against the wind off the Pentland Firth with Orkney on the horizon, remains stubbornly itself.