Castle Menzies is a 16th-century Z-plan castle, the architectural step between the brute defensive towers of the medieval Highlands and the lightly-defended chateaux that came after. The walls are of random rubble, originally harled with roughcast lime. The quoins, turrets, and door and window surrounds are of finely carved blue freestone from a quarry on the south side of Loch Tay. The Menzies chiefs held it for over 500 years. A marriage stone above the original entrance, installed by James Menzies in 1371, records his marriage to Barbara Stewart, daughter of John Stewart, 3rd Earl of Atholl. The castle was the seat of the clan, strategically placed at a Highland crossroads where roads from Aberfeldy and Tummel Valley converged.

Duleep Singh had been deposed at age ten when the British annexed the Punjab in 1849, after winning the Second Anglo-Sikh War. The boy was separated from his mother, converted to Christianity, and sent to live with Sir John and Lady Login. In 1854 the family was moved to Britain itself. By 1855 they had leased Castle Menzies, and the maharajah lived at the castle for three years until 1858. He took to shooting on the estate, learned to manage Scottish weather, and developed the taste for shooting parties and country house life that would define the rest of his ambivalent existence in exile. He never returned to the Punjab. He died in Paris in 1893, having spent much of his adult life trying to reclaim something that the empire he was supposed to belong to would never give back. The young man who hunted across the Menzies grounds in the 1850s left no obvious trace at the castle, but it was for three years his closest thing to home.

On Sunday 21 April 1878 at around 7pm, a fire broke out in the roof of Castle Menzies. A man passing along the road below spotted the smoke and raised the alarm. Sir Robert Menzies, 7th Baronet, and Lady Menzies were away at Farleyer; word was sent, and they returned immediately. Local efforts kept the fire under control until the fire engine arrived from Taymouth Castle around 10pm, by which point the worst was over. The damage to the building was repaired, but the shock proved too much for Lady Menzies. She died the following month. The 19th-century newspaper accounts treated this as cause and effect, the excitement of the fire affecting her health, in the way Victorian medicine often did with women's sudden deaths. Whatever the actual cause, the fire and the death are recorded together in the castle's history, a Sunday evening that took both the roof and the lady of the house.

Sir Neil Menzies, 8th Baronet, inherited the estate in 1903 and died without heirs in 1910. The house contents were auctioned in 1913. The papers went to Sotheby's in 1914 for 1,160 pounds, including a letter of Mary Queen of Scots dated 31 August 1566 that fetched 300 pounds and a 1647 document signed by Charles I that brought 56. The castle and 11,600 acres were sold in 1918 to Francis Willey, later Baron Barnby, for 69,000 pounds. After his death the widow put it up again in 1930. By the 1950s the castle was nearly derelict. The Menzies Clan Society stepped in after 1957 and began the long restoration. A 40,000-pound project starting in 1972, partly funded by a 10,000-pound grant from the Historic Buildings Council, ran for nearly a decade under Dr Bill Dewar MBE. A further 135,900-pound grant in 1991 funded the final 230,000-pound phase. The castle is now open to the public, a category A listed building since 1971, with the Menzies Stone weighing 115 kilograms sitting outside for lifters to attempt. Round and smooth-textured, it has defeated most who have tried.