The lands of Midhope passed through the Graham, Livingstone, and Martin families before Sir James Hamilton of Finnart picked up half of them in 1537. John Martin of Midhope, the laird in 1438, is thought to have built the original tower house; his widow Christian held the right to keep occupying it under a contract with the Livingstones. By the late 16th century the castle had come into the hands of Alexander Drummond, brother of Robert Drummond of Carnock, Master of Work to the Crown of Scotland. A stone over the entrance is carved with AD 1582 MB, commemorating Alexander and his wife Marjory Bruce, who was a daughter of Robert Bruce of Airth. The Drummonds extended the building. They added a painted ceiling, cinquefoils and gold stars on a blue background, that probably celebrated Sir Robert Drummond's marriage to a Hamilton heiress in 1619. Fragments of that ceiling survive in the care of Historic Environment Scotland.

Alexander Drummond was a servant of the Earl of Huntly, who considered him a cousin. On 7 August 1573, on Huntly's behalf, Drummond rode to Holyrood Palace and handed over jewels belonging to Mary, Queen of Scots, to Regent Morton. By then Mary had been imprisoned in England for five years, having fled Scotland after defeat at the Battle of Langside. Her jewels had been used by her supporters as collateral, and now they were being returned to the crown that her own son would soon inherit. Drummond did not record his feelings about the errand. He went home to Midhope, lived out his life, and was buried at Airth Old Kirk. The carved AD 1582 stone above the door of the tower is the most direct trace he left.

In 1678 Midhope was modernised. The entrance tower was removed, the eastern extension was heightened and lengthened, a new doorway was cut, and a small courtyard was laid out to the south. The five-storey tower house with its irregular outline took on something closer to the symmetrical chateau silhouette you see today. A large two-chambered dovecot was built about 140 metres to the south-east; it is still standing. Then the castle drifted. The Hopetoun estate absorbed it. Tenants moved in and out. Repairs lapsed. The roof failed. By the late 20th century Midhope was a roofless shell with empty windows, listed as Category A but visited mostly by sheep and people studying Scottish vernacular architecture.

Starz first aired Outlander in 2014. The production scouted Scotland for a Highland family home that felt both lived-in and slightly fortified. They found Midhope. The exterior shots became Lallybroch, also known as Broch Tuarach in the books, the home of Jamie Fraser. The interior scenes were filmed elsewhere, but the cobbled approach, the worn stone walls, the windows looking out over the Hopetoun fields, those are all Midhope. Visitors started arriving. The estate now opens the grounds for a fee. The castle itself remains a roofless shell. You cannot go inside. But you can stand where the cameras stood, and look at the same stones that the Drummonds raised, the same masonry where Alexander Drummond saddled up to ride to Holyrood with the Queen's jewels in 1573. Most castles have one famous resident. This one has two: a real Renaissance courtier and a fictional 18th-century laird.