The original Huntingtower, now the Eastern Tower, was built at least by the fourteenth century as a free-standing gatehouse - three storeys plus a garret. Around the end of the fifteenth century the Ruthven family raised a second L-shaped tower beside it, with a deliberate gap between the two. The defensive logic was straightforward. Two towers, separately defensible, with a wooden bridge that could be drawn up the moment one was breached. The space between was finally filled in during the late seventeenth century, which is when the castle took on the joined silhouette visitors see today. The Western Tower's windows were enlarged at the same time, turning a defensive stronghold into something closer to a country house.

On the first floor of the Eastern Tower survives one of Scotland's quiet treasures - the earliest substantially complete Scottish Renaissance painted ceiling. Grotesque animals dance along the main beams, including a Green Man peering out from the gloom. Renaissance knotwork patterns curl across the overlying planks. Fragments of wall paintings remain too, with flowers, animals, and Biblical scenes drifting in and out of legibility. In 1912, the chemist Arthur Pillans Laurie of Heriot-Watt University advised the Office of Works to fix the flaking pigment with a weak gelatine solution. The technique held; the ceiling is still there.

On 22 or 23 August 1582, the fifteen-year-old King James VI walked into a trap at what was then called Ruthven Castle. The 1st Earl of Gowrie and his Protestant associates seized the boy king and held him here, then moved him from place to place for ten months. They wanted to control Scottish policy by controlling the king. The Raid of Ruthven, as it became known, lodged itself in James's memory permanently. He eventually escaped from his captors at St Andrews on 27 June 1583. James pardoned Gowrie at first - even returned for a banquet at the House of Ruthven - but when a second plot emerged, Gowrie was executed and his estates seized.

In 1600, the Gowrie story reached its bleak conclusion. The brothers John and Alexander Ruthven were killed at Gowrie House in Perth by an overwhelming force of the king's armed men, accused - some still say falsely - of trying to kidnap James again. This time the king was not merciful. He seized the estates, abolished the name of Ruthven by Act of Parliament, and ordered that any successors be barred from titles or lands. By royal proclamation, Ruthven Castle was renamed Huntingtower. James VI stayed there himself in April 1601, signing letters with the new name as if testing whether it would stick. It did. Within a generation the old name was barely whispered.

The castle's most famous ghost is Lady Greensleeves, identified in local tradition as Dorothea, daughter of the 1st Earl of Gowrie. The story has her in love with a servant who slept in the Eastern Tower, while she lived in the Western. One night her mother climbed the bridge between the towers to catch her in the act. Trapped, Dorothea fled to the roof and leapt across the gap, landing on the Western battlements and slipping into her own bed just in time to be discovered there. The next day she eloped with the servant and disappears from the record. Sightings of a tall woman in green silk have been reported ever since - usually at dusk, occasionally in broad daylight. The figure is said to be an ill omen. A traveller who saw her in the 1930s reportedly drowned the next day, falling from the ferry across the River Tay.