The barony of Yester was granted in the twelfth century by King William the Lion to Hugo de Giffard, a Norman immigrant placed in East Lothian to help anglicise the kingdom. The original stone keep was built before 1267, and tradition attributes it to the famous Sir Hugo - the founder's grandson - who served as a guardian of the young King Alexander III of Scotland. Hugo was, by all medieval accounts, a respected nobleman; he was also, by all medieval accounts, a sorcerer. Walter Bower's fifteenth-century Scotichronicon records the local belief: "Old tales tell that his castle, or at least his cellar and keep, were wrought by witchcraft, for there is there a marvellous underground cavern wonderfully constructed and extending under a large area of ground. It is popularly called Bo' Hall." Sir Walter Scott picked up the story five centuries later in Marmion, identifying Hugo as "the founder of the Goblin-Hall."

Beneath the surface ruins of Yester Castle survives the Goblin Hall itself - a Gothic-vaulted underground chamber some thirteen metres long, ribbed with the kind of pointed stone arches you would expect in a small abbey rather than buried under a hilltop. It is the only complete structure left on the site. The vault is built from massive blocks of carefully cut stone, drained by channels, and accessed by a narrow stair that descends into total darkness. Standing inside it is a peculiar experience: cool, silent, oddly church-like, with the ribs of the vault throwing strange shadows in a head-torch's beam. It does not feel like a cellar. It feels exactly like the sort of place a medieval audience would describe as wrought by witchcraft. In 2021 the chamber had to be closed to the public after thieves stole stone from the supporting walls of the vaults - an incident reported by the BBC and the East Lothian Courier as part of a broader rise in heritage theft.

Sir Hugo had a daughter, Margaret, who married into the Broun of Colstoun family. According to the legend, Hugo gave her a hand-picked pear as a wedding gift, telling her that if anything happened to the fruit, disaster would follow for her descendants. The pear was put in a silver casket and kept untouched for centuries; the Brouns prospered. In 1692, on her wedding night, the fiancee of Sir George Broun took the pear from its casket. It looked perfect, freshly plucked. She could not resist taking a bite. The misfortune began immediately. Sir George accumulated crushing gambling debts and was forced to sell the estate to his brother Robert. Robert and his two sons were drowned shortly afterwards in a flash flood when the River Tyne burst its banks. Sir George died in 1718 without a male heir. The pear, the family say, turned to stone after the bite. It is still at Colstoun House today, the bitemark still visible. It is one of the strangest family heirlooms in Scotland.

Beyond the sorcery, Yester accumulated the routine catastrophes of a Lowland Scottish castle. In 1298 men from the Yester garrison fought at the Battle of Falkirk against Edward I, and Yester briefly passed to the English crown. In 1357 the Giffard male line died out and the castle entered the Hay family through Joanna Giffard's marriage to Sir William de la Haye. The 2nd Lord Hay was killed at Flodden in 1513 in the disaster that took most of Scotland's nobility. The castle was burned by the English in 1544 during the Rough Wooing - Henry VIII's brutal campaign to force the young Mary Queen of Scots into marriage with his son. It was captured again in 1547 and 1548. By 1557 the Hays had abandoned the castle for a new tower house on the site of present-day Yester House, and the medieval castle slowly disappeared, quarried for building stone. But the Goblin Hall, deep underground, was beyond the reach of casual scavenging. The Marquess of Tweeddale's falconer was still living in it as a tenant in 1737 - perhaps the last person to call Sir Hugo's vault home.