The Mowbrays, a Norman family, built the first tower house at Barnbougle in the 13th century. They were also lords of Dalmeny and Inverkeithing, with land all along the southern shore of the Forth. Sir John Mowbray was knighted at the Battle of Homildon Hill in 1402. By 1586 the laird, John Mowbray, was a partisan of Mary, Queen of Scots; his wife was a sister of William Kirkcaldy of Grange, and two of their daughters served the imprisoned queen in England. The Penicuik Jewels - a small but precious set of pieces now held by the National Museum of Scotland - are traditionally thought to be Mary's gift to Gillis Mowbray. In March 1597 the Links of Barnbougle were the site of a state-sanctioned judicial combat: James Carmichael fought Adam Bruntfield in single combat to settle a quarrel over a duel-killing the previous December. Bruntfield killed Carmichael, by some accounts in front of five thousand spectators.

The Mowbrays sold Barnbougle in 1615 to Sir Thomas Hamilton, soon to be Earl of Haddington. Hamilton's grandson sold it in 1662 to Sir Archibald Primrose of Carrington, a senior judge who became Lord Justice General of Scotland in 1676. His son, also Archibald, was created Earl of Rosebery in 1703, and Barnbougle became the Rosebery seat. The tower was rebuilt or replaced at some point in the 17th century. By the early 19th century it was a romantic ruin, and the family had moved into the new Dalmeny House nearby. Then in 1881 came the great rebuilding under Archibald Primrose, 5th Earl of Rosebery, who would briefly serve as British Prime Minister from March 1894 to June 1895.

Rosebery rebuilt Barnbougle specifically to house his private library. He was a serious historian, author of Napoleon: The Last Phase (1900), and he had assembled one of the finest Napoleonic collections in Britain - many of the books and manuscripts later passed to the National Library of Scotland. The castle's purpose-built gallery hall doubled as a speech-practice room, where Rosebery rehearsed the polished oratory that made him one of the most admired speakers of the late Victorian era. He retained his interest in horses too: as Prime Minister he won the Derby twice in office (1894 and 1895) - a feat no other PM has matched. Barnbougle was where he came to think. Today it remains a private property of the Rosebery family, a Category A listed building, used for the family's purposes rather than open to general tourism.

William Fyfe collected the death-bugle legend in 1851 and wrote a romantic poem around it about Sir Roger Mowbray, who took his hound on Crusade to the Holy Land and died there. His ghost - and the hound's - were said to return to the Forth shore whenever a Barnbougle laird was about to die: 'And ever when Barnbougle's lords / Are parting this scene below / Come hound and ghost to this haunted coast / With death notes winding slow.' Other versions of the legend replace the bugle with the baying of the hound itself. Hound Point sits just east of the castle, a small promontory now overshadowed by the Hound Point oil terminal where tankers load Forties crude from the North Sea pipeline. The death-hound, if he still walks, walks past a different kind of industrial monument now. Whether the castle's lords still hear him is, fortunately for them, untested.