The building dates from 1570. The traditional story attaches it to George Gordon, 1st Marquess of Huntly, of Clan Gordon, but the Royal Commission on the Ancient and Historical Monuments of Scotland considers that connection erroneous. What is well established is that in 1647 the Incorporation of Hammermen, the guild of metalworkers, blacksmiths, goldsmiths, and other smiths working in Edinburgh, bought the property as their new headquarters. They hired the architect Robert Mylne to extend it. Mylne was the King's Master Mason, the same man whose work produced the Netherbow Wellhead just up the road outside John Knox House. The Hammermen kept the building for centuries. By the early 20th century it was decaying, and in 1924 the city stepped in and bought it as the building was facing destruction.

One room in the museum is given over to an original copy of the National Covenant, signed at Greyfriars Kirk in February 1638. The Covenant was a public declaration by Scottish Presbyterians refusing to accept the religious innovations of King Charles I, who was trying to impose an English-style prayer book on the Scottish church. The document was first signed on a flat tomb in Greyfriars churchyard. Copies were made and circulated through Scotland, gathering thousands more signatures. The Covenanters' resistance helped trigger the Bishops' Wars, which helped trigger the English Civil War, which ended with Charles I beheaded at Whitehall in 1649. To see one of the original signed copies is to look at a piece of paper that helped take down a king. A different gallery in the building reconstructs Field Marshal Earl Haig's First World War headquarters on the Western Front, with items Haig himself bequeathed to the museum. Two wars, three centuries apart, in the same building.

The museum's main job is to hold the decorative art of Edinburgh, and there is a lot of it. Cut and engraved Edinburgh glass. Hallmarked silver from Edinburgh and the Canongate. Costume. Longcase clocks. Scottish pottery and Scottish porcelain dating from the 1760s, the era when Scotland was developing its own ceramic tradition. The collections show what people who lived in Edinburgh actually owned and used: the ordinary objects that fill households become extraordinary when 250 years pass and you can still see them. The original plans for James Craig's New Town, drawn up in 1767 when the city was about to break out of its medieval ridge and spread north across the drained Nor Loch, are kept here. Among the most prized objects are pieces of Mary Queen of Scots-era silver, the kind of thing James Mosman down the road at John Knox House would have known how to make.

The museum was used as a filming location in Season 3 of Outlander, which is part of why it sometimes shows up on tour itineraries that have nothing to do with museums. Maintained by the City of Edinburgh Council, free to enter, and rarely as crowded as the Royal Mile attractions it sits among, it remains one of the easier ways to slow down and look at Edinburgh from the inside. The locals still call it Huntly House, despite the official name change a hundred years ago. The yellow paint gets refreshed. The Covenant stays under glass. The clocks keep being wound. From the air the Canongate runs in a long bright line from St Giles Cathedral down to Holyrood Palace, and somewhere in the middle of it, a bright yellow building stands out from the grey because it always has.