The old harbour at Portsoy was built in the 17th century, which makes it the oldest on the Moray Firth coast. Sir Walter Ogilvie of Boyne Castle held the burgh of barony charter that allowed it. The new harbour beside it was built in 1825 for the growing herring fishery, which at its peak put fifty-seven boats to sea from this single small village. The whole arrangement is wrapped in stone houses three and four stories high, with steep little streets running down to the water and the Salmon Bothy at the head of the new harbour, now a small museum and community centre open Friday through Monday two until four. The herring fleet is long gone. The harbour walls still hold the way they did in the seventeenth century.

Portsoy marble is the local name for serpentinite, a green and white metamorphic rock that outcrops along this stretch of coast. It is technically not marble at all; the chemistry is magnesium iron silicate hydroxide, in the formula (Mg, Fe)3Si2O5(OH)4, which is the geologist's equivalent of insisting on a long full name in a pub. Whatever it is called, the stone polishes beautifully into a green almost serpentine pattern, and local craftspeople have been making jewellery from it for centuries. Tradition says some of the Portsoy stone made its way to Versailles in the seventeenth century to be used in the palace's chimneypieces. The claim is hard to verify but locally believed. The jewellery is still on sale along Seafield Street.

Jimmy MacBeath was born in Portsoy in 1894 and is buried here. He spent his working life as a wandering singer, tramping the roads of north-east Scotland with no fixed address, singing the bothy ballads of the farm workers and the old shepherding songs at any market or fair that would pay him. Hamish Henderson recorded him in the 1950s for the School of Scottish Studies, and his voice is now in archives that preserve the sound of a Scotland that has almost entirely vanished. Other Portsoy names made bigger noise. William Boyd became a pathologist and wrote medical textbooks used worldwide. Eoin Jess played football for Aberdeen and Scotland. Jim Paterson is the trombonist of Dexys Midnight Runners, which means a small Aberdeenshire fishing village has a credit on Come On Eileen.

The Scottish Traditional Boat Festival started in 1993 to mark the three-hundredth anniversary of the old harbour, and it has run every year since on the last weekend of June. Wooden fishing boats from all along the Scottish coast sail in and tie up at the old harbour, their crews swap stories and the harbour fills with shanty singers and pipe bands. The 2025 festival ran 27 to 29 June. Surrounding the festival, smaller pleasures persist. Portsoy Ice Cream on Seafield Street, daily from half past seven in the morning until eight in the evening. Hook Line and Sinker for fish and chips. The Co-op for everything else. Cullen Skink, the smoked-haddock-and-potato soup invented in the next village west, is on offer in most cafes and pubs; Samuel Johnson, on his 1773 Highland tour with Boswell, loathed it. Most modern visitors disagree.