Long before there was a Dufftown there was Mortlach, an ancient parish where Pictish settlement is visible in the archaeological record. Around 566 AD, St Moluag is said to have established the first Christian church here, on the site of the present Mortlach Parish Church on the southern edge of the modern town. Balvenie Castle, dating from the 13th century, is the second oldest building in the area, restored by John Comyn, Earl of Buchan, and inhabited until 1746 when it was abandoned after the Jacobite Rebellion. The town itself came much later. The 4th Earl of Fife laid it out in 1817 as a planned settlement to help develop the estate and provide housing and employment for soldiers returning from the Napoleonic Wars. The four-road plan converging at the clock tower is his.

The whisky boast has a folk-rhyme origin that includes Keith and Tomintoul along with Glen Grant in the count, depending on which version you hear. What is certain is that Dufftown itself produces more malt whisky than any other town in Scotland, and that the active distilleries within or adjoining the burgh are six: Balvenie, Dufftown (the eponymous distillery), Glendullan, Glenfiddich, Kininvie, and Mortlach. Three former distilleries — Convalmore, Parkmore, and Pittyvaich — sit silent. Glenfiddich is the largest. Mortlach was the first, established in 1823 as the only legal distillery in Dufftown until Glenfiddich was founded in 1887. The whisky museum and heritage centre is small but central, currently undergoing restoration. The Spirit of Speyside festival every spring turns the town into the centre of malt whisky tourism in Scotland.

Built in 1839, the Dufftown Clock Tower stands square and three storeys high with a crenellated roof at the meeting of the four main roads. It was originally a prison. It is now a tourist information centre, which is a fairly precise compression of Scottish small-town history: lock-up turned wayfinding station. The town contains a wealth of Category B and C listed Victorian buildings — the former Clydesdale Bank at 1 Balvenie Street built in 1880, the Police Station and House on York Road from 1897, the B-listed Maravale House at 68 Fife Street from the mid 19th century. The Dufftown War Memorial on Balvenie Street is a bronze gothic cross on a granite ashlar pillar with a small garden of remembrance. It commemorates the men this small town lost in two world wars.

In the third Harry Potter film, Prisoner of Azkaban, Hermione Granger reads aloud from the Daily Prophet that Sirius Black has been spotted near Dufftown. The line is throwaway, but it situates Hogwarts somewhere in this stretch of Scottish Highland country. Dufftown took the joke seriously enough that visitors occasionally ask after the school. The closest real institution is Mortlach Primary School, built in 1902 as the parish school. For secondary education, children attend Speyside High School in Aberlour, about 16 km north. Dufftown was also the birthplace of George Stephen, 1st Baron Mount Stephen, one of the founders of the Canadian Pacific Railway — a Speyside whisky town that exported, along with its malts, a railway baron who built the line that opened the Canadian west.

The mainline railway reached Dufftown in 1862 when the Keith and Dufftown Railway opened a station here. The line carried passengers until 1968 and goods until around 1991. In 2003, the Keith and Dufftown Railway Association reopened the station and the line as a preserved railway, with headquarters at the original station. Today you can ride a heritage diesel through some of the most beautiful countryside in Scotland, between the seven-stills town and the burgh of Keith, past hills and pasture and the occasional pagoda kiln. It is not a fast journey — the line was not built for haste — but it is one of those journeys where slowness is the point.