The bishops of Winchester were among the wealthiest landowners in medieval England, and Farnham was one of their most lucrative manors. Henry de Blois built Farnham Castle in 1138 to give him a place to sleep on the road between Winchester and Westminster. The castle became more than a stopover; it became the bishops' country residence for the next seven centuries, until 1927. The keep is set on a motte that may go back further than 1138 - some elements suggest an earlier ringwork. Three miles west, at Barley Pound, there are the remains of an 11th-century precursor castle. Henry VIII is thought to have spent part of his childhood at Farnham Castle as a ward of Bishop Foxe. The Blind Bishop's Steps - the steps that lead up Castle Street to the gates - were originally built for Foxe, whose failing eyesight made the climb difficult.

Farnham declared for Parliament early in the Civil War. The castle, considered a Royalist rallying point, was given a Roundhead garrison in 1642 - but the garrison commander Captain George Wither, himself a noted poet, decided to evacuate. John Denham, also a noted poet and the new High Sheriff of Surrey, then occupied the empty castle with 100 Royalists. Colonel Sir William Waller arrived from London, blew the castle gates with a petard, and retook it with the loss of one man. The lead from the Town Hall roof was requisitioned to make bullets. On 20 December 1648, King Charles I was taken under military escort through Farnham on his way from the Isle of Wight to his London trial. He lodged that night at Culver Hall (now Vernon House) on West Street and gave his nightcap to its owner Henry Vernon "as a token of Royal favour." Less than a month later, the king was beheaded at Whitehall. Vernon House is now Farnham Library.

William Cobbett was born in a Farnham pub called the Jolly Farmer on 9 March 1763 - the son of a small farmer who taught him to read by writing letters in the dirt. Cobbett grew into one of the most original political voices of his century: a soldier, then a radical journalist, then a Member of Parliament, then a writer of agricultural manuals and Sunday-paper polemics. His Rural Rides, published as a book in 1830, recorded his journeys on horseback through the English countryside in the 1820s and 1830s; the rides were partly reportage on agricultural distress, partly autobiography, partly an unfolding argument about what England was doing to its own countrymen. The Jolly Farmer still stands and is now called the William Cobbett. A plaque marks the room in which he was born. Other Farnham-born writers include the hymn-writer Augustus Toplady, born in 1740, who wrote Rock of Ages.

Daniel Defoe wrote that Farnham had the greatest corn market in England after London. He counted 1,100 fully laden wagons coming into town on a single market day. Hops were grown around Farnham from the early 17th century until the 1970s; the brewing industry built up around them, with the Farnham United Breweries occupying what is now Farnham Maltings. When Courage tried to sell the maltings off for redevelopment in the 1960s, the town raised the money to buy the buildings back and converted them into a community arts centre. Farnham Pottery at Wrecclesham produced "Farnham Greenware" - a distinctive Arts and Crafts ware with a soft green glaze - in close cooperation with Farnham School of Art, which opened in 1866. The pottery closed in 1998 but the school survives as a campus of the University for the Creative Arts.

J. M. Barrie spent the summers of 1901 and 1902 at Black Lake Cottage, a remote woodland retreat just outside Farnham near Tilford. There, with his neighbours the Llewelyn Davies boys, he played pirates and Indians in the woods around the lake and began working out the story that would become Peter Pan, first performed on the London stage in 1904. Other Farnham residents have included the watercolour artist William Herbert Allen, master of Farnham Art School from 1889 to 1927; the illustrator Pauline Baynes, who painted the maps and cover art for C. S. Lewis and J. R. R. Tolkien; the Formula One driver Mike Hawthorn, who was raised in Farnham and lived above the family garage on the West Street; and the England rugby captain Jonny Wilkinson, born in Frimley in 1979 and educated in the town. Nearby Bourne Wood has become a regular film location - the opening battle of Gladiator was filmed there, as were scenes for Thor: The Dark World and Wonder Woman.

About a mile south of Farnham lies the ruin of Waverley Abbey - the first Cistercian abbey in England, founded in 1128 by William Giffard, Bishop of Winchester. The monks of Waverley kept a year-by-year chronicle called the Annals of Waverley, an important primary source for early medieval England. King John visited in 1208. Henry III visited in 1225. The abbey was dissolved by Henry VIII in 1536 with only thirteen monks left in the community. The English Reformation paid for itself in part with the dismantled stone and lead of buildings like these. The ruins are open to visitors today, free of charge, set in a meadow beside the Wey. The North Downs Way long-distance footpath starts at Farnham and runs east 153 miles to Dover. Farnham makes a good starting point. It has, after all, been a stopover on the road to somewhere else for a very long time.