The single largest historical fact about Ampthill is no longer visible. Ampthill Castle, built in the early fifteenth century by Sir John Cornwall (later Lord Fanhope) from the ransoms he had collected as a captain at Agincourt, became one of Henry VIII's favourite hunting lodges in the 1520s. The deer park around it suited him. The castle suited him less. In 1531, Henry banished his first wife Catherine of Aragon to Ampthill, where she lived for two years while he pressed his case for annulment through Cardinal Wolsey, Thomas Cranmer, and finally a clean break with Rome. It was at Ampthill in May 1533 that Catherine received the news that Cranmer's ecclesiastical court at Dunstable Priory had declared her marriage to Henry void from the beginning. She refused to accept the ruling. She continued, until her death three years later, to sign herself the queen of England. The castle was demolished sometime in the seventeenth century. In the 1770s, John Fitzpatrick, the 2nd Earl of Upper Ossory, erected a stone cross on its site with an inscription written by Horace Walpole: "In days of old here Ampthill's towers were seen / The mournful refuge of an injured queen."

Ampthill Great Park - the deer park around the lost castle - was redesigned in the mid-eighteenth century by Lancelot "Capability" Brown, the most influential English landscape architect of the period. He took the formal seventeenth-century park and softened it into Brown's signature rolling pastoral idiom: scattered clumps of trees, a sweeping prospect, a serpentine line where formal alleys had been. When the Earl of Upper Ossory died in 1818, the park passed to Lord Holland, whose Holland House in Kensington became the most famous Whig political salon of the early nineteenth century - the salon at which Macaulay, Sydney Smith, Byron, and a generation of Reform-era politicians argued and ate. Lady Holland planted an avenue of trees at Ampthill that she called the Alameda, inspired by the Almeida promenade in Madrid. The Alameda still runs through the southern part of the park. Lord Upper Ossory had also commissioned a set of thatched cottages on Woburn Street between 1812 and 1816 to house estate workers; they are still occupied. The town also has the unusual feature of a llama on its bowls club emblem - a tribute to founder Sir Anthony Wingfield, who kept a small private zoo on the edge of Ampthill in the early twentieth century.

Ampthill turns up in unexpected places. Richard Nicolls, born in Ampthill in 1624, was the English officer who in 1664 accepted the surrender of the Dutch town of New Amsterdam on behalf of the Duke of York; the city was renamed New York after the Duke, and Nicolls became its first English governor. He died eight years later when a Dutch cannonball struck him at the Battle of Solebay, and the cannonball itself is still preserved on his tomb in St Andrew's Church in the town centre. In 1979, the author and illustrator Kit Williams published Masquerade, a puzzle book whose solution led readers to a buried golden hare. The hare was buried at the foot of Catherine of Aragon's memorial cross in Ampthill Great Park, at the precise spot touched by the tip of the cross's shadow at noon on the spring or autumn equinox. Hundreds of treasure-hunters dug holes across the park before the puzzle was solved in 1982. And from 1968, the British TV producer Brian Clemens lived at Park Farm in Ampthill; an episode of The Avengers ("Noon Doomsday," featuring Steed and Mrs Peel under a high noon countdown) was filmed in the town in July 1968. Ben Chilwell - the Chelsea and England footballer - was raised here. So was Lewis Ludlow of Gloucester Rugby. Sir Albert Richardson, perhaps the leading classical architect of his generation, lived at Avenue House on Church Street from 1919 until his death in 1964 and is buried in the town.