The first owner of the house, around 1514, was probably Sir William Compton. His title was Groom of the Stool to Henry VIII, which sounds quaint until you understand what it actually meant: Compton attended the king at his close-stool, the royal toilet. This was not a humiliation. It was a position of extreme intimacy and trust, reserved for the king's closest confidants, and it gave Compton access and influence that grander courtiers couldn't reach. He used some of that access to acquire the manor of Tottenham, including the Lordship House (as Bruce Castle was then known), and to begin shaping it as a country seat. The house that survives today is mostly red brick with ashlar quoining, its principal façade flanked by symmetrical bays with tall paned windows. The detached round tower, separately listed, is among the earliest substantial uses of brick as the primary material for an English house. Compton was building when brick was still the new technology.

By the early 17th century the house had passed to Richard Sackville, 3rd Earl of Dorset, married to Lady Anne Clifford - an heiress and diarist whose own long battle to claim her inheritance is one of the great stories of the Jacobean court. Sackville gambled. He spent extravagantly. He ran up debts that would eventually swallow much of his estate, and he leased the Lordship House to Thomas Peniston, whose wife Martha (daughter of Sir Thomas Temple) was rumoured to be Sackville's mistress. The arrangement was, by Stuart standards, almost respectable. Eventually Sackville's creditors won, and the house was sold to Hugh Hare, a Norfolk landowner with inherited wealth from his great-uncle Sir Nicholas Hare, Master of the Rolls. Hare bought the manor in 1625, became 1st Baron Coleraine shortly after, and started a dynasty that would shape the house for the next century.

Henry Hare, the second Baron Coleraine, was the one who renamed the place Bruce Castle around the time of his major 1684 remodelling. Much of the south façade dates from then - the heightened end bays, the rebuilt central porch with stone quoins and pilasters, the balustraded top with its small tower and cupola. Hare's romantic life was complicated. He had been engaged in 1661 to Sarah Alston, who instead married John Seymour, the 4th Duke of Somerset. There is evidence the two maintained a close relationship through both their marriages, and when Constantia Hare and the Duke of Somerset were both dead, Henry and Sarah - now the dowager Duchess - married each other. The 1684 remodelling followed. Constantia's ghost, supposedly walking the south wing, didn't appear in print until the Tottenham & Edmonton Advertiser ran the story in 1858, almost two hundred years after her death. The legend has since faded; there have been no recent sightings. Whether Constantia ever walked at all is unclear. What's clear is that Victorian Tottenham wanted her to.

In the early 19th century Bruce Castle became something far stranger than a country house: a progressive boarding school run by Rowland Hill, the future postal reformer. Hill's school operated on the principle that the teacher's job was to instill the desire to learn, not to impart facts. Corporal punishment was abolished. Alleged transgressions were tried by a court of the pupils themselves. The curriculum, radical for the early Victorian period, included foreign languages, science, and engineering. The school attracted the sons of London-based diplomats - particularly from the newly independent South American nations - and the sons of the computing pioneer Charles Babbage. Hill went on, in 1840, to design the penny post: a flat one-penny rate for any letter to anywhere in Britain, paid by the sender with an adhesive stamp. The Royal Mail collection at Bruce Castle Museum exists today on the strength of that connection. The man who taught children without canes also reinvented how an empire communicated.

By the late 19th century Tottenham had transformed. The 1840 railway to Stratford and the 1872 Great Eastern line into Liverpool Street, with subsidised workmen's fares to Bruce Grove, had turned the old farmland into a major rail-served working-class suburb. The grounds of Bruce Castle were among the last undeveloped patches. In 1892 they became Tottenham's first public park. The house opened to the public as Bruce Castle Museum in 1906, was Grade I listed in 1949, and has since accumulated the archives of the London Borough of Haringey, the regimental museum of the Middlesex Regiment (later transferred to the National Army Museum), and a significant collection of paintings by Beatrice Offor (1864-1920), the Edwardian symbolist who lived nearby. Tottenham Hotspur's full club crest depicts the building alongside seven trees and two lions - an unlikely heraldic combination of a Tudor manor, a Saxon legend, and a Northumberland-Percy reference, all proudly representing N17. The Bruces never lived here, but everyone else seems to have.