Brick was almost unheard of in 15th-century England. Castles were stone; manors were timber. When Sir Roger Fiennes - newly returned from fighting in France for Henry V - decided to rebuild the Herst manor, he chose bricks because they were exotic, fashionable in Burgundy and Flanders, and an unmistakable signal of his cosmopolitan ambitions. He died in 1449 before the work was finished, and his son Sir Richard Fiennes (who became the 7th Baron Dacre through marriage to Joan Dacre) completed it. The result is one of the oldest significant brick buildings standing in England. The walls form a square enclosing four internal courtyards, ringed by a wide moat that still mirrors the rose-pink façade. Domesday records that the manor had existed since at least 1086, when a man named Wilbert held it of one of William the Conqueror's followers; the de Monceux family married into ownership around 1200 and gave the place the second half of its name.

By the 18th century the Fiennes line had spent itself out, the 15th Baron Dacre sold the estate in 1708 to a London lawyer named George Naylor, and the castle drifted from one inheritor to the next. In 1776, Robert Hare-Naylor took the advice of architect Samuel Wyatt and gutted the interior. The roof came off, the floors came out, and a 350-year-old castle was deliberately reduced to a picturesque ruin to suit Georgian taste for romantic landscape views. Antiquarian Thomas Lennard, 17th Baron Dacre, was so dismayed that he commissioned the artist James Lambert of Lewes to record the building before it was dismantled. For nearly 140 years afterwards Herstmonceux sat roofless and overgrown, ivy climbing the chimneys and brambles filling what had been the great hall. Then in 1913 Colonel Claude Lowther bought it and began the long restoration that Sir Paul Latham completed in 1933, under the architect Walter Godfrey. Nikolaus Pevsner later judged Godfrey's work "exemplary."

In 1946 the castle and its parkland were sold to the British Admiralty for an unusual purpose: the Royal Observatory was leaving Greenwich. London's lights and smoke had grown too thick for serious astronomy, and a dark rural site was needed. Over the next decade the green fields beside the castle filled with white domes - the so-called Equatorial Group of telescopes, designated A through F - and finally the towering empty silo built for the 100-inch Isaac Newton Telescope. The Royal Greenwich Observatory operated here from 1957 until 1988, when it moved on to Cambridge. The Newton itself had already departed for clearer skies on La Palma in the Canaries during the 1970s, but the domes remain. They are Grade II* listed now, looked after as the Observatory Science Centre, and on a still autumn night you can still walk among them - the Thompson refractor, the Yapp reflector, the Hewitt camera - while the castle's pink walls catch the last of the light beyond the trees.

In 1992 the chemistry pioneer Alfred Bader - an alumnus of Queen's University in Kingston, Ontario - heard that Herstmonceux was empty. He offered to buy it for his wife Isabel. She declined, joking there were "too many rooms to clean." Bader bought it anyway in 1993 and gifted it to Queen's University. After intensive renovations the Bader International Study Centre opened in 1994, hosting Canadian undergraduates studying everything from international law to archaeology in the castle's restored halls. The name became Bader College in 2022. Structural problems closed it in late 2023; the building's twenty-first-century life is, like most of its lives, an act of patient restoration, with reopening planned for 2026.

In September 2025, the Royal Horticultural Society announced that Herstmonceux's collection of hardy water lilies had been recognised as Britain's National Collection of Nymphaea. The lilies live in pools fed by the same springs that fill the moat, opening their pale and rose-coloured cups each summer in front of brickwork roughly contemporary with the Hundred Years' War. The castle also rents itself out for weddings, hosts archery and falconry displays in the grounds, and has stood in over the years for the Narnian palace in the BBC's Silver Chair, for Hogwarts in a Coca-Cola sweepstake, and for Tudor London in the Amazon series My Lady Jane. Six centuries on, the pink bricks of Sussex are doing what they were always meant to do: catching the eye, and outlasting almost everything else.