The site has been occupied since the early 16th century, when an ironmaster and wool merchant named Richard Streatfeild built a timber-framed house here. In 1679 Henry Streatfeild rebuilt the structure in red brick in the Restoration style and named it High Street House - it then faced the village high street directly. The Streatfeilds remained in residence for nearly four centuries. In the early 19th century another Henry Streatfeild commissioned William Atkinson to rebuild the house in the Gothic Revival style; Atkinson's design was not completed, and in 1835 the next Henry Streatfeild engaged the architect Henry Kendal to finish the work. The house was renamed Chiddingstone Castle. The road through the village was diverted to give the new castle its grounds. The family stopped living there in 1900 but kept the freehold until 1938, when they sold to Lord Astor.

Lord Astor never moved in. The Second World War brought members of the Canadian Forces to the castle as part of the substantial Canadian military presence in southeast England preparing for D-Day. After the war the building was used as Long Dene School, an independent boarding school, from the late 1940s until 1954. Then in 1955 Denys Eyre Bower bought the castle to house his collections. Bower had been born in Crich, Derbyshire in 1905. After working as a bank clerk he had taken over the Cavendish Hood antiques business on Baker Street in London in 1943. The redevelopment of Baker Street displaced him; Chiddingstone offered space, country quiet and the possibility of opening collections to the public.

In 1957, two years after he moved in, Bower shot his mistress at Chiddingstone Castle and then attempted to take his own life. Both survived. He was convicted of attempted murder and attempted suicide and sentenced to life imprisonment. The case was not straightforward. A solicitor named Ruth Eldridge took up Bower's cause, working over years to argue that the conviction had been a miscarriage of justice. In 1962 Bower was released. He returned to the castle and, with Eldridge and her sister Mary supporting him, continued to open it to visitors and to display his collections. He never returned to a fully public life, but he was no longer hidden. He lived at Chiddingstone for fifteen more years and died there in 1977.

Bower's Japanese collection focuses on lacquerware - considered among the most important in any private hands - and includes Japanese swords, samurai armour, and haniwa terracotta figures. His Ancient Egyptian holdings cover the full span of the civilisation, from grave goods such as ushabti figures and amulets to vessels for food and drink; in 2013 items toured to the Houston Museum of Natural Science. His Stuart and Jacobite collection reflects his membership of the Royal Stuart Society and his interest in the lost Catholic kings of England - portraits, swords, objects carrying hidden Jacobite symbols, royal manuscripts and a substantial library of antique books. His Buddhist collection - he was a Buddhist himself - is deliberately ecumenical, drawing on multiple schools and countries, with thangkas and Buddha images on display alongside personal objects.

Bower left the castle and his collections to the nation in 1977. The Denys Eyre Bower Bequest, the trust that took over the property, continues to care for both, opening the castle to visitors and providing a venue for weddings. Current trustees include descendants of the Streatfeild family - a small piece of historical continuity stitching the modern museum back to its 19th-century occupants. The castle is Grade II* listed and sits in 35 acres of grounds, themselves on the National Register of Historic Parks and Gardens. The 19th-century kitchen still has its three cake ovens; servants' quarters are preserved. The lake, 3.5 hectares, divides the village from the castle grounds and held the British record for the largest bream caught for 37 years from 1945. Bower's strange, hospitable, idiosyncratic project survives him.