Charles and Emma had been searching for somewhere about 20 miles from London with railway access. They had considered Windsor, and come close to buying a place near Chobham. Downe (then spelled Down) sat in a parish in Kent - it would not become part of the London Borough of Bromley until 1965. Charles wrote to his sister on the Sunday after their first visit. The house, sitting on rather high table-land, had somewhat of a desolate air, but he liked the footpaths: almost every field is intersected by one or more foot-paths - I never saw so many walks in any other country. That detail mattered more than he yet knew. The footpaths would become his thinking-paths. And Emma gave birth to Mary Eleanor on 23 September 1842, nine days after she moved in. The family was getting on well in the new house. But on 16 October, baby Mary died. The Darwins had lost their second child in the place that was meant to be their fresh start.

Darwin made extensive alterations from the beginning. In 1843 he added a large angled bay forming a bow front, extending up through all three storeys of the west elevation - the laying of the first brick was on 27 March that year. By April, work was underway to lower the lane outside by as much as two feet and build new flint boundary walls, so the east garden became more private. A strip of field became a kitchen garden, and later the experimental plots and greenhouses where Darwin would spend decades on his quiet, patient experiments. Between 1845 and 1846 he altered the service wing, adding a butler's pantry and a schoolroom. The Darwins' children kept arriving in the house: Etty in 1843, George in 1845, Bessy in 1847, Francis in 1848, Leonard in 1850, Horace in 1851, and their tenth child Charles Waring in 1856. Then came their hardest loss. Their daughter Anne (Annie), born before the move, died in 1851 at the age of ten of an unspecified illness, and the grief broke something in Darwin that never quite reset. Charles Waring, born in 1856, died in 1858.

In 1846, Darwin rented from his neighbour Sir John William Lubbock a narrow strip of 1.5 acres adjoining the Down House grounds to the southwest. He named it the Sandwalk Wood. One side was shaded by an old shaw with oak trees; the other looked over a hedge to a charming valley. Darwin had a variety of trees planted and ordered a gravel path - the sandwalk - to be created around the perimeter. He walked this loop every day, several circuits at a time, for exercise and for uninterrupted thinking. He set up a number of small stones at one point on the walk so he could kick a stone to the side each time he passed, so he didn't have to interrupt his thoughts by consciously counting. The sandwalk also served as a playground for his children. A wooden summerhouse stood at the lower end. The theory of evolution by natural selection was not invented on this gravel circuit - he had conceived it in London before the move - but it was tested, refined, defended against objections, and built into On the Origin of Species, here, one walked loop at a time. In 1872 Darwin added a verandah to the west side of the drawing room. In 1877 a new billiard room appeared on the east, converted in 1881 into a new study. Darwin spent 16 years growing sundews and other carnivorous plants in his greenhouse, feeding them proteins like roast beef and boiled egg - work that became Insectivorous Plants (1875).

Darwin died in 1882, Emma in 1896. From 1907, the headmistress Olive Willis and her friend Alice Carver established a girls' boarding school in the house - starting with five mistresses and one girl, growing to around 60 pupils. The novelist Elizabeth Bowen was a pupil during the First World War. Returning in the 1930s, she wrote with characteristic Bowen acidity: I have never liked scientific people very much, and it mortifies me to think of them trampling reverently around there on visiting days, thinking of Charles Darwin and ignorant of my own youth. The school moved out in 1922. A second girls' school followed briefly. In 1927 the surgeon Sir George Buckston Browne bought the house from the Darwin heirs for £4,250, spent about £10,000 on repairs, and presented it to the British Association together with a £20,000 endowment to preserve it in perpetuity as a memorial to Darwin. It opened to the public on 7 June 1929. English Heritage took over in 1996. The master bedroom was restored and opened to the public for the first time in June 2016, complete with reconstructions of the Darwins' collection of fine art prints, including Biblical scenes by Raphael, Titian, and Sebastiano del Piombo, and a Leonardo da Vinci self-portrait in Darwin's study together with portraits of Charles Lyell and Erasmus Darwin. The sandwalk is still there. You can walk it. Darwin's small stones are not, but the path is.