Charles Hoare started buying up the Luscombe land in 1788. His sister Henrietta was the widow of Sir Thomas Dyke Acland, 9th Baronet of Killerton, near Exeter, which gave the Hoare family a personal connection to Devon. By 1799 Charles had cleared the existing house and Nash and Repton's commission was under way, with construction beginning in 1800. Repton, characteristically, prepared two sketches: one of a standard symmetrical house, and one of the asymmetrical Picturesque alternative that would settle into the landscape rather than dominate it. Hoare chose the second. The final design drew on Downton Castle in Herefordshire, but with an asymmetrical plan to give panoramic views from every reception room. Nash built it of Portland stone, mostly to two storeys but with the three-storey octagonal tower rising between two wings and a square tower at the porte-cochere. The crenellated parapets, the pinnacles, the turrets, the varying heights, all gave the house the silhouette of a small medieval fortress. Inside, the comfort was thoroughly Georgian.

Through the porte-cochere a visitor enters a circular hall. To the right, the staircase hall. To the left, the dining room, with windows at the far end that open onto the park. Straight ahead is the drawing room, which occupies the ground floor of the octagonal tower, with bookcases on four sides and a white marble chimneypiece designed by the sculptor John Flaxman. The upper sections of the drawing-room windows hold stained glass, and the view through the lower sections runs straight down the valley to the sea. The library is on the second floor, in the north room. Above the drawing room sits a sitting room. Throughout the ground floor hang paintings by Philip James de Loutherbourg, Thomas Lawrence, Alessandro Allori, and Henry Thomson. The servants' quarters were placed in their own wing, less prominent than the principal rooms but with easy access to gardens that ran almost up to the house. Repton's principle was that ground-floor rooms should connect directly to the garden, which meant the hierarchy had to be expressed by architecture rather than by hiding the working life of the house away. The Chapel of St Alban, designed by George Gilbert Scott, was added in 1862 for around 5,000 pounds, despite being too small to hold more than 100 people.

Repton laid out the grounds and John Veitch, founder of the Veitch nursery dynasty at Killerton, actually planted them. The estate extends to 140 hectares, of which 10 hectares are gardens, both formal and informal, with the remainder in parkland and woodland. The main pleasure ground, developed between 1812 and 1814, is called the American Garden, surrounded on two sides by a wall and a ha-ha. It contains oaks that pre-date the house, ornamental shrubs and rhododendrons added from 1890, and a pond from around 1900. At the west end of the American Garden stands a round stone summerhouse with a thatched roof, built originally near the park in 1799 and moved here in 1830. The formal gardens to the east, south, and west replaced the older pleasure grounds in the late nineteenth century. There is a 50-metre lawn east of the house, ending in a stone ha-ha, then gravel walks past the conservatory and the chapel, through rhododendrons, to a series of round gravel areas with a central stone-edged pond. A rose garden, now planted with herbaceous plants, sits up a flight of steps. A rock and water garden, twentieth-century work, completes the circuit back to the house. The whole site is Grade I listed in the National Register of Historic Parks and Gardens.

During the Second World War, Luscombe Castle was used as an evacuation point. A boys' preparatory school was housed in the building, whose pupils included a young William Franklyn, later the British actor known for the Schweppes advertisements. Girls in the care of Barnardo's also lived there. The corridors that had been built for picturesque effect filled with the noise of children sent inland from cities being bombed. After the war the house returned to private use. It is still privately owned today. The park is approached through three gates, each with a nineteenth-century Gothic stone lodge. Luscombe Wood and Summercombe Wood frame the valley with deciduous trees and an eighteenth-century pine plantation. The valley itself slopes down toward Dawlish, two miles south, where the trains of the South Devon Railway still run along the sea wall that Brunel laid out a generation after Nash and Repton finished their castle. Two men collaborated. They never quite collaborated this seamlessly again. The house remains.