Zachariah Button, the wealthy local landowner who commissioned Belmont Castle around 1795, made his money from a chalk quarry next door. The architect was the little-known Thomas Jeffery, recorded in Howard Colvin's Biographical Dictionary of British Architects. Jeffery designed an imitation medieval baronial castle - the kind of fashionable Gothic Revival that swept through late Georgian England as Romanticism replaced Classical austerity. Battlements, towers, oval libraries with fitted bookcases and mouldings, double flights of stone steps descending to terraces. The house had two approaches: a lodge in the Gothic style near the road between West Thurrock and Grays, and another route from the north lawn via the road from Stifford village. Around the house spread extensive pleasure grounds, landscaped with forest trees, shrubs, and plants, terminating on the west at a Gothic temple and on the east with an orchard and paddock. The estate lay over a medieval rural settlement, and several dene holes - the deep flask-shaped chalk pits dug for fertiliser in earlier centuries - were filled in during the landscaping.

During the 1840s the Webb family occupied the property, hosting local cricket matches and a circle of music lovers later mentioned by the pianist and writer Alice Diehl in her memoirs. Edward R. Parker lived at Belmont Castle from 1880 until 1900, serving as chairman of the Orsett Poor Law Union board of governors. Belmont's park was often used for community events. In 1884, between two and three thousand people gathered there for a political rally in support of the Franchise Bill - the legislation that would extend voting rights to most male householders. For a moment, this private estate served as one of the rural meeting points where Victorian democracy was argued into being.

By the 20th century the chalk quarries that had originally funded Belmont were quite literally undermining it. Quarries encroached upon the pleasure grounds, though they were still used for charity fetes into the 1930s. The Dobree family, the last tenants of the entire house, departed in 1930. After that the mansion was divided into individual rented flats and meeting rooms. The remaining land was let to local farmers for grazing cattle, and to the Tilbury Gas Company who used the North Lawn as a sports field and rented part of the mansion to use as changing rooms. The Gothic library, the oval room with its bookcases, was almost certainly broken up by then. The last owner was the Associated Portland Cement Company, who had the mansion demolished in 1943 to make way for yet another chalk quarry.

Wartime Britain, scrambling for industrial materials, was not the moment for sentimental preservation. The cement company needed chalk, and chalk lay directly beneath the mansion. The structure that had presided over a hundred years of cricket, music, political gatherings, and quiet family life vanished into raw material for the wartime construction industry. The only thing that survived was the name. Belmont Castle Academy, a primary school in the area, took the name on, and that is now the only place where the words still appear locally on a building. The Thames still flows past where the four-storey tower once watched it. The dene holes filled in by the Georgian landscapers are now lost twice over, buried under the modern quarry. To anyone passing through Grays today, no trace of the castle remains - but for the children walking into Belmont Castle Academy each morning, who probably think the name is just a name, this is a kind of memorial.