The 1840s were the years Britain stopped exiling its prisoners to Australia and started using them at home. Portland was chosen for the new arrangement because it sat on a mountain of fine limestone next to a strategic anchorage that the Admiralty wanted to fortify. The Admiralty Quarries opened above the prison, and convict labour became the engine of one of the largest civil engineering projects of the Victorian age - the breakwaters of Portland Harbour. The men in the quarries were not a faceless workforce. They were mostly young, mostly poor, transported under sentences that often had little proportion to their crimes: petty theft, vagrancy, a stolen coat. Conditions broke many of them. Deaths from injury and disease were common enough that the prison would later be cited in calls for penal reform across the United Kingdom.

From the moment the prison opened, residents discovered that other people wanted to see it. Homeowners in the new village of The Grove opened upstairs rooms as cafes, selling tea and a view of men at work in the quarries below. The convicts became, in the vocabulary of the day, an attraction. They were ordinary people whose worst moments had been made into someone else's afternoon outing. By 1869 the government formalised what tourism had already assumed: Portland would not be a temporary prison after all. Residents petitioned against permanence and lost. Between 1870 and 1872 the convicts built St Peter's Church just beyond the walls - now Grade II* listed and redundant, a worship space made by men forbidden from leaving.

In 1921 the institution became a Borstal, training school for adolescent offenders, and a different generation took up the rebuilding. Between 1931 and 1935 the Borstal boys transformed an abandoned quarry behind St Peter's Church into a sports stadium. The first sports day was held on 1 August 1936. The community came to watch, a more sympathetic kind of looking than the cafes had offered a century before. Then the war arrived. On 15 August 1940, a German air raid struck the Borstal's Rodney House block. Four boys were killed. Others were severely injured; five admitted to hospital. They had been sent to Portland to be reformed, and instead were caught in a war the rest of the country was still adjusting to.

By the late twentieth century, Portland's reputation had hardened in a different direction. Inspection reports across the 1990s and 2000s described conditions that the country had supposedly left behind: foul toilets, filthy showers, rats in the food service area, and inmates without working sanitation still slopping out - emptying buckets through cell windows. A 2000 inspection led to a siege when twenty-six prisoners barricaded themselves in for eight hours. In 2009 the former footballer Ian Wright filmed Football Behind Bars at Portland, organising twenty-four serious young offenders into a football academy and trying to argue with the cameras that these were boys, not categories. The prison became an adult and young-offender establishment in 2011 and a designated resettlement prison in 2013, charged with preparing inmates for life outside.

The retired officers John Hutton, Steve Ashford and Chris Hunt opened the Grove Prison Museum in March 2014, in the former deputy governor's house across the road from the main gate. They had been collecting memorabilia for two decades. Over a thousand visitors came in the first year - a different kind of curiosity than the Grove cafes once sold, with at least some attempt to understand what happened here. The boundary wall along Grove Road, Grade II listed since 1993, still encloses the village. Alma Terrace, the row of warders' houses built in 1854, still stands. The Grove Lime Kiln, derelict and unloved, sits 320 metres from St Peter's Church, one of the last vestiges of lime production on the island. All of it - the harbour, the church, the walls, the village itself - was made by men whose names mostly nobody remembers.