Grafton had locked people up since 1862, first in a makeshift lockup, then in a flood-prone, overcrowded replacement that proved unfit almost from the start. The permanent prison came only in 1893. In the early 1890s the design of public buildings was thrown open to competition rather than handed to the Government Architect, and a Sydney architect named Henry Austin Wilshire won the commission. He gave Grafton a square brick compound with that single elaborate gatehouse for staff, visitors and prisoners alike, separate ranges for men and women, and a sterile zone between the cells and the outer wall. The Holloway Brothers built it; the gaol was proclaimed on 8 September 1893, and prisoners moved in that November. The materials, brick with sandstone trim and terracotta tiles, were the confident vocabulary of Federation Australia. The intentions inside would prove far darker than the facade.

In 1942 the Department of Corrective Services gave Grafton a particular role: it became the gaol for "intractables," the prisoners deemed too violent or disobedient for ordinary jails. What that designation meant in practice stayed hidden for a generation. Prisoners arrived to a "reception biff" - a beating administered by three or four officers wielding rubber batons, sometimes before the handcuffs were even removed. The man would be taken into a yard, ordered to strip, searched, and then assaulted. "Biff" was a deliberately mild word for what happened. The violence continued throughout a sentence, triggered by any breach of the prison's written or unwritten rules. These were real men, many of them dangerous, but the cruelty done to them was illegal, deliberate, and sanctioned from above.

By the 1970s the rumours had reached Parliament and the press, and the questions could no longer be deflected. The Royal Commission into New South Wales Prisons, led by Justice John Nagle and running from 1976 to 1978, finally laid the system bare. Nagle found that every officer who served at Grafton during its years as a gaol for intractables must have known of the brutal regime, and that most, if not all, had taken part in it. More damning still, he found that the Department and ministers from both major parties had unofficially sanctioned the systematic brutalisation of prisoners. Accepting the Nagle Report in 1978, Neville Wran's Labor government began prison reform under Dr Tony Vinson. A building cannot apologise, but the report it provoked changed how a nation treated the people it imprisoned.

The gaol's later history is quieter but not painless. Grafton was abolished as a full prison by proclamation on 18 December 1991 and reopened as a periodic detention centre the following May. In 2011 its women's wing closed; in 2012 it was downgraded to hold court-bound inmates from the Northern Rivers. It shut for good in August 2019, its role inherited by the vast new Clarence Correctional Centre nearby, opened in July 2020. The original male cell range survives largely intact, and the complex has been heritage-listed on the New South Wales State Heritage Register since 1999 - preserved not as a monument to punishment, but as evidence. Stand before that elegant gatehouse today and you are looking at a reminder of how easily cruelty can hide behind good brickwork.