The cottage was never grand, and it was never meant to be. It was built to house the four government coxswains and their crews, the men who rowed officials and supplies around the young colony's harbour, and they lived here from 1816 until 1845. The building takes its name from John Cadman, a coxswain who occupied it in the 1840s, though by then it had already served a generation of boat crews before him. Its form is plain and honest: a rectangular sandstone block with classical Georgian touches, a plinth and pilasters that hint at architectural ambition above its humble purpose. There are no surviving plans, no record of who designed it or what it cost to build, which has tempted historians to credit the colony's celebrated convict architect Francis Greenway. The evidence is thin. The cottage keeps its own counsel about its origins, as old buildings often do.

Few small buildings have worn so many lives. After the coxswains came the Sydney Water Police, who made the cottage their headquarters from 1845 to 1864, patrolling a harbour thick with shipping and the trouble that follows it. Then it became a Sailor's Home, from 1865 to 1970, a refuge of sorts for the mariners who passed through one of the busiest ports in the southern hemisphere. Around it, the world it had been built to serve was disappearing. In the 1880s the wharves of Circular Quay expanded; the docks and foreshore were filled and built over, severing the cottage's old connection to the water. By 1962 it stood vacant and failing, and in 1964 the Overseas Passenger Terminal rose nearby and blotted out its view of the harbour entirely. A building that had begun at the edge of the sea was now hemmed in by the modern city, forgotten and falling down.

Cadmans Cottage was rescued in the nick of time. In 1970 the Sydney Cove Redevelopment Authority took control of The Rocks, and two years later the cottage was proclaimed a historic site under the National Parks and Wildlife Act, beginning a restoration that arrested decades of decay. In 1988, as Sydney prepared to mark the bicentennial of European settlement, archaeologists dug into the ground around the cottage and into the building itself. What they found mattered precisely because so little had been written down. The reclaimed land out front preserved traces of the Sailor's Home era that exist nowhere in the documents. Inside, old drainage and the scars on the western and northern walls recorded how people had used these rooms across more than a century and a half. For a building with almost no paper trail, the fabric itself became the archive, every layer of stone and sediment a sentence in a story no one had bothered to record.

There is something quietly remarkable about a structure this small surviving in a place this valuable. The Rocks sits on some of the most coveted real estate in Australia, and for most of the twentieth century old buildings here were demolished without ceremony to make way for the new. Cadmans Cottage endured, and today it serves as a gateway into the harbour's history, having housed the Sydney Harbour National Park information centre and welcomed visitors curious about the city's beginnings. To stand before it is to perform a small act of imagination: to subtract the towers and the cruise terminal and the hundred metres of made land, and to picture the water sliding up to the threshold while a coxswain readied his boat. The cottage was added to the New South Wales State Heritage Register in 1999. It earned the place the slow way, by simply refusing to fall.