When Henry Brougham Loch moved into Bemahague in 1863, Castletown was still the island's official capital, but Douglas was booming and the new Lieutenant Governor wanted to live near the action. The old farmhouse was never quite right. In 1871, Tynwald formalised the lease for £200 a year and asked the Daly family to chip in £1,000 toward improvements, then summoned the Liverpool architect Gustavus Hamilton to draw up alterations. While he worked through nine months of reshaping, the Governor camped out at Bishopscourt. Twenty years later, the local builder James Cowle returned to break a wall between the drawing room and dining room, fit folding doors, and rebuild the kitchen wing as a two-storey extension. Each round of work tried to make a farmhouse behave like a residence of state.

By 1900, Tynwald was spending £200 a year just keeping the place in repair, on top of the rent. The arithmetic stopped making sense. In July 1903 a committee proposed buying the house and 112 acres around it for £12,000, and on 24 November that year John Joseph Heywood Daly signed it over. The island now owned its Governor's residence outright. Even then the building wasn't considered satisfactory. Between 1903 and 1906, the Liverpool partnership of William Edward Willink and Philip Coldwell Thicknesse, whose other work included the Cunard Building on the Liverpool waterfront, pulled down the south western end of the house, enlarged the reception rooms and study, built a new hall, porch and main staircase, demolished the stables, and finally connected the place to a main water supply instead of hand pumps drawing from two old wells.

Inside, successive Governors' wives left fingerprints on the rooms. Lady Hill installed the Paris crystal chandeliers that still hang in the reception rooms in the late 1920s. In 1945, Lady Granville, sister of Queen Elizabeth, embroidered a bedspread and satin hangings with the Royal Coat of Arms for the Tynwald room ahead of a visit by King George VI and her sister, the future Queen Mother. Lady Garvey added embroidery to the soft furnishings in the 1960s. A fire in 1914 destroyed the oldest part of the house, with its cramped rooms and low ceilings; the rebuild gave staff better quarters and finally erased the last of the Christian farmhouse. The coat of arms above the entrance is thought to have come from the old Custom House in Peel.

In the entrance hall hangs the bell of HMS Manxman, a Royal Navy minelayer that served through the Second World War. Its inscription points further back than the ship itself: "This bell was given by the people of the Isle of Man in memory of one of its greatest sailors, Captain John Quilliam R.N. who served in HMS Victory at the Battle of Trafalgar, 1805." Quilliam, a Manxman who rose from press-ganged seaman to first lieutenant aboard Nelson's flagship, is remembered each time someone passes the bell on the way to a state dinner. The Lordship of Mann itself has been vested in the Crown since 1765, when London bought out the Duke of Atholl with the Act of Revestment to choke off smuggling that was draining their tax receipts. The political tangle that produced the residence is part of what walks through the door with every guest.

Today, ambassadors, schoolchildren, and Manx residents all pass through Government House for receptions, ceremonies, and lunches. The hill it sits on still looks down toward Douglas Bay, where the ferries arrive and the TT racers thunder past on the A18 a few miles away. The building has been remade so many times that few outside lines still match the 1820s farmhouse beneath it, but the address has stayed steady through nearly thirty Lieutenant Governors. The official residence is now also the island's quiet inventory of its own history: a Liverpool architect's grand staircase, a Trafalgar bell, a sister-of-a-queen's embroidery, all looking out over Governor's Road.