When the architect Benjamin Backhouse drew up Fernberg in 1865, Paddington was barely a suburb at all. The hilltop sat in rough bushland on the edge of settlement, and a sale notice from 1877 marvelled that 'go where you will the house is seen towering aloft above every tenement in the neighborhood.' Backhouse was a busy colonial architect, his name attached to villas across Queensland and New South Wales, and he gave Heussler a brick-and-stone house several storeys tall with broad verandahs built for shade and breeze. But a grand house demands a grand income. Heussler could not sustain it. The property changed hands repeatedly, owned at times by men who never bothered to live in it at all, a fine address waiting for someone who could afford its ambitions.

That someone arrived in the 1880s. John Stevenson, a pastoralist and member of the Legislative Assembly, bought Fernberg flush with the confidence of Queensland's boom years and hired the architect Richard Gailey to remake it. Gailey more than doubled the house, turning Backhouse's 1860s villa into a full Victorian Italianate mansion, complete with a prominent tower, faceted bays and a new entrance facing north. Stevenson's wealth bought stables, an aviary, a glasshouse, even a tennis court. Then the 1890s depression arrived and swept his fortune away. By 1895 the property was mortgaged. The mansion that boom money had built now stood as a monument to how quickly that money could vanish.

Government House came to Fernberg almost by accident. In 1909 Queensland handed its original vice-regal residence to the brand-new University of Queensland, leaving the governor without a home. Plans for a purpose-built Government House at Victoria Park got no further than the footings. So in 1910 the government leased Fernberg as a stopgap, and in 1911 bought it outright, despite complaints that the house was too small. The 'temporary' arrangement never ended. Wing by wing the place grew to fit its new role: formal gardens for vice-regal garden parties in 1910, a major eastern wing in 1937, administration buildings, staff cottages and the slow accretion of a working official residence around the old merchant's villa.

Step into the central foyer and you meet the house's strangest flourish. A life-size figure of Robert the Bruce, the medieval Scottish king, glows down from a large stained-glass window above an intricately carved timber staircase. Around the ground floor are the formal reception and dining rooms and the governor's study, some still furnished with pieces carried over from the old Government House. The grounds matter as much as the rooms. Of the 15 hectares, only about a quarter is formal garden; the rest is genuine remnant eucalypt bushland threaded with paths and gullies. In 1928 Governor Sir John Goodwin and Lady Goodwin cut 'woodland walks' through it, planting jacarandas and wattles among the gums.

More than thirty governors have lived here, and the house quietly records them all. The early arrivals were Scottish, Irish and English appointees of the Crown. Sir John Lavarack, who took up residence in 1946, was the first Australian-born governor. The list turns another corner in 1992 with Leneen Forde, Queensland's first woman in the role, and again in 2003 with Dame Quentin Bryce, who went on to become Australia's first female governor-general. Fernberg has survived university handovers, depressions, royal tours and the 1992 widening of a road that took a hectare of its grounds. The merchant who named it for a distant mountain is long forgotten by most. His name, and his house, endure.