The name Sai Kung means something close to 'market' in the classical sense — the characters 西貢 can be read as a place where goods were exchanged, though the etymology has been debated. Oddly, historian Professor David Faure found no record of a market at Sai Kung in either the Kangxi edition (1688) or the Jiaqing edition (1819) of the Qing dynasty's Xin'an County Chronicles. The settlement may have developed gradually from fishing anchorages rather than from a planned market. The name first appeared in Western publications in the early 1900s, describing it then only as 'the village of Sai Kung.' Whatever its precise origins, the town had crystallized into a recognizable market centre by the time the New Territories were ceded to Hong Kong in 1898, serving as a hub for the surrounding coastal villages and the small islands of Port Shelter.

Sai Kung Town's most significant period of expansion came not from commerce or development planning but from infrastructure. When the High Island Reservoir project began in 1971, it displaced villagers and fishermen from the more remote corners of the Sai Kung Peninsula — people whose communities had been settled for generations in places that now either sat beneath the reservoir or were cut off by the new dam roads. Many of them were relocated to Sai Kung Town. The Hong Kong government funded new residential and commercial development in the town centre, followed by the Tui Min Hoi district — the name means 'over the harbour' — built on land opposite the original settlement. Tui Min Hoi Chuen, completed between 1984 and 1986, was the first rural public housing estate developed by Hong Kong Housing Society. The relocation changed the town's scale without entirely altering its texture: it remained, at its core, a place of fishing families and seafood trade.

Before Hong Kong International Airport moved from Kai Tak in Kowloon to Chek Lap Kok on Lantau Island in 1998, Sai Kung was a favoured residential address for the international airline crews and airport employees who needed to be near Kowloon without living in its density. The town acquired a cluster of Western pubs, restaurants, and small shops that persist to this day — a trace of that era visible in the mix of Cantonese seafood restaurants and Irish-style bars that share the same narrow streets. The expat community shrank after Kai Tak closed, but the cafes and Western-facing businesses stayed, absorbed into the town's broader identity as a weekend destination for Hong Kong residents in search of something that feels slightly unhurried.

The harbour at Sai Kung Hoi was a fishing harbour; today it functions as a typhoon shelter, and the motorized junks moored there are mostly hired for day trips rather than working the nets. Boat tours run to nearby islands and to the isolated coastal villages of Sai Kung Hoi that have no road access — a reminder that the town still serves as a departure point as much as a destination. The seafood restaurants are the town's most famous feature, and they operate with an unceremonious directness: tanks of live fish and shellfish line the pavement in front of each restaurant, customers point at what they want, and the kitchen takes it from there. The town has no MTR link. Buses and minibuses connect it to Hang Hau, Mong Kok, and Choi Hung stations, though on busy weekends the queues can be long. This minor inconvenience is, for regular visitors, half the charm.