The word "factory" is misleading. These were not manufacturing sites. The term derived from the Portuguese feitoria — meaning trading post — since Portugal was the first Western nation to establish regular commerce with China. A factory in this context was the combined office, warehouse, and residence of foreign merchants, who were called factors or supercargos. Their hierarchy was meticulous: chief supercargo, second supercargo, and so on, each managing a specific category of trade — tea purchases, silk purchases, sales to the Chinese side. The bookkeepers were called writers. The officers who verified accounts aboard ships were called pursers. The entire apparatus ran according to a set of customs and protocols that had evolved over generations of seasonal trade, governed by the Hoppo, the imperial customs superintendent whose formal title was Canton Sea Customs Minister and whose control over port fees gave him significant personal wealth.

On the Chinese side of the trade, a licensed guild of merchants called the Cohong held a monopoly on transactions with foreign buyers. Established formally in 1760, the Cohong's members included some of the richest individuals in the world at the time. The most prominent was Howqua — Wu Bingjian in full — whose personal fortune was estimated at an extraordinary scale and whose business relationships extended to American trading houses in Boston and New York. The Cohong merchants not only conducted trade but owned the factory buildings themselves, making them landlords to their own customers. Prices for goods flowing along China Street — a thoroughfare that got its name from being lined with porcelain dealers — were negotiated between these Chinese merchants and the foreign supercargos within a system of mutual guarantee: the Hong merchants vouched for the behavior and tax obligations of their foreign counterparts. The arrangement created interdependence, and occasionally genuine friendships, across a boundary that was otherwise strictly policed.

The Thirteen Factories district was destroyed by fire three times. In 1822, an accidental fire swept through the neighborhood. The factories were rebuilt. In 1841, during the First Opium War, fire returned amid the fighting. They were rebuilt again. In 1856, at the onset of the Second Opium War, the factories burned for the third and final time. By then, the geopolitical ground had shifted enough that reconstruction made no sense. The 1842 Treaty of Nanking, which ended the First Opium War, had opened Shanghai, Ningbo, Xiamen, and Fuzhou as treaty ports. The Canton System was over. The trade that had been funneled through a single Guangzhou neighborhood for nearly a century dispersed across the Chinese coast. After the Second Opium War, foreign merchants relocated first to Honam Island across the Pearl River, then to the sandbar settlement of Shamian, west of the old city. The former site of the Thirteen Factories is now part of Guangzhou Cultural Park.

For all the grand commercial history, the daily life of the factory quarter had its own vivid texture. Hog Lane ran through the area, its open-fronted shops catering to sailors on shore leave with food, drink, clothing, and novelties the records call "chowchows." Sampan women worked the river near the ships at Whampoa, doing laundry and odd jobs. Enslaved people were sometimes brought by senior supercargos as personal staff. Peter Parker opened an eye hospital in the area in 1835 — the institution that became the Canton Hospital — and commissioned the painter Lam Qua, who had a studio nearby, to paint pre-operative portraits of patients with major tumors or deformities. Those portraits, unsettling and humane in equal measure, survive in collections today as documents of both medical history and the particular cosmopolitan strangeness of a place where Chinese and Western commerce, medicine, religion, and daily life overlapped in ways that neither side had fully anticipated when the arrangement began.