The garden began in 1894 as the Japanese Village and Tea Garden, built for the California Midwinter International Exposition by Australian-born entrepreneur George Turner Marsh, who hired Japanese craftsmen to construct the site. When the fair closed, Marsh sold his concession to San Francisco for $4,500. The city hired Makoto Hagiwara, a Japanese immigrant and landscape designer, to manage the property. What Hagiwara did over the next three decades was extraordinary. He personally oversaw the transformation of a temporary fair exhibit into a permanent five-acre garden, importing plants, birds, and the now-famous koi fish from Japan, more than tripling its size. After the 1915 Panama-Pacific International Exposition, the South Gate, Temple Gate, and Pagoda were acquired from that fair's Japanese exhibits and added to the garden. By the time of Hagiwara's death in 1925, the garden had become an institution.

Hagiwara's daughter Takano and her children took over the garden after his death, maintaining the family tradition of personal stewardship. Then came Pearl Harbor. With the onset of World War II and rising anti-Japanese sentiment, the Hagiwara family was evicted from their home within the garden and sent to an internment camp. The garden was stripped of its Japanese identity and renamed the Oriental Tea Garden. The Shinto shrine was demolished. Even the family's prized dwarf trees were taken; they ended up in the hands of Dr. Hugh and Audrey Fraser. The garden the Hagiwaras had spent decades cultivating was used as a symbol of exactly the kind of erasure being inflicted on Japanese Americans across the West Coast.

The postwar years brought slow repair. In 1952, the name Japanese Tea Garden was restored. The Hagiwara family was offered only minimal assistance in the restoration for which they had volunteered their expertise. But gestures of reconciliation followed. In 1949, the Gump family donated a bronze Buddha. Because the 1951 Japanese Peace Treaty was signed in San Francisco, the Japanese Consul General presented a 9,000-pound Lantern of Peace on January 8, 1953, commissioned through small donations by the children of Japan. Landscape architect Nagao Sakurai designed a Peace Garden and a karesansui, a dry landscape garden where raked gravel represents water and tortoise-shaped stone islands symbolize immortality. In 1974, artist Ruth Asawa created a plaque honoring Makoto Hagiwara and his family. The road bordering the garden was named Hagiwara Tea Garden Drive in 1986.

Every element in the garden carries meaning rooted in Buddhist and Shinto traditions. The taiko bashi, or drum bridge, arches so steeply that its reflection in the water completes a perfect circle resembling a drum. Its purpose is practical as well as aesthetic: the steep arch forces you to slow down, which is the entire point of a Japanese garden. Rocks serve as what tradition calls the backbone of the garden, arranged asymmetrically in clusters where a large base stone sets the mood and smaller stones agree with it. Tall upright stones evoke masculinity in Shinto belief; low flat stones represent femininity, reflecting the principle that all things in nature contain gender equilibrium. Stone lanterns represent the five elements of Buddhism: earth, water, fire, air, and spirit. Every three years, arborists climb the sixty-foot Monterey pines with ropes to prune them into the zig-zagged planes favored in traditional Japanese landscapes. The garden teaches patience, if you let it.