The Tainan Prefectural Hall was built in 1916, during the Japanese colonial period, as a government administrative building. Its architecture reflects the Western-influenced style common to Japanese institutional buildings of the era. When American bombers targeted Tainan during World War II, parts of the structure were completely destroyed by incendiary fire. After the war, the building was patched together and repurposed, first as the Air Force Supply Command Headquarters, then as the Tainan City Government Hall. The repairs restored the historic section largely to its Japanese-period appearance, but decades of government use left the building worn. In 1992, scholars and cultural officials chose this battered survivor as the home for something Taiwan had never had: a national institution dedicated to its own literature.

The idea began at the 1990 National Culture Conference, convened by the Executive Yuan's Cultural Construction Committee. Scholars and experts argued that Taiwan lacked an authoritative institution for collecting and researching its literary materials. This was not an abstract complaint. Taiwanese literature had been written in at least four languages, suppressed under multiple regimes, and scattered across private collections, university archives, and personal libraries. Works in Taiwanese Minnan had been marginalized during periods of Mandarin-only policy. Japanese-language literature from the colonial period occupied an ambiguous cultural space. Tainan was chosen as the museum's location for its historical significance as Taiwan's oldest cultural center, and after years of administrative coordination and scholarly advocacy, the museum opened on October 17, 2003. It was designated a national historic site that same November.

The museum today combines the restored historic building with modern construction, including two underground floors and two above-ground floors added after the war. The result is a deliberate act of architectural layering: Japanese-period facades sit alongside contemporary gallery spaces, each era visible and acknowledged rather than concealed. Inside, the museum researches, catalogs, preserves, and exhibits literary artifacts spanning the full range of Taiwan's linguistic and ethnic diversity. In 2021, it was elevated to a third-level central institution under the Ministry of Culture, a bureaucratic designation that reflects growing recognition of Taiwanese literature as a distinct field. A satellite facility, the Taiwan Literature Base, operates out of Japanese-style dormitories on Qidong Street in Taipei. Walking distance from Tainan Station, the museum sits at the heart of a city that has been writing its own story since the 17th century.