The mosque's origins lie in the British colonial military. In 1896, the Hong Kong Regiment established the first mosque on this site to serve the Indian Muslim troops stationed at Whitfield Barracks nearby — the same barracks whose grounds were eventually transformed into Kowloon Park. The soldiers who worshipped here had come from the subcontinent as part of the garrison that Britain maintained across its empire, and their religious needs required a dedicated space. That original building stood for decades, an expression of the layered realities of colonial Hong Kong: British authority, South Asian labor, and a community maintaining its faith far from home. The mosque served its congregation faithfully until progress intervened in the most literal way — underground construction for the Mass Transit Railway in the late 1970s damaged the old structure's foundations beyond repair.

Compensation from the MTR Corporation, combined with donations from the local Muslim community, funded a new building on the present site at 105 Nathan Road. It opened on 11 May 1984. The architect, I.M. Kadri, designed it to embody the identity of Hong Kong's Muslim community — to be unambiguously itself among the glass towers and signage around it. The traditional Islamic architecture stands as a deliberate contrast. White marble covers both the facade and the paving. The four minarets mark the corners of the upper terrace. Inside, three prayer halls and a community hall share the building with a medical clinic and a library. The main prayer hall on the first floor holds 1,000 people. Above it, the women's prayer hall is surrounded by a terrace and topped by a dome 5 meters in diameter and 9 meters high — modest in scale, refined in proportion.

The mosque today primarily serves Muslims from South Asia and Indonesia, many of whom live in Tsim Sha Tsui and the surrounding neighborhoods. Across Nathan Road, Chungking Mansions — the famously labyrinthine tower block that has housed generations of traders, travelers, and immigrants — is a neighbor that tells its own story of Hong Kong's diversity. The mosque functions not only as a place of worship but as an anchor for non-Chinese Muslims in the city: a cultural institution, a social center, a place where sermons are delivered in Urdu, English, and Arabic to reflect the congregation's range. The chief imam delivers Friday khutbahs in all three languages. Free Quranic education is available for students of all ages. The mosque has, over its long life, become something larger than architecture: a community in permanent session.

On 20 October 2019, during the protests against the extradition bill that had convulsed Hong Kong for months, a police water cannon vehicle sprayed the Kowloon Mosque twice with stinging blue dye. More than ten people near the entrance were hit and hospitalized, including former Indian Association of Hong Kong chairman Mohan Chugani and Legislative Councillor Jeremy Tam. The police described the dousing as a mistake. What followed told a different story about the city's capacity for solidarity: masked protesters, pedestrians, and mosque-goers rushed to the building to help wash away the dye. Chief Executive Carrie Lam came in person the next day to apologize. The image of strangers cleaning a sacred site together — hands in blue-stained water, working alongside worshippers they may never have met — became one of the defining moments of an extraordinarily difficult year. The mosque had stood through a great deal in its long history. It would stand through this too.