The site began as the wooden manor of the Raczkowicz family. In the sixteenth century, the architect Hiob Bretfus built a brick two-story Gothic-Renaissance castle for Piotr Wiesiołowski the Younger, complete with moat, earth ramparts, and a tower converted to a staircase. Branicki kept the bones and rebuilt almost everything else. From 1728, the Saxon architect Johann Sigmund Deybel directed the transformation. The fence between the avant cour and the courtyard of honor was crowned in 1757 with two monumental sculptures by Johann Chrysostomus Redler: Hercules fighting the dragon, Hercules fighting the hydra. Inside, painters Szymon Czechowicz, Ludwik Marteau, Augustyn Mirys, and the celebrated French Rococo decorator Jean-Baptiste Pillement worked on the rooms. Frescoes came from Georg Wilhelm Neunhertz in 1738 and Antoni Herliczka. Stucco from Samuel Contesse and Antoni Vogt. The result was a building meant to host envoys from France, England, Turkey, and Russia, and to suggest that its owner could properly receive them as a sovereign.

The palace grounds matter as much as the palace. A straight tree-lined avenue centers on the building, crossing the river on a three-arched stone bridge to the paved forecourt. The central block has two stories above an arcaded basement, with a pediment displaying Branicki's coat of arms and end pavilions topped by domed roofs in two tiers. An Italianate balustrade hides a low attic. A heroic Atlas group crowns everything. On the garden front, a colonnaded terrace looks out over a French parterre of clipped hedges and gravel allées punctuated by sphinxes, with a later English landscape garden wrapping the whole composition in naturalistic groves. Outbuildings include the Arsenal of 1755, an Orangery, and Italian and Tuscan pavilions. Branicki also designed the city itself, laying out Białystok's central triangular market and threading streets to align with palace views.

After the Third Partition of Poland in 1795, Białystok went to Prussia, then in 1807 to the Russian Empire. The palace's furnishings began to disappear. After Izabella Branicka's death, the Potocki family carried off most of the paintings and furniture, scattering them across estates that would themselves later be destroyed. Tsar Alexander I purchased some pieces. Garden sculptures went to Saint Petersburg to decorate imperial residences. In 1837, the building was repurposed as an Institute of Noble Maidens, a Russian girls' school. In 1884, undistinguished additions were grafted onto the palace for the school's needs, defacing the Baroque structure. The Germans turned it into a military hospital in 1916, which led to further damage. In 1919, the newly independent Second Polish Republic gave the building to its army. In 1920, during the Polish-Soviet War, the palace briefly served as the headquarters of the Soviet-installed Provisional Polish Revolutionary Committee.

The catastrophe came in July 1944. As the Wehrmacht retreated before the Red Army, German Brandkommando units methodically torched central Białystok. The Branicki Palace burned with the rest of the city. Damage was assessed at about seventy percent. The shell stood. Reconstruction began almost immediately, treated as a matter of national pride. The engineer Stanisław Bukowski drew up the plans between 1947 and 1949, with input from the leading conservator Jan Zachowatowicz, aiming to recreate the mid-eighteenth-century exterior. In 1949 the planned use changed: Białystok's first university, the Medical Academy, would occupy the rebuilt palace. Architect Andrzej Nitch made adaptations that Bukowski objected to. Bukowski returned in 1951 as architectural director, and by 1955 the complex was substantially rebuilt. The gardens were restored in the 1950s by Gerard Ciołek. Bukowski supervised the painting of polychromes in the main hall and staircase as late as the 1970s. The Versailles of Poland today is at once a faithful reconstruction and a building twice removed from its original. Medical students walk hallways that Branicki's envoys once received guests in. The Atlas group still crowns the roofline. The sphinxes still guard the parterre. What was lost was not the form.