The Ohlone people, who gathered bird eggs on the rocky outcrop, never lived there permanently. The Spanish called it La Isla de los Alcatraces, the Island of the Pelicans. Mexican governor Pio Pico gave it to William Workman in June 1846, expecting a lighthouse in return. Months later, the Mexican-American War made such arrangements irrelevant. President Millard Fillmore designated the island for military use in 1850, and by 1859, the Army had completed a citadel fortress at the summit, eleven cannons pointed at the Golden Gate. This was Third System fortification, though unlike most such works, the builders let the natural sandstone outcrops shape the design rather than imposing a standard pattern.

The island held criminal soldiers as early as 1859. By 1861, it was the military prison for the entire Department of the Pacific, housing Civil War prisoners of war within its first year. When President Lincoln suspended habeas corpus in 1863, private citizens accused of treason joined the soldiers behind bars. Confederate sympathizer Asbury Harpending landed there after outfitting a schooner as a privateer to prey on Pacific Mail ships carrying gold. By the war's end, more than one hundred cannons bristled from the island, and the prison held Confederate sympathizers who had made the fatal mistake of celebrating Lincoln's assassination too publicly.

After the Civil War came the Indian Wars. Paiute Tom became the first Native American imprisoned on Alcatraz in 1873, transferred from Camp McDermit. In the 1870s, Major George Mendell ordered prisoners and mules to reshape the island itself, leveling the top and dumping debris into the coves. They were quite literally building the prison that held them. The 1906 San Francisco earthquake barely touched Alcatraz while devastating the city, and prisoners flooded in from damaged mainland facilities. When the original citadel collapsed in 1908, military prisoners built its replacement between 1909 and 1911, a concrete structure that earned the nickname that would follow the island into infamy: The Rock.

The prison rebuilt in concrete from 1910 to 1912 under Colonel Reuben Turner, at a cost of $250,000, using granite blocks from the original gun mounts as wharf bulkheads. In 1915, the facility was renamed the Pacific Branch, U.S. Disciplinary Barracks, signaling rehabilitation alongside punishment. Some reformed prisoners returned to active duty. But twenty-nine escape attempts involving eighty convicts marked the military years, with sixty-two caught and the rest vanishing into the bay or beyond. In November 1918, four prisoners escaped on rafts and were spotted at Sutro Forest; only one was recaptured. The others simply disappeared, leaving behind a mystery the island would repeat.

By 1933, the Army was ready to transfer its troublesome island. The Bureau of Prisons wanted a high-security site to combat the Prohibition-era crime wave, and Alcatraz fit the bill. Modernization began in October 1933, and the Alcatraz Federal Penitentiary opened in August 1934, ending eighty years of Army occupation. Thirty-two hardened Army prisoners stayed behind while the rest scattered to Fort Leavenworth and Atlanta. The tunnels and chambers of the original 1850s fort still exist beneath the concrete, accessible to those who know where to look. The prison closed in 1963, but the fort's bones remain, layer upon layer of confinement stretching back to the first soldier locked away in 1859.