The original village of Seaham was tiny, an agricultural settlement near St Mary's Church and Seaham Hall, north of the present town. Anne Isabella Milbanke was the local landowner's daughter, an intelligent and sheltered young woman who would later be remembered as a serious mathematician. Byron, already notorious, accepted her proposal and came north to marry her. He was thirty years old at the time. He found Seaham wintry, lonely, and frankly tedious, but the North Sea pulled him to the shore for long walks. Their daughter Ada was born in December 1815. Augusta Ada Byron, later Countess of Lovelace, would write the first known computer algorithm in 1843 in her notes on Charles Babbage's Analytical Engine. The Byron marriage was effectively over by 1816. Annabella raised Ada on a strict regime of mathematics to keep her away from her father's poetic temperament. Whether this strategy worked is debatable. What is certain is that one of the foundational moments of computer science traces back to the heir of a Seaham estate, conceived during a winter wedding her father wished had not happened.

In 1821, the Milbankes sold their estate to the 3rd Marquess of Londonderry, who saw what the North Sea coast and the County Durham coalfields could become together. He built a harbour in 1828 to ship coal from the mines he was about to encourage. The first colliery opened in 1845. The harbour proved too small almost immediately, and the 6th Marquess commissioned engineers Patrick and Charles Meik to reclaim land, deepen, and extend it. The new harbour opened officially in 1905. Its design uses a series of interconnecting locks rather than the typical two-wall construction, allowing ships to manoeuvre in a sheltered basin. The 3rd Marquess had also approached the architect John Dobson in 1823 to plan a model town around the harbour, but funding failed and the town grew piecemeal. It was first called Seaham Harbour, to distinguish it from the ancient village; over time the whole settlement became known simply as Seaham.

John Candlish's Londonderry Bottleworks opened in 1853 and survived until 1921, the largest glass bottle works in Britain. Waste glass, dumped at sea throughout the operating life of the factory, has been ground by tides and wave action into smooth, frosted pebbles for more than a century. Local beaches, especially at Blast Beach and around Dawdon, are now famous for sea glass: green, white, blue, occasionally the rare red of railway lantern glass. Collectors come from across Britain to walk the tideline at low water. Three deep coal mines defined twentieth-century Seaham: Dawdon Colliery, Vane Tempest, and Seaham Colliery, locally known as the Knack. Seaham Colliery suffered an underground explosion in 1880 that killed more than 160 men and boys, including surface workers and the rescuers who went in afterward. All three pits closed by 1992 during the long decline that accelerated through the 1984-85 miners' strike. The closures gutted the local economy. Then, on 17 November 1962, the RNLI lifeboat George Elmy was lost off Seaham, with eight men and a boy drowning. The new coast road is named George Elmy Lifeboat Way in their memory.

In 2014, the local artist Ray Lonsdale installed a steel statue on the Seaham seafront. Officially titled 1101, locally known as Tommy, it depicts a First World War soldier sitting on a bench in the moments after the Armistice was signed, contemplating the war's end. It was originally temporary, on display for three months only. A community fundraising campaign raised the money to keep it permanently. It now sits where the harbour, the colliery legacy, and the North Sea meet. The town's other landmarks include the Londonderry Institute in Tempest Road, built in 1853-55 by Thomas Oliver in a monumental Greek style; the former Londonderry Offices on the seafront, headquarters of the Marquess's mining empire; and the parish church of St Mary the Virgin, with its Anglian nave dating from the late seventh century. Several films have used Seaham as a location: the opening scene of Alien 3 (1992) was shot on Blast Beach at Dawdon, and Billy Elliot (2000) drew on the look and feel of these post-pit towns. Ramsay MacDonald, Labour Prime Minister, represented Seaham in Parliament between 1929 and 1935.