Dartmouth had wanted a real fort for decades. The crisis that finally produced one was a financial deal. The town agreed to develop the existing castle into a proper artillery position in return for a royal subsidy of thirty pounds a year, later raised to forty. The new gun tower went up at the end of the fifteenth century on a rocky outcrop overlooking the entrance to the harbour, opposite Saint Petrox Church. The design is unmistakably built around gunpowder. The basement held the artillery, firing through primitive gunports protected from the sea by wooden shutters. The ground floor combined offices, garrison accommodation, and the chain room with its pulley and capstan for raising the harbour chain. The first floor was lodging fitted with an oven. Gunloops for handguns ran around basement, ground, and first floors for close defence, and lighter artillery sat on the roof. The battlements were raised in height shortly after construction to deal with the threat of musket fire from the higher ground behind. The wall is up to twenty-three feet high and seven feet thick. The two main rooms on each level are circular and rectangular respectively, the result of changes during construction. It is a building that admits its own mistakes.

In 1543, Henry VIII allied with Spain against France, and when the Spanish made a separate peace the following year England suddenly faced a French invasion with no ally and a hostile Scotland on its northern border. The king ordered the south coast defences improved. Dartmouth Castle was reinforced with three new gun batteries flanking the tower, plus a fourth platform called Lamberd's Bulwarke in the south-east corner. Lamberd's Bulwarke caused trouble. Sir Peter Carew, soldier, adventurer, and Member of Parliament for Dartmouth, claimed the new platform trespassed on his family's house within the castle precinct. He seized the castle outright and threw the town's officers out. The argument was eventually resolved, but it shows how complicated possession of these places could be: half royal fortress, half private property, with national defence running through somebody's drawing room. In 1597, with another Spanish scare brewing, the gun tower was improved again and Lamberd's Bulwarke repaired.

The Civil War exposed the castle's weak point. Dartmouth initially sided with Parliament in 1642, and the castle was guarded by just five men. In 1643, Prince Maurice besieged the town for the Royalists and discovered that he could shell the castle from the higher ground at its back, where its guns could not effectively reply. The castle was overcome. An earthwork fort called Gallants Bower was probably built afterwards to protect that vulnerable hill, though it may have existed since 1627. In January 1646, Sir Thomas Fairfax marched on Dartmouth for Parliament and demonstrated the lesson in reverse. He took the town first, then Gallants Bower, and only after both had fallen forced the surrender of Sir Hugh Pollard, the castle's Royalist commander, the following day. Under the Interregnum, Gallants Bower was decommissioned, the Carews' house in the castle pulled down, and Sir John Fowell appointed as governor to run the defences and prevent smuggling. He stayed in the post until 1677, outlasting the king he served and the king who came back.

By 1820 the castle had two serviceable guns. In 1847 the writer Clarkson Stanfield observed it was "not spacious, and mounts but a few guns." Two technological revolutions changed that. The arrival of shell guns and steamships in the 1840s, followed by rifled cannon and iron-clad warships in the 1850s, persuaded the Royal Commission on the Defence of the United Kingdom in 1859 that the south coast was again exposed. Lamberd's Bulwarke was rebuilt and renamed the Grand Battery, then in 1861 rebuilt again as the Dartmouth Point Battery, with shell-proof granite defences housing three traversing emplacements for 68-pounder guns and two ten-inch eighty-six-hundredweight guns on the roof. It was manned by three regulars and over fifty-five reservists from the new Sixth Devonshire Artillery Volunteer Corps. In the Second World War it was rearmed again with two 4.7-inch quick-firing guns dating from the First World War, housed in concrete gun houses, to protect merchant convoys and the Philips shipyard. Officers lived in a cottage; non-commissioned officers in the castle itself; other ranks in the Victorian fortifications and later Nissen huts.

In 1955 the castle was transferred back to the Ministry of Works and repaired. The Dartmouth Point Battery, by now generally known as the Old Battery, was leased back to the town for use as a restaurant; the site reopened to the public; a 19th-century tower behind the battery that had served as a lighthouse from 1856 to 1886 was preserved. English Heritage manages it now, with the Grand Battery displayed as it would have appeared in the nineteenth century; it drew 37,940 visitors in 2007. The 14th-century gun tower is a Grade I listed building, the 19th-century battery Grade II*. Just behind the gun tower stands Saint Petrox Church, a three-aisled building that took its current form in 1641. From the parapet of the gun tower you can still see exactly what the original engineers were aiming at: the narrow throat of the estuary where the iron chain once ran, the matching tower at Kingswear opposite, the open Channel beyond. The view that built this castle is still the view that defines it.