The station was established in 1883 by Charles Fisher and Maurice Lyons, who stocked the property with 20,000 head of cattle overlanded from Wilmot in Queensland by the legendary drover Nat Buchanan. The South Australian government had granted the lease in December 1879, covering nearly 16,000 square miles. Within a decade, financial trouble sent the property to Goldsbrough Mort & Co, who sold it in 1900 for £27,500 to a syndicate including the Kidman Brothers. By 1894 the station was running 30,000 head and 500 horses across 21,000 square kilometres. Cattle went by sea to Batavia via Singapore. By 1907, the herd had grown to an estimated 69,350 head. In 1902, when Sidney Kidman and the Emanuel brothers acquired the property, it was openly described as one of the largest cattle stations in the world.

The parade of managers at Victoria River Downs was colourful, sometimes violently so. In March 1895, Goldsbrough Mort & Co employed the notorious Jack Watson to run the station — the same man who had briefly managed neighbouring Auvergne after its manager was shot dead in a card game. Watson didn't last long at Victoria River Downs either: he drowned in the Katherine River in 1896. His brother Robert Molesworth Watson took over the management. Sidney Kidman eventually sold the station in 1909 to Bovril Australian Estates for £200,000, bundling it with two other properties. Later owners included media magnate Kerry Packer's rival bidders — the Northern Territory government vetoed Packer's purchase in 1984, steering approval to Peter Sherwin for $11.6 million. Packer went on to acquire Newcastle Waters Station instead.

The station takes its name from the Victoria River, which cuts through country of genuine drama — wide floodplains that transform during the wet season into vast temporary lakes, red loam banks that crumble into the flow during heavy rain, tributary creeks with names like Battle Creek, Depot Creek, and Jasper Creek threading through the property. In 2010, the station recorded its highest ever monthly rainfall total: 747 millimetres in a single February, shattering a 120-year record. In 2014, the property endured 88 consecutive days with temperatures in excess of 35 degrees Celsius between September and November. This is not easy country.

Victoria River Downs has shed significant parcels over its history. The Kidman Springs research station — carved out in 1963 and now a 314 km² pastoral research facility — sits within its former boundaries. Pigeon Hole Station, once an outstation of 181,100 hectares, became a separate Aboriginal settlement inhabited by Gurindji and Ngarinyman people. In August 2024, Pigeon Hole was formally renamed Nitjpurru, its Bilinarra language name, after the billabong just north of the community. When catastrophic floods struck in early 2023, residents of Nitjpurru (Pigeon Hole), Daguragu, and Kalkarindji were evacuated by Australian Defence Force aircraft to Katherine, then by bus to Darwin. The river that defines this landscape remains capable of overwhelming everything built beside it.