After dismissing Botany Bay as a site for the colony, Governor Phillip sailed the First Fleet into what James Cook had named Port Jackson (Warran/Warrane in the local language) and dropped anchor at a horseshoe bay with an all-important freshwater stream running into it. Phillip christened the bay Sydney Cove after the British Home Secretary, Baron Sydney of Chislehurst, who was responsible for the colonies.

The socioeconomic divide of the future city was foreshadowed when the convicts were allocated the rocky land to the west of the stream (known unimaginatively as the Rocks), while the governor and other officials pitched their tents to the east.

Built with convict labour between 1837 and 1844, Circular Quay was originally (and more accurately) called Semicircular Quay, and acted as the main port of Sydney. In the 1850s it was extended further, covering over the by then festering Tank Stream, which ran into the middle of the cove.

As time went on, whalers and sailors joined the ex-convicts at the Rocks – and inns and brothels sprang up to entertain them. With the settlement filthy and overcrowded, the nouveau riche started building houses on the upper slopes, their sewage flowing to the slums below. Residents sloshed through open sewers, and alleys festered with disease and drunken lawlessness. Thus began a long, steady decline.

Bubonic plague broke out in 1900, leading to the razing of entire streets, and Harbour Bridge construction in the 1920s wiped out even more. It wasn’t until the 1970s that the Rocks’ cultural and architectural heritage was finally recognised. Redevelopment saved many old buildings by converting them into tourist-focused businesses. Shops hocking Opera House keyrings proliferate, but gritty history is never far below the surface if you know where to look.

Much of Millers Point has been public housing for over a century, originally for dock workers. With property prices so high, the State government from 2016 began a series of phases of auctioning off these homes to the highest bidder, evicting one of Sydney's oldest communities in the process and ignoring significant public protest.