The town of Nantucket (called 'Town' by the locals) is the island's only real population center. Once home port to the world's largest whaling fleet, the town's storied past is reflected in the gracious period buildings lining its leafy streets. It boasts the nation's largest concentration of houses built prior to 1850 and is the only place in the US where the entire town is a National Historic Landmark. It's a thoroughly enjoyable place to just amble about and soak up the atmosphere.
There are two ferry terminals: Straight Wharf and Steamboat Wharf. Walk off Straight Wharf and you're on Main St; Steamboat Wharf is just a few blocks north. The majority of restaurants, inns and other visitor facilities are within a 10-minute walk of the wharves.