Two buses, 93 and X93, run south to Scarborough (£6.20, one hour, every 30 minutes), with every second bus going via Robin Hood's Bay (£4.30, 15 minutes, hourly); and north to Middlesborough (£6.20, one hour, hourly), with fewer services on Sunday. The bus station is next to Whitby's train station.
The Coastliner service 840 runs from Leeds to Whitby (£14, 3¼ hours, four times daily Monday to Saturday, twice on Sundays, though you'll need to change at Malton) via York and Pickering.
Coming from the north, you can get to Whitby by train along the Esk Valley Railway from Middlesbrough (£7.10, 1½ hours, four daily), with connections from Durham and Newcastle. From the south, it's easier to get a train from York to Scarborough, and then a bus from Scarborough to Whitby.