There are actually several onsen in Tokyo itself. The LP book can help you.
There is an onsen that you can find by walking up the street that is to the east of the zoo. It parallels the boundary of the zoo. Keep walking till you pass a hotel then you need to make a left down a side street. You can ask when you get to the hotel as the onsen is famous.
How about a sento ? Almost as good.
From the Nippori Station on the Yamanote Line exit the south exit and walk up the hill past the Suzuki Ryokan. Go through the cemetary till you get to a street with traffic lights on them. Cross the street and make a left.
Keep walking till you pass the police box (noted with it's red light out front.) about 50 feet past that look for the place with the little wooden lockers on your left. You can buy a towel and soap there. The entire cost will be 600 yen.
enjoy !