How long to spend in Budapest

Yes, 2-3 days is enough. The Szechenyi Baths are on the Yellow line metro at the next to last stop going north. They're worth a visit. They're open to men and women. If you go, I advise entering at the main entrance and NOT the side entrance which is closest to the metro. Walk around to the left past the side entrance to get to the front and the main entrance. It's a long story, but basically if you go through the main entrance you won't have any problems and you may (or may not) have some at the side entrance.

The Zoo is not one of the greatest in the world, but it is nice enough. It's next to the famous Gundel restaurant which is Budapest's best known, but not necessarily the best, restaurant. They have a lunch buffet on Sunday and lunch on other days has a relaxed dress code (jacket and tie not required for lunch, but don't show up looking like a beach bum either) and is lower priced than dinner, which is quite expensive. The main art museum is good in Budapest.

I advise getting either a Touristajegy card or a Budapest card. The Touristajegy card gives you 72 hours of public transportation. The Budapest card gives you 48 or 72 hours of public transporation (depends on which card you get) and some reduced museum admission. There are ticket inspectors everywhere on public transportation, especially the metro, and the metro has 4 different tickets. You have to figure out which ticket you need before getting on it as the combination of stops and line changes you need to take determines which of the 4 tickets you need. Frankly, the system is rather confusing if you don't live there and a bit of a hassle for tourists, so I just advise getting one of those cards I mentioned as they are valid for the metro. The Budapest card can be bought anywhere but the Touristajegy can only be bought at bus stations and some metro stops.

I spent 3 days without daytrips and that felt OK.

I would say that 3 days are sufficient.
If you want to see something different, you might visit the Statue Park in the south of the city. That's where the Hungarians mothballed their communistic monuments after 1990. On entering the premises you will hear the Internationale and other political songs, then you can stroll around the statues of Marx, Lenin, and others. A veritable oddity!