Spending a day seeing sights including Vikingskipsmuseet (Viking Ship Museum) in Oslo is okay, but then I'd get out of the Oslo area and go west and north quick-fast as the western part of Norway is far more interesting if history, nature and wildlife is your cup of tea. I would certainly not pick Vestfold over Bergen!
Bryggen is admittedly a bit hyped, but Håkonshallen, Edvard Grieg's amazingly well-preserved home at Troldhaugen and the view from Fløyen in a cozy almost South European atmosphere (hopefully helped by passable weather...) make it all worth the while. A extra tip is going to the history of Natural History Museum to behold their jaw-dropping exhibit of a mounted blue whale skeleton in full size.
For fjords, if you're in a hurry, Sognefjorden and Geirangerfjorden are safe bets and relatively easily combined with stave church visits at Urnes and Borgund for history. A hidden jewel is the mountain road from Hellesylt by Geirangerfjorden to Leknes in Hjørundfjorden - perhaps the most dramatic road in Norway. Get a ride or hire a car if possible.
Break the bank to stay at Union Hotel Øye - kaiser Wilhelm II's favourite during his numerous stay in Norwegian fjords - and then catch jugend architecture at Ålesund and penguin feeding at the city's aquarium. Wild life is covered by visiting the amazing bird nesting mountain at Runde - again, an underrated gem.
If you do go North, I think Helgelandskysten is your best bet - and by plane. Nice scenery, not that far north (in terms of travelling time) and you can have midnight sun. But it's a part of Norway best enjoyed by a two- or three-day drive, or even better boat cruise...