You are perhaps falsely transferring your idea of pretty Irish fishing villages to a country where it isn't like that. Icelandic villages are mostly not very interesting, modern houses in a grid pattern. There is also no pub culture - beer was only legalised about 20-odd years ago - so that other factor you are used to from your own country, the lively country hostelry, doesn't really exist either. And if they are operational fishing villages, there might be a strongly smelling fish processing plant within smelling distance. Nearly everyone lived on country farms - even fishermen, for fishing had a short season - until the 20th century got going, so there is very little history in most villages, they are just small dormitories in the countryside. Nevertheless there are some pretty villages, Stykkisholmur being perhaps the clearest case. And there are plenty of villages which at least have a nice setting.
The interest in Iceland lies mainly in its scenery, and therefore the question is, what would you do when you got to some village like, say, Stykkisholmur with no transport? There are scenic attractions in the area, but how would you get to them? Even in summer, the bus service is barely adequate to do what you suggest, ie hop on and hop off buses to visit attractions - how do you do that when there is only one or two buses per day - you aren't going to see very much in one day if that is all there is, and it is hardly suitable for those sights that only justify an hour. You could perhaps deal with some tour operators to put on some kind of customised tour for you, but that could be rather expensive. There is some possibility of going on off-season sea tours in the area to see killer whales, but they may run from the next village Grundarfjordur, which has recently obtained attention from tourists as it has some great scenery in the vicinity, although the village itself is nothing much.
One area where there is a concentration of sites in a small area is Myvatn, where things are close enough together you can hire a bike to ride around them. Whether you could tolerate the weather conditions for cycling around that time of year, I don't know, it could be like cycling in the W of Ireland in November/December. You would fly up to Akureyri, but then you'd still have to get to Myvatn 60km away, there will be the occasional bus (a few per week) to do that, but it may not connect well with the plane. And there are some tour operators in Myvatn, it's a major tourist centre, but with winter interest still not arrived I wonder if there would be very much on for you that time of year.
Another place you could go to, though it isn't far from Reykjavik at all, is Hveragerdi, as there is a variety of walking that can be done from there without transport. But, being close to Reykjavik, it can also be visited by bus from Reykjavik as a daytrip, as the bus service that far is adequate for day trips, giving you time to go on a decent walk also. Though there are also some sights in the area, like the lava tubes, which you'd really want transport to get to.
There is therefore something to be said for just staying in Reykjavik at this time of year, if you have no car. This is where the main off-season day tour offer is concentrated, so you would be able to find a variety of day tours to go on. And you could go on day trips by transport, without requiring a tour, to places like Hveragerdi (bus) and the Westmann Islands (fly).
