I'm sorry to disagree with you no. 1, but if you land at 8pm, your not going to get to Big Milly's much before 11.00-12.00 once you've cleared customs, got your luggage and made your way there - and thats assuming your flight is on time. Seems a bit of a waste of time if your planning on leaving early the next morning to go to Cape Coast. Just to balance that comment, let me just add that I don't have a great view of Big Milly's - I stayed with my best friend (we're late 30's) and I really wasn't that impressed, the room was pretty grotty and the hassle we got from the local men was really tiresome - maybe you'll get less as part of a couple but I wouldn't necessarily count on it. My husband (a Ghanaian rasta) hates the place as he says it gives all of the rasta's a bad name due to the way some of them behave...... Given that, I guess I'm not going to recommend putting the effort in to go there - horses for course though, I'm aware its lots of people's favourite place here, just really not my cup of tea.
I would rather suggest you stay near the airport for the first night - there's plenty of hotels nearby, both higher end bog standard (Holiday Inn, Golden Tulip, Fiesta Royale) and boutique (Villa Monticello) or more middle of the range (Airport View, Alisa hotel, Afia Hotel etc). Maybe a little more expensive, but will be a whole lot more convenient. And if you stay close to the airport, you can get the hotel shuttle and only start with your driver the next morning - this may help justify a more expensive hotel room as you'll have to pay for an extra day of the driver if you get him to meet you at the airport.
Going to Cape Coast on a Saturday morning, traffic shouldn't be too much of an issue as long as your away before 9 am - its more the Sunday night traffic coming back into Accra that you need to be careful of especially if your catching a plane that evening. We tend to leave around 12, stop for lunch on the way and get back home to Accra (we live 5 mins from the airport) by 4-5pm. If you do come earlier, well you will have more days anyway, but getting out of the city in the morning isn't generally too much of an issue as long as it isn't a public holiday (google it to find out) and you leave by 7 ish. Would definitely recommend getting a car with a driver, I do drive here but can find it incredibly stressful even after living here for a few years - and the road to Cape Coast can be a nightmare of police stops and speed checks especially if your coming in the run up to Christmas.
Can't comment on the other places you mention as I haven't stayed there, but can, as a couple of alternatives recommend Anamabo (even if just for a meal as the restaurant is lovely) and Elmina Bay. The canopy walk is good fun and be sure to visit at least one of the slave forts - Elmina is the best tour IMO, but Cape Coast is the most stunning visually. Also if you do stay for more days, look up Women in Progress to do some great tours if thats something you'd be into.