I've recently been out to Orange and Millthorpe and while they are lovely towns, I don't think they are worth it on this time scale, or in comparison to the coastal attractions.
After leaving the Blue Mountains I would head back down to the city, aiming for Engadine, then Grand Pacific Drive through the Royal National Park all the way down to Kiama on day one.
Stay there 1-2 days and use as a base to see Minamurra Falls, Jamberoo water park for the kids (If open, and if you think this worthwhile), Kiama, and nearby beaches like Gerringong and Seven Mile Beach.
Next day, stick to the coastal route until it ends at Berry. I'd probably skip the diversion to Fitzroy Falls and Kangaroo Valley with toddlers as this is a steep and winding road. Head on to Huskisson, or, even better, if you are camping, Green Patch, and enjoy the stunning scenery of Jervis Bay. Hyams Beach, Green Patch have the stunning white sands, Murrays at the end is fabulous, and there is wildlife everywhere.
A good next stop could be the tourist park at Murramarang. This is a great set up for kids, good beaches, and kangaroos everywhere - but during school holidays could be booked out, so check. If it is, Ulladulla or Bateman's Bay may be good, but still make the diversion to Murramarang, or, better, Pebbly Beach.
After that you have 2 choices
- Either head up the mountain at Bateman's Bay to Braidwood (great old town) and then Bungendore and Canberra, and after a day or 2 there on to Melbourne via the Barton and Hume Highways (7 hours no stop, or, divert via Beechworth) or, if time permits perhaps via Thredbo and take a trip the chairlift to get a glimpse of Mt Kosciusko and stay here a night, then on to Melbourne.
- Or, continue down the coast. Lots of beaches, but make the tiny diversion into Tilba which is a lovely time capsule village. Merimbula, Lakes Entrance are good places to aim for but, again, this is school holidays, so try to book ahead.
The only reason I did not mention Wilson's Promontory and Phillip Island is because you'd already made clear you were going there.
I've not been to Wilson's, but Phillip island is good for kids - koalas, seals and the little penguin parade.