Good advice.
Generally driving across the border with a rental is not an issue, as mentioned, it's the one way rental. There are often big surcharges for one way rentals even within Canada or the US; you also need to read the fine print carefully as an unlimited mileage rental in the US will often turn to pay per km in Canada. Companies will sometimes waive one fees between large markets (i.e. Seattle to Vancouver), but Montana is not a big rental car hub.
If you are from the UK/South Africa/Ireland/Australia you might be able to find a one way deal without a drop off fee through a company called Canadian Affair.
The differences between cars are actually very small - lots of people move between Canada/US and take their cars. The speedometer differences are becoming less of a problem with digital dashboard displays. The bigger issue tends to be in the winter when rules about snow tires are different in parts of Canada.
I'm not sure I would say there is plenty of public transport in Canada - in Alberta & BC, there is generally decent public transport within the major cities. Outside of the cities, it's basically Greyhound and occasionally another long distance bus company. There is one cross-Canada train route farther north (through Japser and Edmonton) with trains a few times per week. In southern Alberta, Greyhound is basically the only public transport and it's not one I'd choose over other options.
I think your options are going to one of a few:
1) Return the car in Montana, fly to Calgary and rent another car for the Canadian leg of your trip
2) Head over to Seattle. From there you can either return the car and fly, ferry or take the train to Vancouver. Rent a car there and head to Calgary. You might also be able to find a one way rental from Seattle to Calgary - a lot more people cross the border there, so it's easier for rental companies to deal with getting cars returned between countries.
3) You could probably find a patchwork of buses from Montana to Calgary, but you'd be going via Vancouver so it would take 2-3 days