I'm not a language expert; I'm just speaking as a native Spanish speaker and relating my own experience, so take it as face value:
I work with Brazilians on a daily basis, not only executives, who are (as a general rule) very familiar with Spanish, but often with blue collar warehouse workers who speak only Brazilian Portuguese. I've never had a problem understanding what they say or making myself understood. Some people have heavy local accents (i.e. Bahiano or Cearês) but nevertheless, if we speak slow enough and without slang we have no problems holding a conversation, even on the phone. (BTW I've never had a Portuguese lesson in my life but once you spot the very standard differences in grammar and phonetics, it all becomes very obvious and easy to grasp). I'm fluent in English so if we both speak conversational English we might switch to it for some short periods or isolated phrases but otherwise there is no problem and we both end up speaking Portuñol. I think that if your goal is to have a perfect Spanish or Portuguese there might be some interference, but otherwise, for conversational purposes, I'd think that speaking Spanish would be a huge step into learning Portuguese (or vice versa).
Having said that, I wouldn't miss the opportunity to learn Mandarin but that's a personal decision.
Mis dos centavos/ meus dois cêntimos.