What decides if Imposter Bowser will just take all of your stars on King Browser's Keep?

Just had a game with my sister where we landed on Bowser a couple of times early on and he stole our star out right LITERALLY EVERY TIME. We're talking four or five times in a row he stole a star. And this was our first time on this board so we assumed that was just what he does on this bored. Then a CPU was going to get an ally and land on a Bowser space, so we let him win the ally so he would lose 2 stars - AND THEN HE GOT TO SPIN THE WHEEL TWICE AND ONLY LOST LIKE 30 COINS. Our plans were foiled and he got away with the win because of it. Online it says it's just random is that true?