Are there any setup differences for different player counts?
Yes
The only difference in the setup is the number of "Reveal a Secret Number" Clue cards used in the game. In a four-player game you will return 2 such cards back to the game box, leaving only 5 of them present in the game.