At the start of the game, the Invaders are at Terror Level 1. They’re not afraid of the Island and probably aren’t even aware of the Spirits’ existence.
As you earn Fear Cards you will reach new Terror Levels with easier victory conditions. Anytime you meet the current victory condition, you win after resolving the current effect — the Invaders have had more than they can take, and abandon the Island entirely.