Champions of Midgard

If I'm attacking a troll with 4 dice and roll 3 blank ones and decide to use Favor token to throw the dice again, do I lose the 3 dice and get to reroll just the 1 die?

You don't lose any Warriors before the reroll. So you can choose to reroll all the 4 dice (or fewer, if you keep some from the first roll).