Your score is equal to all Favor you have earned minus any Disgrace you have suffered (usually due to Problems erupting and other game effects).
Certain Personality cards, like the Financial Heir, are played face down in front of you during the game. These often affect your score.