Lose Glory based on your reputation with the townsfolk. Glory loss is calculated according to this chart: