Phillotson is eminently the respectable man. Though he fails to achieve the same goals Jude pursues, his bearing and view of things do not change much. Even when Arabella encounters him on the road to Alfredston, now down on his luck and teaching at Marygreen because it's the only place that will have him, this air of respectability remains. It must be this which Sue can't stand about him, the respectability plus the legal right to make love to her.
Sue's opinion of him does not make him any less decent. He is like Jude in many ways: he is goodhearted and honorable-, he allows instinct to overrule reason; he is too accommodating for his own good; he is intelligent. Like Jude he is ill-equipped to get what he wants in life and soon resigns himself to mediocrity. However, unlike Jude he no longer is dazzled by ideals, perhaps because he is older. Maybe too late he learns to act on the basis of calculation, estimating that Sue's return will be worth the benefits it may bring.
Phillotson, in short, is a man whom it is easy neither to like nor to dislike; he goes largely unnoticed.