The Master said, "One who has reached the age of forty and is still disliked will be so to the end."--Analects, book 17 (Waley tr.).
-- Earlier in the same book, it's said that anyone can change except the very wisest and the very stupidest people. So whether one is disliked is not just a function of one's virtues. We also can gather whether Confucius thought that public opinion was smarter than "the stupidest."