How I see it is that he knows certain people would be very frosty towards him, or worse.
The Wales family would not attend at all.
He knows just a few would be civil towards him - a very few.
He knows his presence would be a dampener on the event.
He is being realistic, regardless of what the King might wish.