It wouldn't surprise me either. Reminds me of the death of Queen Victoria. King Edward VII couldn't oust John Brown fast enough nor her Indian aide.
The Queen, up to her last day, didn't suffer fools gladly and she was, I believe, a keen judge of character. HM and Angela Kelly clicked from day one. If HM had caught onto something not kosher with Angela, she would have not kept Angela on for very long. I think that envy/jealousy among the Queen's staff may be the root and I'm sorry to say, Ms. Kelly was put down due to her upbringing and not belonging to the aristocracy.
I can see Charles being swayed by the late Queen's staff or he may have not liked her himself.