I've read that the fact that Lee was divorced was a minor problem in comparison to her being married to Stanislaw Radziwill who was a member of the Polish Govt. in exile, this being at the height of the Cold War and Poland being Communist plus a staunch satellite of the Soviet Union. Stanilaw's father Janusz had been imprisoned in Russia in 1939, released and then arrested again after the War.
Both Poland and the Soviets were squawking at any inclusion of Stanislaw and Lee being included at a Palace function and put the UK Govt. plus the Queen in an awkward position.