After officially stepping away from their senior royal duties in April, the Sussexes have plenty of realistic reasons for not seeing the Queen—the major one being that they now live in California. But now Prince Harry and Meghan Markle might cancel visits to Queen Elizabeth that were previously scheduled to take place this summer for another reason altogether.