Kai,
I believe you're using F/X CAD 2020. It looks like other AutoCAD users have had similar issues before, and there are a few possible causes and solutions for this.
1. Here's an AutoCAD documentation that suggests the PDF could be placed too far from the origin: PDF Underlay keeps shifting in AutoCAD
You can try an easier fix for this using Land F/X's UCS tool to set a closer origin to your plan: Creating a User Coordinate System (UCS)
2. Some of them resolved the PDF underlay shifting positions while zooming issue by either reverting or updating their graphics driver. You can try that: PDF shifting limits during zooming
3. Another user resolved the same issue by unloading and reloading a few times. I might suggest also trying to restart your computer in there: PDF's Jumping when Xref'd
Try those out and let me know if any of them also help you. If not, we'll need to go to a ticket where you can send me the drawing and PDF to help troubleshoot further. How to Submit a Technical Support Ticket
-Amanda