Geeze, finally figured it out.
It was the HDR setting from the DP cable. I must have enabled it when setting up the new PC. As soon as I disabled it, everything worked fine. Side note, using an HDMI cable instead of the DP cable fixes this, as it automatically disables HDR.
Thanks for your help, that was a beast to figure out.