Known issues
There is a known issue with NVIDIA GPUs in combination with the Full Screen video codec, H.265 encoding and resolutions higher than 4096x4096.
Refer here for more information.
As a work around, use the default Use video codec for compression” graphics policy setting or explicitly set the policy to either “Let the system decide” or for “Actively changing regions.