I have no issue with 4K HDR content with my Samsung QLED properly registering and displaying the correct format in Windows or Big Sur, it just works. However, you will have to mess with the gamma settings on your LG manually to get the picture looking normal and not washed out. For me, I had to adjust the BT profile 1 notch (to +1) and the gamma was fixed.
Try swapping HDMI ports on your R 580 as unfortunately there is no hack or option available in Mac OS to for forcing HDR. As well load up an HDR YouTube video and see if that triggers your TV to enable HDR. Safari supports 4K HDR natively now and I must say the playback performance puts Windows to shame.