After a lot of tries, finally installed MacOS Sonoma (on Ryzen 7 5800X3D and RTX 4080). MacOS doesn't feel very smooth (feels like slow old computer), but its usable.
1. Downloaded files provided in original post
2. Installed VMware Player 17.0.1
3. Configured VM as mentioned in original post (with Opencore 4 cores), and added 8gb ram (16gb didn't make a difference)
also added these 2 lines to the .vmx file (felt like it made performance bit better but maybe no difference):
ulm.disableMitigations = "TRUE"
smc.version = "0"
4. Started the VM installation - First you will see white text going, that's all good, wait for a while (for me 30min to 1h) even if the text is the same and looks like it's going nowhere.
Then you will see apple logo and then options to access disk utility etc, erase the disk and start MacOS install. White text will go again for quite a while, just leave it. After some time I noticed that setup changes and displays 4 icons (opencore, your disk, install Sonoma, Recovery), first time I didn't manage to click anything and it continued with the white text. After some time again those 4 icons appeared, I clicked on my disk/partition (called MacOS), then again white text and I guess I got to the next step of installation process (GUI interface etc.).
Also one time when I first saw those 4 icons I disconnected the .iso and still finished the installation. So just let the installation process run for good time.
Tried to install system updates, first 2 vm's got broken, 3rd installed updates successfully.
Next thing I noticed that installation process is like 100x faster using Opencore 4 cores compared to Opencore 8 cores (which comes with unlocker).
When I was first testing installation it didn't work with 4 core .vmdk, it only worked with 8 core. Idk why. Then again it didn't work at all, then it worked with 4 core, etc. I don't know why there was such inconsistency.
Anyway, installed everything with 4 core, tried to shut down and replace to 8 core (also changed vm settings to 8 core) but it looks like it will not work as the boot is taking now more than 1h already and it's slow. (attached screenshot mby that helps)