The Oslo Setup team looked into this and had a couple of suggestions.
First, try a passive install as described in the readme:
OsloSetup.exe install /passive /ae
You can also try disabling "Accelerate 3D Graphics" as described in the thread http://communities.vmware.com/message/1184328?tstart=0. That is, disable DirectX support in VMWare.
Note also that if the UI in the setup program doesn't work right, then it's possible that "Intellipad" and "Quadrant" won't either, so disabling DirectX is probably a good thing in any case.
Let us know if either of these options work.
.Kraig