Some further troubleshooting seems to rule out the HDTV, vga cable, and VGA port.
I can see the pc on a 19" LCD over vga.
I can see a pc image from my laptop on my Samsung tv over VGA
Both using the same cable.
The vertical scan rate input field is limited to just 5 characters. I can type 59943 or 59.93, but not 59.934
The Horizontal scan rate does not have this limitation, but cannot be changed manually.
In the System32 folder, there appears to be two versions of the utilities. I am not sure if one is for 32bit and one 64, but I have tried both.
CustomModeApp.exe and CustomModeApp2_0.exe
I have tried running them as an administrator, and in compatibility mode.