EDIT: No comfort, but I am not the only one... Vogon related 'machine=vgaonly' thread
Darth gave me a hint toward thinking differently. It is using machine=vgaonly as the root cause of the problem. The initial game I made the custom config file was for Zool 2. The broken cracked version that is on site allows you to play, without intro and cutscene animations. Once I fixed that, the game did not run without setting up a virtual drive or using something like Daemon tools for a virtual drive. This was the main reason for the quest: I wanted to capture an AVI of the animated intro for Zool 2 in 320x200 and decrease the file size.
Trying to get a virtual drive to work with D-Box (not DOSBox) to test was causing more problems to get it to work than what I wanted to fix. However, in the process, I was using a modified version of the customized config for Zool 2. so all of the games I was adding new custom config files for had machine=vgaonly.
I don't know why, I would think it would do the opposite, but commenting out that line or changing it to machine=svga_s3 makes the captures turn out in 320x200 - 256 colors. That would be the last place I would look since I usually don't equate anything that states "vga only" with 24 bit colors.
My main goal is thwarted. Since the game for the AVI will only run if machine=vgaonly is in place, I guess I leave that one at 640x400, but at least I learned something. At least that's my disclaimer.
Thanks everyone!