I don't know about that video problem. The codec used by DOSBox doesn't modify the video, it just zips it. Try re-installing it, it can be done from a shortcut in the DOSBox folder in the Start menu.
It may be due to scaler settings, how come the size is 668x418? What game is it? If you take screenshots instead of video, are they OK or do they look the same? What size are the screenshots? Try running DOSBox with its default settings.
For merging, clipping, etc. I've used the free edition of this program in the past:
http://www.microsoft.com/expression/..._Overview.aspx
It's better than Movie Maker in that it allows you to control resolution, scaling, stretching, letterboxing etc. precisely.