[The Fifth Horseman]

I think I'd go with an AMD 486 DX4/100, perhaps slightly overclocked, on a Shuttle HOT-433 motherboard. (Incidentally, this happens to be the very setup of my first PC, but the mobo broke in 2004). Though, I'd try and find replacement cache modules to expand the motherboard's cache size from 256 Kb to full 1 MB.
50 ns EDO modules for RAM memory, 16 MB total.
For soundcard, I'd probably go with something for ISA slot and compatibile with Sound Blaster PRO 2. Two CD-Audio connectors would be welcome.
VGA can be about any PCI graphics card with at least 1 MB of memory, altough I'd prefer a 4 MB one if possible.
Everything in a midi-size Tower case. The mini-size ones are horribly cramped.

Beyond that, I'd go for an A4 Tech 4d++ mouse (combo-type COM/PS2 one) rather than a regular one (it's much more ergonomic).
A PCI USB controller may not be a bad idea, since there are USB drivers for DOS and it would help a lot with transferring stuff to and from the computer, and a CD-R/RW drive is another option worth considering.

I'd probably take two identical Western Digital drives from their Caviar line (exact size is not specifically important, I'd say about 1 GB each will be sufficient), both set to Master and attached to separate IDE channels in drive racks so they can be swapped - it's essentially a hardware-based method of dual-booting.
Though, I'm tempted to experiment with ramdrives and SSD modules, the latter being a particularly tempting idea. Considering they offer read and write speeds much beyond what the old HDDs were capable of, that would be a major speed boost for the entire system.

After that, I'd go for the following modifications:
Replacing the regular ATA, COM and LPT cables with slightly modified ones (re-shaped to take up less space inside the case)
Placing noise reduction mats into the case (doable, done that before in fact, but requires some extra work cutting everything to fit)
Installing a PC-Speaker-to-CD-Audio adapter (done that before)
Replacing the PSU fan with a ball-bearing one to reduce the noise.
Replacing entire CPU heat sink with a passive one to further reduce the noise (perhaps a modded sink taken out of an old PII).

I think that would essentially be it.