The problem you're describing isn't a function of resolution. The fuzzyness is due to MilkDrop utilising a 256x256 texture to render the frames. Increasing the texture size would fix it, but I'm not sure if a Voodoo3 can do more than 256x256 (I run on a GeForce2 Pro, and anything above 512x512 starts to run slow)
The blockyness is a result of the grid size. MilkDrop calculates vectors for a number of points in the image (The grid) and then interpolates values between them, rather than having to calculate for every pixel. This is the main reason the plugin runs so fast. By increasing the grid size, more points in the image will actually get calculated (rather than approximated) and you should smoother curves and less blocky zooms.
The trade off is speed, of course, but I wouldn't run on anything less than max grid size if you can get away with it.