Aspect Ratio for Preset Sampler Images

[NeonAngel]

I have a problem trying to get sampler images used in presets to display without any stretching. In the example below (see .milk file attached), the image displays correctly if it is a SQUARE image - so it will just touch each side, with no distortion, and the top and bottom are correctly cropped. (see first image - the white area illustrates what screen will look like in widescreen). Perfect.



If it is a wide or tall image though, it will be horribly distorted and will only look perfect when the image's aspect ratio matches EXACTLY the screen's aspect ratio. What am I doing wrong?

warp_1=`sampler sampler_mandala;
warp_2=`
warp_3=`shader_body
warp_4=`{
warp_5=` // sample previous frame
warp_6=` ret = tex2D( sampler_mandala, uv);
warp_7=`}
comp_1=`shader_body
comp_2=`{
comp_3=` ret = tex2D(sampler_main, (uv-0.5)*aspect.xy+0.5).xyz; //Works OK only if image ratio is SQUARE
comp_4=` ret *= 1.00; //gamma
comp_5=`}

Image is very distorted on a square screen:


Original tall image - the white area shows what it should look like on a square screen:


Widescreen image - the white area shows what it should look like on a square screen:


I am busy preparing a comprehensive set of sampler presets for a new MilkDrop based VJ application called NestDrop. It will also include a new sprite function that is going surprise quite a few people! 8196x8196 at 60fps x 4 screens is possible - all with full spout output to any compatible application - and free!

[Nitorami]

Read the milkdrop preset authoring guide to understand how to map Images to the screen.
Btw you better sample your image in the comp shader, unless you have a specific reason to take the detour via the warp shader.

[NeonAngel]

Apologies for previous post - for some reason the images didn't show up. So i'll have another go...

In the first attached image (square green mandala), the left side is the original image - the right shows what it currently looks like in widescreen output. It's perfect as it's centred, not distorted, and just touches the edges. The code in the previous post works great here.

In the second attached image, however, (tall huichol mask), the left side shows the original tall image, but instead of keeping the correct proportions, it will be horribly stretched and distorted horizontally in widescreen as shown on the right. The original image will actually only look perfect when the image's aspect ratio matches EXACTLY the screen's aspect ratio.

Similarly for the third attached image (bosch painting), again with the original on the left, when in widescreen it now stretched vertically as shown on the right. Also just like the second example, the original image will actually only look perfect when the image's aspect ratio matches EXACTLY the screen's aspect ratio.

[Nitorami]

Just read! the! preset! authoring! guide!

[NeonAngel]

Thanks Nitorami. I am collaborating on a project where I am gathering every preset I can find that uses sampler images, around 40-50 different ones in total, and then setting about changing the images and making some minor code tweaks, usually just by trial and error. Otherwise I have no skills at all in preset authoring and find the code extremely difficult to understand. Nonetheless, the results I am getting are beautiful, it's only this issue that's causing a problem for some of the images and presets.

I have read the preset authoring guide re aspect ratios, and managed to make the change using 'aspect.xy', but it only works for square images. The only other solution I can think of is to convert the tall or wide images to square-shaped, so they stay in the correct proportions whatever the screen ratio. Any ideas how to change the code in the example to solve this?

By the way, you may be interested in work I am also doing on sprites, which don't seem to have had much attention over the years. I am about to release a package based around high-quality, high-resolution images and a reworked 'milk2_img.ini' file. Normally these images would be impossible to use as presets usually freeze for long periods of time as the images are loaded into memory. A programmer I am collaborating with on a project called NestDrop has come up with a solution so this is no longer an issue. The results are absolutely amazing and will give Milkdrop a new lease of life. This is an almost untapped area of Milkdrop and I have set about to change that.

[Nitorami]

Hi NeonAngel
I myself have never dealt with sprites, and am curious what you will come up with.

Nestdrop was announced recently on this forum in a different thread.

For your texture mapping:
You don't need the warp shader. Do it in the comp shader directly. The comp shader's "ret" is effectivels what you'll see on the screen.
What you need is the aspect ratio of your image; just define it above the shader body, and then use it for sampling. Here is the relevant part of the authoring guide:

If your shader wants to know how big the texture is, declare this
(also above the shader_body section):

float4 texsize_billy; // .xy = (w,h); .zw = (1/w, 1/h)

MilkDrop will see the "texsize_" prefix and automatically know what
to do. (You don't have to include the //comment, of course.)

To stretch this texture to cover the screen, do this (in the shader
body):

ret = tex2D(sampler_billy, uv).xyz;

Or to map it fitted to the screen, aspect-aware:

ret = tex2D(sampler_billy, uv * aspect.xy).xyz;

Or to tile it so the pixels are represented 1:1:

ret = tex2D(sampler_billy, uv * texsize.xy * texsize_billy.zw).xyz;

Or to map it tiled exactly 5 times:

ret = tex2D(sampler_billy, uv * 5).xyz;

Or to zoom into the center 20% of the image:

ret = tex2D(sampler_billy, (uv-0.5)*0.2 + 0.5 ).xyz;

Of course, you could also declare sampler_pw_billy, to do point
sampling, or sampler_fc_billy, for clamping, and so on.

[NeonAngel]

Thank you the references. I already read them over and over and tried various things before joining the forum, but description of what should happen when using 'aspect.xy' in the statement "ret = tex2D(sampler_main, (uv-0.5)*aspect.xy+0.5).xyz;" does not seem to match what the documentation promises, ie.

"The overall design goal in MilkDrop, concerning aspect ratio, is to fit the preset to the long axis of the window, and to crop the rest, but to do all of this without any stretching or zooming (so all internal canvas pixels map 1:1 to screen pixels).
-per-frame/per-vertex equations:
* multiply XY coords by the values "aspectx" and "aspecty", respectively.
-shader code:
* multiply UV coordinates by 'aspect.xy', prior to using them
to sample a texture, to make the texture fit on the screen properly.
(For example, if the screen is wide, the image will be fitted to cover
the width of the screen, and it will be cropped at the top and bottom.)"

It works perfectly if the source image is square, whatever the output screen ratio, but not for tall or wide images - they are still stretched to fit as if 'aspect.xy' wasn't there. This leads me to suspect there is a bug in the MilkDrop code itself that needs correcting. I am having that checked out.

If I'm wrong, any ideas how to correct the very simple preset I initially attached Nitorami, so it will work with 'texsize.xy' or whatever, for both tall and wide images? I would be very grateful if you could show me how I should change the code in the file.

By the way Nitorami you asked abount the sprites I was working on. I have attached four examples of the sort of things possible with any MilkDrop-based installation. The poor sprite has not had a lot of attention at all it seems. (Not you Prenso!). I 'discovered' it just a year ago - tried a few images, changed the code around a bit and I was staggered at how fabulous it looked in 4K/60fps. Checked out this forum and YouTube and so on - not a great deal there at all. So I set about trying to get the best out of it myself with high-resolution imagery all alone over several months. Just by luck eventually I was introduced to a programmer who happened to be buliding a GUI over MilkDrop, with high-res spout output. I transferred everything I learnt to the new system. We have now worked on the use of sprites to a point where their full potential is being realised.

If you want to know more I wil give more details of how to use Sprites. The message function has also been modernised using some great OFL fonts.

[Nitorami]

NeonAngel
Listen. This is not a bug. Aspect is just the screen aspect, not the image's.

uv goes from (0;0), top left, to (1;1) bottom right. tex2D(sampler_image, uv) will stretch any image to the entire screen.

Multiplying uv by aspect.xy (which is the screen aspect, i.e. 16:9) just means that your image will be squeezed accordingly; a square image will then not be stretched over the entire screen but appear in correct proportions. But ONLY a square image !

You need to use the proportions of your IMAGE to scale it. This is what matters.

See the attached example. Hope that helps.

[NeonAngel]

OK I've used your code in another preset by Zylot (attached), using the image of the tall mask (first image attached).

Now the tall aspect ratio of the image is correctly transferred to the preset as you can see in the second attached screenshot image. It is tiled of course, so the screen is filled. As you suggested tiling can be overridden - I disabled that as it leaves much of the screen blank. Anyway this looks really good. Great so far and I will use this technique for some preset modifications thank you.

However it's not quite what I was first asking about. The third attached image shows what the documentation I quoted suggests can happen - ie regardless of screen ratio, the image will be fitted so it:

"fits the preset to the long axis of the window, and to crop the rest, but to do all of this without any stretching or zooming".

Is this actually possible for anything that isn't square-shaped - can you show me this please? This is an important technique for me to learn, as the image regardless of dimensions, will be centred automatically, and fitted to width (or height) automatically, allowing randomly selected images to fit well without any tailoring. Please remember I have very little knowledge of preset authoring.

[Nitorami]

Bit tricky. Like that ?

[NeonAngel]

Thank you so much Nitorami. Apologies for not answering sooner, but I wanted to check things out thoroughly before replying and have been short of time.

I have used your example with every combination of screen ratio and image ratio - 18 in all! First I used the code in the solution you sent me and it looked fantastic, with all types of image and screen - always centred, with any necessary tiling at 50% brightness to emphasise the main image in the centre. Scaling was set by the following code:

scale = texsize_DeepSpace_FineArt_GardenOfEarthlyDelights_1505_HieronymusBosch_PUBD.zw*texsize.xy * min(sz.x,sz.y);

However, it still wasn't exactly what I asked for, ie image fitted in centre with no distortion or tiling, so I looked at your code and thought what happens if I change min(sz.x,sz.y) to max(sz.x,sz.y) - and it did exactly that! So now I have two great solutions.

To demononstrate I have attached three images showing each main screen ratio - square, wide and tall, and what happens if a square, wide, or tall image is used with either min or max in the scaling equation for each screen ratio.

In all cases the results are controlled and look good. So, I'd say the problem is solved. Thank you so much!

PS There is no way I could have worked this out from the manual - it's way too difficult for me to understand.

[Nitorami]

Hi NeonAngel
Good to hear it works now. I don't know why you used "min"; my original code uses "max (sz.x,sz.y)".

I can't pull code like that out of the hat either, but it is not difficult; although some trial and error will certainly be required.
Scaling the image aspect correctly is easy, but that only means the image will be displayed undistorted, not that it will also fit the screen. That would only be the case for images with exactly the same aspect ratio as the screen.

So in a next step we need to scale the image, depending on whether it is too tall or too high. We need a decision, and that is done by max() which simply selects the axis with the largest misfit, and then scales both axes accordingly, maintaining the already corrected aspect.

[NeonAngel]

Yes I got max/min wrong way round, but didn't notice until after posting. I am now going to rework the presets using images I've been working on using this code. A while ago I searched for every preset I could find that already used sampler images (60 or so) and set about exchanging the source images and tweaking the code. It's quite possible to use quite high resolution images nowadays, even 2048 pixels or more without too many loading delays. The use of very high quality images makes a lot of difference I've found, especially at 4K or higher. I am currently able to run Milkdrop presets at more than 12,000x12,000 pixels at up to 60 fps believe it or not and send it all to Spout-enabled systems such as Resolume, so it can be mixed with other content as well.

[NeonAngel]

Well Nitorami I hope you are keeping well. Just to let you know I did finally complete the work on those presets and it turned out pretty good I think. I searched through 10,000's of examples to pick out those that directly reference images - many of which are yours it seems - and carefully worked on them as best as I could.

I have created a new thread in the 'Presets' section and provided three links to new videos which demonstrate the results. I hope you like them.

By the way, it was quite a technical feat to get full 2K quality on YouTube. My computer was not capable of compressing such resolution to MP4 in real-time, so I had to record it losslessly (at 10Gb/Minute) using OBS, then compress it later to 1GB/Minute. It looked almost identical to MilkDrop running in real-time, although unfortunately there are occasional light shudders. I also found running MilkDrop on one disk drive and recording OBS to a separate drive help reduce any glitches.