Wiimote-reactive Milkdrop visuals

[ORB 13]

Sweet!!! What a cool idea.

Don't have a dongle so tried it out on my laptop. Got some ideas of presets I can make.

[Flexi]

http://de.youtube.com/watch?v=0XriZLCxybs

[indaVizz]

fantastic! it looks like painted art. can you post this preset here?

btw you can also make both moving points ir-sensitive, one for the left and one for the right.

cheers

[Flexi]

IR fun for Two

[Saint Goody]

Here's another idea... might get a lil' tricky...

webcam motion detection...

[Flexi]

i know what you mean. i still have not soldered my IR LEDs, but be sure i will try some funny ideas.

[indaVizz]

hmm i think you should be able to do something with the latest miilkdrop release, using one wiimote to point towards you and a multi-LED IR source to illuminate you.

this guy, johnny chung, showed how you can use a wiimote to create a minority report-like effect: youtube.com/watch?v=0awjPUkBXOU

you point a wiimote and a source of IR towards you, and put some reflective tape on your fingers, now the wiimote that is still sees two moving IR points whose x,y coordinates it sends to the pc.

so with the v0.2.0 miilkdrop you can write a preset that reads positions of the two IR points by reading these vars:

these are subject to change:
---------------------------
wii_aux1: [0.0 | 1.0] visibility of the first IR dot as seen by the LEFT wiimote
wii_aux2: [0.0..1.0] X coordinate of the first IR dot as seen by the LEFT wiimote
wii_aux3: [0.0..1.0] Y coordinate of the first IR dot as seen by the LEFT wiimote
wii_aux4: [0.0..20.0] size of the first IR dot as seen by the LEFT wiimote
wii_aux{5..8}: same but for the second IR dot as seen by the LEFT wiimote

you can imagine you move your left hand more to the left and the preset zooms in faster (or maybe even you move it closer to the wiimote and it means more zoom), you move left and right fingers closer and it gets more blue etc.

i got this IR source from amazon: amazon.com/gp/product/B0009IB8RM though it may be a bit too strong.

the only problem is that the wiimote has a narrow viewing angle, around 40 degrees. i wonder if it would help to put a wide lens on it like the one you find in a door peephole.

also it may be more reliable to use a ping-pong ball covered in a reflective tape instead.

anybody wants to try it out?

[Flexi]

http://graffitiresearchlab.com/?page_id=6#video

[carmatic]

like, can you make it so that the IR will track more dots as they come into, and go out of, the field of view of the IR camera? in a demo-like situation, its much more feasible to surround yourself with IR sources than it is in the living room where the Wii's are normally used...
so instead of tracking 2 dots, it tracks a variable number of dots , or maybe just the 2 dots closest to the upper left corner of its view or something

also , how much of the wiimote usage aspect can be used in VJ mode? like where the viz goes on one big screen for the audience and theres a second screen with all the windows and stuff that the operator looks at, can the VJ mode be modified to show stuff like the raw numbers or rolling graphs?

[indaVizz]

well each wiimote has an IR camera and firmware in it that recognizes and tracks IR sources, up to 4. (right now only two are exposed in preset vars, but could do all 4.) but -- you can use two or more wiimotes and then track many IR dots.

you'd probably need some procedural code, i.e. change in milkdrop source or a dll to do something meaningful with all that info, it'd be hard to do it from a preset. (i would think, though i'm sure some presets do things ryan geiss never thought were possible. :-)

or, you could change milkdrop to get frames from a real IR webcam and do anything you want; the wii IR was just the easiest thing to do, not that i know how i'd do optical recognition anyway but it's all doable. if you have a killer app in mind we can do anything.

[indaVizz]

about VJ mode... i haven't thought of that, what i had in mind is a performer on stage who makes the visuals on the spot. but i guess the wiimotes could be used by a VJ as well. i don't have any experience with VJing, if you have some ideas go ahead.

btw here's a video i just made that shows how this could be used with a big screen for the audience at a larger event, it uses some stuff that's not been released yet. i think this one is the best so far, hope you guys like it.

youtube.com/watch?v=9P4Bs_lQOVU

[carmatic]

well i thought that the performer is effectively the VJ when it comes to being on stage in this case, or the performer works very closely with a VJ who does the normal milkdrop command stuff... either way i think that being able to have a feedback other than what the audience is seeing is a good idea...

[indaVizz]

that's right, the performer is the vj, and he changes presets using cursor buttons on the wiimote. the acceleration values are fairly unhelpful for direct reading, whatever the performer does with them or with the regular wiimote buttons (A and B) is just food for the preset code. i had those graphs before modifying milkdrop and would just stare at them, without getting any idea what i could do with them. only when i hooked the values up to preset code i was able to see what made sense.

as for preset changing... right now you have left and right cursor keys on the right wiimote for that, and also a button to randomize or order the preset list. but it's not enough: sometimes you know the song is picking up tempo or power or is more flowy etc. so you'd like to use a specific kind of preset.

so what i was thinking of is if there's some sort of sorted preset list text file in the preset directory, then miilkdrop would parse the list and then the up and down wiimote keys would switch between categories of presets (explosive, subtle, staccato, etc.) while left and right would switch within the category.

btw in the video above i hand picked a sequence of presets in advance that i thought fit the mood of the song.

...besides preset changing, what else does a real vj do today with milkdrop?

[carmatic]

this might sound abit more wiimote than milkdrop, but if you accidentally press the A and B buttons together and reset the connection between the wiimote and the computer, can the program detect that and automatically pick up the sync again?

also, i have a suggestion... instead of just pressing a button to change the preset, how about if you hold the button, then move the wiimote while the button is held down... have multiple combinations of moves, so that for example front-up switches to one preset, back-left switches to another preset, down-front-right is another preset, etc... and put large tresholds before it registers so its all about making 90 degree movements

[indaVizz]

i like that idea -- use not just buttons but gestures to select presets. actually i'd make a modification of it: use "analog buttons" of the wiimote to select presets. wiimote has no analog buttons but you simulate them in miilkdrop by pressing a button (on or off) and then twist the wiimote; the further you twist, the "harder" the button is pressed.

so maybe you hold a button and twist to the right, depending on how much you twist you get a more explosive preset in the list, you tilt the wiimote up and you get a more flowy preset etc. when you release the button.

resetting the controler: i don't think you can break the connection by pressing any buttons, and if you lost the connection for some reason you'd have to go back to the bluetooth stack software and reconnect. but i never had that happen, so i guess it's pretty safe for performance. we'll see. :-)

[carmatic]

no i mean like, if you press a+b at the same time, the wiimote's lights should start blinking and all that, which means that its searching for a bluetooth host or something, then you have to press a button on the wii in a certain amount of time to get them to connect to each other again

this might sound more milkdrop than wiimote, but is it possible to control the transitions? like instead of fully transitioning from one preset to the next in a fixed amount of time, the progress of the transition is determined by an input...
i think this could go very well with the wiimote twisting idea, for example if you put a pair of IR leds horizontally , the wiimote can use this to determine the horizontal angle you are holding it at... the more rotation you give the wiimote, the further the transition... also, you can have a 'key in a lock' effect if the transition is activated by 2 things, one is if the pair of leds are within a certain area of the IR camera, secondly when a large forward motion is detected, and both of these have to happen at the same time... so if you want to change the scene, you put the 'key' into the 'keyhole' and start 'turning' the key...

[indaVizz]

only if you press (1)+(2) the wiimote gets in discoverable mode, but it doesn't lose the connection (and you can't press those two accidentally).

transitions... interesting idea, i've always been using a hard cut, but maybe can use a wiimote pitch angle to control the duration. i'll play with that a bit.

i try to keep changes to milkdrop to a minimum, i'm dreaming that some day milkdrop 2.0 will be open sourced so i want to be able to port the wii stuff to it quickly. i'll test that theory when i start porting to redi jedi's beta...

[carmatic]

i am guessing that it is already possible to combine sound response with wiimote control? im thinking like, a basic effect is generated by the sound , and this effect fades away slowly ... the wiimotes will then control some other functions that act on the result of the sound, kind of like passing your hands through smoke or something
the advantage of this is that the visuals will be more responsive than you can ever be without knowing the music beforehand, the disadvantage is that you wont be the star of the show anymore

[indaVizz]

you're right on both counts.

but there's something else besides being the star of the show: i'm totally convinced that people don't want to see any "artificial intelligence" in a live performance. if the computer does something in reaction to sound, or does something randomly, people will not know if it's you the performer doing that or is it the computer, and they'll hate the confusion.

from their perspective, if the visuals totally match the mood of the song and the beat, we (the audience) want to be sure that's because you (the performer) are good and you gave your best. and if they don't, we'll know that you suck. but we don't want the computer to help you in any way.

that's my theory anyhow. i think that's not different from singing (don't you hate lip syncing) or playing a guitar live or dancing with fire or whatever -- it's about people and the way they connect through art.

[carmatic]

yeah, but i guess what im thinking of was more in terms of the versatility of the setup, imagine where the wiimotes are chained to a booth which the public have access to, that kind of thing

one thing i am pretty sure of is that in an interactive setting, the number of presets can be much much much lower because people dont get bored when they are playing with the visuals... which is a good thing i guess, since you have to account for both the original sound response of milkdrop as well as the wiimote input, more variables, complexity, and work for each preset
what combination of sound reactivity and wiimote control is totally up to the presets of course, but it can be presented as wiimote-focused or sound-focused.. to be sound focused can be, for example, the wiimotes can control the position of some points on the screen, and the sound waveform can then be centered on these points... if the waveforms fade slowly, it would be as if you are holding sound-reactive flames on-screen
i havent had time to set up all that bluetooth stuff on my computer yet, but im guessing that a serious wiimote-focused preset can have absolutely no sound input at all, and the functions which affect the waveform and fade are entirely derived from the speed, acceleration, etc etc of the wiimote input, or if there is sound reactivity, it is subtle and sublime stuff like the colour of the background or something... the kind of thing that would be used if there is going to be a dedicated performer on the booth

[indaVizz]

i see. that feature is supported already. normally (for the wii milkdrop) bass, bass_att and other sound preset vars are a function of wiimote acceleration, angles and buttons, so that when you don't move, these vars are 0, regardless of the music played in winamp. then winamp + miilkdrop is just a graphical engine.

but if you press certain wiimote key combo (left (A) + left (1)), those vars are reconnected back to sound analysis for the music played in winamp, so an unmodified preset behaves as in the original milkdrop. but the wii-specific vars are still available (wii_l_f etc.), so you could write a preset that does exactly what you described -- use sound vars and waveforms for subtle reaction and wii-only vars for not-so-subtle reaction.

... i could also have each preset decide whether sound vars source should be winamp sound or wiimotes, by having it define fWiiAnalyzeSound=1 (default 0).

[carmatic]

yaay just had a go with milkdrop, it was as simple as plug and play on my xp x64, plug my belkin dongle in, wait for windows to install the drivers, sync the wiimote, and when i started the test program it recognized it straight away... makes me sleep easy at night when stuff like that happens

anyway, i have an idea... if there is a way to control the sensitivity of milkdrop's sound reactivity using the wiimote, you could 'reward' the participant with a large reaction on the visuals if they get it right, dance dance revolution-style...
or how about 'transforming' the input of the wiimotes to different frequencies of audio, like long sweeps (large constant acceleration) create high frequency sounds and sudden jerks (large change in acceleration) cause pulses of bass... make it so that when the wiimotes are still, there will be constant white noise, and as the wiimotes feel acceleration, the tone of the sound changes... this can either be fed into the visuals as it is, or mixed with the audio stream for some 'relevance' to the visuals

i dont know how relevant this is, but when i was browsing wikipedia i came across http://en.wikipedia.org/wiki/Linear_response_function and somehow it reminded me of this thread, can anyone think of anything?

[Flexi]

hehe i didn't know this formula has a well known name, but in fact i used it in quite alot presets (all the ones, that take advance of the elastic spring and some more)

rewarding is definetly the way to go, when you want to get serious with it. (i just got 48 points on the wii sports sandbag)
If i find something cool, i will post it here! but don't you wait on me, my creativity is unpredictable

[carmatic]

like, how difficult is it to make the wiimotes control some kind of 'noise generator' like i described above? the noise will then be fed into the visualization and processed as if it is the music...
in addition to amplifying certain frequencies depending on acceleration, maybe the x-axis wiimote acceleration can change the phase of the affected frequencies as well, the y-axis wiimote acceleration can change the y-axis offset of the waveform, and z-axis acceleration can change the bandwidth of the processing, so you can have direct control over the waveform using the wiimotes...you can 'roll' the waveform around by moving the wiimote side to side, you can 'push' the waveform up and down, and you can create different interference patterns by moving the wiimote forwards or backwards... this potentially makes unmodified milkdrop presets much much much more useful, and we can use all 5542 of them

[Abydos]

I love you.
I'm buying a 1080 hd projector soon and just happened to stumble acroos this, I'm going to have soooo much more fun now, cheers!

[carmatic]

like, what i think could use some work is like, when the wiimote is moved, the bass waveform is identical and repeating, which looks kind of unnatural, that is why i was thinking of using the white noise... or if the sound variables can be reconnected back to the sound input, but not directly, like if the wiimote actually controls some kind of DSP using its movements, and the resulting processed sound is fed into the sound variables... that way you get a good waveform, plus you get a good response from the wiimote too

which sound variables are controlled by which wiimote variable?

[liam9519]

Is it just me or is the 0.2.0 link not working? Really would like to try it out!

Keep up the good work!
liam

[Flexi]

right, but the 0.1.0 downloads works...

http://rapidshare.com/files/133826115/vis_miilk.dll
simply replace this file - that's all it needs for the IR support

[liam9519]

ah cheers

[ShadowHarlequin]

can you get an attatchment usb or otherwise that will give my laptop bluetooth capability? it has a switch on the front but its real stiff and theres no software installed for it so i guess my model doesnt have bluetooth

also i cant find a dl link for the software, the download page isnt there any more
[edit, scrolled down on the page and got it :P]

what parameters can you control with the wiimote? could you control any variable?

[Flexi]

yes you can control any variable. you can either use the raw force values or roll/pitch(+yaw with the infrared bar) + some button press status.

i bought a usb BT dongle for my PC but not all are working with the wiimotes. It's all written in the Miilkdrop readme. check it out

[ShadowHarlequin]

cool, i cant afford any hardware at the moment though :P lookng into using a gamepad with this instead for the moment which might work

[Flexi]

by private request, a reupload of the Miilkdrop dll + documentation and preset pack