Wiimote-reactive Milkdrop visuals

[indaVizz]

Hello developers, users, preset artists,

I just released the first version version of a Milkdrop mod that I think you'll like. I modified Milkdrop so that all the preset graphics is drawn only in response to you moving two Wiimotes in your hands. Move your right hand to the right, for example, and the fireball on the screen glows more blue. Move it faster forward, and it becomes hot white and zoomed in closer.

If you do it to the rhythm of the music (whether played by Winamp or from some other source -- an iPod in your room or a DJ in the nightclub), and if you hook up a projector to your PC, the feeling of the feedback when you flick the Wiimote and see the preset/Milkdrop instantly react is spectacular.

Here's the best part: it uses many existing Milkdrop presets unchanged. I disabled all sound analysis in Milkdrop, and replaced it with Wiimote readings. So the "bass" variable that a preset reads is really the acceleration on the X axis of the right Wiimote, "bass_att" is the X acceleration of the left Wiimote, "treb_att" is Z acceleration of left and right combined, and so on. In other words the stronger you move the Wiimote on the X axis, the stronger the "bass" the preset thinks is, and some shape on the screen gets bigger or greener or whatever in response.

I put on YouTube a demo of the software in action at

     http://youtube.com/watch?v=bCVF6EZFR3g

-- I'm still struggling to find the best way to describe this but I hope the video will make it clear. And fun, too.

You can download the software (predictably named Miilkdrop, packaged as a Winamp vis plugin) and setup instructions, from http://indaviz.com/download/. It's free, and I'll post the source code as well, once I clean it up.

I included in the installation hundreds of fantastic presets that react well to Wiimotes. I also modified one of them (Zylot's "Wisps") to directly read some new Wii-related preset variables I introduced.

There's other software out there that connects Wiimotes to visuals, for example via MIDI which a VJ-ing program reads. But everything I've seen looked too slow: for the feedback to feel good, both for you and the people watching, the image must be drawn no later than around 30-40ms (roughly the same time as one frame of video) after you move the Wiimote. Milkdrop is a speed demon that's up to the task.

Thanks for reading this, if you'd like to give it a shot I'd appreciate any feedback (and/or Wii-customized presets that I'd include in the next release).

Cheers,
Davor/indaVizz

[indaVizz]

... just noticed the URLs didn't come through, so here they are in the raw form:

YouTube video: youtube.com/watch?v=bCVF6EZFR3g

Download page: indaviz.com/download

[redi jedi]

I SOOOO have to check this out when I get home, Hope my bluetooth is compatable...

which version of milkdrop did you make this from?
1.04
1.04d
or on of the 1.05 betas?

I'm assuming its not 2 since I dont think the source is out there...

[indaVizz]

It's 1.04b I think, the only source I could find. (I SOOOO wish I had 2... :-) But even the 1.04 source is priceless.)

As for USB bluetooth, I can recommend the one I got from SparkFun Electronics for $17, link below. It worked for me on 3 different laptops, two of which had bluetooth built in but built-ins rarely work with Wiimotes. The SparkFun one was also recommended by a well-known Wiimote hacker (well known in the Wiimote hacking circles anyway), Johnny Chung.

You'll also definitely want two Wiimotes. Circuit City and the like usually have plenty in stock for $40 apiece even though you can't buy the Wii console itself. (I don't have one either.)

wiili.org/index.php/Compatible_Bluetooth_Devices
sparkfun.com/commerce/product_info.php?products_id=150

[redi jedi]

hey thats pretty fun... what are the available variables from the wii?

here's a quick remix of your test preset...
the left wiimote controls the zoom and the rot, tilt it to the sides to rotate and tilt foward and back to zoom, right one does the same as it use to, but for both "dots"

[indaVizz]

This is great!! I love it how you can change the depth of the tunnel when you want to, and then send the fireballs down so they "fall" longer. The feedback is awesome, the best so far.

Here's the list of all the variables:
indaviz.com/download/miilkdrop.html#8

wii_l_x, for example, is the acceleration on the X axis of the left Wiimote. The X axis is not in space but relative to the Wiimote -- it's in the direction and orientation of the left cursor button on the Wiimote. It's range is -3.0 to +3.0, close to 0.0 if the Wiimote is not moving (on its X axis). The Z axis is in the direction of the face of the Wiimote, and the Y axis is the direction of the up cursor button.

The roll and pitch (wii_*_r and wii_*_p) are the only ones that don't drop to 0 when you hold the Wiimote still so they can be used to change something slowly, like this depth and rotation of the Wisps' tunnel.

Were you able to connect the hardware already?

[indaVizz]

... here's a video of the remixed preset:

youtube.com/watch?v=AVqtB0tJkuo

[redi jedi]

ya hardware hook up just fine, already had a bluetooth adapter and the wiimotes... prolly have some more remixes for ya soon...

[Flexi]

holy crap! this is awesome!
and hey i've got all the necessary hardware (beamer, wiimotes, BT dongle)
and as you crawled my mind, about two weeks ago i've been to Berlin where i discussed this idea with some friends who want to organize a demo-party.
Someone has to call Ryan Geiss for the Milkdrop2 source :D. I want to dive into brilliant shader fractals with my wiimotes.
okay this is dreams of the future, but imagine a headtracking implementation..

:up: ...something very cool is going on...

[indaVizz]

I just saw 1.05 beta on redi jedi's site... don't know how I missed it. I'll take it and Wiify it, probably in a week or so.

As for a party, you can imagine 5-6 people holding a Wiimote each and each controlling a different part of the preset. Each would be in charge of visualizing a different instrument or vocals in the song, kind of like playing in a visual orchestra.

(Would be easy to make, Miilkdrop-wise, too, I'd just need to add vars wii_1_x, wii_2_x, wii_3_x etc, and you'd need a good preset to show it all well.)

[Flexi]

oh no, i can't find my bluetooth dongle - i fear this is the moment, where i have to tidy up my room :hang:

[indaVizz]

redi jedi,

here's a minor edit to show the use of the "A" buttons on wiimotes for effects ("B" buttons are for preset navigation):

replace
rot=-wii_l_r*wii_l_f
with
rot=-wii_l_r*wii_l_f*(1-wii_l_b)

so that if the left button (A) is pressed, it blocks the rotation. (i also added *.3 to reduce sensitivity.) wii_l_b is either 0.0 or 1.0.

and also replace
zoom=.9+sin(wii_l_p)*.1
with
zoom=.9+sin(wii_l_p*(1-wii_r_b))*.1
so that the right A button blocks the zoom.

i like it how it looks when i start with small visual changes and then expand as the music goes, using the most dramatic effects sparingly. in this case it's start with both zoom and rot blocked, just activating one dot, than another, and so on until at one point everything explodes at once.

thanks again btw, this remix is fantastic.

davor/indaVizz

[Flexi]

i borrowed my dongle to a friend and will get it back next weekend. say what's the physical maximum number of wiimotes you can connect to a pc resp. milkdrop?
My hypothesis: if you average all input of a serious number of 'pilots' and map it on the right selection of visualization, some very strange psychic effects will appear.
You know the Ideomotor effect? ( http://en.wikipedia.org/wiki/Ideomotor_effect )
I hope, steering the motion will end up in flowing and surfing the viz and finally in a perfect mindfuck. :D at least it's worth a try :p

[indaVizz]

a friend of mine suggested exact same thing...

i read somewhere that the maximum number of wiimotes per PC is 7 (some master-slave thing, part of the protocol). but there's nothing that says that you can't have another PC that reads 7 more, and so on, and they can all send updates the one running miilkdrop over the network. the averaging could be done by miilkdrop and given to presets as input vars (wii_avg_x etc.)

what would the preset look like?

[Flexi]

i would really love to see a trip through one of the chaotic "skin dot" presets, but... there's no go with Milkdrop 1.04/1.05.
I guess a really fast ride into a starfield should work fine too.
i will do my own experiments and give feedback!

[ryan]

Wiitards.

:p

[Flexi]

setup successfully completed!

Biiig Thanks for bringing me to this point, it's like a litte dream come true.
Now let's hack some stunning presets! :igor: :winamp: :up:

(@ryan: :blah: )

[Flexi]

oh i really forgot about the poor image quality of Milkdrop 1.05 :weird:
(i want pixelshader :cry: )

i think, i need the acceleration values inclusive gravity. i can do the smoothing and roll pitch calculation in milkdrop code as well.
and may it be possible, that the number of variables shrunk? i just tried to remix an old preset, which already used the maximum number. (but hey - i remember, this number was raised with the 1.05beta ;))

here are first results. this is fun :p

edit:
it's probably not the best preset for dancing action.i tried a really soft dampening - inclusive overshooting :p - so all peaks are filtered out, but if you're lucky you get into resonance with the loop controller. you might edit the first three vars...)

[Flexi]

man this is insane...
...press A ;)

this preset obviously uses the maximum amount of variables. it's only 12 of em? thats far to few for most presets...

[indaVizz]

this is like you're navigating a spaceship through a field of acid shapes. the colors are incredible.

here's a remix that includes beat reaction: the right wiimote paints shapes, the stronger the move the bigger the shapes, while the left wiimote navigates the "spaceship" as before, plus any force applied to it makes it move faster. (i.e. zoom depends on wii_l_f). left and right buttons are for two speeds (couldn't think of anything better).

youtube.com/watch?v=u78m9TfL9l4

[Flexi]

still practicing.
this is the code i would rather see in the per-frame section.

enjoy!:p

[indaVizz]

the physics of it is awesome. it feels like you're moving a live worm with the wiimotes.

can you add more direct feedback to it but still keep the lifelikeness of it? e.g. you squish it with one wiimote acceleration and stretch with another, but it's standing still when you're not moving the wiimotes? (almost still, it would be good that it still wiggles a bit but very little.) the roll and pitch of one wiimote could control its angle or how close the two ends are.

what i'm thinking is, it would be cool to make the feedback so that you always know what action your arm movement is going to make (like you do with your real arms and hands).

[indaVizz]

here's a wii remix of Rovastar's Eye on Reality mix. i just copied what redi jedi did with the wisps wii remix: left wiimote roll controls rotation, pitch controls zoom, right wiimote force (acceleration) controls the size and colors of the eye, and left wiimote force does something that i don't know what it is but looks really cool. buttons supress roll/pitch->rot/zoom.

it has great feedback (the only thing i talk about) but i think the colors and shapes are great too (that came with the package). hope you like it!

davor

p.s. i'll base the new release off of latest MD beta; let me know if you think some other vars should be supported. (i was planning to add wii_{1..7}_* values for up-to-7 wiimote support.) as for gravity-including x, y, and z acceleration, you can derive them from wii_{l,r}_{x,y,z} and roll/pitch:

wii_l_gx = wii_l_x + sin( wii_l_r );
wii_l_gy = wii_l_y - sin( wii_l_p );
wii_l_gz = wii_l_z + cos( wii_l_p );

and same thing for wii_r_*. the gx, gy, gz are the acceleration values as measured by the wiimote directly and they are most accurate. roll and pitch is calculated from there (and not quite accurate when you move the wiimotes too fast), and so are x, y, z gravity-less accelerations.

[Flexi]

no way, that roll and pitch values have most accuracy jet.
see how the the other value freaks out, when one axis is rotated by 90°. place your wiimote on a table and roll it from left to right.
(red is pitch and blue is roll)
can't you pass through the acceleration values right before gravity is subtracted, or am i totally off the mark?

and another thing:
what an effort would it be to give me input from the IR sensor? i read "Wiimote detects and transfers up to four IR hotspots" (just imagine, all i need for headtracking is two points;))

[redi jedi]

Gotta check these out when I get home... damn this is so cool.

hey is there any way to get the litebar supported so we can get exact positional data, and or mabye a point that the user is point the mote at? that could be fun..

hey indaVizz, are you any good with c#? I don't care much for the fact that I dont have the V2 source(maybe saying this will help with that) so I've decided to rebuild it in C#.. mainly cause my new job is c#/vb6 based so I need the practice.. but once i get the md features implemented I'm prolly gonna switch the preset format(xml based most likely) or at least extend it alot... can you do the same thing in c#?!?

btw.. using Lua as the scripting engin(current md format executes fine) so the extendability is gonna be wicked(built in strings/arrays/classes anyone?)

[Flexi]

:D i have just been to our electronics store and buyed some IR-LEDs and a button cell.
Who's faster - me organizing a soldering gun or you implementing? ;)

this will get into serious performance art!

@jedi redi: :up: nice to hear. I have quite a lot experience with Java and need C# practice as well. Say if you need a helping hand.

[redi jedi]

helping hands are always good..

I'm thinking that at least up to being md2 feature comparable(I don't expect to match the speed) it will be open source, then I might break it off into another project(leaving the source still available for others of course) for further development but I may start totally over at that point to, I'm not sure(nor do i care right now) cause I have some other ideas I want to explore after gaining the experience of this.

But anyway I'm trying to make it fully OOP so I can reuse parts of it in other projects, right now I'm mostly doing the design parts, are you any good at class design?

@indaVizz: btw thanks for reintegrating it into my beta, thats cool.. Few requests for the next version:
if you have any more variables from the controller, expose them.

put the bass,mid,and treb back, or add a var we can set that say to turn them off, or a menu option, something..

up the min eval user variables(think its in one of the header files), are you adding to that when you add your variables? as its just a total, not the avail custom variables. its needs to be 30 over all the built in ones for md1.04 and 40-45+ for betas.

[Flexi]

I'm studying computer science major in software-technics ;) UML is my milk (or should be :rolleyes: ), but i must admit i have not worked too much with it yet. I'm also quite proficient in XML (but again more in an academic manner).
Re-engineering and model-based software engineering are exactly my topics. :p

this is curious: I'm still waiting for my processor for my new state-of-the-art PC and i also resolved to finally switch to Vista and going into demo-stuff like wiimotes and DX10.
I'm a little delayed with my studies, but this semester i don't have very much courses... (but therefore more examinations :hang: )

[redi jedi]

Well heres the main design questions as I see them(remember this inst exactly suppose to be a direct copy of MD2, just have AT LEAST the features in it):

do you go with a fixed or semi-fixed render loop(like MD, this is what allows transitions to work the way they do)

or do you go with a preset driven render(presets make render calls directly) loop(possible with Lua) to give more control but resolve to bitmap based scripted transitions? then write a milkdrop translator to fill a template with the code from an md file...


I personally like transitions so I'm prone to go with a semi-fixed(kinda like the beta where you have some predetermined control over the loop, but its essentially static) loop, any thoughts on that?

Lua presents the ability to add some really cool shit as far as presets are concerned.. for instance Events, you could create an Event handler in C# that would call functions that have been registered from Lua, With a built in beat detection system you could register some code to run On Beat, maybe some other code to run OnMouseMove... that could be cool..
anyway do ya add things like events or maybe custom renders(humm.. fixed render line with scriptable renders on top? is that feaseable..)

also do you build a beat detection system into the main shell, since it will be faster there, and most real-time BD algos need some "warm up" time, plus a standard BD output will make remixing presets easier, and if they have standard out put you could switch between them aswell, mabye let the user pick the one they think works the best

[indaVizz]

hey guys, thanks for the feedback, first of all!

based on the last few messages here's the list of new features i plan to add:

1. based on redi jedi's latest beta.

2. IR support -- that's a great idea! i was obsessed with the wii-wristbands (with camera soldered out) and totally forgot about it. more about IR in the next message.

3. more wii vars -- though the only ones missing at the moment are wii_*_g{x,y,z}, which are raw accelerator values. (flexi: more on that in the next message), and also wii_*_y for "yaw", the angle at which the wiimote tils in (its) horizontal plane; that one works only when IR is supported.

4. a preset-writable var to determine whether bass/mid/treb come from sound or from wii.

5. bumping up user eval vars -- i didn't know such thing existed, thanks. anyone has a preset where this number is exceeded (or i can make it easy to exceed it) so i can test?

6. supporting up to 7 wiimotes at the time: wii_{1,2,3,4,5,6,7}_* (in addition to wii_l_* and wii_r_*) for collective performance. that should be pretty straightforward, although tedious, to add.

btw could adding 100+ new input vars make running presets slow? i would guess/hope not, but i don't know how MD really works.

7. add an option that if no wiimotes are found, behaves like a regular milkdrop. (also when you press certain key combos you can turn off wii-reactivity and put back regular sound-reactivity. then if you play your music through winamp you can switch from user-driven to music-driven visuals.)

8. make left wiimote's cursor buttons decide what the right wiimote's cursor buttons do, e.g. if the last left cursor button pressed was "up", then right cursor buttons control sensitivity; if the last left button was "down" then the right cursor buttons do something else, like send their values to the preset. (it's a terrible user interface but in a performance you may need to do as much as possible without going back to the PC).


let me know if something is missing or you think is unnecessary. if you want IR support right now, i could do that first and then switch to beta.

i'll also release the source soon, the cleanup is almost done. all the changes in the code are marked with [wii], and there's about 40 small blocks of code added to the MD source and just one block commented out, plus there's a couple of wii-only files. so i'm hoping that the transition to the beta should be fairly smooth.

[indaVizz]

now let's talk about IR. once again, great idea.

each wiimote can detect up to 4 IR spots with its camera, and process the data and send it to the host at 10ms intervals.

the "sensor" bar next to the wii console has a group of two IR LEDs on each side, which i think the wii still sees as two bright spots b/c the LEDs on each side are close. which probably means you could use up to two sensor bars. (you can buy a wii sensor bar for $10 which looks pretty neat.)

the viewing angle of the wiimote is pretty narrow, only 40 degrees. but if you hold a sensor bar relatively close to it you could control it pretty smoothly. or, you could use two sensor bars (or some other source, like an IR floodlight far away).

if you want everything IR related, you'd have

- wii_{l,r}_ir{1,2,3,4}_{x,y,o,s}, where
1..4 are up to 4 spots, x y are its coordinates,
o is the order of the dot on the x axis, and s is its size.
(assume size 0 means it's out of the viewing angle.)
- wii_{l,r}_ir_{x,y,d,z} which is data derived from the
4 hotspots. x and y are the calculated virtual "cursor"
coordinate (0 - 1.0), d is distance between first two
dots (i.e. one on the sensor bar), and z is the
calculated distance between the wiimote and the sensor
bar; if the sensor bar is not far, z could also be put
to some pretty good use (e.g. zoom)

... which is a lot of vars (especially if supported for 1..7 wiis: wii_{l,r,1..7}_ir{{1,2,3,4}_{x,y,o,s},ir_{x,y,d,z}} is 180 new vars. but then again, maybe it doesn't matter.)

but i'm thinking maybe it would be ok for now just to pass on the calculated virtual cursor data to presets, i.e. wii_{l,r}_ir_{x,y,d,z}. what do you think?

also i'll try to see if i can detect whether IR is used by a preset (i.e. if it references a wii_*_ir* var, to turn IR reading off when not needed to save battery; if it's too much change in the sources, a preset that uses Wii IR can set fUsesWiiIR=1).

the great thing about the ir is it lets you make a slow, continuous change, unlike the accelerometers which can only detect the force of your strike at the moment. roll and pitch are also ok for slow change (as you guys used them in the presets you made), but are inaccurate when closer to right angles.

[indaVizz]

"It's like Speed 2, only with a bus instead of a boat!" -- Milhouse

redi jedi, i don't know c#, never used .net. but i'm sure it shouldn't be hard to add wii support to MD[2] even if it were c#: i basically have one external .c file that fires up threads that set up and read wii stuff and write the readings into some global vars. then the rest of md is changed in a few places to read those global vars and do something extra. (the MD functions' code is changed in less than a dozen places, and those are almost all added blocks of code -- the rest is added #defines and help/text etc.). so the changes are few and isolated and i guess c# can be linked with C code in some form.

i only wonder if c# is fast enough to do what MD[2] does... i have a feeling that a java MD wouldn't be, and i've heard somewhere that c# is similar to java, don't know if its bytecode too.

what i've measured (unscientifically) is that if the delay between music -- beat for example -- and the visual effect that matches it is more than 50 or so ms, but under 150 or so ms, you can't directly see the delay but the visuals feel kind of flat. so super fantastic visuals that change 90ms after the beat may feel less good that more ordinary visuals that show 20ms after.

maybe i'm just paranoid about speed, there's all these games that draw full 3d scenery 30 times a second and, i doubt they are all written in super-optimized c/c++.

as for changing the preset to use some scripting, i think that would be terrific!

cheers

[Flexi]

@indavizz: :up: :up: :up: :D

@rj: a built-in BD sounds very nice! I guess you should open up a new thread on this topic.
Transitions are a must-have feature and in my opinion they have to be scriptable same as ordinary presets, but maybe only with one color channel as filter. btw: filter is good headword, i'd love to see full layer control.

[redi jedi]

@indavis: adding 170 variables to evalLib(the expression evaluator) *might* slow things down.. lol it kinda depends. all I can tell you for sure it that it WILL increase runtime memory usage, although thats not rocket science.. I never saw any real slow downs from adding variables(never added that many though) just from what the variables implemented. If you do have any problems with speed, I have a possible solution for ya.. you can add functions to eval lib, so add like a get_wi_var(controller_number, var_name); this will add no variables at all(except your control var) so that should clear up the speed issues if any preset them selves..

for an example of how to go about that and be able to get data to-from the main MD(eval has no access to MD) look at the sound(low,high) function I added to the beta, the main rendering loop calls a function in eval lib to update its copy of the music frame data, which the callable function actually grabs.. so it might be best(since your dealing with more data here) to move the data reading/storage to be inside of eval lib...

@flexi -- ya I think a new thread is warranted.. maybe we can stir some more interested devlopers..

Oh and about c#'s speed..
It ALL depends on what your doing.
saw a trade study the other day where they did a prime number counter in c++,vb6,managed c++,c#,vb.net

one guy rote the test in c++ then the other dude coppied it into c# and the like, right out of the box managed c++ won, and even c# beat normal c++(vb.net lost horriably)

for the c++ to win the guy had to go through 6 revisions and write 5 or 6 low level classes to override c++'s built in(string for instance, but also his own memory allocator)

so.. c++ *can* be made faster than the .net languages, but I for one(giess might be an exception) cant compete with the programers that write compilers.. let a lone the CLR.. there are so many optimizations there its not funny, given the quicker dev cycle of c# it seems like a good trade off to me...

[indaVizz]

hey guys,

i just finished adding IR support and making lots of buttons available as inputs to presets to control preset actions. the IR thing looks fantastic and opens up great opportunities for intelligent visualization. thanks for the suggestion!

here's a video demo:
youtube.com/watch?v=A0dS7ll9e48

i'll post the new release and the source tomorrow.

cheers

[Flexi]

oh i really see a lot of fun coming up.

thx so much! :up:

[indaVizz]

... the video above is long, here's a short clip with the best effects:

youtube.com/watch?v=j8QE4Awn4mM

[indaVizz]

miilkdrop update to v.0.2.0 is at

indaviz.com/download/milkdrop_0.2.0_update.zip

just copy the contents of "milkdrop 0.2.0 update" into winamp\Plugins. (it's 176K, just the dll and some new wii presets.)

i haven't updated the documentation (i'm leaving soon and i'll have no internet for a week) so i'll put the most important documentation update right here:

* IR is now supported!
* plenty more wii buttons now available for presets to read
* more wii vars
* changed how you navigate presets
* some more wii and wii-IR specific presets



let's start with the IR support first:
-----------------------------------------------
variables wii_l_irx and wii_l_iry show the position of the virtual cursor on the screen, calculated from the IR source(s) to which the left wiimote points. so this is not the direct location of the sources but it's derived from them. (i made those direct coordinates available as well but the virtual cursor seems more useful.) variable wii_l_irz is a distance from the IR source, in some arbitrary units, and around the 0..1 range. variable wii_l_yaw is the horizontal tilt angle of the wiimote. and same thing for wii_r_* vars.

this is great for remixing presets where x and y of something is determined by music.

in the Miilkdrop directory where presets are there are several (attempts at) remixes of presets to use the IR. only one is serious, "[] wisps [wii-IR].milk", but the pac-man is funny too.

these presets now need instruction manuals, or you can reverse engineer them, but here's the "[] wisps [wii-IR].milk" in short:
- left wiimote looks into IR, right wiimote is used for beat (as in the video)
- right wiimote's (B) button increases the glow of wisps, left (B) changes the bg color and uses distance from the sensor bar for zoom; left (A) turns the blood color of the wisps, right (A) turns off the IR and essentially changes the preset into redi jedi's original remix.
- but wait, there's more: right wiimote keys control the shape of the formation of the wisps -- one, two, three, four, square; (see note on cursors keys mode below)

(i find it the presets are most instrument-like when the left wiimote controls some slow-changing value -- rotation, zoom, or position, while the right wiimote controls the beat through force. the left wiimote force can be also applied sometimes, and then there's all the buttons.)

NOTE: now both Wiimotes have IR reading turned on, which drains batteries faster (twice as fast). i use the Nyko wiimote charger and it works great, never have to change batteries. in the 0.2.1 release i'll change it so that IR is turned off for the current preset if it doesn't use IR.





buttons and navigation
--------------------------------
so now plenty of wiimote buttons are available to presets to change their mode of operation: wii_l_b is the (B) button (it used to be (A), so the few Wii-enabled presets we had that used button (A) will now expect button (B), this is the only "compatibility break"), wii_l_a is the (A) button, and same for wii_r_*.

the preset can read four extra buttons: the cursor keys on the right wiimote: wii_r_cup, wii_r_cdo, wii_r_cle, wii_r_cri. for that the right wiimote must be in the "send cursor keys to preset mode" which is controlled by the left wiimote cursor keys:

* when the left wiimote cursor key DOWN is pressed, right wiimote gets in the mode where right wiimote cursor keys are sent to preset
* when the left wiimote cursor key LEFT is pressed, right wiimote gets in the mode where right wiimote cursor keys navigate presets: left changes to previous, and right changes to next. i have some interesting plans for up and down right cursors keys for preset navigation for live performance.
* when the left wiimote cursor key RIGHT is pressed, right wiimote gets in the mode where right wiimote cursor keys control wii sensitivity (the only thing right wiimote cursor keys used to do in 0.1.0)
* when the left wiimote cursor key UP is pressed, nothing happens. any suggestions for what that mode could be?

also now because presets are changed with cursor keys, there won't be that annoying accidental preset switch when you put the wiimote on the table and lose all your in-preset editing.

i also added a couple of "system" modes:
- holding left wiimote DOWN cursor key and then pressing the left (A) key will reload the preset; useful if you're editing the preset in a text editor
- holding left wiimote (A) key and then pressing (1) key will turn off wii-reactivity and basically switch to normal Milkdrop, where presets are controlled by music (unless they are wii presets, of course). good if you want to see how the preset originally works, or to take a break, or show how much better you are in driving the preset than the beat detection algorithm. :-)
- holding left wiimote (A) key and then pressing (1) key turns on Wii reactivity again.

F1 for help can remind you of all these.



new vars
------------

here's a complete list of wii-specific variables:

wii_{l,r}_* vars available to presets, where l=left wiimote, r=right wiimote, *=var:
------------------------------------------------------------------------------------

gx: [3.0..3.0] - acceleration on wiimote X axis as measured by wiimote, includes gravity
gy: [3.0..3.0] - acceleration on wiimote Y axis as measured by wiimote, includes gravity
gz: [3.0..3.0] - acceleration on wiimote Z axis as measured by wiimote, includes gravity
r: [-PI..+PI] - roll (e.g. twist) of the wiimote, r = 0.0 when wiimote is horizontal and face up
p: [-PI..+PI] - pitch (e.g. vertical angle ) of the wiimote, p = 0.0 when wiimote is horizontal and face up
x: [3.0..3.0] - acceleration on wiimote X axis, gravity excluded: x = gx - sin(r)
y: [3.0..3.0] - acceleration on wiimote Y axis, gravity excluded: y = gy + sin(p)
z: [3.0..3.0] - acceleration on wiimote Z axis, gravity excluded: z = gz - cos(p)
f: [0.0..5.9] - total force of the acceleration vector without gravity (f = sqrt(x^2 + y^2 + z^2)
g: [0.0..5.9] - total force of the acceleration vector with gravity (g = sqrt(gx^2 + gy^2 + gz^2)
a: [0.0 | 1.0] - indicates whether wiimote A button is being pressed
b: [0.0 | 1.0] - indicates whether wiimote B button is being pressed
m: [0.0 | 1.0] - motion detection (currently pathetic: m = (f < 0.9 || f > 1.1)
irx: [0.0..1.0] - X coordinate of the cursor calculated from the IR source (sensor bar)
iry: [0.0..1.0] - Y coordinate of the cursor calculated from the IR source (sensor bar)
irz: [0.2..11.0+] - distance from the IR source (sensor bar), 0.8 seems to be "normal" distance
yaw: [-PI/9..+PI/9] - yaw (horizontal tilt angle), -20..+20 degrees is wiimote's viewing angle

the following are right-wiimote only:
------------------------------------
wii_r_cup: [0.0 | 1.0] - indicates whether the UP cursor button on the right wiimote is pressed
wii_r_cdo: [0.0 | 1.0] - indicates whether the DOWN cursor button on the right wiimote is pressed
wii_r_cle: [0.0 | 1.0] - indicates whether the LEFT cursor button on the right wiimote is pressed
wii_r_cri: [0.0 | 1.0] - indicates whether the RIGHT cursor button on the right wiimote is pressed

miscellaneous:
-------------
wii_inc: [0..infinity] incrementer -- increases by 1.0 every 10ms (in lieu of "time" which is a function of wiimote force now)

these are subject to change:
---------------------------
wii_aux1: [0.0 | 1.0] visibility of the first IR dot as seen by the LEFT wiimote
wii_aux2: [0.0..1.0] X coordinate of the first IR dot as seen by the LEFT wiimote
wii_aux3: [0.0..1.0] Y coordinate of the first IR dot as seen by the LEFT wiimote
wii_aux4: [0.0..20.0] size of the first IR dot as seen by the LEFT wiimote
wii_aux{5..8}: same but for the second IR dot as seen by the LEFT wiimote



that's all i can think of for now. thanks a lot for giving this a shot, you guys are helping me make this way cooler than it was originally.

enjoy!

[indaVizz]

here's the source!

indaviz.com/download/miilkdrop-src.v0.2.0.zip

this too has no documentation but if you want to rebuild it i'm sure you can figure it out: all you need in terms of initial info is to checkout the README in the wii\ directory for the software you need to install to build for Wiimotes (WINDDK), and you'll need to muck with the include/link paths a bit but that should be all.

if you just want to take a look at what was changed, look for the [wii] tag in the sources. the changes are rather small and contained i believe, but any feedback on this is welcome. also it needs a little bit more cleanup and comments.

release 0.2.1 would just be
- turning on IR only for presets that need them
- adding wii_1..7 and wii_avg vars for collective trips
- documentation update

once again i'll be away for a week with no internet but will continue work the first day i'm back.

some nice new wii and wii-IR presets would be much appreciated!

cheers
indaVizz

[Flexi]

OMFG i never believed it would be that cool.
i tuned and tuned and tuned and finally it was just ...wow

hope you like it.
(let http://www.youtube.com/watch?v=SQq_XmhBTgg run in background when trying this out.)

[ORB 13]

Sweet!!! What a cool idea.

Don't have a dongle so tried it out on my laptop. Got some ideas of presets I can make.

[Flexi]

http://de.youtube.com/watch?v=0XriZLCxybs
:rolleyes: :cool: :rolleyes:

[indaVizz]

fantastic! it looks like painted art. can you post this preset here?

btw you can also make both moving points ir-sensitive, one for the left and one for the right.

cheers

[Flexi]

IR fun for Two :D

[Saint Goody]

Here's another idea... might get a lil' tricky...

webcam motion detection...

[Flexi]

i know what you mean. i still have not soldered my IR LEDs, but be sure i will try some funny ideas. :)

[indaVizz]

hmm i think you should be able to do something with the latest miilkdrop release, using one wiimote to point towards you and a multi-LED IR source to illuminate you.

this guy, johnny chung, showed how you can use a wiimote to create a minority report-like effect: youtube.com/watch?v=0awjPUkBXOU

you point a wiimote and a source of IR towards you, and put some reflective tape on your fingers, now the wiimote that is still sees two moving IR points whose x,y coordinates it sends to the pc.

so with the v0.2.0 miilkdrop you can write a preset that reads positions of the two IR points by reading these vars:

these are subject to change:
---------------------------
wii_aux1: [0.0 | 1.0] visibility of the first IR dot as seen by the LEFT wiimote
wii_aux2: [0.0..1.0] X coordinate of the first IR dot as seen by the LEFT wiimote
wii_aux3: [0.0..1.0] Y coordinate of the first IR dot as seen by the LEFT wiimote
wii_aux4: [0.0..20.0] size of the first IR dot as seen by the LEFT wiimote
wii_aux{5..8}: same but for the second IR dot as seen by the LEFT wiimote

you can imagine you move your left hand more to the left and the preset zooms in faster (or maybe even you move it closer to the wiimote and it means more zoom), you move left and right fingers closer and it gets more blue etc.

i got this IR source from amazon: amazon.com/gp/product/B0009IB8RM though it may be a bit too strong.

the only problem is that the wiimote has a narrow viewing angle, around 40 degrees. i wonder if it would help to put a wide lens on it like the one you find in a door peephole.

also it may be more reliable to use a ping-pong ball covered in a reflective tape instead.

anybody wants to try it out?

[Flexi]

http://graffitiresearchlab.com/?page_id=6#video ;)

[carmatic]

like, can you make it so that the IR will track more dots as they come into, and go out of, the field of view of the IR camera? in a demo-like situation, its much more feasible to surround yourself with IR sources than it is in the living room where the Wii's are normally used...
so instead of tracking 2 dots, it tracks a variable number of dots , or maybe just the 2 dots closest to the upper left corner of its view or something

also , how much of the wiimote usage aspect can be used in VJ mode? like where the viz goes on one big screen for the audience and theres a second screen with all the windows and stuff that the operator looks at, can the VJ mode be modified to show stuff like the raw numbers or rolling graphs?

[indaVizz]

well each wiimote has an IR camera and firmware in it that recognizes and tracks IR sources, up to 4. (right now only two are exposed in preset vars, but could do all 4.) but -- you can use two or more wiimotes and then track many IR dots.

you'd probably need some procedural code, i.e. change in milkdrop source or a dll to do something meaningful with all that info, it'd be hard to do it from a preset. (i would think, though i'm sure some presets do things ryan geiss never thought were possible. :-)

or, you could change milkdrop to get frames from a real IR webcam and do anything you want; the wii IR was just the easiest thing to do, not that i know how i'd do optical recognition anyway but it's all doable. if you have a killer app in mind we can do anything.

[indaVizz]

about VJ mode... i haven't thought of that, what i had in mind is a performer on stage who makes the visuals on the spot. but i guess the wiimotes could be used by a VJ as well. i don't have any experience with VJing, if you have some ideas go ahead.

btw here's a video i just made that shows how this could be used with a big screen for the audience at a larger event, it uses some stuff that's not been released yet. i think this one is the best so far, hope you guys like it.

youtube.com/watch?v=9P4Bs_lQOVU

[carmatic]

well i thought that the performer is effectively the VJ when it comes to being on stage in this case, or the performer works very closely with a VJ who does the normal milkdrop command stuff... either way i think that being able to have a feedback other than what the audience is seeing is a good idea...

[indaVizz]

that's right, the performer is the vj, and he changes presets using cursor buttons on the wiimote. the acceleration values are fairly unhelpful for direct reading, whatever the performer does with them or with the regular wiimote buttons (A and B) is just food for the preset code. i had those graphs before modifying milkdrop and would just stare at them, without getting any idea what i could do with them. only when i hooked the values up to preset code i was able to see what made sense.

as for preset changing... right now you have left and right cursor keys on the right wiimote for that, and also a button to randomize or order the preset list. but it's not enough: sometimes you know the song is picking up tempo or power or is more flowy etc. so you'd like to use a specific kind of preset.

so what i was thinking of is if there's some sort of sorted preset list text file in the preset directory, then miilkdrop would parse the list and then the up and down wiimote keys would switch between categories of presets (explosive, subtle, staccato, etc.) while left and right would switch within the category.

btw in the video above i hand picked a sequence of presets in advance that i thought fit the mood of the song.

...besides preset changing, what else does a real vj do today with milkdrop?

[carmatic]

this might sound abit more wiimote than milkdrop, but if you accidentally press the A and B buttons together and reset the connection between the wiimote and the computer, can the program detect that and automatically pick up the sync again?

also, i have a suggestion... instead of just pressing a button to change the preset, how about if you hold the button, then move the wiimote while the button is held down... have multiple combinations of moves, so that for example front-up switches to one preset, back-left switches to another preset, down-front-right is another preset, etc... and put large tresholds before it registers so its all about making 90 degree movements

[indaVizz]

i like that idea -- use not just buttons but gestures to select presets. actually i'd make a modification of it: use "analog buttons" of the wiimote to select presets. wiimote has no analog buttons but you simulate them in miilkdrop by pressing a button (on or off) and then twist the wiimote; the further you twist, the "harder" the button is pressed.

so maybe you hold a button and twist to the right, depending on how much you twist you get a more explosive preset in the list, you tilt the wiimote up and you get a more flowy preset etc. when you release the button.

resetting the controler: i don't think you can break the connection by pressing any buttons, and if you lost the connection for some reason you'd have to go back to the bluetooth stack software and reconnect. but i never had that happen, so i guess it's pretty safe for performance. we'll see. :-)

[carmatic]

no i mean like, if you press a+b at the same time, the wiimote's lights should start blinking and all that, which means that its searching for a bluetooth host or something, then you have to press a button on the wii in a certain amount of time to get them to connect to each other again

this might sound more milkdrop than wiimote, but is it possible to control the transitions? like instead of fully transitioning from one preset to the next in a fixed amount of time, the progress of the transition is determined by an input...
i think this could go very well with the wiimote twisting idea, for example if you put a pair of IR leds horizontally , the wiimote can use this to determine the horizontal angle you are holding it at... the more rotation you give the wiimote, the further the transition... also, you can have a 'key in a lock' effect if the transition is activated by 2 things, one is if the pair of leds are within a certain area of the IR camera, secondly when a large forward motion is detected, and both of these have to happen at the same time... so if you want to change the scene, you put the 'key' into the 'keyhole' and start 'turning' the key...

[indaVizz]

only if you press (1)+(2) the wiimote gets in discoverable mode, but it doesn't lose the connection (and you can't press those two accidentally).

transitions... interesting idea, i've always been using a hard cut, but maybe can use a wiimote pitch angle to control the duration. i'll play with that a bit.

i try to keep changes to milkdrop to a minimum, i'm dreaming that some day milkdrop 2.0 will be open sourced so i want to be able to port the wii stuff to it quickly. i'll test that theory when i start porting to redi jedi's beta...

[carmatic]

i am guessing that it is already possible to combine sound response with wiimote control? im thinking like, a basic effect is generated by the sound , and this effect fades away slowly ... the wiimotes will then control some other functions that act on the result of the sound, kind of like passing your hands through smoke or something
the advantage of this is that the visuals will be more responsive than you can ever be without knowing the music beforehand, the disadvantage is that you wont be the star of the show anymore

[indaVizz]

you're right on both counts.

but there's something else besides being the star of the show: i'm totally convinced that people don't want to see any "artificial intelligence" in a live performance. if the computer does something in reaction to sound, or does something randomly, people will not know if it's you the performer doing that or is it the computer, and they'll hate the confusion.

from their perspective, if the visuals totally match the mood of the song and the beat, we (the audience) want to be sure that's because you (the performer) are good and you gave your best. and if they don't, we'll know that you suck. but we don't want the computer to help you in any way.

that's my theory anyhow. i think that's not different from singing (don't you hate lip syncing) or playing a guitar live or dancing with fire or whatever -- it's about people and the way they connect through art.

[carmatic]

yeah, but i guess what im thinking of was more in terms of the versatility of the setup, imagine where the wiimotes are chained to a booth which the public have access to, that kind of thing

one thing i am pretty sure of is that in an interactive setting, the number of presets can be much much much lower because people dont get bored when they are playing with the visuals... which is a good thing i guess, since you have to account for both the original sound response of milkdrop as well as the wiimote input, more variables, complexity, and work for each preset
what combination of sound reactivity and wiimote control is totally up to the presets of course, but it can be presented as wiimote-focused or sound-focused.. to be sound focused can be, for example, the wiimotes can control the position of some points on the screen, and the sound waveform can then be centered on these points... if the waveforms fade slowly, it would be as if you are holding sound-reactive flames on-screen
i havent had time to set up all that bluetooth stuff on my computer yet, but im guessing that a serious wiimote-focused preset can have absolutely no sound input at all, and the functions which affect the waveform and fade are entirely derived from the speed, acceleration, etc etc of the wiimote input, or if there is sound reactivity, it is subtle and sublime stuff like the colour of the background or something... the kind of thing that would be used if there is going to be a dedicated performer on the booth

[indaVizz]

i see. that feature is supported already. normally (for the wii milkdrop) bass, bass_att and other sound preset vars are a function of wiimote acceleration, angles and buttons, so that when you don't move, these vars are 0, regardless of the music played in winamp. then winamp + miilkdrop is just a graphical engine.

but if you press certain wiimote key combo (left (A) + left (1)), those vars are reconnected back to sound analysis for the music played in winamp, so an unmodified preset behaves as in the original milkdrop. but the wii-specific vars are still available (wii_l_f etc.), so you could write a preset that does exactly what you described -- use sound vars and waveforms for subtle reaction and wii-only vars for not-so-subtle reaction.

... i could also have each preset decide whether sound vars source should be winamp sound or wiimotes, by having it define fWiiAnalyzeSound=1 (default 0).

[carmatic]

yaay just had a go with milkdrop, it was as simple as plug and play on my xp x64, plug my belkin dongle in, wait for windows to install the drivers, sync the wiimote, and when i started the test program it recognized it straight away... makes me sleep easy at night when stuff like that happens

anyway, i have an idea... if there is a way to control the sensitivity of milkdrop's sound reactivity using the wiimote, you could 'reward' the participant with a large reaction on the visuals if they get it right, dance dance revolution-style...
or how about 'transforming' the input of the wiimotes to different frequencies of audio, like long sweeps (large constant acceleration) create high frequency sounds and sudden jerks (large change in acceleration) cause pulses of bass... make it so that when the wiimotes are still, there will be constant white noise, and as the wiimotes feel acceleration, the tone of the sound changes... this can either be fed into the visuals as it is, or mixed with the audio stream for some 'relevance' to the visuals

i dont know how relevant this is, but when i was browsing wikipedia i came across http://en.wikipedia.org/wiki/Linear_response_function and somehow it reminded me of this thread, can anyone think of anything?

[Flexi]

hehe i didn't know this formula has a well known name, but in fact i used it in quite alot presets (all the ones, that take advance of the elastic spring and some more)

rewarding is definetly the way to go, when you want to get serious with it. (i just got 48 points on the wii sports sandbag:p)
If i find something cool, i will post it here! but don't you wait on me, my creativity is unpredictable ;)

[carmatic]

like, how difficult is it to make the wiimotes control some kind of 'noise generator' like i described above? the noise will then be fed into the visualization and processed as if it is the music...
in addition to amplifying certain frequencies depending on acceleration, maybe the x-axis wiimote acceleration can change the phase of the affected frequencies as well, the y-axis wiimote acceleration can change the y-axis offset of the waveform, and z-axis acceleration can change the bandwidth of the processing, so you can have direct control over the waveform using the wiimotes...you can 'roll' the waveform around by moving the wiimote side to side, you can 'push' the waveform up and down, and you can create different interference patterns by moving the wiimote forwards or backwards... this potentially makes unmodified milkdrop presets much much much more useful, and we can use all 5542 of them

[Abydos]

I love you.
I'm buying a 1080 hd projector soon and just happened to stumble acroos this, I'm going to have soooo much more fun now, cheers!

[carmatic]

like, what i think could use some work is like, when the wiimote is moved, the bass waveform is identical and repeating, which looks kind of unnatural, that is why i was thinking of using the white noise... or if the sound variables can be reconnected back to the sound input, but not directly, like if the wiimote actually controls some kind of DSP using its movements, and the resulting processed sound is fed into the sound variables... that way you get a good waveform, plus you get a good response from the wiimote too

which sound variables are controlled by which wiimote variable?

[liam9519]

Is it just me or is the 0.2.0 link not working? Really would like to try it out!

Keep up the good work!
liam

[Flexi]

right, but the 0.1.0 downloads works...

http://rapidshare.com/files/133826115/vis_miilk.dll
simply replace this file - that's all it needs for the IR support

[liam9519]

ah cheers

[ShadowHarlequin]

can you get an attatchment usb or otherwise that will give my laptop bluetooth capability? it has a switch on the front but its real stiff and theres no software installed for it so i guess my model doesnt have bluetooth

also i cant find a dl link for the software, the download page isnt there any more
[edit, scrolled down on the page and got it :P]

what parameters can you control with the wiimote? could you control any variable?

[Flexi]

yes you can control any variable. you can either use the raw force values or roll/pitch(+yaw with the infrared bar) + some button press status.

i bought a usb BT dongle for my PC but not all are working with the wiimotes. It's all written in the Miilkdrop readme. check it out :p

[ShadowHarlequin]

cool, i cant afford any hardware at the moment though :P lookng into using a gamepad with this instead for the moment which might work

[Flexi]

by private request, a reupload of the Miilkdrop dll + documentation and preset pack