Wiimote-reactive Milkdrop visuals

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • indaVizz
    Junior Member
    • Mar 2008
    • 27

    #31
    IR

    now let's talk about IR. once again, great idea.

    each wiimote can detect up to 4 IR spots with its camera, and process the data and send it to the host at 10ms intervals.

    the "sensor" bar next to the wii console has a group of two IR LEDs on each side, which i think the wii still sees as two bright spots b/c the LEDs on each side are close. which probably means you could use up to two sensor bars. (you can buy a wii sensor bar for $10 which looks pretty neat.)

    the viewing angle of the wiimote is pretty narrow, only 40 degrees. but if you hold a sensor bar relatively close to it you could control it pretty smoothly. or, you could use two sensor bars (or some other source, like an IR floodlight far away).

    if you want everything IR related, you'd have

    - wii_{l,r}_ir{1,2,3,4}_{x,y,o,s}, where
    1..4 are up to 4 spots, x y are its coordinates,
    o is the order of the dot on the x axis, and s is its size.
    (assume size 0 means it's out of the viewing angle.)
    - wii_{l,r}_ir_{x,y,d,z} which is data derived from the
    4 hotspots. x and y are the calculated virtual "cursor"
    coordinate (0 - 1.0), d is distance between first two
    dots (i.e. one on the sensor bar), and z is the
    calculated distance between the wiimote and the sensor
    bar; if the sensor bar is not far, z could also be put
    to some pretty good use (e.g. zoom)

    ... which is a lot of vars (especially if supported for 1..7 wiis: wii_{l,r,1..7}_ir{{1,2,3,4}_{x,y,o,s},ir_{x,y,d,z}} is 180 new vars. but then again, maybe it doesn't matter.)

    but i'm thinking maybe it would be ok for now just to pass on the calculated virtual cursor data to presets, i.e. wii_{l,r}_ir_{x,y,d,z}. what do you think?

    also i'll try to see if i can detect whether IR is used by a preset (i.e. if it references a wii_*_ir* var, to turn IR reading off when not needed to save battery; if it's too much change in the sources, a preset that uses Wii IR can set fUsesWiiIR=1).

    the great thing about the ir is it lets you make a slow, continuous change, unlike the accelerometers which can only detect the force of your strike at the moment. roll and pitch are also ok for slow change (as you guys used them in the presets you made), but are inaccurate when closer to right angles.

    Comment

    • indaVizz
      Junior Member
      • Mar 2008
      • 27

      #32
      speed

      "It's like Speed 2, only with a bus instead of a boat!" -- Milhouse

      redi jedi, i don't know c#, never used .net. but i'm sure it shouldn't be hard to add wii support to MD[2] even if it were c#: i basically have one external .c file that fires up threads that set up and read wii stuff and write the readings into some global vars. then the rest of md is changed in a few places to read those global vars and do something extra. (the MD functions' code is changed in less than a dozen places, and those are almost all added blocks of code -- the rest is added #defines and help/text etc.). so the changes are few and isolated and i guess c# can be linked with C code in some form.

      i only wonder if c# is fast enough to do what MD[2] does... i have a feeling that a java MD wouldn't be, and i've heard somewhere that c# is similar to java, don't know if its bytecode too.

      what i've measured (unscientifically) is that if the delay between music -- beat for example -- and the visual effect that matches it is more than 50 or so ms, but under 150 or so ms, you can't directly see the delay but the visuals feel kind of flat. so super fantastic visuals that change 90ms after the beat may feel less good that more ordinary visuals that show 20ms after.

      maybe i'm just paranoid about speed, there's all these games that draw full 3d scenery 30 times a second and, i doubt they are all written in super-optimized c/c++.

      as for changing the preset to use some scripting, i think that would be terrific!

      cheers

      Comment

      • Flexi
        wellspring of milk
        Major Dude
        • Apr 2007
        • 2058

        #33
        @indavizz:

        @rj: a built-in BD sounds very nice! I guess you should open up a new thread on this topic.
        Transitions are a must-have feature and in my opinion they have to be scriptable same as ordinary presets, but maybe only with one color channel as filter. btw: filter is good headword, i'd love to see full layer control.
        WebSocket Hub for Kinect SDK 2.0 with Milkdrop shader pipeline in VanillaJS and glsl
        Codepen | Shadertoy | OpenProcessing | studio sketchpad
        Twitter @ Google+ @ YouTube @ Facebook

        Comment

        • redi jedi
          Will code for food
          • Mar 2005
          • 521

          #34
          @indavis: adding 170 variables to evalLib(the expression evaluator) *might* slow things down.. lol it kinda depends. all I can tell you for sure it that it WILL increase runtime memory usage, although thats not rocket science.. I never saw any real slow downs from adding variables(never added that many though) just from what the variables implemented. If you do have any problems with speed, I have a possible solution for ya.. you can add functions to eval lib, so add like a get_wi_var(controller_number, var_name); this will add no variables at all(except your control var) so that should clear up the speed issues if any preset them selves..

          for an example of how to go about that and be able to get data to-from the main MD(eval has no access to MD) look at the sound(low,high) function I added to the beta, the main rendering loop calls a function in eval lib to update its copy of the music frame data, which the callable function actually grabs.. so it might be best(since your dealing with more data here) to move the data reading/storage to be inside of eval lib...

          @flexi -- ya I think a new thread is warranted.. maybe we can stir some more interested devlopers..

          Oh and about c#'s speed..
          It ALL depends on what your doing.
          saw a trade study the other day where they did a prime number counter in c++,vb6,managed c++,c#,vb.net

          one guy rote the test in c++ then the other dude coppied it into c# and the like, right out of the box managed c++ won, and even c# beat normal c++(vb.net lost horriably)

          for the c++ to win the guy had to go through 6 revisions and write 5 or 6 low level classes to override c++'s built in(string for instance, but also his own memory allocator)

          so.. c++ *can* be made faster than the .net languages, but I for one(giess might be an exception) cant compete with the programers that write compilers.. let a lone the CLR.. there are so many optimizations there its not funny, given the quicker dev cycle of c# it seems like a good trade off to me...
          Blah!

          Comment

          • indaVizz
            Junior Member
            • Mar 2008
            • 27

            #35
            IR support!

            hey guys,

            i just finished adding IR support and making lots of buttons available as inputs to presets to control preset actions. the IR thing looks fantastic and opens up great opportunities for intelligent visualization. thanks for the suggestion!

            here's a video demo:
            youtube.com/watch?v=A0dS7ll9e48

            i'll post the new release and the source tomorrow.

            cheers

            Comment

            • Flexi
              wellspring of milk
              Major Dude
              • Apr 2007
              • 2058

              #36
              oh i really see a lot of fun coming up.

              thx so much!
              WebSocket Hub for Kinect SDK 2.0 with Milkdrop shader pipeline in VanillaJS and glsl
              Codepen | Shadertoy | OpenProcessing | studio sketchpad
              Twitter @ Google+ @ YouTube @ Facebook

              Comment

              • indaVizz
                Junior Member
                • Mar 2008
                • 27

                #37
                ... the video above is long, here's a short clip with the best effects:

                youtube.com/watch?v=j8QE4Awn4mM

                Comment

                • indaVizz
                  Junior Member
                  • Mar 2008
                  • 27

                  #38
                  Miilkdrop 0.2.0

                  miilkdrop update to v.0.2.0 is at

                  indaviz.com/download/milkdrop_0.2.0_update.zip

                  just copy the contents of "milkdrop 0.2.0 update" into winamp\Plugins. (it's 176K, just the dll and some new wii presets.)

                  i haven't updated the documentation (i'm leaving soon and i'll have no internet for a week) so i'll put the most important documentation update right here:

                  * IR is now supported!
                  * plenty more wii buttons now available for presets to read
                  * more wii vars
                  * changed how you navigate presets
                  * some more wii and wii-IR specific presets



                  let's start with the IR support first:
                  -----------------------------------------------
                  variables wii_l_irx and wii_l_iry show the position of the virtual cursor on the screen, calculated from the IR source(s) to which the left wiimote points. so this is not the direct location of the sources but it's derived from them. (i made those direct coordinates available as well but the virtual cursor seems more useful.) variable wii_l_irz is a distance from the IR source, in some arbitrary units, and around the 0..1 range. variable wii_l_yaw is the horizontal tilt angle of the wiimote. and same thing for wii_r_* vars.

                  this is great for remixing presets where x and y of something is determined by music.

                  in the Miilkdrop directory where presets are there are several (attempts at) remixes of presets to use the IR. only one is serious, "[] wisps [wii-IR].milk", but the pac-man is funny too.

                  these presets now need instruction manuals, or you can reverse engineer them, but here's the "[] wisps [wii-IR].milk" in short:
                  - left wiimote looks into IR, right wiimote is used for beat (as in the video)
                  - right wiimote's (B) button increases the glow of wisps, left (B) changes the bg color and uses distance from the sensor bar for zoom; left (A) turns the blood color of the wisps, right (A) turns off the IR and essentially changes the preset into redi jedi's original remix.
                  - but wait, there's more: right wiimote keys control the shape of the formation of the wisps -- one, two, three, four, square; (see note on cursors keys mode below)

                  (i find it the presets are most instrument-like when the left wiimote controls some slow-changing value -- rotation, zoom, or position, while the right wiimote controls the beat through force. the left wiimote force can be also applied sometimes, and then there's all the buttons.)

                  NOTE: now both Wiimotes have IR reading turned on, which drains batteries faster (twice as fast). i use the Nyko wiimote charger and it works great, never have to change batteries. in the 0.2.1 release i'll change it so that IR is turned off for the current preset if it doesn't use IR.





                  buttons and navigation
                  --------------------------------
                  so now plenty of wiimote buttons are available to presets to change their mode of operation: wii_l_b is the (B) button (it used to be (A), so the few Wii-enabled presets we had that used button (A) will now expect button (B), this is the only "compatibility break"), wii_l_a is the (A) button, and same for wii_r_*.

                  the preset can read four extra buttons: the cursor keys on the right wiimote: wii_r_cup, wii_r_cdo, wii_r_cle, wii_r_cri. for that the right wiimote must be in the "send cursor keys to preset mode" which is controlled by the left wiimote cursor keys:

                  * when the left wiimote cursor key DOWN is pressed, right wiimote gets in the mode where right wiimote cursor keys are sent to preset
                  * when the left wiimote cursor key LEFT is pressed, right wiimote gets in the mode where right wiimote cursor keys navigate presets: left changes to previous, and right changes to next. i have some interesting plans for up and down right cursors keys for preset navigation for live performance.
                  * when the left wiimote cursor key RIGHT is pressed, right wiimote gets in the mode where right wiimote cursor keys control wii sensitivity (the only thing right wiimote cursor keys used to do in 0.1.0)
                  * when the left wiimote cursor key UP is pressed, nothing happens. any suggestions for what that mode could be?

                  also now because presets are changed with cursor keys, there won't be that annoying accidental preset switch when you put the wiimote on the table and lose all your in-preset editing.

                  i also added a couple of "system" modes:
                  - holding left wiimote DOWN cursor key and then pressing the left (A) key will reload the preset; useful if you're editing the preset in a text editor
                  - holding left wiimote (A) key and then pressing (1) key will turn off wii-reactivity and basically switch to normal Milkdrop, where presets are controlled by music (unless they are wii presets, of course). good if you want to see how the preset originally works, or to take a break, or show how much better you are in driving the preset than the beat detection algorithm. :-)
                  - holding left wiimote (A) key and then pressing (1) key turns on Wii reactivity again.

                  F1 for help can remind you of all these.



                  new vars
                  ------------

                  here's a complete list of wii-specific variables:

                  wii_{l,r}_* vars available to presets, where l=left wiimote, r=right wiimote, *=var:
                  ------------------------------------------------------------------------------------

                  gx: [3.0..3.0] - acceleration on wiimote X axis as measured by wiimote, includes gravity
                  gy: [3.0..3.0] - acceleration on wiimote Y axis as measured by wiimote, includes gravity
                  gz: [3.0..3.0] - acceleration on wiimote Z axis as measured by wiimote, includes gravity
                  r: [-PI..+PI] - roll (e.g. twist) of the wiimote, r = 0.0 when wiimote is horizontal and face up
                  p: [-PI..+PI] - pitch (e.g. vertical angle ) of the wiimote, p = 0.0 when wiimote is horizontal and face up
                  x: [3.0..3.0] - acceleration on wiimote X axis, gravity excluded: x = gx - sin(r)
                  y: [3.0..3.0] - acceleration on wiimote Y axis, gravity excluded: y = gy + sin(p)
                  z: [3.0..3.0] - acceleration on wiimote Z axis, gravity excluded: z = gz - cos(p)
                  f: [0.0..5.9] - total force of the acceleration vector without gravity (f = sqrt(x^2 + y^2 + z^2)
                  g: [0.0..5.9] - total force of the acceleration vector with gravity (g = sqrt(gx^2 + gy^2 + gz^2)
                  a: [0.0 | 1.0] - indicates whether wiimote A button is being pressed
                  b: [0.0 | 1.0] - indicates whether wiimote B button is being pressed
                  m: [0.0 | 1.0] - motion detection (currently pathetic: m = (f < 0.9 || f > 1.1)
                  irx: [0.0..1.0] - X coordinate of the cursor calculated from the IR source (sensor bar)
                  iry: [0.0..1.0] - Y coordinate of the cursor calculated from the IR source (sensor bar)
                  irz: [0.2..11.0+] - distance from the IR source (sensor bar), 0.8 seems to be "normal" distance
                  yaw: [-PI/9..+PI/9] - yaw (horizontal tilt angle), -20..+20 degrees is wiimote's viewing angle

                  the following are right-wiimote only:
                  ------------------------------------
                  wii_r_cup: [0.0 | 1.0] - indicates whether the UP cursor button on the right wiimote is pressed
                  wii_r_cdo: [0.0 | 1.0] - indicates whether the DOWN cursor button on the right wiimote is pressed
                  wii_r_cle: [0.0 | 1.0] - indicates whether the LEFT cursor button on the right wiimote is pressed
                  wii_r_cri: [0.0 | 1.0] - indicates whether the RIGHT cursor button on the right wiimote is pressed

                  miscellaneous:
                  -------------
                  wii_inc: [0..infinity] incrementer -- increases by 1.0 every 10ms (in lieu of "time" which is a function of wiimote force now)

                  these are subject to change:
                  ---------------------------
                  wii_aux1: [0.0 | 1.0] visibility of the first IR dot as seen by the LEFT wiimote
                  wii_aux2: [0.0..1.0] X coordinate of the first IR dot as seen by the LEFT wiimote
                  wii_aux3: [0.0..1.0] Y coordinate of the first IR dot as seen by the LEFT wiimote
                  wii_aux4: [0.0..20.0] size of the first IR dot as seen by the LEFT wiimote
                  wii_aux{5..8}: same but for the second IR dot as seen by the LEFT wiimote



                  that's all i can think of for now. thanks a lot for giving this a shot, you guys are helping me make this way cooler than it was originally.

                  enjoy!

                  Comment

                  • indaVizz
                    Junior Member
                    • Mar 2008
                    • 27

                    #39
                    Miilkdrop 0.2.0 source

                    here's the source!

                    indaviz.com/download/miilkdrop-src.v0.2.0.zip

                    this too has no documentation but if you want to rebuild it i'm sure you can figure it out: all you need in terms of initial info is to checkout the README in the wii\ directory for the software you need to install to build for Wiimotes (WINDDK), and you'll need to muck with the include/link paths a bit but that should be all.

                    if you just want to take a look at what was changed, look for the [wii] tag in the sources. the changes are rather small and contained i believe, but any feedback on this is welcome. also it needs a little bit more cleanup and comments.

                    release 0.2.1 would just be
                    - turning on IR only for presets that need them
                    - adding wii_1..7 and wii_avg vars for collective trips
                    - documentation update

                    once again i'll be away for a week with no internet but will continue work the first day i'm back.

                    some nice new wii and wii-IR presets would be much appreciated!

                    cheers
                    indaVizz

                    Comment

                    • Flexi
                      wellspring of milk
                      Major Dude
                      • Apr 2007
                      • 2058

                      #40
                      OMFG i never believed it would be that cool.
                      i tuned and tuned and tuned and finally it was just ...wow

                      hope you like it.
                      (let http://www.youtube.com/watch?v=SQq_XmhBTgg run in background when trying this out.)
                      Attached Files
                      WebSocket Hub for Kinect SDK 2.0 with Milkdrop shader pipeline in VanillaJS and glsl
                      Codepen | Shadertoy | OpenProcessing | studio sketchpad
                      Twitter @ Google+ @ YouTube @ Facebook

                      Comment

                      • ORB 13
                        Member
                        • Feb 2006
                        • 79

                        #41
                        Sweet!!! What a cool idea.

                        Don't have a dongle so tried it out on my laptop. Got some ideas of presets I can make.

                        Comment

                        • Flexi
                          wellspring of milk
                          Major Dude
                          • Apr 2007
                          • 2058

                          #42
                          a preset i wrote for the Milkdrop Mod "Miilkdrop"visithttp://forums.winamp.com/showthread.php?s=&threadid=289007for the full story

                          WebSocket Hub for Kinect SDK 2.0 with Milkdrop shader pipeline in VanillaJS and glsl
                          Codepen | Shadertoy | OpenProcessing | studio sketchpad
                          Twitter @ Google+ @ YouTube @ Facebook

                          Comment

                          • indaVizz
                            Junior Member
                            • Mar 2008
                            • 27

                            #43
                            fantastic! it looks like painted art. can you post this preset here?

                            btw you can also make both moving points ir-sensitive, one for the left and one for the right.

                            cheers

                            Comment

                            • Saint Goody
                              Senior Member
                              • Apr 2008
                              • 184

                              #45
                              Here's another idea... might get a lil' tricky...

                              webcam motion detection...
                              I am the purple heathen.

                              Comment

                              Working...
                              X