Kinect Effects Overlaid on Milkdrop

[JohanSebastian]

Hello:

Doing special effects for bands has been a dream of mine since I was a kid. I got a chance to see Roger Waters do The Wall in Dec of 2010. The computer graphics shown at that concert were mind blowing. That concert inspired me to really start pursuing my dream of doing special effects.

In January of last year, I noticed that my favorite nightclub, called The Candelight in Portland, OR, had a projector and a drop down screen, and yet I never saw them use it. I met up with Joe Shore, the owner and said to him "lets do some computer graphics on your projector". He replied "show me what you can do".

So I went out in search of hardware/software to do just that. Since all the computers I had were outdated, I wound up building a new computer system with an NVidia 580 GPU.

There are a few good packages for Vjing, and yet I found myself wanting more. When I stumbled across Winamp and Milkdrop I was very pleased. The visual effects which are generated by Milkdrop are outstanding, especially when you couple it with some of the latest presets, written by folks such as Natorami (Martin) etc.

Last summer a friend of mine told me about the Kinect, so I picked one up, hooked it up to my VJ system, started learning about it, and writing software for it. So far I've been able to come up with some pretty good effects for it.

In the back of my mind, one of my dreams was to be able to overlay my kinect effects on top of milkdrop.

After about a year of research and head scratchin', my dream of overlaying kinect effects on top of Milkdrop is finally limping along.

I've posted a few videos on youtube, which were recorded at the Candlelight. They can be viewed at:

http://www.youtube.com/user/DesignerVIsuals

Or on my web site, http://www.designervisuals.com

Enjoy!
Johan

[Nitorami]

That is quite cool, and actually I often wondered why no viz was played in the bars and nightclubs I know, but the answer is probably that in my small city the guys simply don't know about that or they don't bother. These are mostly poorly paid students and why should they give a toss. Pity, and I can very well imagine these videos with the sort of music they play. It requires a sensible DJ though because the presets need to be quite well selected to fit the music. I would not just have milkdrop running and select the presets on randon.

However I personally would play visuals with whatever easy going, fun disco style CDs, rather than live bands. I find the large screen background somewhat overwhelming and, well, impolite to the band... I trust it is ok with them but still feel it distracts from the persons, particularly with that sort of music. But I concede I do not know how it comes across live.

Would you mind to look up candyrats promo videos for Ewan Dobson on youtube ? I would be curious to know what you think about these.

Martin

[JohanSebastian]

> It requires a sensible DJ though because the presets need to be quite well selected to fit the music. I would not just have milkdrop running and select the presets on randon.

I Definitely relate to what you are sayin' Martin. I have been going through the 5k MD presets I've downloaded from various sources, and most of them are, well not that great, or they are variations/experiments on a given theme.

I started off by letting MD pick them at random, and whenever I saw a preset I really thought was kewl, I looked back on my 'puter screen and jotted down the name of the preset. I then started moving them into a Favs directory.

Now that I have the kinect effects overlaid, I immediately saw where I needed to put together another subset of presets which have certain attributes (like originating from the middle of the screen, and coming out towards the viewer, and really drawing the viewers eye to the singer(s) in the band).

> I find the large screen background somewhat overwhelming and, well, impolite to the band... I trust it is ok with them but still feel it distracts from the persons, particularly with that sort of music. But I concede I do not know how it comes across live.

The reaction from various bands/band members has been varied. Some of them felt 'threatened' by what I was doing. Most though LOVE it, ie the ones that have a vision for the 'bigger picture'. I mean come on, the more successful bands (like Roger Waters) not only have graphics, its a BIG part of their concerts (such as The Wall concert I saw a year ago. The graphics were mind blowing!).

Now here's a VERY interesting thing that happened. When I started using the kinect effects, it was like the band members (and the audience both) suddenly took notice, in a BIG way. Before I started using the kinect, the band seldom mentioned my name when introducing the band members.

The 1st night I started using the kinect effects (the fire effect in particular) the lead singer of the band mentioned/thanked me 3-4 times during the show. Its been that way ever since.

While the MD effects, by themselves (even with using the MD sprites, showing the band members pics with their names alongside etc) were pretty cool, and got some attention, when I introduced the kinect effects, things Radically Changed! Note: I have the kinect hanging in front of the stage, and there is a small area in front of the band where people can dance. They absolutely LOVE seeing themselves up on the screen while they are dancing.

The other thing is when I goto the club, on nights Im not doing a show, I have people walking up to me constantly asking me "what? no graphics show tonight?"

> Would you mind to look up candyrats promo videos for Ewan Dobson on youtube ? I would be curious to know what you think about these.

Oh thanks for pointing me to these vids Martin. Those are very kewl! Thats pretty much what I envisioned a year ago, to have a combo of the band members in the foreground, with MD in the background.

I imagine the way these vids were done was in post processing/editing. My goal is to do it in real time (and also do effects on the band members bodies, like my kinect effects do).

One of the things I started working on is to put together YAE (yet another effect) which uses the depth data in conjunction with the RGB data (the kinect has both an rgb camera AND the depth camera/sensor in one unit. It also has 4 mics built into it). The goal is to do exactly what I am seeing in these videos to which you pointed me. In other words "cut out" the video of the band members and overlay the rgb video on top of MD.

I ran into one snag, which is the two streams of data are not calibrated, so they dont line up. I will figure out how to register the 2 though, I just need to figure it out.

I've been a programmer for 35 years, so its gettin' harder for 'puters to fool me, for very long.

[JoeFerg]

Milkdrop question

Johan,

Have you looked into making some type of microphone "input" function for Milkdrop? There has been a now-closed forum discussing the potential to turn MD into a real-time, mic-triggered visualization tool. Please indulde me with your feelings on this, oh wise one... 35 years means that you dealt with IBM punch cards. Cheers to that!

I talked to Ryan Geiss, the author, years ago about Geiss' screensaver mode. Milkdrop was not available back then (2001 or so), but I wonder if I/we could bug him about this as well.

-JF

[JoeFerg]

Rephrased question

Johan, would you qant to help me??

Right now, my musical act uses a homemade lighting rig (e.g. http://vimeo.com/35124669), but I've always wanted to include Milkdrop, or some kind of real-time graphics dispay in the backdrop, kind of like you've done. Please send a message and we'll talk... We have a programmer and a physicist in the group; we can communicate the technical details I'm sure!

[Flexi]

you can use a linein for winamp and milkdrop, that's easy: Press Ctrl+L and type in "linein://" (without the quotation marks)

btw, someone has ported my Milkdrop-inspired shader experiment from WebGL to OSX/Kinect recently. There is also a link to the source code: http://rel.me/2012/05/05/turing-flui...ct-integration

[JohanSebastian]

Joe
I just saw your posts. For some reason this BB doesnt send me email when others reply to threads (where ive posted. Strange). This BB is also very difficult to use on my phone (no mobile app?).

I also dont see how to send you an email directly.

So how about if you goto my web site (DesignerVisuals.com) and shoot me an email thru my site?

Im always curious to see/talk about what others are doing in the area of special effects.

In the meantime im continuing to do R&D with the kinect and milkdrop.

I nade a stab at getting the old MD code to compile, without much success. I talkwd to ryan geiss and he in turn talked to nullsoft about making the newer version of MD open source. No go yet.

So im looking at projectM to see if it will work.i was able to get it to compile. Thats as far as i got..so far.

Johan