Sure, playing video games is fun, but what if we could have even more fun with it?
The goal of this project is to modify the video output in fun ways á la data moshing and audio-visual feedback.
Check out some more smosh videos on Instagram.
So far I’ve managed to create a nice system for capturing the switch video, mess with it’s chroma- and alpha- levels, and sync them up to some brutish sound detection triggers.
The Max ‘control panel’ of the video manipulation looks like this:
You may be able to see there are a few setup steps, then some controls for switching color planes, selecting colors for chroma-blending, as well as music controls.
The video can be triggered by sound by detecting when the amplitude of a song passes a certain threshold as defined by the yellow bar above it:
I’ve now also modified the patcher to use the beloved-by-community
jit.world object in an attempt to reduce lag and create an opportunity to use shaders & graphics-intensive video effects.
Overall, the whole setup now looks like this:
As you may be able to see, even with jit.world, the modulated output is not very high-res, though it is real (or near-real) time and I had fun playing with the live effects running on their own.
I think this issue may be due to the video capture card I am using, or how I am getting the signal into Max specifically.
More technical details below. Stay tuned for more updates as I progress with this project!
Here’s what the full setup looks like:
…and here’s a diagram of what exactly is going on there:
If you want to try this out for yourself, you can start with project’s Max patches from this GitHub folder. The current best file to use is the
Any and all feedback is welcome, contact me here.
First viable iteration (
handsfree-with-sound patch in the repo)
Credit where credit is due
None of this would be possible without the resources available on the web and through my education. Specifically:
- Matt Romein who taught me in Live Image Processing & Performance. You can find some resources and tutorials here.
- Jeremy Sykes for feedback & support of this particular project’s approach & hardware.
- MH Rahmini for play-testing & feedback
Other resources that helped me and may help you: