Ambient Light TV with a Raspberry Pi and a Webcam (Part 2)

in

In this post I'll go into more details about how I built the ambient light rig. Here is the previous part in case you missed it. I'm not the first one to make one of these, so this is not going to be a perfect instructable. Instead, I'll go deeper into the things I did differently (like the camera), and link to the resources I used for all the other stuff. Here we go!

Most rigs out there use an Arduino linked to a pc, which has the benefit of being relatively easy to implement but the downside is that you can only use it with a video stream directly from the pc's graphics card (for most people, this is acceptable). You can buy ready-made systems that work in a similar way: LiveLight.

But for myself, I wanted every feed to work: PC, Cable TV, Apple TV and my gaming consoles.

For this to work, I would need access to all the feeds going into my TV. Intercepting a HDMI feed before it goes into the TV is technically possible, but the gear I needed would have been way too expensive. So the only other cheap alternative was to use a webcam, knowing that frame rate, lag and color inaccuracies were going to be an issue.

The camera is best placed pointing directly at the screen to avoid glare and viewing angle color inaccuracies. In my case, I only had two viable options:

As you see in the second picture, I had to close the curtains because at that angle the TV reflects everything from the window adjacent to it. This wasn't a big problem since I prefer to close the curtains when watching a movie or playing games anyway.

Pure blacks tend to be more blueish too, but I can live with that. This actually has a nice effect when the screen goes dark, as the whole rig glows blue providing some light in the room.

For the webcam, I used a Logitech C250 (because I already owned one). It can capture at different resolutions and has a manual focus ring. The manual focus ring is great, because you can intentionally set it to out of focuse to get color averaging for free.

The Adalight tutorial was a good guide for building the frame. This is what I used:

The following gallery shows my process making the frame.

At this point, I started to work on the server code for the Raspberry Pi. The first implementation was a Python program running in Debian linux LXDE gui mode. This was a failed approach for two reasons:

  1. The GUI mode was using too many resources on the Pi, so the experience would be quite sluggish
  2. The Python GPIO library (wiringPi) I was using wasn't fast enough to update 50LEDs.

So I decided to go with a custom C implementation using the wiringPi C++ library. I wrote all the code on my pc and synced it to the pi using rsync. The source code for the server is hosted on github. I'm working on a readme, but the code should be pretty comprehensive if you're familiar with C++.

I used OpenCV to capture frames from the webcam, jansson as a JSON library for saving settings, Log4CPlus as a logging library, and Boost for network communication with a client program running on my PC.

The client program is used for tweaking camera color settings and to set the capture bounds of the frame. It communicates directly with the server on the raspberry Pi. I'll go into more details of the client app in Part3. Here's how it looks:

At this point I had everything pretty much working, and the main task now was to clean everything up and attach the Pi to the frame.

Capturing at full 640x480 resolution was really slow, I expect this is due to the limited USB bandwidth on the Pi. So I had to go all the way down to the lowest capture size: 160x120. This gives me a relatively stable capture framerate of 20FPS.

The LED frame is lagging around 2 frames behind the TV. This is noticeable, but I've found that you eventually get used to it. Blending previous frames together with the current one also helps a lot for smoother color transitions on the LEDs.

The other reason for the slow framerate is the Auto Exposure setting on the webcam. Basically, when the image on screen is dark, the webcam takes longer to capture the frame (longer exposure). The driver I'm currently using on the Pi (using Debian Linux) doesn't allow me to set this to a fixed exposure. I'm still trying alternatives and will post once I have a solution.

All in all, I'm pretty pleased with the results. If I manage to turn off Auto Exposure, I think I might be able to hit 30FPS. (Edit: Confirmed! I managed to set a fixed Exposure setting, and I'm getting a smooth 30fps now.)

I'm eventually going to write Part3, which is more about the client. All the code is on github, so don't wait up.

If you're interested in building one of these, here are some resources I used to get me started:

Ambient Light TV with a Raspberry Pi and a Webcam (Part 1)

in

After finishing my Arduino 8x8 Led Display, I got a Raspberry Pi, and this is what I made with it: A 50 Pixel RGB ambient light rig for my TV.

The colors are sampled from the edges of the TV screen using a webcam. After sampling from the captured frame, each RGB led is updated with the appropriate color on screen. Watching movies and playing games with it turned on is awesome!

The initial prototype used an Arduino. The color sampling was done on the PC and sent to the Arduino with serial output. However, with that setup I could only use it when my computer was the input feed (cloned monitor to TV). I also managed to break my Arduino, so that forced me to use the Pi instead.

With the webcam approach, I can use it with any feed: cable TV, XBox, Apple TV, Steam Big Picture and the Raspberry Pi itself. I briefly looked into HDMI splitters and frame grabbers but that was all way too expensive.

I took a bunch of work-in-progress pictures while making it, I'll post those in part 2. Overall it took about 6 weekends to complete, over a period of 7 months.

Read part 2 here

Arduino 8x8 LED Display

in

Last year, one of my colleagues at work was showing off his awesome automatic driving RC car project. He introduced me to Arduino and showed me where to get it. I got the SparkFun Arduino Starter Kit with a multimeter. It comes with lots of little components and a bunch of tutorials. The tutorials were a great start, but honestly I quickly got bored. For me, the best way to learn is to just get my hands dirty and think about issues as they occur. YouTube is full of great Arduino projects, so I quickly found something I wanted to make: A 8x8 LED screen that I could potentially expand and make mini games for. 

I found a similar project on Instructables so it was easy to figure out what components were needed. I ordered them from various websites like: Hobby Tronics, Cool Components and Rapid Online.

Here are some work in progress shots. It took about 2 months to complete, including wait delivery wait times and project planning.

After all the soldering, most of the work was C++ code, but for me that was the easy bit! You can browse it on GitHub.

ArduinoLEDMatrixSmall

And there you have it, a retina display for the Arduino! (viewing distance 100 meters)

Eventually, I wanted to expand the screen to a 24x16 matrix (adding 5 more of the same blocks), but noticed I was going to need a separate power supply. All the power was currently coming directly from the Arduino, which is just enough to drive 64 leds. Adding more, would mean I needed to change the circuit layout, or add a whole new breadboard.

So I decided to park this project and move on to more exciting things.

Blender Keyboard Shortcut Explorer

in

I've been learning Blender on and off lately, and one thing I learned early on is that shortcut keys are super important in Blender. It has been difficult to memorize the ton of shortcuts without looking at user made cheat-sheets and keyboard layouts. But most of them are never up to date with the latest version so I usually have to resort searching community forum posts. Last weekend I was messing around with python in Blender and I saw that it was possible to access all the keyconfig data. I exported all the data to JSON and had fun making a html keyboard that had all the shortcuts on it. It even updates live with modifier key presses on the keyboard.

Check it out here: waldobronchart.github.io/ShortcutMapper/

blender-shortcuts

It's a lot easier to keep up to date with the latest Blender because of the awesome data-driven-ness of it.

Depending on community response, I'm thinking of adding things like: search functionality, export custom key configs, upload code to Github and release it as a plugin for Blender.