In this article I would like to share what I learned about colours while working on the algorithm of Hue Camera. This article is directed to those who are interested in how Hue Camera works and those who would like to understand more about some colour related concepts. Because of this I will only explain how the concepts of the algorithm work and will not show how this is done using code samples. Please use this article for informational and educational purposes. Please don’t use it to exactly copy it, but to understand it and make great new stuff.

Please let me know if you have questions or improvements to the article in the comments!

The goal of this algorithm is to find the most important colour for the overall ambiance in a picture. This cannot simple be done by calculating the average value of all the pixels, because you will likely end up with a greyish dull colour. Let me give you an example: in the image below, you immediately notice the purple sunset over the city. This is very important to the overall ambiance. The image below gives an comparison of the result of the average colour and the colour calculated by the Hue Camera’s algorithm:

As you can see the average is a dark brown colour, because the purple/orange colours get blended with the black, green and yellow of the city. Hue Camera ignores these other colours because they are not significant enough for the ambiance.

How is this done?

Step 1: Categorise colours

As the name suggests, RGB colours consist of 3 values: red, green and blue. This means we can easily visualise the colours in 3D space using the 3 available axis: x, y and z. What you end up with is a cube which contains all available colours:

The cube is then divided in multiple smaller cubes which represents a specific range of colours. The goal of the next steps is to find out which of these smaller cubes is the most important. It’s important to find the right balance in the size of the cubes: smaller cubes will result in a more specific but unstable result, while bigger cubes will result in a more stable but a less specific result.

Step 2: Prioritise colors

To determine which of the cubes is most important to the overall ambiance of the picture, the groups are sorted based on the occurrence as wel as the saturation and the luminance of the color.

Saturation
The saturation is a measure to describe the intensity of the colour. Colours with 0% saturation are black, white or a shade of grey in between. The higher the saturation value, the more colourful it is. This is illustrated by the image below: the saturation starts at 0% at the inner circle and will increase to the edges of the circle:

In this algorithm, higher saturated colours are more important because they will also stand out in the picture.

Luminance
Luminance is a measure to describe the perceived brightness of a color. It’s value is not only based on the lightness of a colour, but also on the hue because each hue naturally has an individual luminance value. This can be seen in the illustration below where all colours have a saturation level of 100% and a brightness of 50%:

You can for example see that a fully saturated blue colour might appear to be darker than a fully saturated yellow colour. This is important to take in consideration when finding the colour which stands out in a picture.

Occurence
Even if the luminance and/or saturation value are ideal, it won’t stand out if it’s only 1 pixel. The more a colour appears, the more important it will be to the overall ambiance. Also here the balance is crucial: the more important the occurrence will be on the result, the more it will become like the average color because that’s only based on occurrence. On the other hand you will get unexpected results when you will ignore the occurrence for very dark or very light scenes.

Importance
The combination of the factors above determine on the overall importance of the colour. A highly saturated colour is still black when the luminance is at 0% and if the saturation is 0% you will end up with a shade of grey, so the importance is the right balance between these factors. The most important colours are those around 50% luminance and a high saturation value. Since this is a very important part of the algorithm, I won’t tell you the exact equation, but it will be a lot like the following:

I = (1.0f - abs(L - 0.5f) * 2.0f) * S + C

Where:

I = importance
L = luminance (0.0 – 1.0)
S = saturation (0.0 – 1.0)
C = a constant value > 0

Step 3: Finding the most important group

So now we have multiple buckets (the color groups) containing pixels of which we determined the importance. The most important bucket is the one of which the combined importance of the pixels in the bucket is highest. So after inspecting all buckets on their contents, we know which range of colours is the most important to the overall ambiance. The other groups can be ignored from now on.

Step 4: Find the specific colour of a group

We are now left with 1 bucket of pixels, but although the colours are much alike since they are in the same range, they are not exactly the same. To determine the specific colour this group represents, all we have to do is take all the pixels in the bucket and calculate the average value (still taking the importance of the specific pixel in consideration).

Repeat

Now we have found the most important colour in this frame but unfortunately we don’t send this colour to the lights and call it day. That’s because the captured video stream captures quite a lot of frames each second (probably 30), so this frame might be completely different from it’s neighbours. So we repeat steps 1 to 4 for a couple of frames and calculate the average colour found for all these frames and then we send it to the lights. This is also a matter of finding the right balance: the result will be less stable when you use less frames, but the algorithm will become slower and chances are that the colour will really change during these frame when using to many.

Please let me know if you have questions or improvements to the article in the comments!