Colour-based target tracking

I have a new project idea, which involves tracking targets in a video stream. Starting off with just tracking colours: