Debugging feature extraction of images
A big part when writing code for visual recognition in cognitive computing is to extract information aka. features from images. For example we need to understand colour information and want to see a histogram chart on a certain RGB or HSV channel. OpenCV offers a nice way to combine image analytics and image manipulation. In combination with the library mathplot it is relatively easy to analyse the picture, extract features and write debug information as a chart to the picture itself.
Getting a test picture
For this demo I will choose a picture from a webcam of the Observatory Friolzheim. On astrohd.de we can get a life picture taken from the observatory dome. Beside the webcam pointing to the building there are also weather information (very helpful when mapping images features) and a AllSky cam.
Getting the software and libraries
As described in this blog article there is a nice docker container for openCV available. We can use this one and just add missing libs and tools with a minimalistic Dockerfile:
|
1 2 |
from victorhcm/opencv:latest RUN pip install matplotlib |
and build it with
|
1 |
docker build -t mathplot-opencv:latest . |
Python file to get the chart for debug visual features
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
import cv2 import glob import numpy import matplotlib matplotlib.use('Agg') import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(111) ax.set_xlim([0, 255]) for fullname in glob.glob("data/*.jpg"): filename = fullname.split('/')[1] name = filename.split('.')[0] image = cv2.imread(fullname, cv2.IMREAD_COLOR) histogram = numpy.bincount(image.ravel(), minlength=256) histogram[:2] = 0 histogram[250:] = 0 weights = [0.3, 0.4, 0.3] histogram = numpy.convolve(histogram, numpy.array(weights)[::-1], 'same') maxindex = numpy.argmax(histogram) print maxindex ax.plot(histogram, 'blue') fig.canvas.draw() imga = numpy.fromstring(fig.canvas.tostring_rgb(), dtype=numpy.uint8, sep='') imga = imga.reshape(fig.canvas.get_width_height()[::-1] + (3,)) imga = cv2.cvtColor(imga, cv2.COLOR_RGB2BGR) imga = cv2.addWeighted(image, 0.7, imga, 0.3, 0) cv2.imwrite("debug/%s.png" % name, imga) plt.cla() |
Line 5
This one is very important if we want to run this matplotlib “headless”. Meaning without a graphic export attached to the runtime. Like in a docker container or on a remote server.
Line 8-10
We create a matplot lib figure with subplot here where we are going to add the chart.
Line 16
As openCV uses numpy to work with images, the image itself is displayed in a multidimensional array and image.ravel() flattens this to a one dimensional array. Bincount just counts the amount of unique numbers used in this array. So with this line we just create a new 1 dimensional array of all the RGB colours in the image which is basically a histogram of the image. By doing so we mix all colours together, which is ok in this sample or a classic histogram. However if we want to see only one channel we can use
|
1 2 |
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) h, s, v = cv2.split(hsv) |
to extract a single channel.
Line 18-19
If the picture contains very shiny or very black parts the high and low values like <10 or >250 are over proportionally high. This will distort the histogram sometimes. These two lines just resets the histogram array on the upper and lower end to 0.
Line 21-22
This just smooth the histogram.
Line 23-24
Argmax returns the index of the highest number in a array. This one can be considered as a feature of the image. For example if we want to rank or sort our webcam pictures by daytime regarding to the sunlight this features gives us an indication. If we are using this feature the lines 18-19 becomes very important here to eliminate the reflexions in a picture. For example in the demo image the observation dome. The upper pictures contains the chart which already blanked out parts. The original chart looks like the one on the right.
Line 26-27
The matplotlib chart is drawn here into the subplot. Still the matplotlib chart and the image itself are “stored” in different libraries.
Line 29
Here the magic happens. fig.canvas.tostring_rgb exports the chart from matplotlib into a string and numpy.fromstring imports this string back to a numpy array which is like the picture we imported at the beginning.
Line 30
Here we reshapes the newly created image to the same size as the imported image.
Line 31
OpenCV can handle all kind of different image colour representations, like RGB, HSV aso. The fromstring importer reads in RGB but our image is in BGR, so this line just converts the colour representation.
Line 32
Here we combine both image with a weighted parameter of 70% of the original image and 30% of the chart.
Line 33
Finally we save this image in a debug folder with the same filename but just because we can as a png file.
Line 35
Finally we clear up the used memory. This is very important when we convert a huge amount of images.
How to use this for visual recognition
By extracting features in images this little piece of code helps us to add the feature directly in a debug image. In our example we only used the histogram and extracted the colour with the highest number, but by having the histogram in hand we could see what was the underlying information the sorting algorithm used. Sorting images in night, dusk/dawn, daylight can be a way of preparing pictures for further processing in neuronal networks. By browsing through the images in one of this sorting folders we can easily see what went wrong in a picture which does not belong in this group. Typical problem is night and day pictures are missorted because of reflexions.