What can you do with OpenCV?
A walkthrough of the most frequently used and fundamental functions of the OpenCV library in Python.
What is OpenCV? Put simply, it is an image processing library.
That single sentence is enough to sum up OpenCV, yet the library offers far more. It was first written in C++ as a very fast, open-source computer vision library for operations on images. Today OpenCV can be used from Python and Java too, and it is popular in machine learning for pre-processing and post-processing, as well as in traditional computer vision algorithms. With it you can read, edit, convert and write digital images, apply morphological and geometric transformations, and even follow objects or extract features using classical computer vision methods.
This tutorial walks through the most frequently used and fundamental OpenCV functions in Python.
Installation
With venv:
pip install opencv-python
With a mamba environment:
mamba install -c conda-forge opencv
With poetry:
poetry add opencv-python
Whichever tool you use, you can pin a particular OpenCV version like this:
opencv-python==4.6.0.66
Image Loading and Basic Manipulations
Let’s begin with the basic functions for reading an image, showing it, and writing it to a file. Throughout the tutorial we will work with an image every computer vision practitioner knows: lenna.png.
import cv2
import numpy as np
import sys
# load the image
file_path: str = "lenna.png"
img: np.ndarray = cv2.imread(filename=file_path,flags=cv2.IMREAD_UNCHANGED)
# check whether image was correctly loaded
if img is None:
sys.exit(f"Could not read the image {file_path}.")
# display image and wait for keyboard input
cv2.imshow(winname="Display image", mat=img)
# wait for the user key press and hide the window
key = cv2.waitKey(delay=0)
# if the `s` was pressed save the image
if key == ord("s"):
cv2.imwrite(filename="lenna_out.png", img=img)
The flags passed to cv2.imread let you read an image in various modes, such as IMREAD_UNCHANGED, IMREAD_GRAYSCALE, IMREAD_COLOR and others. The complete list of modes is available under cv2.ImreadModes.
The cv2.waitKey function waits x milliseconds for a key to be pressed in an OpenCV window. With the delay parameter set to 0, it waits for keyboard input indefinitely. That is why 0 is the usual choice, since it waits until the user reacts. If you pass any other value, the window closes either when that time runs out or when a key is pressed, whichever comes first.
Next, we can read the shape of our image:
h, w, c = img.shape
BGR vs. RGB
Keep in mind that OpenCV loads digital images in BGR format by default. In other words, the first channel of OpenCV’s 3-channel array is Blue rather than Red. The cv.imread function takes this into account and the images are shown correctly, but once we try to show an image loaded by OpenCV using matplotlib, something odd happens, as the output in Figure 1 shows.
import matplotlib.pyplot as plt
import matplotlib.image as AxesImage
imgplot:AxesImage = plt.imshow(X=img)
plt.show()
Figure 1. The Lenna image in BGR format.
This happens because the Red and Blue channels are swapped by default. You can undo the swap with cvtColor.
rgb_image: np.ndarray = cv2.cvtColor(src=img, code=cv2.COLOR_BGR2RGB)
Figure 2. The Lenna image in RGB format.
Now everything looks as it should again.
OpenCV image types
The simplest OpenCV image type is np.uint8. This is a 1-byte unsigned integer, so it holds values between 0 and 255. A grayscale image uses 1 byte per pixel, a color image uses 3 bytes per pixel, and an image with an extra alpha channel uses 4 bytes. Note that jpg images never carry an alpha channel, so if your image needs transparency, go with png.
By default, an image loaded with OpenCV has the np.uint8 type:
>>> print(img.dtype)
uint8
Before a digital image is passed to a neural network, it has to be converted to float, and it is normally normalized as well. The following code does this:
normalized_img: np.ndarray = img.astype(np.float32)/255.0
or
normalized_img: np.ndarray = cv2.normalize(img.astype(np.float32), dst=None, alpha=1.0, beta=0.0, norm_type=cv2.NORM_INF
To normalize an image so that its smallest value becomes zero and its largest value becomes 1, we can write:
float_img: np.ndarray = img.astype(np.float32)
normalized_img: np.ndarray = (float_img - np.min(float_img))/(np.max(float_img) - np.min(float_img))
or
normalized_img: np.ndarray = cv2.normalize(img.astype(np.float32), dst=None, alpha=1.0, beta=0.0, norm_type=cv2.NORM_MINMAX)
In this case the image ends up in the 0-1 range. Quite often, though, we want to normalize an image using its mean and standard deviation. Both numpy and OpenCV can do that:
mean: tuple[float] = (0.485, 0.456, 0.406)
std: tuple[float] = (0.229, 0.224, 0.225)
float_img: np.ndarray = img.astype(np.float32)
normalized_img: np.ndarray = (float_img/255.0 - mean)/std
You can also do it straight in a deep learning framework, for instance torchvision. This approach is the better one, because it typically benefits from GPU acceleration.
import torch
from torchvision import transforms
transforms = torch.nn.Sequential(
transforms.Normalize(mean, std),
)
normalized_img: torch.tensor = transforms(torch.tensor(float_img/255.0).permute(2,0,1))
Returning to image types: medical images such as x-rays, as well as depth images, are typically stored as 16-bit grayscale maps. Once such an image is loaded into memory, it has the uint16 type, with values in the 0-65536 range.
>>> import pydicom
>>> dicom = pydicom.read_file("./heart.DCM")
>>> data = dicom.pixel_array
>>> print(data.dtype)
uint16
To display such an image, we have to convert it to uint8.
normalized_img: np.ndarray = cv2.normalize(data.astype(np.float32), dst=None, alpha=256.0, beta=0.0, norm_type=cv2.NORM_INF).astype(np.uint8)
cv2.imshow("dicom_image", np.transpose(normalized_img[:3],(1,2,0)))
cv2.waitKey(0)
Figure 3. Displaying a dicom image stored on 16 bits. One plane of a knee MR scan. Source: Figshare
The final image type, occasionally used to store binary masks, is bool.
h, w, c = img.shape
mask: np.ndarray = np.ones_like(a=img,dtype=bool)
mask[h//2-20:h//2+20,w//2-20:w//2+20] = 0
img *= mask
cv2.imshow(winname="masked_image",mat=img)
cv2.waitKey(delay=0)
Figure 4. The Lenna image with a binary mask applied.
Basic image operations and image processing
Image histogram
A histogram shows how the intensities of an image are distributed. For every possible pixel value, it counts how many pixels have that intensity. For uint8, the possible values are 0-255. A histogram reveals useful properties of an image, such as its intensity distribution, contrast and brightness. Since a histogram can only be built for a single-channel (grayscale) image, an image with several channels requires a separate histogram for each channel.
colors: tuple[str] = ('b','g','r')
for i,color in enumerate(colors):
histogram: np.ndarray = cv2.calcHist(images=[img], channels=[i], mask=None, histSize=[256], ranges=[0,256])
plt.plot(histogram, color=color)
plt.xlim([0,256])
plt.show()
Figure 5. Histogram of the Lenna image.
The histogram shows that the Lena image contains many high-intensity red pixels. This matches what we see, since the image does look red.
Histogram equalization
Histogram equalization spreads out the histogram to boost the contrast of an image. When the pixels fall only into the lower half of the intensity range (0-127) or only into the upper part (128-255), contrast is low, because all pixels have similar intensities. Once the histogram is equalized, the pixels of the image span all available intensity levels, which raises the contrast.
Figure 6. Histogram equalization applied to the Lenna image.
equalized_channels: np.ndarray = [cv2.equalizeHist(src=img[:,:,channel]) for channel in range(img.shape[2])]
equalized_img: np.ndarray = np.concatenate(np.expand_dims(equalized_channels, axis=3), axis=2)
colors: tuple[str] = ('r','g','b')
fig = plt.figure()
ax1 = fig.add_subplot(221)
ax2 = fig.add_subplot(222)
ax3 = fig.add_subplot(223)
ax4 = fig.add_subplot(224)
ax1.title.set_text('Original histogram')
ax2.title.set_text('Equalized histogram')
ax3.title.set_text('Original image')
ax4.title.set_text('Equalized image')
plt.subplot(2,2,4)
rgb_equalized_img: np.ndarray = cv2.cvtColor(src=equalized_img, code=cv2.COLOR_BGR2RGB)
plt.imshow(rgb_equalized_img)
plt.subplot(2,2,2)
image_histogram(img=rgb_equalized_img, colors=colors)
plt.subplot(2,2,3)
rgb_img: np.ndarray = cv2.cvtColor(src=img, code=cv2.COLOR_BGR2RGB)
plt.imshow(rgb_img)
plt.subplot(2,2,1)
image_histogram(img=rgb_img, colors=colors)
plt.show()
Contrast and brightness adjustment
Brightness
Changing brightness simply means adding a constant to, or subtracting it from, the entire image. This corresponds to moving the histogram to the left or right, with values clamped at 255 or 0 depending on the direction of the shift.
shift: np.ndarray = np.array([0,0,-40], dtype=int)
img = img.astype(int)
img += shift
img = np.clip(a=img, a_min=0, a_max=255)
img = img.astype(np.uint8)
for i, color in enumerate(colors):
histogram: np.ndarray = cv2.calcHist(images=[img], channels=[i], mask=None, histSize=[256], ranges=[0,256])
plt.plot(histogram, color=color)
plt.xlim([0,256])
rgb_image: np.ndarray = cv2.cvtColor(src=img, code=cv2.COLOR_BGR2RGB)
plt.subplot(2,2,2)
plt.imshow(rgb_image)
plt.show()
Figure 7. Brightness adjustment applied to the Lenna image.
Figure 7 shows that subtracting the constant = 20 from the red channels moves the entire histogram to the left, so the image appears less red. If, instead, we shift every pixel, we get the effect of a brightness adjustment.
Brightness adjustment requires converting the image to int64 beforehand, because a plain shift on uint8 values would wrap around.
Contrast
Adjusting contrast amounts to scaling the histogram, i.e. multiplying the values.
scale: np.ndarray = np.array([1.4, 1.4, 1.0], dtype=float)
img = img.astype(float)
img *= scale
img = np.clip(a=img, a_min=0, a_max=255)
img = img.astype(np.uint8)
for i, color in enumerate(colors):
histogram: np.ndarray = cv2.calcHist(images=[img], channels=[i], mask=None, histSize=[256], ranges=[0,256])
plt.plot(histogram, color=color)
plt.xlim([0,256])
rgb_image: np.ndarray = cv2.cvtColor(src=img, code=cv2.COLOR_BGR2RGB)
plt.subplot(2,2,2)
plt.imshow(rgb_image)
plt.show()
Figure 8. Contrast adjustment applied to the Lenna image.
Gamma correction
Gamma correction adjusts image brightness in a non-linear way.
It brightens the dark parts of an image without saturating the rest. This is particularly handy for photos captured in poor lighting, such as the one shown below.
def image_histogram(img: np.ndarray,colors: tuple[str]) -> None:
for i, color in enumerate(colors):
histogram: np.ndarray = cv2.calcHist(images=[img], channels=[i], mask=None, histSize=[256], ranges=[0,256])
plt.plot(histogram, color=color)
plt.xlim([0,256])
colors: tuple[str] = ('b','g','r')
plt.subplot(2,2,3)
plt.imshow(img)
plt.subplot(2,2,1)
image_histogram(img=img, colors=colors)
# gamma correction
plt.subplot(2,2,2)
gamma: float = 0.4
img = img.astype(float)
img = np.clip(pow(img / 255.0, gamma) * 255.0, 0, 255)
img = np.clip(a=img, a_min=0, a_max=255)
img = img.astype(np.uint8)
image_histogram(img=img, colors=colors)
rgb_image: np.ndarray = cv2.cvtColor(src=img, code=cv2.COLOR_BGR2RGB)
plt.subplot(2,2,4)
plt.imshow(rgb_image)
plt.show()
Figure 9. Gamma correction applied to an image with uneven brightness. Source: Canon
With gamma correction the whole image becomes brighter, but no area gets saturated. Figure 9 shows the outcome.
OpenCV in computer vision applications
OpenCV works best for image pre- and post-processing and for classical computer vision tasks. Below we take a quick look at detecting rectangles with OpenCV.
We start from an image that contains two rectangles.
Figure 10. Image of cards used for the rectangle detection task. Source: Sostrenegrene
The first step is to read the image in grayscale mode. This matters, because the thresholding operation we are going to use only works on single-channel images.
# load the image in grayscale
file_path: str = "./cards.png"
img: np.ndarray = cv2.imread(filename=file_path, flags=cv2.IMREAD_GRAYSCALE)
Before thresholding, the image needs a slight blur. This removes noise and gives a smoother thresholding result.
img = cv2.medianBlur(src=img, ksize=5)
We use adaptive thresholding here. Figure 10 contains a shadow, and adaptive thresholding helps compensate for the uneven brightness across different regions of the image.
thresholded_image: np.ndarray = cv2.adaptiveThreshold(
src=img,
maxValue=255,
adaptiveMethod=cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
thresholdType=cv2.THRESH_BINARY,
blockSize=11,
C=2,
)
Thresholding alone already produces decent contours, as Figure 11 shows.
Figure 11. The cards after adaptive gaussian thresholding.
Still, we want to remove the noise before the next step - contour detection. For that, we run a morphological opening on the thresholded image.
kernel5x5: np.ndarray = np.ones(shape=(5,5), dtype=np.uint8)
opened_image: np.ndarray = cv2.morphologyEx(
src=thresholded_image,
op=cv2.MORPH_OPEN,
kernel=kernel5x5,
)
Figure 12. The cards after the opening operation.
Good. The result is somewhat cleaner now, so we can proceed to detecting contours.
contours, hierarchy = cv2.findContours(
image=opened_image,
mode=cv2.RETR_LIST,
method=cv2.CHAIN_APPROX_TC89_L1,
)
Once we have the contours from Figure 12, we move on to filtering them. We want to keep only the contours shaped like rectangles and squares. First we set MIN_ALLOWED_AREA, which discards every contour that is too small (image noise). Then, looping over the contours one at a time, we calculate the minimum enclosing rectangle of each. Next we calculate the area of the contour and compare the two areas. When they are close, the contour may be a rectangle, so we mark it on the image with a rectangle and a label. The complete code is below.
Figure 13 shows the final result. Every rectangle we were looking for was detected correctly.
Figure 13. Final result of the rectangle detection task.
# load the image in grayscale
file_path: str = "./cards.png"
img: np.ndarray = cv2.imread(filename=file_path, flags=cv2.IMREAD_GRAYSCALE)
# check whether image was correctly loaded
if img is None:
sys.exit(f"Could not read the image {file_path}.")
img = cv2.medianBlur(src=img, ksize=5)
thresholded_image: np.ndarray = cv2.adaptiveThreshold(
src=img,
maxValue=255,
adaptiveMethod=cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
thresholdType=cv2.THRESH_BINARY,
blockSize=11,
C=2,
)
kernel5x5: np.ndarray = np.ones(shape=(5,5), dtype=np.uint8)
opened_image: np.ndarray = cv2.morphologyEx(
src=thresholded_image,
op=cv2.MORPH_OPEN,
kernel=kernel5x5,
)
contours, hierarchy = cv2.findContours(
image=opened_image,
mode=cv2.RETR_LIST,
method=cv2.CHAIN_APPROX_TC89_L1,
)
h, w = img.shape
MIN_ALLOWED_AREA: Final[float] = 0.1*h*w
for cnt in contours:
rect: np.ndarray = cv2.minAreaRect(points=cnt)
box: np.ndarray = cv2.boxPoints(box=rect)
box = np.int0(box)
box_area: float = cv2.contourArea(contour=box)
area: float = cv2.contourArea(contour=cnt)
if box_area 0 or area 0:
continue
areas_ratio: float = (min(area, box_area)/max(area, box_area))
if area > MIN_ALLOWED_AREA and areas_ratio < 1.2 and areas_ratio > 0.8:
x1, y1 = cnt[0][0]
# label rectangle detected on the image
cv2.putText(img, 'Rectangle', (x1, y1), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
sorted_bbox: np.ndarray = box[np.argsort(np.linalg.norm(box, axis=1))]
# corners are in order bottom left, bottom right or top left, top left or bottom right, top right
# to be in the drawing order last two corners need to switch places
sorted_bbox_tmp: np.ndarray = sorted_bbox[2].copy()
sorted_bbox[2] = sorted_bbox[3]
sorted_bbox[3] = sorted_bbox_tmp
# draw detected rectangle on the image
img = cv2.drawContours(
image=img,
contours=[sorted_bbox],
contourIdx=-1,
color=(0,255,0),
thickness=3,
)
cv2.imshow(winname="result", mat=img)
cv2.waitKey(delay=0)
Summary
For classical computer vision tasks and image processing, OpenCV is an excellent library. Its biggest strengths are its speed, its support for several languages, and the wide range of image processing functions it provides. This large open-source library was first built in c++, and some implementation choices inherited from c++ are still visible in both python and java. Before neural networks took over, it was also one of the leading libraries for tackling computer vision problems such as text detection, object detection, object classification, object recognition, feature extraction, face detection, image segmentation, motion tracking and object tracking. In my view, OpenCV is an excellent companion to neural networks for processing input images, for example for image filtering, or for camera calibration.