Horizon straightening

When you install a panoramic multisensor camera you typically tilt its sensors to capture more of the ground and less of the sky. But because of how the image is projected, the tilt causes the horizon in the resulting wide-angle, panoramic overview image to be bent.

Horizon straightening is a feature in multisensor cameras that compensates for the tilt. By straightening the horizon and removing distortion, the feature enhances the viewing experience and provides an image that is perceived to be straight. Objects that are vertical in real life stay vertical also in the image.

Due to the projection, the corners of the straightened image will lack some sensor information. These areas can be preserved (displayed as black pixels) or gently stretched to fill in the corners, depending on user preference. In both cases, maximum coverage is maintained along the horizon.

Horizon straightening is a feature in panoramic multisensor cameras for providing a visually appealing image that is perceived to be straight. The feature compensates for any tilt of the camera, which would otherwise bend the horizon. Side effects of this compensation are automatically taken care of to maintain coverage and preserve a rectangular image.

This white paper explains briefly how horizon straightening works and how it can benefit panoramic camera use.

Panoramic multisensor cameras use multiple sensors to provide a single, wide-angle panoramic overview. Thanks to recent progress in the development of stitching algorithms, the images from all the sensors are seamlessly stitched together to form one continuous image without any visible borders, gaps, overlaps, or color differences between each sensor’s contribution to the whole image.

When you install a multisensor camera, you typically tilt the camera (or rather the sensors) to capture more of the ground and less of the sky. This causes the horizon to be bent because it is not in the middle of the image.

The way the image is projected in a panoramic camera requires everything in the view to be bent around a center line. To maximize the use of each image sensor, the image is usually projected so that this line is in the center of the image. If the horizon happens to be located on the center line, it will be straight. But if the camera has been tilted so that the horizon is not in the middle of the image, the horizon will be bent.

The above panoramic image is fully functional but the viewing experience could be further improved. The fact that the image content is not straight may also affect the results of object detection analytics, if any such video analytics are applied.

Horizon straightening compensates for any physical tilt of the camera, thereby removing distortion and enabling the horizon to be straight even though it is not in the middle of the image. All objects and lines that are vertical in real life stay vertical also in the image. Horizon straightening thereby enhances the viewing experience.

Horizon straightening maintains the 180-degree coverage at the horizon and the camera’s vertical coverage at the vertical center line. The pixel density on the horizon is slightly affected after adjusting the horizon from a curved to a straight line with the same horizontal width.

1. Left: Without horizon straightening, the image is bent around the camera view’s center line (marked in yellow). Due to camera downtilt, this line is not at the horizon.
2. Right: With horizon straightening, the camera downtilt is compensated and the image is projected around the horizon.

When horizon straightening bends the original, rectangular image, it produces a non-rectangular image. In this process, the corners of the original image are cropped out, which means that the sensor information from the corners of the camera view are lost. This is usually not an issue, since the camera’s corner view areas typically represent a rather small number of pixels of sky and wall (if the camera is wall mounted).

The straightened image will lack sensor information in its corners. This is because the camera’s coverage is not enough to fill all the pixels of the non-rectangular image. You can choose to preserve the corners as black, possibly combined with cropping the image so the corners are not visible anyway. But horizon straightening also has the ability to create a good visual result by gently and smartly stretching the available data to fill in the corners and provide a complete picture without black corners. Maximum coverage is kept along the horizon both in a stretched image and in an image with preserved, black corners.

The stretching works differently depending on what amount of stretching you choose. Set to maximum, stretching takes place mainly in the bottom part of the image and may have a large effect on the appearance of objects that are located there. Set to minimum, stretching affects a larger part of the image, but with smaller effects throughout. Minimum stretch will also keep the noise level down, which may be especially important in low light scenes.