The Crucial Role of Cameras in Autonomous Driving Systems

In autonomous driving systems, cameras play a role as crucial as the eyes of a human driver, arguably being the most essential sensor in the entire autonomous driving setup.

Compared to millimeter-wave radar and LiDAR, cameras offer numerous advantages, including lower costs, rich color information, and the ability to provide semantic details. Due to these advantages, cameras are prevalent in smart vehicles, with a single vehicle equipped with 3-5 cameras at a minimum and sometimes even more.

How much do you really know about this widely used and particularly vital sensor?

Basic Structure of Cameras

Let's start by examining the basic structure of cameras.

Automotive cameras mainly consist of optical lenses, image sensors, image signal processors (ISPs), serializers, and connectors.

Optical Lens (Lens)

The optical lens focuses light and projects objects in the field of view onto the surface of the imaging medium. The lens also includes an infrared filter (IR Filter) that filters out infrared light, invisible to the human eye, leaving only visible light.

The focal length of the lens determines the field of view (FOV) of the captured image. A longer focal length results in a narrower field of view. For instance, a long focal length lens may be added for forward-facing views to capture objects like traffic lights at a distance of 150 meters, typically with a FOV of around 30 degrees.

Conversely, a shorter focal length provides a larger field of view, such as fisheye cameras used in 360-degree surround view systems (AVM). Fisheye cameras require a very large field of view for image stitching, with a focal length generally less than 16mm. The horizontal FOV of fisheye cameras is usually greater than 170 degrees, sometimes exceeding 180 degrees. However, the trade-off for a large field of view is image distortion, especially near the edges, requiring distortion correction.

Major optical lens suppliers include Sunny Optical, Largan, and OFILM.

Image Sensor (Sensor)

The image sensor utilizes the photoelectric conversion characteristics of photoelectric devices to convert light signals on its sensitive surface into electrical signals.

Automotive cameras commonly use CMOS (Complementary Metal Oxide Semiconductor), which is the most critical and expensive part, accounting for approximately 50% of the cost.

CMOS determines the most important parameter of the camera - pixels. The commonly mentioned 2 million or 8 million pixels mean there are 2 million or 8 million photosensitive points, indicating the number of photodiodes responsible for photoelectric conversion.

Another crucial parameter determined by CMOS is the frame rate. The frame rate represents the number of times CMOS updates per second. A higher frame rate results in clearer images, with common rates being 30 or 25 frames per second.

Currently, the automotive CMOS market is mainly dominated by two major manufacturers: ON Semiconductor (Onsemi) and OmniVision Technologies (OmniVision), with high market concentration.

Image Signal Processor (ISP)

The ISP is used to process RAWDATA format image data output by CMOS, enhancing image quality by removing noise, controlling exposure, and performing automatic white balance, ultimately outputting RGB or YUV formats.

The ISP can be integrated into the camera module, either directly within the image sensor or as a standalone chip (integrated into the domain controller).

If integrated into the camera, the output is the ISP-processed RGB/YUV image. Otherwise, the output is RAWDATA (raw data). RAWDATA is transferred to the domain controller, where the integrated ISP is used for processing.

Additionally, there is a serializer inside the camera because video signals need to be transmitted using standard protocols like GSML, requiring conversion to serial data before transmission.