Exposure control mechanisms for CMOS image sensors fall into one of two categories: global shutter or rolling shutter. A global shutter operates by exposing all the pixels in the active area at a single instant in time, while a rolling shutter exposes different areas of the pixel array at different points in time. Specifically, each row on the pixel array is exposed in sequence. One way to easily visualize how a rolling shutter works is to imagine that the active area is covered by a completely opaque screen. This screen has a single slit which aligns with the rows of the pixel array. During operation, the screen moves in discrete steps, stopping such that each row in the pixel array is directly underneath the slit for a predetermined amount of time (figure 1).

Electrically, there are two signals, readout and reset, which control this process. The reset signal erases the values currently stored in a row and thus triggers the beginning of the exposure period. The readout signal trails the reset by a number of rows, which corresponds to the exposure time. This signal stops the integration period and begins the process of reading the values stored in each pixel. Ideally, any exposure time between 0 and the frame period minus Tr is possible.

One of the most powerful advantages of using this technique is that exposure and pixel read-out can occur simultaneously such that the effective exposure time per time period is increased. Since only certain rows in the array are being exposed at any given instant, the rest of the pixels in the array can be read out.

In contrast, global shutter techniques must stagger exposure and read-out stages, resulting in a net reduction in exposure duty cycle. The exposure advantage provided by the rolling shutter becomes significant especially for high-resolution imagers, since total read-out time increases with total pixel count. Furthermore, cost limitations in consumer-grade CMOS sensors drive manufacturers to fit more, smaller pixels into the same available silicon area. Since smaller pixels require greater exposure time, the added duty cycle advantage of a rolling shutter becomes even more valuable for these devices. Of course, there is no such thing as a free lunch, and this advantage comes with several tradeoffs.

Figure 2. Intensity Flicker Example. Pixel brightness is proportional to the intensity integral, I(t). For a rolling shutter, different rows in the image are exposed at different times, resulting in visible brightness variations.

Since a rolling shutter exposes different areas of the pixel array at different moments in time, it is easy to predict that this technique may produce some distortions or artifacts. We will focus on the phenomenon of intensity flicker, which occurs when the light source used to illuminate the scene has a time-varying intensity function.

What is intensity flicker and why does it happen? Imagine taking a picture of a scene using a light source with voltage-dependent brightness. If we use a periodic voltage source to power this device, we would obtain a time-varying light intensity of the same fundamental frequency. Ignoring non-linearity in the response, the output of each pixel will be proportional to the intensity integral:

Where:

T_e = the exposure time
i(t) = the positive-valued intensity function

Intensity flicker occurs when different rows in the pixel array are exposed at different times such that I(t) has a different value. This results in an alternating pattern of bright and dark rows, which corresponds to the maxima and minima of I(t) (figure 2). Figure 2 shows a portion of an image taken with a light source powered by a 60 Hz voltage input.

With this understanding of what fundamentally causes intensity flicker, we can develop a model to further characterize this phenomenon.

One of the tell-tale signs of intensity flicker is dark/bright horizontal banding appearing across the image, as shown in figure 2. How do you calculate the spatial frequency of these bands? Suppose we are using rolling shutter exposure on a sensor with an active area containing a rows and b columns, with k rows of inactive “blanking.” Given the period of the intensity integral TI(t) = 1/fI(t), and the row exposure time Te, we can calculate the spatial frequency, in band-pairs/row of the intensity flicker, since the frequency of the light intensity integral function, I(t) is the same as the frequency of the light intensity function, i(t).

For example, if we were using a 1080p sensor (1920×1080) and an exposure time of 1/120 seconds to capture a scene illuminated with a 60 Hz periodic light source, the flicker spatial frequency would be approximately 1/2 band-pairs/row, that is, alternating dark/bright rows. Since most periodic light sources have a fixed frequency, it is possible to predict the exact spatial frequency of any resulting intensity flicker. This characteristic frequency can be used to effectively identify and discriminate brightness variations in the image that are likely caused by flicker rather than by real objects in the scene.

Another common characteristic of intensity flicker is that it appears to shift or “crawl” across the image during streaming video. The alternating bright/dark bands appear to move orthogonally to the sensor rows. What is the cause of this apparent movement?

Figure 3. Intensity flicker drift. A mismatch between exposure time and intensity integral period causes each successive frame in a video stream to
shift with respect to the intensity integral function. This results in apparent movement or “crawling” of the intensity bands in the direction shown.

The crawling effect in intensity flicker bands is caused by a mismatch between the frame and intensity integral periods. Suppose, for example, that the frame period is not a multiple of the intensity integral period, but instead exceeds the closest integer multiple by some time z. For simplicity, assume that the multiple is 1 and z is equal to 1/8 of the intensity integral period. Each frame in the video stream will then finish integration at a different point in the intensity integral curve than the previous frame. But, since the intensity integral is a periodic function, the net effect for each frame in the sequence is equivalent to a time-shift to the left by the excess time z. The position of the band-pairs must also shift accordingly (figure 3). Figure 3 contains a diagram, which illustrates this effect.

The speed of this flicker drift is given by:

Where:

r = the number of rows per frame
fps = the frame rate at which the sensor is operated

Values of flicker drift speed, which have different signs, indicate drift in opposite directions (for example, top to bottom versus bottom to top). It is important to note that under operational conditions, this quantity will only vanish if the exposure time is equal to an integer multiple of the intensity integral period. This condition is indeed very meaningful, as we will discuss shortly.

The descriptions we have developed can be used to engineer approaches to reduce the effects of intensity flicker. In many cases the exact frequency of the light intensity function is known, from which we can calculate the expected spatial frequency of the resulting intensity flicker. We can then use a spatial notch filter, which removes all components in the expected frequency band, effectively removing the effect of flicker. Of course, this “atomic bomb” of a technique would also remove real scene components, which happen to fall near the flicker spatial frequency. A possible improvement on this technique would be to use a video stream and only compensate for components that have both the predicted frequency and drift at the predicted velocity. This refinement would almost guarantee that no real scene components are incorrectly targeted. Alternatively, we can use a mathematical property of all periodic functions and adjust the exposure time such that flicker does not happen at all.

The integral over an integer multiple of the period of any integrable periodic function f(t) with period T is constant:

and is independent of the starting time t. We can apply this identity by choosing the exposure time to be an integer multiple of the period of the intensity function. Then equation 1 becomes:

Since the variation in the intensity integral is the fundamental cause of flicker, a constant intensity integral implies no flicker. From our investigation into flicker drift, we know that this condition is the only one which results in zero drift velocity. Thus, we have only two possible cases: Te is an integer multiple of Ti(t) and there is no flicker, or, flicker occurs and it must drift! Implementing this technique for flicker reduction boils down to creating an algorithm which nudges the exposure time towards the closest integer multiple of Ti(t), thus eliminating flicker at the cost of exposure time resolution.

Flicker is only one of many artifacts caused by rolling shutter exposure. Some of these artifacts, such as wobble, skew and partial exposure are not easily correctable. Of course, we could avoid these problems altogether by choosing a different exposure technique. Several CMOS image sensor manufacturers are now enabling the use of global exposure by requiring an external mechanical shutter. Unfortunately, the use of external mechanical shutters just adds to the cost of implementation for the module/system manufacturer. This is on top of added design concerns regarding size, reliability and power consumption. Nonetheless, not all shutter technologies are built equally, and some companies have recently started releasing MEMS-based shutters. Usually embedded inside the lens structure, these devices allow all the functions of a traditional mechanical shutter, but with the advantages that come with greatly reduced mass and size. Using a combination of a CMOS sensor capable of global exposure with a MEMS-based shutter is a great tradeoff between cost/size/reliability and image quality.

Tessera
www.tessera.com

::Design World::