![Udacity Self-Driving Car: Term-1 Bill Kromydas
Project 4: Advanced Lane Finding December 31, 2017
3
system to detect lane curvature sooner near the horizon and also maximizes the opportunity for
detecting an adequate number of “dashed” line segments which is important for a robust curve fit.
Image Filtering
In order to detect lane lines from camera images two thresholding approaches were recommended in
the lecture notes (color filtering and gradient filtering). Assuming all lane lines are either yellow or
white as in all the test images and videos, I looked for color channels in various color spaces that
might perform well for detecting either of these colors under various lighting conditions. The images
on the following page show some of the better color channels for detecting yellow and white lane
lines. I considered four color spaces: RGB, HLS, LAB and HSV and experimented with each of the
channels across a range of images. There are a number of channels that appear to perform equally
well for detecting white (RGB, HSV-V, LAB-L). However, LAB-B stands out as the best channel for
detecting yellow, especially in low light conditions.
I also experimented with gradient thresholds as recommended in the lecture notes but I found that
color thresholding seemed to be the dominate factor. For gradients, I used Sobel-x as well as gradient
magnitude and direction. I was able to reproduce (almost exactly) the binary filtered image for
test5.jpg as shown in the lecture notes using a combined color and gradient filter with an OR
operation between color and gradients.
combined_binary[(color_binary == 1) | (grad_binary == 1)] = 1
This filter performed well for detecting many lane lines with nice precision, but it failed for cases in
the challenge video where seams in the road surface could not be masked due to the strong gradient
and linear nature of such features. I therefore opted to use an AND operation with different threshold
values for combining color and gradients to minimize false detections due to gradients.
combined_binary[(color_binary == 1) & (grad_binary == 1)] = 1](https://image.slidesharecdn.com/p4advancedlanefinding-180220035550/85/Advanced-Lane-Finding-3-320.jpg)
Advanced Lane Finding
- 1. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 1 Advanced Lane Finding Summary The objective of this project is to build a lane detection system based on the video feed from a camera mounted on the front of a car. Image frames from the video are processed to detect left and right lane lines which are used to define the lane location and orientation. The image above shows an example of the lane rendering for a single image frame. The processing pipeline for this system includes the following steps: - Camera calibration - Distortion correction - Color / gradient pre-processing to filter potential lane lines - Perspective transformation - Lane line search and track functions - Lane metric computations (curvature, width, parallel degree) - Lane rendering on original image frame Camera Calibration Camera calibration is necessary because it will provide us with the parameters that are required to remove radial and tangential image distortion inherent in camera images. The camera is calibrated using a series of chessboard images. Chessboard images are useful for this task because they have a well defined, high contrast shape that is easy to detect. Calibration is accomplished by first defining two sets of points: “object” points and “image” points. The mapping between these two sets of points will enable us to compute the camera matrix and distortion coefficients using the Open CV function calibrateCamera(). The object points and image points are defined to be the intersecting corners of the chessboard. The object points represent the actual (known) corner points of the physical calibration chessboard and the image points are the same corners extracted from each of the test images. Defining the object points is straight forward because they correspond to the 54 corners of the physical chess board. The image points are extracted form each calibration image using the Open CV function findChessboardCorners(). The first image below is one of the calibration images. The second
- 2. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 2 image shows the identified image points that were detected using findChessboardCorners(). The last image is the undistorted image that was transformed using the camera matrix and distortion coefficients. Each calibration image can be used to compute a camera matrix and the distortion coefficients. Once these are computed for each calibration image they can be averaged across all calibration images to compute a final camera matrix and the associated distortion coefficients for the camera. Once the camera is calibrated these parameters can be used to undistorted the images from the video frames prior to image processing. calibration image corner detection undistorted image Perspective Transform Perspective transformations be very useful for transforming images into a space that is easier to work with. Consider the camera image below to the left. If we transform that image to a bird’s eye view from above then the computations for lane curvature and other metrics will be greatly simplified and easier to visualize. This perspective is also helpful for comparing a car’s location directly with a map. In order to define a bird’s eye view perspective transformation we need to define a rectangle on a plane in the original image. We will call these points “source” points. They define the points of interest in the original image that lie in a plane in the physical world. Next we need to map those points to a set of four destination points in a warped image. Once we have defined both sets of points we can use the Open CV function getPerspectiveTransform() to compute the perspective transformation matrix (and its inverse). Once we have the transformation matrix (M) we can use the Open CV function warpPerspective() to transform an image to a bird’s eye view perspective. The source and destination points that I selected are listed in the table below. Source points Destination Points x y x y 567 470 314 0 720 470 986 0 1100 720 986 720 205 720 314 720 The examples below show the undistorted camera image to the left with a red trapezoid that defines a “rectangular” region on the road surface. We assume the road is flat and that the lane lines in these images are parallel which allow us to define the source points that represent a rectangle in real word space. The images to the right show the transformed (warped) image as a bird’s eye view with the expected result (i.e., the trapezoid is now transformed to a rectangle in the warped image). See get_src_dst_points() in lane_line_helpers.py. For the selected source and destination points the lane width in pixel space is about 690 pixels. This calibrated value will be used to estimate the lane width when processing images. It is important to select the source points so the top of the trapezoid is high enough in the image frame to capture lane line detections close to the road horizon. This allows the
- 3. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 3 system to detect lane curvature sooner near the horizon and also maximizes the opportunity for detecting an adequate number of “dashed” line segments which is important for a robust curve fit. Image Filtering In order to detect lane lines from camera images two thresholding approaches were recommended in the lecture notes (color filtering and gradient filtering). Assuming all lane lines are either yellow or white as in all the test images and videos, I looked for color channels in various color spaces that might perform well for detecting either of these colors under various lighting conditions. The images on the following page show some of the better color channels for detecting yellow and white lane lines. I considered four color spaces: RGB, HLS, LAB and HSV and experimented with each of the channels across a range of images. There are a number of channels that appear to perform equally well for detecting white (RGB, HSV-V, LAB-L). However, LAB-B stands out as the best channel for detecting yellow, especially in low light conditions. I also experimented with gradient thresholds as recommended in the lecture notes but I found that color thresholding seemed to be the dominate factor. For gradients, I used Sobel-x as well as gradient magnitude and direction. I was able to reproduce (almost exactly) the binary filtered image for test5.jpg as shown in the lecture notes using a combined color and gradient filter with an OR operation between color and gradients. combined_binary[(color_binary == 1) | (grad_binary == 1)] = 1 This filter performed well for detecting many lane lines with nice precision, but it failed for cases in the challenge video where seams in the road surface could not be masked due to the strong gradient and linear nature of such features. I therefore opted to use an AND operation with different threshold values for combining color and gradients to minimize false detections due to gradients. combined_binary[(color_binary == 1) & (grad_binary == 1)] = 1
- 4. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 4 undistorted image undistorted image LAB-B Channel LAB-B Channel HSV-V Channel HSV-V Channel RGB-R Channel RGB-R Channel LAB-L Channel LAB-L Channel
- 5. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 5 Image Pre-Processing The two sets of four images below illustrate the effect of combining color and gradient with an AND operation between them in the final filtered image (lower right in each quadrant). In the first set of images either color or gradient alone would have been adequate for lane detection. However, in the second set of images it is clear that the combined filter does the best job at eliminating clutter. undistorted image color filter only gradient filter only combined (color & gradient) undistorted image color filter only gradient filter only combined (color & gradient)
- 6. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 6 Processing Pipeline The processing pipeline for lane detection consists of the following steps: - image un-distortion - image filtering (color and gradient) - image warping - image region of interest clipping - lane acquisition via sliding window search - lane tracking via band search for previously detected lanes - polynomial fitting to the detected line points - computation of lane line metrics (curvature, width, parallel) - final lane rendering on original image The following images illustrate the key processing steps in the pipeline. undistorted image color and gradient filtered binary warped image (clipped) sliding window search polynomial fit tracking final lane rendering projected on original image
- 7. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 7 Search Modes I experimented with the sliding window search mode and the convolution centroid approach. I found the sliding window search mode to be more robust at identifying lane lines and I therefore focused on that implementation for this project. I used 18 sliding windows with a pixel margin of +/- 80 pixels. This seemed to provide a good resolution for initial detection of lane lines. In order to minimize the effect of clutter from the center of the lane I created a keep-out zone so the histogram computation for computing the peaks ignores the keep-out region in the center of the lane. I also made a configurable parameter (HIST_FRACTION = .8) that allows the histogram to examine data through a greater portion of the image frame (i.e., greater than the lower half). This is useful for acquiring lane detections when there is no detectable data in the lower half of the image frame. Lane Tracking Once the lane has been detected using the sliding window search mode it is maintained in a track mode which uses a search band around the best fit polynomial for each lane line. The best fit polynomial is computed as the running average of the last N image frames which helps to smooth the transition from frame to frame (N_FRAMES_SMOOTH = 3). The search band in the current frame is defined by the best fit polynomial from previous frames. The search band is used to find new line detections which are used to compute a new current polynomial fit. If a given frame has no detections or does not pass the lane metric criteria for lane detection then the current frame is skipped and the previous best fit polynomials are used to render the lane in the original image. The number of consecutive frames that can be skipped in track mode before the sliding window search mode is executed is a configurable parameter (MAX_FRAMES_SKIPPED = 5). Lane Metrics Both search and track modes described above use a final confirmation step to determine if the lane has been detected. The confirmation step uses three separate metrics: (1) minimum lane curvature, (2) lane width, (3) parallel lane lines. Each of these metrics must be satisfied within configurable threshold limits in order to declare the lane detected. The lane line curvature of both lane lines must be greater than a minimum configurable threshold. The lane width is measures at the base of the image frame and must be within a +/- threshold value to the standard lane width of 3.7 meters. Parallel lines are checked by comparing the first two coefficients (quadratic and liner terms) of each polynomial fit to threshold values to check for their similarity. Examples The series of images on the following page show two of the more challenging image frames. The image on the left is challenging due to the white colored pavement and an abundance of clutter which both compete with detecting the lane lines. In spite of the relatively poor quality of the filtered image the sliding window search method does a good job of identifying the lane lines. The image on the right shows the car emerging from the shadow of the bridge. In this case the system lost track of the lane lines while under the bridge and was attempting to reacquire the lines using a sliding window search. Rather than using only the bottom half of the image frame to compute the histogram peaks I made that a configurable parameter so that situations like this could be handled by allowing the algorithm to use more of the data in the upper portion of the image frame to kick start the detection process sooner. I also added a keep out zone in the center of the image which is also configurable. The purpose of the keep-out zone is to mask clutter in the center of the lane from the peak histogram computations so that the clutter does cause false lane line detections.
- 8. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 8
- 9. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 9 Lane Rendering and Diagnostics For each image frame the detected lane is rendered on the original image as a transparent green swath between the two polynomial lane line fits. The left and right lane boundaries are outline in yellow. Several diagnostics are displayed in the upper left portion of the frame to identify the status of the lane detection which is very useful for debugging. Reflection This project presented an opportunity to solve an interesting and challenging problem using a wide range of methods. In retrospect I would have spent more time up-front investigating more robust methods for image filtering and creating diagnostics to improve the experimentation process. The main thing I would spend more time exploring would be adaptive thresholding techniques. All of the color and gradient thresholding I implemented for this project consisted of fixed threshold values which can work in some cases and not in others. Manually tuning threshold parameters is tedious to begin with and is not robust due to fixed values. I believe an adaptive thresholding technique that depends on the local conditions (lighting, texture, etc…) would have been a much more robust approach. Other image processing techniques may have also worked better to isolate the lane lines. I experimented with a technique to clone a lane line when the detections from one line were significantly unbalance compared to the other line. This proved to be only marginally useful in the current implementation and sometimes buggy, so it was not included in the default configuration. I also did not make use of the lane center offset that was computed. This information could be used to dynamically adjust the lane detection and tracking functions.
- 10. Udacity Self-Driving Car: Term-1 Bill Kromydas Project 4: Advanced Lane Finding December 31, 2017 10 Files Submitted This project includes the following source files: Files Description lane_finder_driver.py Test driver camera_calibration.py Camera calibration image_pipeline.py Main processing routine for each image filter_lane_lines.py Filter images lane_line_helpers.py Convenience functions find_lines_from_sliding_window.py Sliding window search algorithm find_lines_from_fit.py Lane tracking algorithm Line.py Line class Lane.py Lane class params.py Tuning parameters Default Tuning Parameters MAX_FRAMES_SKIPPED = 5 # max number of skipped frames before sliding window search must be performed N_FRAMES_SMOOTH = 3 # number of image frames to smooth MIN_CURVATURE_METERS = 350 # tightest lane curvature allowed for an individual frame NUM_SLIDING_WINDOWS = 18 # number of (vertically stacked) sliding windows MIN_WINDOW_PIX = 20 # min number of pixels in a window to trigger a detection CENTER_KEEPOUT = 100 # ignore keep-out region in center of image frame HIST_FRACTION = .8 # percentage of frame (from bottom) to use for histogram WINDOW_MARGIN = 80 # sliding window margin (+/- width) for detections FIT_MARGIN = 60 # pixel margin around fitted lane line LANE_WIDTH_PIX = 690 # from calibrated strait line image (base of image frame) LANE_WIDTH_METERS = 3.7 # standard (actual) distance between lane lines LANE_WIDTH_MARGIN = .9 # lane width margin threshold SOBEL_KERNEL = 9 LANE_WIDTH_TRESH = (LANE_WIDTH_METERS - LANE_WIDTH_MARGIN, LANE_WIDTH_METERS + LANE_WIDTH_MARGIN) PARALLEL_THRESH = (0.0005, # squared term 0.09) # linear term FILTER_GRADIENT = 'all' # = 'all', combine sobel-x and (dir, mag), otherwise just sobel-x FILTER_COMBINED = 'all' # = 'all', combine color and gradient above, otherwise just color YM_PER_PIX = 30/720 # meters per pixel in y dimension XM_PER_PIX = LANE_WIDTH_METERS/LANE_WIDTH_PIX # meters per pixel in x dimension