AV Perception

Build perception systems that detect objects, lanes, and traffic elements - giving autonomous vehicles the ability to understand the driving environment.

Perception Tasks

Sensor Processing

Camera-Based Detection

import torch
from ultralytics import YOLO

# Load a model trained on driving datasets
model = YOLO('yolov8l.pt')

# Detect objects in a driving scene
results = model('driving_scene.jpg')

# Process detections
for result in results:
    for box in result.boxes:
        cls = result.names[int(box.cls)]
        conf = float(box.conf)
        x1, y1, x2, y2 = box.xyxy[0].tolist()
        print(f'{cls}: {conf:.2f} at [{x1:.0f},{y1:.0f},{x2:.0f},{y2:.0f}]')

LiDAR Point Cloud Processing

import numpy as np
import open3d as o3d

# Load LiDAR point cloud
pcd = o3d.io.read_point_cloud('lidar_scan.pcd')

# Ground plane removal using RANSAC
plane_model, inliers = pcd.segment_plane(
    distance_threshold=0.2, ransac_n=3, num_iterations=1000
)
obstacles = pcd.select_by_index(inliers, invert=True)

# Cluster remaining points into objects
labels = np.array(obstacles.cluster_dbscan(eps=0.5, min_points=10))
print(f'Detected {labels.max() + 1} obstacle clusters')

Sensor Fusion for AVs

Autonomous vehicles fuse data from multiple sensors to overcome individual sensor limitations:

Modern Perception Architectures

• BEVFormer: Transforms multi-camera inputs into a bird's-eye-view representation for unified 3D detection
• PointPillars: Efficient 3D object detection from LiDAR point clouds
• CenterPoint: Anchor-free 3D detection and tracking for LiDAR data
• BEVFusion: Fuses camera and LiDAR features in bird's-eye-view space
• UniAD: Unified autonomous driving model combining perception, prediction, and planning