LiDAR Self-Driving in CARLA

Problem and motivation

Most teams in the class trained models to label LiDAR points (car, road, pedestrian). We wanted the harder version of the problem: a model that actually drives the car. Labeling points tells you what is around the car; driving means deciding what to do about it, every twentieth of a second, with your own mistakes changing what you see next.

Approach

Learning by copying an expert

CARLA, an open-source driving simulator, has a built-in autopilot. We drove it around two training towns and recorded about 25,000 moments of LiDAR alongside what the autopilot did with the steering, throttle and brake. The models learn to predict those controls from the LiDAR, the car's speed and the traffic-light state. This is called imitation learning, or behavior cloning.

Testing in towns the models never saw

Training, validation and testing each use different towns, so a model can't pass the test by memorizing streets. The test is driving, not prediction error: each driver runs the same 20 seeded scenarios (about 400 m each) across two test towns, and we count how each one ends. Prediction error on recorded data is reported only as a secondary number, because it is known to say little about how well a model drives.

A rule-based baseline to beat

The baseline uses no learning: it finds the ground plane in the point cloud (RANSAC), groups what's left into obstacles (DBSCAN), and brakes for anything inside a lane-wide corridor along its route. It sees the same LiDAR, speed and light state as the models.

Fair conditions

Every driver gets the same speed cap (50 km/h), the same start state, and the same post-processing of its controls. In the first milestone, the learned models control only throttle and brake, and the same route follower steers for them and for the baseline. The full-control models steer themselves, so they are compared with each other rather than with the baseline.

Outcome and results

Results over all 20 test scenarios (Town02 and Town05):

Swipe the table sideways to see every column.

^*19 of 20 scenarios so far. On those same 19, the model without the route input had 7 collisions and 6 completions, so the route input made little overall difference, though it hit vehicles less often (2 times instead of 5).

The baseline is by far the safest; its single collision is a pedestrian who steps out a few seconds into one scenario, which every driver hits. Its weakness is getting stuck: 13 of 20 runs end with it stopped for 30 seconds. Seven of those are correct waits (in red-light queues, and behind a crashed bus); the other six are the same large rock beside a Town02 road, which its detector mistakes for an obstacle.

The biggest improvement in the learned model came from the labels, not the network. The autopilot brakes in short single-frame pulses, so on most frames near a car the "right answer" was no brake, and the model learned to rarely brake. Stretching each brake pulse over its neighboring frames cut collisions from 18 to 8. The cost is caution: that model drives a half to a third as fast on average and runs out of time more often.

The full-control CNN steers well on recorded data but never finishes a route: once it drifts a little, it sees situations the expert never recorded, and the errors compound. That is a well-known weakness of pure imitation learning.

The transformer never moves at all: in every scenario it holds full brake from the first moment. In the training data the expert almost never accelerates from exactly zero speed, so the model learned that a stopped car stays stopped, and it reads its own speed rather than the road. This is a known failure of imitation learning, sometimes called the inertia problem, and retraining with noise on the speed input didn't remove it, because at low speeds the expert is still almost always braking in our recordings.

Skills practiced

• Building and training CNNs in PyTorch on LiDAR data
• Designing a fair closed-loop test: fixed seeded scenarios, held-out towns, per-scenario reporting
• Running a simulator headless on a shared SLURM GPU cluster
• Debugging a model one hypothesis at a time: our first car never moved, and it took a chain of tests to find that a brake value of about 0.002 was enough for the simulator to hold it still

Challenges and what I'd do next

The hardest part was that a model can look good on recorded data and still fail on the road. Our first models matched the expert's labels closely and then refused to move, or never braked for the car ahead. Every fix came from watching the failures and testing one explanation at a time.

Next: give the speed-only model the planned route as an extra input, so it can tell a car in its path from one beside it after a turn, which is where most of its crashes happen. For the full-control models, collect training data from the model's own mistakes (DAgger), the standard remedy for compounding errors.

Tools

• CARLA 0.9.16 (simulator)
• PyTorch
• Python
• NumPy
• SLURM GPU cluster