Autonomous Robotic Foosball

C++ / Nvidia Jetson / TensorRT / Arduino

A standard foosball table retrofitted to play one side by itself. An overhead camera tracks the ball, an Nvidia Jetson decides where each rod needs to be, and stepper motors translate and rotate the rods fast enough to block and score against a human opponent. Built with Aditya Gaur in Duke's Cyber-Physical Systems Lab (CPSL).

Live demonstration of the autonomous system scoring twice and conceding zero goals against a Duke I&E Fair attendee.

Hardware Implementation

3D Rendering — Fig 1. Initial 3D CAD design for the assembly, based on a standard Tornado table.

Lab Setup — Fig 2. The constructed system in the Duke CPSL. The frame supports the NEMA-17 stepper motors and belt drives required for simultaneous lateral and rotational control.

The hardware architecture splits high-level perception and low-level motor actuation. An Nvidia Jetson AGX Orin handles vision processing via a Logitech camera stream (720p at 30fps) cropped to the playfield. We convert frames to the HSV color space and apply color range thresholding to isolate the blue ball and yellow end-of-rod markers; a centroid calculation on the largest contour provides each object's position. Velocity is computed from the delta over a sliding window of recent frames, and the linear trajectory is projected to the intersection coordinate with each rod's x-plane (for multi-player rods, the figure closest to the intercept is selected to intercept).

The Jetson transmits MOVE and KICK commands over serial to an Arduino Mega 2560, which drives the eight NEMA-17 steppers (four lateral, four rotational) through AccelStepper acceleration profiles; four limit switches provide the physical zero reference for the lateral homing sequence.

Vision Pipeline — Fig 3. The dual-path perception pipeline. The ball tracking runs via traditional CV (HSV Thresholding) for low latency, while rod rotation is inferred via a quantized neural network.

State Estimation & Control

The biggest issue was knowing what rotation each player rod was at. As steppers are stateless, if the rod hit the ball, some momentum would be transferred to the ball and the rod would end up at a rotated position that didn't correspond to the kick command. We needed a visual connection to close the control loop.

For rotational state estimation, we trained four EfficientNet-B1 regression models, one per rod. We collected the data to tune the models ourselves: a collection script moved the lateral rods against to the limit switches, then steps the motors through a grid of every rotational and lateral position combination. At each grid point it pauses for, captures a 1080p frame, and encodes the ground truth directly in the filename (e.g. rot_s14_lat_p40.jpg), so the dataset comes out pre-labeled with no manual annotation. We then augmented the images with randomized lighting, blur, noise, and shift/scale transforms to cover gameplay conditions like motion blur and camera jitter. The models reach a test MAE of 0.82 steps (about 1.5° of rotational error). We quantized them to INT8 with Post-Training Static Quantization (14.5MB down to 7.9MB each) and exported them via ONNX to TensorRT engines; inference for all four rods runs on parallel CUDA streams and finishes in about 14ms per frame, comfortably inside the 30fps frame budget.

To solve translational stepper drift (again, the motors are stateless), the vision system periodically compares the y-position of the yellow tape markers against the position implied by the motor step count. If the error exceeds a pixel threshold, the Jetson sends a RESETPOS command to the Arduino to overwrite its internal step counter, effectively closing the loop without hardware encoders.

Camera Stream Debug — Fig 4. Live debug feed of the perception engine. The green circle isolates the ball via color filtering. Red dots track the yellow tape markers, which serve as the anchor to infer the blue player positions. The overlays display the regression model's output for rotational state (0-200 steps). Here, the strikers register ~110 steps (indicative of being inverted) while other rods hold near 0 or 200 (upright).