Language-Steered Drones

Thu, 01 Jan 2026 00:00:00 +0000

Drone navigating to a leaf blower

Developed a vision-language navigation (VLN) policy for autonomous drone flight in photorealistic 3D Gaussian Splatting environments. Given a natural language instruction like “go to the green leafblower,” the drone autonomously identifies and navigates to the target — collision-free.

The video shows the drone’s onboard view: RGB (left) and semantic similarity field (right) for the query “green and pink leafblower.” In the right view, red indicates high similarity with the query and blue indicates low similarity — the drone navigates towards the high-similarity region while avoiding obstacles. The system first encodes the language instruction via CLIP embeddings, localizes the target using CLIPSeg semantic segmentation, and generates real-time control commands to navigate through a cluttered indoor environment.

The control policy is a lightweight neural network (SqueezeNet Commander MLP) trained via Behavioral Cloning from an ACADOS-based MPC expert. A key contribution is the design and implementation of a full DAgger (Dataset Aggregation) pipeline — including mixed-policy rollouts, expert annotation filtering, iterative retraining with best-model checkpointing, and automated benchmarking — to systematically correct for compounding errors under distribution shift. A second key contribution is the introduction of explicit geometric features — bearing and elevation — extracted from the CLIPSeg heatmap centroid, providing the policy with a direct spatial signal for goal-directed control. This replaces the previous approach where target localization had to be implicitly learned from visual embeddings alone.

Results: 90% navigation success rate (up from 52%), collision rate reduced from 20% to 8%, with generalization to unseen RRT-planned trajectories.

This research is conducted under Prof. Mac Schwager (Multi-Robot Systems Lab, Stanford) in collaboration with PhD student Maximilian Adang (MSL Stanford).

GitHub Repository

Tech Stack

PyTorch · 3D Gaussian Splatting (gsplat) · CLIPSeg · ACADOS optimal control · CUDA

Autonomous Spacecraft Docking

Thu, 01 Jun 2023 00:00:00 +0000

Docking demonstration

This project was conducted as an undergraduate research project in collaboration with Nyx Exploration Company and Antoine Pallois, focusing on autonomous spacecraft docking.

The work investigated guidance, navigation, and control (GNC) strategies for close-proximity operations, with a particular emphasis on vision-based relative navigation and closed-loop docking maneuvers. Experimental validation was performed using a quadcopter drone as a hardware test platform.

Technical report

📄 Final technical report (PDF)