Software Subsystem
2026 Software Team
Raina Hu, Albert Wang, Allison Hutchings, William Zhang, Heidi Bax, Negar Badr, Luca Geuzaine, Arthur Guihaire, Sohaib Kaidali, Avyan Kandasamy, Panth Kiklawala, Lara Libertad Landauro, Owen Le Sann, Justine Lin, Ariyan Nagratha, Parsa Rahimnia, Rachel Ruddy, Lauren Spee, Sara Yang, Salma Yousry, Chantal Zhang
Software Architecture
The software stack is constructed around the Robot Operating System (ROS2), an open-source subscriber-publisher model that allows for multiple independent processes called nodes to communicate easily. Most of our codebase is written using Python 3. For latency-sensitive nodes, such as the republishing of sensor data, C++ is used. This software stack runs on Nvidia L4T 36.4.3 (Ubuntu 22.04) with Docker. Processes are executed and managed by Docker containers on a local Docker network. Docker containers are configured with Ubuntu 22.04, PyTorch, CUDA, and ROS2 to enable cross-platform compatibility independent of development and production environments.
An on-board NVIDIA Jetson AGX Orin provides the necessary computational power to run all packages concurrently within the Docker containers. Additionally, the Jetson is equipped with a 2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores, which allows us to run our YOLOv11 and RF-DETR models for object detection in real time.
The main focus for this year was on thoroughly testing previous code that has only been tested in simulation and ensuring we have reliable state estimation, controls and propulsion systems. One of the previous problems from last year was the lack of a thorough testing procedure and framework. A significant portion of our efforts this year was focused on testing, debugging, and standardizing procedures to improve pre-competition preparation.
Modules in our software stack are modularized such that modifications to a specific subsystem could be done in branches which do not involve overarching changes to the software architecture.
Vision
The vision package interprets the environment using a front-facing stereo RBGD camera along with a down-cam. Both of the front-facing camera feeds are processed using fine-tuned YOLO-v11 and RF-DETR models for robust object detection. The model is fine-tuned over 300 images of Robosub objects labelled with Roboflow, which have then been augmented to more than 10,000+ images. For the downward-facing camera, state estimation and camera intrinsics data help calculate direction vectors to detected objects, enabling rough positional estimates on the pool floor. By correlating the 2D bounding box detections with the 3D camera distance information, we are able to accurately infer the 3D positions of detected objects in the robot's reference frame.
This year we introduced a new model training pipeline supplemented with synthetic data from our Unity simulation. YOLO-v11 and RF-DETR models trained on synthetic data were then fine-tuned on pool data using Roboflow. Models were also optimized for our Jetson with TensorRT, leading to accurate and real-time object detection.
For tasks involving the grabber which require precise manipulation, a basic visual servoing setup has been implemented. This system leverages the real-time visual feedback from the cameras to adapt the AUV's pose dynamically, guiding the grabber to its target. Since this is our first year integrating a grabber, this system will need further refinements in subsequent years. The software team collaborated closely with the mechanical team to find a proper placement for the grabber that will ensure the grabber is within the field of view of the down camera.
Sensors
Douglas is equipped with an Inertial Measurement Unit (IMU), a Doppler Velocity Log (DVL) and a depth sensor. In previous years, the data streams from the sensors were fused and processed by an Extended Kalman Filter (EKF). This year, to maximize reliability, we traded that complexity for a highly deterministic state_aggregator node. It fuses our Movella IMU, Waterlinked DVL A50, and depth sensor into a stable, zero-latency state estimate without the erratic jumps of a complex SLAM backend.
Planning & Controls
High-level mission planning is managed by a dedicated package that employs behaviour trees to effectively navigate through each competition task. We construct missions based on the requirements of the competition and our strategy. Each competition task is assigned a corresponding subtree, which can asynchronously dispatch navigation commands to the controls package via ROS2 action servers . These missions evaluate the environment as a discrete set of observable objects and preempts maneuvers dynamically.
The controls and propulsion packages ensure precise AUV movement using a Proportional-Integral-Derivative (PID) Controller. The controls package receives target positions and orientations, which are then used in multiple PID loops to generate body-framed forces and torques. The propulsion package then decomposes these forces and torques into thruster Pulse Width Modulation (PWM) speeds, which are communicated to the Electronic Speed Controllers (ESCs) via a micro-ROS node.
Telemetry
To facilitate the tuning of PID control loops and debug the vision pipeline in real time, the visualization tool Foxglove was used. This tool provides an intuitive interface for visualizing sensor data, control inputs, and AUV behavior in real-time, greatly streamlining the process of optimizing the control system during pool testing and development. Peripheral layouts for monitor electrical status and sensor health were additionally implemented.
Simulation
Comprehensive testing of the Douglas AUV's software stack is facilitated by a custom Unity simulation environment that replicates the competition setting and allows for easy testing of new planner code. This robust environment allows for code testing and the simulation of full competition runs in a controlled and repeatable manner. A software endpoint in the ROS software stack in setup, which utilizes sockets to perform real-time communication through a network. The network simplifies communication with Docker containers as the network is managed through the use of Docker containers.
Furthermore, a continuous integration (CI) pipeline on GitHub has been established to perform regular automated tests using Unity. Work has been done to de-bloat the previous simulation builds. In the future, the team plans to implement simulated sensor data capabilities within this environment, which will enable thorough testing of state estimation algorithms without the need for physical pool tests, thereby accelerating development and validation cycles.