AUTOLAB Github Open Code and Data Sources

Check out what we are currently working on, and find opportunities to contribute! Includes open source projects and AUTOLab-specific utilities.

  • Dex-Net–A research project for generating datasets of synthetic point clouds, robot parallel-jaw grasps and metrics of grasp robustness to train machine learning-based methods to plan robot grasps.
  • Debridement–Fast and reliable autonomous surgical debridement.
  • DART— DART (Disturbances for Augmenting Robot Trajectories) collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot’s trained policy during data collection.
  • GQ-CNN–Our Python package for grasp quality convolutional neural networks
  • YuMiPy–Our Python package for interfacing with ABB’s YuMi robot
  • MeshRender–A set of Python utilities for rendering 3D meshes with OpenGL
  • Perception–AUTOLab’s toolkit for robot perception tasks
  • Malasakit–A collaboration with Berkeley’s CITRIS and the National University in the Philippines for assessing disaster risk reduction strategies

For more projects, visit our Github page.

Dexterity Network (Dex-Net)

A Cloud-Based Network of 3D Objects for Robust Grasp Planning: A data-driven approach to robust robot grasping and manipulation based on a new dataset of 3D object models that currently includes over 10,000 unique 3D object models and 2.5 million parallel-jaw grasps. Dex-Net includes a Multi-Armed Bandit algorithm with correlated rewards from prior grasps to estimate the probability of force closure under sampled uncertainty in object and gripper pose and friction.

Dex-Net as a Service

Dex-Net as a Service (DNaaS) is a prototype cloud-based grasp planning system. Anyone can upload a 3D object mesh (in .obj format with triangular faces) and visualize candidate grasp axes using a parallel-jaw gripper, each ranked by their robustness (green for most robust to perturbations).


Surgical debridement is the process of removing dead or damaged tissue to allow the remaining parts to heal. Automating this procedure could reduce surgical fatigue and facilitate teleoperation, but doing so is challenging for Robotic Surgical Assistants due to inherent non-linearities in cable-driven systems. Consequently, we propose and evaluate a two-phase calibration process. Using an endoscopic stereo camera with standard edge detection, experiments with 120 trials achieved success rates of 91.7% to 99.2%, slightly exceeding prior results (89.4%) and more than 2.1x faster, decreasing the time per fragment from 15.8 seconds to 7.3 seconds.

Cloud Robotics

What if robots and automation systems had unlimited computation and memory and could share data and code? This is becoming practical with wireless networking and data centers. Working with Google, Siemens, Cloudminds, and Cisco, we are developing new Cloud Robotics algorithms and platforms such as the Dexterity Network.


One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy. A known problem with this “off-policy” approach is that the robot’s errors compound when drifting away from the supervisor’s demonstrations. On-policy, techniques alleviate this by iteratively collecting corrective actions for the current robot policy. However, these techniques can be tedious for human supervisors, add significant computation burden, and may visit dangerous states during training. We propose a new algorithm, DART (Disturbances for Augmenting Robot Trajectories), that collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot’s trained policy during data collection. For high dimensional tasks like Humanoid, DART can be up to 3x faster in computation time and only decreases the supervisor’s cumulative reward by 5% during training. On the grasping in clutter task, DART obtains on average a 62% performance increase over Behavior Cloning.


Robot-Assisted Precision Irrigation Delivery (RAPID) is a co-robotic approach where a team of humans and robots move through the fields to adjust low-cost adjustable drip irrigation emitters at the plant level.


A collaboration with the CITRIS Data and Democracy Initiative in developing a new social media platform that directly engages citizens and communities from developing regions to collectively assess conditions, needs, and outcomes related to engineering innovations in humanitarian assistance, solar energy, democracy and governance, and global health.

SCHool Project

To be useful in warehouses, homes, and other environments from schools to retail stores, robots will need to learn how to robustly manipulate a wide variety of objects. For instance, to enhance the productivity of human workers, service and factory robots could keep specified surfaces clear by identifying, grasping, and relocating objects to appropriate locations. Pre-programming robots to perform such complex manipulation tasks is not feasible; instead this project will investigate scalable robot manipulation, where multiple robots collaboratively learn from multiple humans. The project will contribute new models, algorithms, software, and experimental data to advance the state-of-the-art in deep learning, human–robot interaction, and cloud robotics. To broadly convey the results of this research to students and the public, the project will create a book and video with the Lawrence Hall of Science and the African Robotics Network.

Computer Assisted Surgery

To improve patient care and more accurately target treatment within the human body, we’re developing new geometric models and algorithms for automating surgical subtasks, such as suturing and debridement using the da Vinci surgical robot. We are also developing models of soft tissues and new methods for dose planning, brachytherapy, and planning algorithms for steering flexible needles.

Automated Manufacturing

To produce the high quality, rapidly evolving products of the future, we’re establishing a science base for automated assembly by analyzing its basic components. We develop efficient geometric algorithms for feeding, fixturing, and grasping industrial parts.

New Media Artforms

To discover what can be expressed with new technologies such as networks, robots, digital cameras, and sensors that could not previously be expressed, we’re designing art installations that explore issues related to cultural history, privacy, and “telepistemology: what is knowable at a distance.”

Prior Projects


Internet-based “online robots” now provide public access to remote locations such as museums and laboratories. The Tele-Actor is a collaborative online teleoperation system for distance learning that allows many students to simultaneously share control of a single mobile resource.

Jester 5.0

Jester is a joke recommender system designed to study social information filtering. It uses a collaborative filtering algorithm that uses universal queries to elicit real-valued user ratings on a common set of items and applies principal component analysis (PCA) to the resulting dense subset of the ratings matrix. PCA facilitates dimensionality reduction for offline clustering of users and rapid computation of recommendations.


(Automated) Collaborative Observatory for Natural Environments


Anytime Nonparametric A* (ANA) Algorithm

Motion Planning in Medicine

Optimization and Simulation Algorithms for Image-Guided Procedures (Monograph)


Assembly Line Adaptive Netbot (ALAN), a proposed practical robot system for applications in manufacturing, material handling, and food production that is emerging from discussions between leaders from industry and academic research.