RESEARCH

Monday, Oct. 2

A Gaussian Variational Inference Approach to Motion Planning

Hongzhe Yu, Yongxin Chen

We propose a Gaussian variational inference framework for the motion planning problem. In this framework, motion planning is formulated as an optimization over the distribution of the trajectories to approximate the desired trajectory distribution by a tractable Gaussian distribution. Equivalently, the proposed framework can be viewed as a standard motion planning with an entropy regularization. Thus, the solution obtained is a transition from an optimal deterministic solution to a stochastic one, and the proposed framework can recover the deterministic solution by controlling the level of stochasticity. To solve this optimization, we adopt the natural gradient descent scheme. The sparsity structure of the proposed formulation induced by factorized objective functions is further leveraged to improve the scalability of the algorithm. We evaluate our method on several robot systems in simulated environments, and show that it achieves collision avoidance with smooth trajectories, and meanwhile brings robustness to the deterministic baseline results, especially in challenging environments and tasks. [Motion and Path Planning, Planning under Uncertainty, Optimization and Optimal Control]

ARMP: Autoregressive Motion Planning for Quadruped Locomotion and Navigation in Complex Indoor Environments

Jeonghwan Kim, Tianyu Li, Sehoon Ha

Generating natural and physically feasible motions for legged robots has been a challenging problem due to its complex dynamics. In this work, we introduce a novel learning-based framework of autoregressive motion planner (ARMP) for quadruped locomotion and navigation. Our method can generate motion plans with an arbitrary length in an autoregressive fashion, unlike most offline trajectory optimization algorithms for a fixed trajectory length. To this end, we first construct the motion library by solving a dense set of trajectory optimization problems for diverse scenarios and parameter settings. Then we learn the motion manifold from the dataset in a supervised learning fashion. We show that the proposed ARMP can generate physically plausible motions for various tasks and situations. We also showcase that our method can be successfully integrated with the recent robot navigation frameworks as a low-level controller and unleash the full capability of legged robots for complex indoor navigation. [Legged Robots, Task and Motion Planning, Simulation and Animation]

ConSOR: A Context-Aware Semantic Object Rearrangement Framework for Partially Arranged Scenes

Kartik Ramachandruni, Max Zuo, Sonia Chernova

Object rearrangement is the problem of enabling a robot to identify the correct object placement in a complex environment. Prior work on object rearrangement has explored a diverse set of techniques for following user instructions to achieve some desired goal state. Logical predicates, images of the goal scene, and natural language descriptions have all been used to instruct a robot in how to arrange objects. In this work, we argue that burdening the user with specifying goal scenes is not necessary in partially-arranged environments, such as common household settings. Instead, we show that contextual cues from partially arranged scenes (i.e., the placement of some number of pre-arranged objects in the environment) provide sufficient context to enable robots to perform object rearrangement without any explicit user goal specification. We introduce ConSOR, a Context-aware Semantic Object Rearrangement framework that utilizes contextual cues from a partially arranged initial state of the environment to complete the arrangement of new objects, without explicit goal specification from the user. We demonstrate that ConSOR strongly outperforms two baselines in generalizing to novel object arrangements and unseen object categories. The code and data are available at https://github.com/kartikvrama [Semantic Scene Understanding, Deep Learning Methods]

Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning

Xiaoyu Huang, Zhongyu Li, Yanzhen Xiang, Yiming Ni, Yufeng Chi, Yunhao Li, Lizhi Yang, Xue Bin Peng, Koushil Sreenath

We present a reinforcement learning (RL) framework that enables quadrupedal robots to perform soccer goalkeeping tasks in the real world. Soccer goalkeeping with quadrupeds is a challenging problem, that combines highly dynamic locomotion with precise and fast non-prehensile object (ball) manipulation. The robot needs to react to and intercept a potentially flying ball using dynamic locomotion maneuvers in a very short amount of time, usually less than one second. In this paper, we propose to address this problem using a hierarchical model-free RL framework. The first component of the framework contains multiple control policies for distinct locomotion skills, which can be used to cover different regions of the goal. Each control policy enables the robot to track random parametric end-effector trajectories while performing one specific locomotion skill, such as jump, dive, and sidestep. These skills are then utilized by the second part of the framework which is a high-level planner to determine a desired skill and end-effector trajectory in order to intercept a ball flying to different regions of the goal. We deploy the proposed framework on a Mini Cheetah quadrupedal robot and demonstrate the effectiveness of our framework for various agile interceptions of a fast-moving ball in the real world. [Legged Robots, Reinforcement Learning, Whole-Body Motion Planning and Control]

KGNv2: Separating Scale and Pose Prediction for Keypoint-Based Grasp Synthesis on RGB-D Input

Yiye Chen, Ruinian Xu, Yunzhi Lin, Hongyi Chen, Patricio Vela

We propose an improved keypoint approach for 6-DoF grasp pose synthesis from RGB-D input. Keypoint-based grasp detection from image input demonstrated promising results in a previous study, where the visual information provided by color imagery compensates for noisy or imprecise depth measurements. However, it relies heavily on accurate keypoint prediction in image space. We devise a new grasp generation network that reduces the dependency on precise keypoint estimation. Given an RGB-D input, the network estimates both the grasp pose and the camera-grasp length scale. Re-design of the keypoint output space mitigates the impact of keypoint prediction noise on Perspective-n-Point (PnP) algorithm solutions. Experiments show that the proposed method outperforms the baseline by a large margin, validating its design. Though trained only on simple synthetic objects, our method demonstrates sim-to-real capacity through competitive results in real-world robot experiments. [Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation, Grasping]

Learning a Single Policy for Diverse Behaviors on a Quadrupedal Robot Using Scalable Motion Imitation

Arnaud Klipfel, Nitish Rajnish Sontakke, Ren Liu, Sehoon Ha

Learning various motor skills for quadrupedal robots is a challenging problem that requires careful design of task-specific mathematical models or reward descriptions. In this work, we propose to learn a single capable policy using deep reinforcement learning by imitating a large number of reference motions, including walking, turning, pacing, jumping, sitting, and lying. On top of the existing motion imitation framework, we first carefully design the observation space, the action space, and the reward function to improve the scalability of the learning as well as the robustness of the final policy. In addition, we adopt a novel adaptive motion sampling (AMS) method, which maintains a balance between successful and unsuccessful behaviors. This technique allows the learning algorithm to focus on challenging motor skills and avoid catastrophic forgetting. We demonstrate that the learned policy can exhibit diverse behaviors in simulation by successfully tracking both the training dataset and out-of-distribution trajectories. We also validate the importance of the proposed learning formulation and the adaptive motion sampling scheme by conducting experiments. [Legged Robots, Reinforcement Learning, Imitation Learning]

Mechanical Intelligence in Undulatory Locomotors

Tianyu Wang, Christopher Pierce, Velin Kojouharov, Baxi Chong, Kelimar Diaz, Hang Lu, Daniel Goldman

In the study of biological limbless locomotion, the role of “mechanical intelligence” — passive processes controlled by physical properties — is often overlooked, which limits the effectiveness of robotic models aiming to replicate these creatures’ locomotion performance. This work demonstrates the significance of mechanical intelligence in limbless locomotion in complex terrain, using a comparative study of a nematode worm, Caenorhabditis elegans, and a robot developed to resemble the bilateral actuation mechanism in limbless organisms. Through experiments in laboratory models of complex environments, We found that the robot effectively models nematodes’ kinematics and locomotion performance with open-loop control, suggesting that mechanical intelligence reduces the requirement for active sensing and feedback during obstacle navigation. Moreover, we demonstrated that mechanical intelligence facilitates effective open-loop robotic locomotion in diverse indoor and outdoor environments. This research not only presents the general principles of mechanical intelligence in terrestrial limbless locomotion across biological and robotic systems, but also offers a novel design and control paradigm for limbless robots for applications such as search-and-rescue operations and extraterrestrial explorations. [Biologically-Inspired Robots, Biomimetics, Redundant Robots]

Multi-Gait Locomotion Planning and Tracking for Tendon-Actuated Terrestrial Soft Robot (TerreSoRo)

Arun Niddish Mahendran, Caitlin Freeman, Alexander Chang, Michael McDougall, Vishesh Vikas, Patricio Vela

The adaptability of soft robots makes them ideal candidates to maneuver through unstructured environments. However, locomotion challenges arise due to complexities in modeling the body mechanics, actuation, and robot-environment dynamics. These factors contribute to the gap between their potential and actual autonomous field deployment. A closed-loop path planning framework for soft robot locomotion is critical to close the real-world realization gap. This paper presents a generic path planning framework applied to TerreSoRo (Tetra-Limb Terrestrial Soft Robot) with pose feedback. It employs a gait-based, lattice trajectory planner to facilitate navigation in the presence of obstacles. The locomotion gaits are synthesized using a data-driven optimization approach that allows for learning from the environment. The trajectory planner employs a greedy breadth-first search strategy to obtain a collision-free trajectory. The synthesized trajectory is a sequence of rotate-then-translate gait pairs. The control architecture integrates high-level and low-level controllers with real-time localization (using an overhead webcam). TerreSoRo successfully navigates environments with obstacles where path re-planning is performed. To best of our knowledge, this is the first instance of real-time, closed-loop path planning of a non-pneumatic soft robot [Modeling, Control, and Learning for Soft Robots, Soft Robot Applications, Motion and Path Planning]

On Designing a Learning Robot: Improving Morphology for Enhanced Task Performance and Learning

Maks Sorokin, Chuyuan Fu, Jie Tan, Karen Liu, Yunfei Bai, Wenlong Lu, Sehoon Ha, Mohi Khansari

As robots become more prevalent, optimizing their design for better performance and efficiency is becoming increasingly important. However, current robot design practices overlook the impact of perception and design choices on a robot’s learning capabilities. To address this gap, we propose a comprehensive methodology that accounts for the interplay between the robot’s perception, hardware characteristics, and task requirements. Our approach optimizes the robot’s morphology holistically, leading to improved learning and task execution proficiency. To achieve this, we introduce a Morphology-AGnostIc Controller (MAGIC), which helps with the rapid assessment of different robot designs. The MAGIC policy is efficiently trained through a novel PRIvileged Single-stage learning via latent alignMent (PRISM) framework, which also encourages behaviors that are typical of robot onboard observation. Our simulation-based results demonstrate that morphologies optimized holistically improve the robot performance by 15-20% on various manipulation tasks, and require 25x less data to match human-expert made morphology performance. In summary, our work contributes to the growing trend of learning-based approaches in robotics and emphasizes the potential in designing robots that facilitate better learning. [Mechanism Design, Visual Learning, Evolutionary Robotics]

Residual Physics Learning and System Identification for Sim-To-Real Transfer of Policies on Buoyancy Assisted Legged Robots

Nitish Rajnish Sontakke, Hosik Chae, Sangjoon Lee, Tianle Huang, Dennis Hong, Sehoon Ha

The light and soft characteristics of Buoyancy Assisted Lightweight Legged Unit (BALLU) robots have a great potential to provide intrinsically safe interactions in environments involving humans, unlike many heavy and rigid robots. However, their unique and sensitive dynamics impose challenges to obtaining robust control policies in the real world. In this work, we demonstrate robust sim-to-real transfer of control policies on the BALLU robots via system identification and our novel residual physics learning method, Environment Mimic (EnvMimic). First, we model the nonlinear dynamics of the actuators by collecting hardware data and optimizing the simulation parameters. Rather than relying on standard supervised learning formulations, we utilize deep reinforcement learning to train an external force policy to match real-world trajectories, which enables us to model residual physics with greater fidelity. We analyze the improved simulation fidelity by comparing the simulation trajectories against the real-world ones. We finally demonstrate that the improved simulator allows us to learn better walking and turning policies that can be successfully deployed on the hardware of BALLU. [Model Learning for Control, Reinforcement Learning, Legged Robots]

The Design, Education and Evolution of a Robotic Baby 

Hanqing Zhu, Sean Wilson, Eric Feron

Inspired by Alan Turing’s idea of a child machine, we introduce the formal definition of a robotic baby, an integrated system with minimal world knowledge at birth, capable of learning incrementally and interactively, and adapting to the world. Within the definition, fundamental capabilities and system characteristics of the robotic baby are identified and presented as the system-level requirements. As a minimal viable prototype, the Baby architecture is proposed with a systems engineering design approach to satisfy the system-level requirements, which has been verified and validated with simulations and experiments on a robotic system. We demonstrate the capabilities of the robotic baby in natural language acquisition and semantic parsing in English and Chinese, as well as in natural language grounding, natural language reinforcement learning, natural language programming and system introspection for explainability. The education and evolution of the robotic baby are illustrated with real world robotic demonstrations. Inspired by the genetic inheritance in human beings, knowledge inheritance in robotic babies and its benefits regarding evolution are discussed. [Learning and Adaptive Systems, AI-Based Methods, Control Architectures and Programming, Natural Language Acquisition and Programming]

Tuesday, Oct. 3

Comparing the Effectiveness of Control Methodologies of a Hip-Knee-Ankle Exoskeleton During Squatting

Jared Li, Owen Winship, Stefan Fasano, Bridget Longo, Nicole Esposito, Gregory Sawicki, Robert J. Griffin, Gwendolyn Bryan

Manual materials handling occupations often involve repetitive lifting, lowering, and carrying motions, which can lead to muscular fatigue and/or injury. The risk increases when loads must be worn on the body for the entirety of a job shift. Exoskeletons have been developed to assist these types of motions, but require the user to bear the weight of a load through their body. Load carriage exoskeletons have been developed to offload worn mass from the user to the ground through the device structure, but they have had limited success and have not been well studied in manual materials handling tasks. In this paper, we introduce a hip-knee-ankle exoskeleton and two control methods: virtual model control and gravity compensation. We compared the ability of each controller to reduce lower-limb muscle activity during squatting. Because the virtual model controller is tailored to squatting, we hypothesized that it would outperform gravity compensation. Both controllers were able to reduce the activity of major lower-limb muscle groups during squatting when compared to squatting with the exoskeleton turned off. Contrary to our original hypothesis, the gravity compensation controller generally outperformed the virtual model controller, which may have been caused by the gravity compensation controller having more consistent knee torque application. These results indicate the efficacy of both controllers in reducing injury risk in the lower limbs during squatting. [Prosthetics and Exoskeletons, Human-Centered Robotics, Wearable Robotics]

Construction & Implementation of a Soft Continuum Manipulator

Yenamala Reddy, Chase Lolley, Konrad Ahlin, Stephen Balakirsky

In the field of chimeric antigen receptor (CAR) T-cell cancer therapy, ensuring production quality is crucial in bioprocessing. This paper presents an innovative approach utilizing a hollow hyper-redundant robot for sampling and visual inspection of biological substances within vertical wheel bioreactors [Industrial Robots, Soft Robot Applications]

Control of Cart-Like Nonholonomic Systems Using a Mobile Manipulator

Sergio Aguilera, Seth Hutchinson

This work focuses on the capability for Mobile Manipulators to effectively control and maneuver cart-like nonholonomic systems. These cart-like systems are passive wheeled objects with nonholonomic constraints with varying inertial parameters. We derive the dynamic equations of the cart-like system using a constrained Euler-Lagrange formulation and propose a Linear Quadratic Regulator controller to move the cart along a desired trajectory using external forces (applied by the MM) at a given contact point. For the MM, we present a control architecture to i) control the mobile base to keep the cart inside the workspace of the manipulator and ii) a control Lyapunov function formulation to control the manipulator in torque control, while decoupling the motion of the base from the arm and applying the required wrench onto the object. We validate our approach experimentally, using a MM to push a shopping cart and track a desired trajectories. These experiments show the accuracy of the control architecture to track the desired trajectories for carts with different inertial parameters and improve of controllability of the system by changing the contact point on the cart. [Mobile Manipulation, Force Control, Dexterous Manipulation]

Convex Approach to Data-Driven Off-Road Navigation Via Linear Transfer Operators

Joseph Moyalan, Yongxin Chen, Umesh Vaidya

We consider the problem of optimal control design for navigation on off-road terrain. We use a traversability measure to characterize the difficulty of navigation on off-road terrain. The traversability measure captures terrain properties essential for navigation, such as elevation maps, roughness, slope, and texture. The terrain with the presence or absence of obstacles becomes a particular case of the proposed traversability measure. We provide a convex formulation to the off-road navigation problem by lifting the problem to the density space using the linear Perron-Frobenius (P-F) operator. The convex formulation leads to an infinite-dimensional optimal navigation problem for control synthesis. We construct the finite-dimensional approximation of the optimization problem using data. We use a computational framework based on the data-driven approximation of the Koopman operator. This makes the proposed approach data-driven and applicable to cases where an explicit system model is unavailable. Finally, we apply the proposed navigation framework with single integrator dynamics and Dubin’s car model. [Motion and Path Planning, Optimization and Optimal Control, Model Learning for Control]

D-ITAGS: A Dynamic Interleaved Approach to Resilient Task Allocation, Scheduling, and Motion Planning

Glen Neville, Sonia Chernova, Harish Ravichandar

“Complex, multi-objective missions require the coordination of heterogeneous robots at multiple inter-connected levels, such as coalition formation, scheduling, and motion planning. This challenge is exacerbated by dynamic changes, such as sensor and actuator failures, communication loss, and unexpected delays. We introduce Dynamic Iterative Task Allocation Graph Search (D-ITAGS) to simultaneously address coalition formation, scheduling, and motion planning in dynamic settings involving heterogeneous teams. D-ITAGS achieves resilience via two key characteristics: i) interleaved execution, and ii) targeted repair. Interleaved execution enables an effective search for solutions at each layer while avoiding incompatibility with other layers. Targeted repair identifies and repairs parts of the existing solution impacted by a given disruption, while conserving the rest. In addition to algorithmic contributions, we provide theoretical insights into the inherent trade-off between time and resource optimality in these settings and derive meaningful bounds on schedule suboptimality. Our experiments reveal that i) D-ITAGS is significantly faster than recomputation from scratch in dynamic settings, with little to no loss in solution quality, and ii) the theoretical suboptimality bounds consistently hold in practice.” [Multi-Robot Systems, Cooperating Robots, Distributed Robot Systems]

Game-Theoretical Approach to Multi-Robot Task Allocation Using a Bio-Inspired Optimization Strategy

Shengkang Chen, Tony X. Lin, Fumin Zhang

This paper introduces a game-theoretical approach to the multi-robot task allocation problem, where each robot is considered as self-interested and cannot share its personal utility functions. We consider the case where each robot can execute multiple tasks and each task requires only one robot. For real-world applications with mobile robots, we design a utility function that includes both assignment conflict penalties and path-dependent execution cost. For a robot to maximize its own utility, it needs to select a subset of conflict-free tasks that minimizes its total travel distance. Our approaches utilize a consensus communication scheme to share robots’ task selection and the Speeding-Up and Slowing-Down (SUSD) strategy to search in a combinatorial action (task selection) space for a subset of tasks that can achieve a higher utility at each iteration. The SUSD strategy can perform a gradient-like search without calculating the derivatives, which allows robots to improve upon on their current task selections. Simulation results show that robots using the proposed algorithms can successfully find Nash equilibria for effective coordination. [Multi-Robot Systems]

Lidar Panoptic Segmentation and Tracking without Bells and Whistles

Abhinav Agarwalla, Xuhua Huang, Jason Ziglar, Francesco Ferroni, Laura Leal-Taixe, James Hays, Aljosa Osep, Deva Ramanan

State-of-the-art lidar panoptic segmentation (LPS) methods follow “bottom-up” segmentation-centric fashion wherein they build upon semantic segmentation networks by utilizing clustering to obtain object instances. In this paper, we re-think this approach and propose a surprisingly simple yet effective detection-centric network for both LPS and tracking. Our network is modular by design and optimized for all aspects of both the panoptic segmentation and tracking task. One of the core components of our network is the object instance detection branch, which we train using point-level (modal) annotations, as available in segmentation-centric datasets. In the absence of amodal (cuboid) annotations, we regress modal centroids and object extent using trajectory-level supervision that provides information about object size, which cannot be inferred from single scans due to occlusions and the sparse nature of the lidar data. We obtain fine-grained instance segments by learning to associate lidar points with detected centroids. We evaluate our method on several 3D/4D LPS benchmarks and observe that our model establishes a new state-of-the-art among open-sourced models, outperforming recent query-based models. [Object Detection, Segmentation and Categorization, Recognition, AI-Based Methods]

Risk-Tolerant Task Allocation and Scheduling in Heterogeneous Multi-Robot Teams

Jinwoo Park, Andrew Messing, Harish Ravichandar, Seth Hutchinson

Effective coordination of heterogeneous multi-robot teams requires optimizing allocations, schedules, and motion plans in order to satisfy complex multi-dimensional task requirements. This challenge is exacerbated by the fact that real-world applications inevitably introduce uncertainties into robot capabilities and task requirements. In this paper, we extend our previous work on trait-based time-extended task allocation to account for such uncertainties. Specifically, we leverage the Sequential Probability Ratio Test to develop an algorithm that can guarantee that the probability of failing to satisfy task requirements is below a user-specified threshold. We also improve upon our prior approach by accounting for temporal deadlines in addition to synchronization and precedence constraints in a Mixed-Integer Linear Programming model. We evaluate our approach by benchmarking it against three baselines in a simulated battle domain in a city environment and compare its performance against a state-of-the-art framework in a pandemic-inspired multi-robot service coordination problem. Results demonstrate the effectiveness and advantages of our approach, which leverages redundancies to manage risk while simultaneously minimizing makespan. [Multi-Robot Systems, Task and Motion Planning]

Task-Oriented Grasp Prediction with Visual-Language Inputs

Chao Tang, Dehao Huang, Lingxiao Meng, Weiyu Liu, Hong Zhang

To perform household tasks, assistive robots receive commands in the form of user language instructions for tool manipulation. The initial stage involves selecting the intended tool (i.e., object grounding) and grasping it in a task-oriented manner (i.e., task grounding). Nevertheless, prior researches on visual-language grasping (VLG) focus on object grounding, while disregarding the fine-grained impact of tasks on object grasping. Task-incompatible grasping of a tool will inevitably limit the success of subsequent manipulation steps. Motivated by this problem, this paper proposes GraspCLIP, which addresses the challenge of task grounding in addition to object grounding to enable task-oriented grasp prediction with visual-language inputs. Evaluation on a custom dataset demonstrates that GraspCLIP achieves superior performance over established baselines with object grounding only. The effectiveness of the proposed method is further validated on an assistive robotic arm platform for grasping previously unseen kitchen tools given the task specification. Our presentation video is available at: https://www.youtube.com/watch?v=e1wfYQPeAXU. [Grasping, Deep Learning in Grasping and Manipulation, Perception for Grasping and Manipulation]

Wednesday, Oct. 4

An Implantable Variable Length Actuator for Modulating in Vivo Musculo-Tendon Force in a Bipedal Animal Model

Sean Thomas, Ravin Joshi, Bo Cheng, Huanyu Cheng, Michael C. Aynardi, Gregory Sawicki, Jonas Rubenson

Mobility, a critical factor in quality of life, is often rehabilitated using simplistic solutions, such as walkers. Exoskeletons (wearable robotics) offer a more sophisticated rehabilitation approach. However, non-adherence to externally worn mobility aids limits their efficacy. Here, we present the concept of a fully implantable assistive limb actuator that overcomes non-adherence constraints, and which can provide high-precision assistive force. In a bipedal animal model (fowl), we have developed a variable length isometric actuator (measuring ­9 × 30 mm) that is able to be directly implanted within the leg via a bone anchor and tendon fixation, replacing the lateral gastrocnemius muscle belly. The actuator is able to generate isometric force similar to the in vivo force of the native muscle, designed to generate assistive torque at the ankle and reduce muscular demand at no additional energy cost. The device has a stroke of 10 mm that operates up to 770 mm/s (77 stroke lengths/s), capable of acting as a clutch (disengaging when needed) and with a tunable slack length to modulate the timing and level of assistive force during gait. Surgical techniques to attach the actuator to the biological system, the Achilles tendon and tibia, have been established and validated using survival surgeries and cadaveric specimens. [Humanoid and Bipedal Locomotion, Rehabilitation Robotics, Wearable Robotics]

Athletic Mobile Manipulator System for Robotic Wheelchair Tennis

Zulfiqar Zaidi, Daniel Martin, Nathaniel Belles, Viacheslav Zakharov, Arjun Krishna, Kin Man Lee, Peter Wagstaff, Sumedh Naik, Matthew Sklar, Sugju Choi, Yoshiki Kakehi, Ruturaj Patil, Divya Mallemadugula, Florian Pesce, Peter Wilson, Wendell Hom, Matan Diamond, Bryan Zhao, Nina Moorman, Rohan Paleja, Letian Chen, Esmaeil Seraj, Matthew Gombolay

Athletics are a quintessential and universal expression of humanity.From French monks who in the 12th century invented jeu de paume, the precursor to modern lawn tennis, back to the K’iche’ people who played the Maya Ballgame as a form of religious expression over three thousand years ago, humans have sought to train their minds and bodies to excel in sporting contests. Advances in robotics are opening up the possibility of robots in sports. Yet, key challenges remain, as most prior works in robotics for sports are limited to pristine sensing environments, do not require significant force generation, or are on miniaturized scales unsuited for joint human-robot play. In this paper, we propose the first open-source, autonomous robot for playing regulation wheelchair tennis. We demonstrate the performance of our full-stack system in executing ground strokes and evaluate each of the system’s hardware and software components. The goal of this paper is to (1) inspire more research in human-scale robot athletics and (2) establish the first baseline for a reproducible wheelchair tennis robot for regulation singles play. Our paper contributes to the science of systems design and poses a set of key challenges for the robotics community to address in striving towards robots that can match human capabilities in sports. [Engineering for Robotic Systems]

Cognition Difference-Based Dynamic Trust Network for Distributed Bayesian Data Fusion

Yingke Li, Ziqiao Zhang, Junkai Wang, Huibo Zhang, Enlu Zhou, Fumin Zhang

Distributed Data Fusion (DDF), as a prevalent technique that empowers scalable, flexible, and robust information fusing, has been employed in various multi-sensor networks operating in uncertain and dynamic environments. This paper proposes a cognition difference-based mechanism to construct a dynamic trust network for real-time DDF, where the cognition difference is defined as the statistical difference between the sensors’ estimated probability distributions. Distinguished by the mutual correlation between trust and cognition difference, two principles of determining trust are investigated, and their performances are analyzed by conducting simulations in the scenarios of source seeking. Our simulation and experiment results show that the proposed approach is effective in providing comprehensive and robust performance in general and unstructured environments. [Sensor Fusion, Distributed Robot Systems, Sensor Networks]

Fluoroscopic Image-Based 3-D Environment Reconstruction and Automated Path Planning for a Robotically Steerable Guidewire

Sharan Ravigopal, Timothy A. Brumfiel, Achraj Sarma, Jaydev P. Desai

“Cardiovascular diseases are the leading cause of death globally and surgical treatments for these often begin with the manual placement of a long compliant wire, called a guidewire, through different vasculature. To improve procedure outcomes and reduce radiation exposure, we propose steps towards a fully automated approach for steerable guidewire navigation within vessels. In this paper, we utilize fluoroscopic images to fully reconstruct 3-D printed phantom vasculature models by using a shape-from-silhouette algorithm. The reconstruction is subsequently de-noised using a deep learning-based encoder-decoder network and morphological filtering. This volume is used to model the environment for guidewire traversal. Following this, we present a novel method to plan an optimal path for guidewire traversal in three-dimensional vascular models through the use of slice planes and a modified hybrid A-star algorithm. Finally, the developed reconstruction and planning approaches are applied to an ex vivo porcine aorta, and navigation is demonstrated through the use of a tendon-actuated COaxially Aligned STeerable guidewire (COAST).” [Surgical Robotics: Steerable Catheters/Needles, Computer Vision for Medical Robotics, Surgical Robotics: Planning]

Generating Scenarios from High-Level Specifications for Object Rearrangement Tasks

Sanne van Waveren, Christian Pek, Iolanda Leite, Jana Tumova, Danica Kragic

Rearranging objects is an essential skill for robots. To quickly teach robots new rearrangements tasks, we would like to generate training scenarios from high-level specifications that define the relative placement of objects for the task at hand. Ideally, to guide the robot’s learning we also want to be able to rank these scenarios according to their difficulty. Prior work has shown how diverse scenario generation from specifications and providing the robot with easy-to-difficult samples can improve the learning. Yet, existing scenario generation methods typically cannot generate diverse scenarios while controlling their difficulty. We address this challenge by conditioning generative models on spatial logic specifications to generate spatially-structured scenarios that meet the specification and desired difficulty level. Our experiments showed that generative models are more effective and data-efficient than rejection sampling and that the spatially-structured scenarios can drastically improve training of downstream tasks by orders of magnitude. [Data Sets for Robot Learning]

Multi-Legged Matter Transport: A Framework for Locomotion on Noisy Landscapes

Baxi Chong, Juntao He, Daniel Soto, Tianyu Wang, Daniel Irvine, Grigoriy Blekherman, Daniel Goldman

While the transport of matter by wheeled vehicles or legged robots can be guaranteed in engineered landscapes like roads or rails, locomotion prediction in complex environments like collapsed buildings or crop fields remains challenging. Inspired by principles of information transmission which allow signals to be reliably transmitted over noisy channels, we develop a “matter transport” framework demonstrating that non-inertial locomotion can be provably generated over “noisy” rugose landscapes (heterogeneities on the scale of locomotor dimensions). Experiments confirm that sufficient spatial redundancy in the form of serially-connected legged robots leads to reliable transport on such terrain without requiring sensing and control. Further analogies from communication theory coupled to advances in gaits (coding) and sensor-based feedback control (error detection/correction) can lead to agile locomotion in complex terradynamic regimes. [Redundant Robots, Motion Control, Mechanism Design]

Natural Language Specification of Reinforcement Learning Policies through Differentiable Decision Trees

Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, Matthew Gombolay

Human-AI policy specification is a novel procedure we define in which humans can collaboratively warm-start a robot’s reinforcement learning policy. This procedure is comprised of two steps; (1) Policy Specification, i.e. humans specifying the behavior they would like their companion robot to accomplish, and (2) Policy Optimization, i.e. the robot applying reinforcement learning to improve the initial policy. Existing approaches to enabling collaborative policy specification are often unintelligible black-box methods, and are not catered towards making the autonomous system accessible to a novice end-user. In this paper, we develop a novel collaborative framework to enable humans to initialize and interpret an autonomous agent’s behavior. Through our framework, we enable humans to specify an initial behavior model via unstructured, natural language, which we convert to lexical decision trees. Next, we leverage these translated human-specifications, to warm-start reinforcement learning and allow the agent to further optimize these potentially suboptimal policies. Our approach warm-starts an RL agent by utilizing non-expert natural language specifications without incurring the additional domain exploration costs. We validate our approach by showing that our model is able to produce >80% translation accuracy, and that policies initialized by a human are able match the performance of relevant RL baselines in two differing domains. [Human-Centered Automation, Human-Centered Robotics]

Next-Best-View Selection from Observation Viewpoint Statistics

Stephanie Aravecchia, Antoine Richard, Marianne Clausel, Cedric Pradalier

This paper discusses the problem of autonomously constructing a qualitative map of an unknown 3D environment using a 3D-Lidar. In this case, how can we effectively integrate the quality of the 3D-reconstruction into the selection of the Next-Best-View? Here, we address the challenge of estimating the quality of the currently reconstructed map in order to guide the exploration policy, in the absence of ground truth, which is typically the case in exploration scenarios. Our key contribution is a method to build a prior on the quality of the reconstruction from the data itself. Indeed, we not only prove that this quality depends on statistics from the observation viewpoints, but we also demonstrate that we can enhance the quality of the reconstruction by leveraging these statistics during the exploration. To do so, we propose to integrate them into Next-Best-View selection policies, in which the information gain is directly computed based on these statistics. Finally, we demonstrate the robustness of our approach, even in challenging environments, with noise in the robot localization, and we further validate it through a real-world experiment. [Mapping, Reactive and Sensor-Based Planning, Field Robots]

Simultaneous Shape and Tip Force Sensing for the COAST Guidewire Robot

Nancy Joanna Deaton, Timothy A. Brumfiel, Achraj Sarma, Jaydev P. Desai

Placement of catheters in minimally invasive cardiovascular procedures is preceded by navigating to the target lesion with a guidewire. Traversing through tortuous vascular pathways can be challenging without precise tip control, potentially resulting in the damage or perforation of blood vessels. To improve guidewire navigation, this paper presents 3D shape reconstruction and tip force sensing for the COaxially Aligned STeerable (COAST) guidewire robot using a triplet of adhered single core fiber Bragg grating sensors routed centrally through the robot’s slender structure. Additionally, several shape reconstruction algorithms are compared, and shape measurements are utilized to enable tip force sensing. Demonstration of the capabilities of the robot is shown in free air where the shape of the robot is reconstructed with average errors less than 2mm at the guidewire tip, and the magnitudes of forces applied to the tip are estimated with an RMSE of 0.027N or less. [Tendon/Wire Mechanism, Compliant Joints and Mechanisms]

Telerobotic Transcatheter Delivery System for Mitral Valve Implant

Ronghuai Qi, Namrata Unnikrishnan Nayar, Jaydev P. Desai

Mitral regurgitation (MR) is the most common type of valvular heart disease, affecting over 2% of the world population, and the gold-standard treatment is surgical mitral valve repair/replacement. Compared to open-heart surgeries, minimally invasive surgeries (MIS) using transcatheter approaches have become popular because of their notable benefits such as less postoperative pain, shorter hospital stay, and faster recovery time. However, commercially available catheters are manually actuated, causing over-exposure of clinical staff to radiation and increased risk of human error during medical interventions. To tackle this problem, in this letter, we propose a telerobotic transcatheter delivery system, which consists of a robotic catheter (5.7mm OD), a reinforced guide tube (1.11m length), and an actuation system. We present the robotic system design, fabrication of key components, and static model of reinforced quadlumen tube. The robot interface design enables the user to intuitively control the robot. We demonstrate the effectiveness of the telerobotic transcatheter delivery system and reinforced quadlumen tube in a realistic human cardiovascular phantom for preclinical evaluation. [Medical Robots and Systems, Mechanism Design, Product Design, Development and Prototyping]

The Effects of Robot Motion on Comfort Dynamics of Novice Users in Close-Proximity Human-Robot Interaction

Pierce Howell, Jack Kolb, Yifan Liu, Harish Ravichandar

Effective and fluent close-proximity human-robot interaction requires understanding how humans get habituated to robots and how robot motion affects human comfort. While prior work has identified humans’ preferences over robot motion characteristics and studied their influence on comfort, we are yet to understand how novice first-time robot users get habituated to robots and how robot motion impacts the dynamics of comfort over repeated interactions. To take the first step towards such understanding, we carry out a user study to investigate the connections between robot motion and user comfort and habituation. Specifically, we study the influence of workspace overlap, end-effector speed, and robot motion legibility on overall comfort and its evolution over repeated interactions. Our analyses reveal that workspace overlap, in contrast to speed and legibility, has a significant impact on users’ perceived comfort and habituation. In particular, lower workspace overlap leads to users reporting significantly higher overall comfort, lower variations in comfort, and fewer fluctuations in comfort levels during habituation. [Human-Robot Collaboration, Physical Human-Robot Interaction, Human-Centered Robotics]

Visual Contact Pressure Estimation for Grippers in the Wild

Jeremy Collins, Cody Houff, Patrick Grady, Charles C. Kemp

Sensing contact pressure applied by a gripper can benefit autonomous and teleoperated robotic manipulation, but adding tactile sensors to a gripper’s surface can be difficult or impractical. If a gripper visibly deforms, contact pressure can be visually estimated using images from an external camera that observes the gripper. While researchers have demonstrated this capability in controlled laboratory settings, prior work has not addressed challenges associated with visual pressure estimation in the wild, where lighting, surfaces, and other factors vary widely. We present a deep learning model and associated methods that enable visual pressure estimation under widely varying conditions. Our model, Visual Pressure Estimation for Robots (ViPER), takes an image from an eye-in-hand camera as input and outputs an image representing the pressure applied by a soft gripper. Our key insight is that force/torque sensing can be used as a weak label to efficiently collect training data in settings where pressure measurements would be difficult to obtain. When trained on this weakly labeled data combined with fully labeled data that includes pressure measurements, ViPER outperforms prior methods, enables precision manipulation in cluttered settings, and provides accurate estimates for unseen conditions relevant to in-home use. [Contact Modeling, Modeling, Control, and Learning for Soft Robots, Perception for Grasping and Manipulation]

See you in Detroit!