Robotic Caregivers – Fall 2021

BMED 4833 ROB & BMED 8813 ROB at Georgia Tech

Lectures: Monday & Wednesday, 11 am- 12:15 pm, Whitaker Room 260

Overview

Robotics researchers and futurists have long dreamed of robots that can serve as caregivers. In this project-based course, you’ll learn about future opportunities and present realities for robots that contribute to caregiving. You’ll gain hands-on experience with teleoperation, autonomy, perception, navigation, manipulation, human-robot interaction, and machine learning. You’ll also learn about robot design, healthcare robotics, and entrepreneurship.

Instructor

Prof. Charlie Kemp

Conflict of Interest Statement: This class uses the Stretch RE1 robot from Hello Robot Inc. In addition to being an associate professor at Georgia Tech, Dr. Kemp is a co-founder and the chief technology officer (CTO) of Hello Robot Inc. where he works part time. He owns equity in Hello Robot and is an inventor of Georgia Tech intellectual property (IP) licensed by Hello Robot. Consequently, he receives royalties through Georgia Tech for sales made by Hello Robot. He also benefits from increases in the value of Hello Robot.

Final Projects

At the end of the term, the three teams presented their projects via videos and slides. If all the team members were in favor of it, the team provided materials to be posted here.

Each team had members with robotics expertise, including at least one Robotics PhD student, and biomedical expertise, including at least two biomedical engineering undergraduates. Teams also had members from the new Robotics MS program and the Biomedical Engineering MS program. All students had experience with the Python programming language.

The Bottle Butler by Team Red

Video, Google Slides, PDF of Slides, Code

Still from Bottle Butler Video Still from Bottle Butler Video

Midnight Stretch by Team Green

Video, PDF of Slides

Midnight Stretch Still from Video Midnight Stretch Still from Video

Midnight Stretch Still from Video Midnight Stretch Still from Video

Stretch with Stretch by Team Blue

Video, PDF of Slides

Team member demonstrates Stretch with Stretch.  Team member demonstrates Stretch with Stretch.

Instagram Post from their Trip to a Physical Therapy Center

Image from Instagram PostI Image from Instagram PostImage from Instagram Post

Course Materials

A Robot for Each Team

The three project robots in their boxes ready to be opened by the teams.

Each team had exclusive use of a Stretch RE1 mobile manipulator throughout the term that they could access outside of class. This promoted a sense of ownership, including responsibility and care, and avoided the potential for one team to break another team’s work. It also enabled them to add custom hardware to their robots and leave it on. This approach also helps with management of the robots, since instructors can wipe and refresh the robots before next year’s class.

To better work together as a team, each team had its own 32″ monitor that they connected to their robot with an HDMI cable. The large size enabled multiple team members to work with the robot at the same time. Using a smaller monitor proved difficult during an early programming exercise. Specifically, each team used a Samsung S32D850T LCD Monitor, which has a resolution of 2560 x 1440 (WQHD). Each team also had a wired keyboard and mouse that they connected to their robot to avoid the potential for interference.

Classes

  • Class 1 – Class Overview
  • Class 2 – Team Formation & Robot Teleoperation via a Game Controller
    • “From One to Many: My Personal Quest for Meaningful Mobile Manipulation”
      by Prof. Kemp [Video] [Google Slides] [PDF]
    • Activities
      • Broke up into teams to achieve comparable mixes of skills and degree levels
      • Team members got to know each other with icebreaker questions
      • Each 5-person team spent 10 minutes teleoperating the Stretch RE1 using the game controller with each team member getting a turn
    • Assignments
      • Carefully read the Problem Statement
      • Propose 3 to 5 tasks for your team’s midterm project and provide evidence for their merit
  • Class 3 – Teleoperation Continued
  • Class 4 – Ideation on Tasks & Programming an Autonomous Action
    • “Tasks & Teleop Exercise” by Prof. Kemp [Google Slides] [PDF]
    • Activities
      • Each team
        • 30 minutes
          • Discussed tasks proposed by the team members
          • Selected top three proposed tasks
        • 15 minutes
          • Implemented and tested their teleop code on the robot
          • All of the teams succeeded!
        • 5 minutes
          • Presented top three tasks to the entire class
          • Received feedback on their tasks from Prof. Kemp
      • While two teams discussed tasks, the third team worked with the robot
    • Assignments
      • Get started on midterm proposal slides
      • Prepare project related questions for the expert guest lecturers
  • Class 5 – Guest Lecture from Expert on Professional Caregiving
  • Class 6 – Guest Lecture from Experts on Robots for Older Adults
  • Class 7 – Guest Lecture from Experts on Robots for People with Disabilities
    • “Evaluating Assistive Devices” by Henry Evans [Video]
    • Discussion with Henry & Jane Evans
      • Henry and Jane Evans are recognized experts on robots for people with disabilities. They have unmatched experience using assistive robots in their home, and a passion for improving the lives of people with disabilities. This was a deeply meaningful guest lecture and an excellent opportunity to benefit from their insights.
  • Class 8 – Team Day Focused on Midterm Proposals
    • Each team worked on its midterm proposal for the next class.
    • Prof. Kemp spent approximately 18 minutes with each team to answer their questions and provide guidance.
  • Class 9 – Midterm Proposal Presentations
  • Class 10 – Robot Unboxing & Getting to Know Your Robot, Part 1
    • “Getting to Know Your Robot” by Prof. Kemp [Google Slides][PDF]
    •  Activities
      • Each team discussed the feedback they received for their proposal presentation and Prof. Kemp spent time with each team to discuss the feedback
      • Teams began unboxing their robots!
        • The three project robots in their boxes ready to be opened by the teams.
        • To better work together as a team, each team has its own 32″ monitor that they connect to their robot with an HDMI cable. Specifically, they’re using a Samsung S32D850T LCD Monitor, which has a resolution of 2560 x 1440 (WQHD).
        • Note for next time: Leave more time so that each team can unbox and teleoperate the robot using the game controller interface
  • Class 11 – Getting to Know Your Robot, Part 2
    • “Getting to Know Your Robot – Part 2” by Prof. Kemp [Google Slides][PDF]
    •  Activities
      • Teams began working with their robots
      • Teams also carefully stored their robots for the next class
    • Assignments
      • Gain access to the room, so that your team can work with its robot outside of class
      • Begin implementing projects
  • Class 12 – Team Day Focused on Midterm Projects
  • Class 13 – Team Day Focused on Midterm Projects
    • The class met via MS Teams.
    • After a brief full-class discussion, the teams went to their own video conference meetings (breakout groups).
      • Prof. Kemp emphasized that each team owns its robot over the term, which implies that each team has freedom in how it works with its robot and the responsibility to take care of its robot.
      • During the full-class discussion, class members raised issues and solutions that they thought might be relevant to other teams. This was a welcome development, since the teams are supposed to be collaborating rather than competing.
    • Prof. Kemp then spent time with each group in the reverse order as the previous class.
      • He let teams know up front that it was fine to not take advantage of his visit and instead focus on working together.
      • Prof. Kemp focused on discussing any questions the team had.
      • Prof. Kemp also asked teams if they were happy with the class and if they were having fun. He encouraged class members to let him know if they had feedback or ideas for the class.
  • Class 14 – Team Day Focused on Midterm Projects
    • Teams worked with their robots to implement their projects.
    • One team wanted to send commands to their robot from ROS.
    • Another team was attempting to install the basic web interface on GitHub. They were encountering difficulties. One of the other teams had successfully installed it and overcome challenges that occurred. A representative from that team began helping them and volunteered to meet with them outside of class.
    • The third team took their robot to a different location so that they could use the game controller to teleoperate it to perform manipulation tasks with a special object and test out a custom 3D-printed attachment they developed.
  • Class 15 – Team Day Focused on Midterm Projects
    • Prof. Kemp spent time with each team.
    • He got to try out a team’s systems, which enabled him to provide detailed feedback from the perspective of both an expert and a naive user. He was impressed! The demonstration was the most compelling autonomous interaction he had personally experienced with the Stretch RE1.
    • He visited another team, which was working in another room better matched to their project. He learned more about the team’s approach and difficulties. He also let them know that the default force limit on the arm was likely creating challenges for them. The team had come up with a creative solution to their problem that involved minor modification to the environment.
    • He encouraged a team to be critical of their own work and actively look for weaknesses. He emphasized that unlike research and business in the real world, this class encourages negative results!
  • Class 16 – Team Day Focused on Midterm Projects
    • Due to ROS World 2021, Prof. Kemp was unable to attend class. Teams worked on their own preparing for their midterm project presentations. The following day, Prof. Kemp held open office hours via MS Teams. Only one team attended.
  • Class 17 – Midterm Project Presentations
    • Each team presented their midterm project for 10 minutes and then had 10 minutes to answer questions from a panel of experts. Questions were restricted to the panel of experts and graders. The next class will be devoted to in-class discussion about each of the projects, so all class members will have time to ask questions.
    • Expert Panel
      • Henry and Jane Evans,  Dr. Vy Nguyen, and Julian Cheng served on the expert panel via MS Teams on a Linux laptop connected to a large monitor, a sound bar, and a high-quality wide-angle webcam placed in the middle of the room.
      • Julian Cheng added his expertise on commercialization, startups, and fundraising to the panel. He is a venture capitalist (VC) and the Founder and Managing Partner of Gen 1 Capital, which is an investor in Hello Robot. Julian is an engineer by training, active in higher education, and familiar with the Stretch RE1.
      • Panelists were informed ahead of time that they need to consider the teams’ presentations as confidential, since the teams own any intellectual property (IP) they create.
    • Prof. Kemp, Henry Clever, and Patrick Grady graded the presentations using the midterm project rubric.
    • All of the projects were distinct and exciting! Each team showed a compelling video and one of the teams gave a successful live demonstration!
  • Class 18 – Class Discussion about Midterm Projects
    • “Final Project Discussion” by Prof. Kemp
    • Activities (see Prof. Kemp’s slides for additional details)
      • Team members read the graded rubrics that were posted in the morning.
      • Prof. Kemp provided general feedback on the midterm projects.
      • The class discussed the final project and final project proposal.
        • Final project video
          • Students were in favor of each team creating a short video (30 sec to 1.5 min)  suitable for posting on the Internet.
          • Students were also in favor of Prof. Kemp actually posting the three videos, but they will have the option to change their minds.
          • Deciding on this now gives teams a clear goal and a better chance of creating videos that they’re happy to share with the world.
        • Students were in favor of Prof. Kemp creating a new final proposal rubric even though there is not much time until their presentations.
      • For the last 30 minutes, the class discussed each team’s project for 10 minutes with an emphasis on next steps. This was enjoyable and highly valuable!
  • Class 19 – Class Discussion about Final Projects
  • Class 20 – Final Proposal Presentations
    • Each team had 10 minutes to present their project plans for the rest of the term followed by a 10 minute discussion.
    • All teams chose to continue to work on their current projects.
    • Prof. Kemp, Henry Clever, and Patrick Grady provided their assessments and feedback using the final proposal rubric.
    • The presentations and discussions were insightful.
  • Class 21 – Commercialization, Part 1
    • “Commercialization” by Prof. Kemp [Google Slides][PDF]
      • Prof. Kemp should have left more time for the Q&A session at the end.
  • Class 22 – Commercialization, Part 2
    • “Commercialization Follow-up with Q&A” by Prof. Kemp [Google Slides][PDF]
      • After the Q&A session, Prof. Kemp used an opt-in rather than an opt-out approach to team consultations, since the teams were highly engaged in their own discussions.
  • Class 23 – Team Day Focused on Final Projects
    • Team day without Prof. Kemp.
  • Class 24 – Team Day Focused on Final Projects
  • Class 25 – Team Day Focused on Final Projects
    • Team day with consultations by Prof. Kemp.
      • Students asked about speech recognition, laser range finders, and object grasping among other topics.
  • Class 26 – Team Day Focused on Final Projects
    • Last team day before the video viewing session and final presentations! Everyone was hard at work.
  • Class 27 – Video Viewing Session
    • Teams showed their videos to the class and the class provided feedback. The time was divided among the three teams and fully utilized.
      • The primary goal was to help teams prepare videos for public dissemination on the Internet. All team members indicated they would like to make their videos available to the general public.
      • The topics of discussion included the intended audience, introductions to provide context, incorporation of technical details, opportunities to improve clarity, opportunities to improve quality, and attribution via team member participation in the videos and credits at the end of the videos.
  • Class 28 – Final Project Presentations
    • Each team presented its final project for 10 minutes and then had 10 minutes to answer questions from a panel of experts. Questions were restricted to the panel of experts and graders. Throughout the term, the students had asked each other questions and given feedback.
    • Prof. Kemp, Henry Clever, and Patrick Grady graded the presentations using the final project rubric.
    • Expert Panel
      • We were fortunate to have a distinguished panel of experts attend virtually. Julian Cheng, Prof. Sonia Chernova, Prof. Zackory Erickson, Henry and Jane Evans, Prof. Seth Hutchinson, Dr. Travis Kadylak, Dr. Vy Nguyen, and Prof. Wendy Rogers served on the panel.
        • Panelists were informed ahead of time that they would need to consider the teams’ presentations to be confidential, since the teams own any intellectual property (IP) they create.
        • The panel members participated via MS Teams on a Linux laptop connected to a large monitor, a USB-powered sound bar, and a wide-angle webcam (Logitech C920) placed in the middle of the classroom on a rolling overbed table (see image below).
          • Video Conferencing Equipment for the Expert Panel
      • Based on our experience from the mid-term project presentations and suggestions from the students, we conducted this hybrid class as follows.
        • During each team’s presentation
          • Each team presented in the front of the classroom with its laptop display projected onto a large screen.
          • While a team was presenting, Prof. Kemp muted the expert panel laptop and set its sound output volume to zero. This was to avoid audio feedback between the expert panel laptop and the presenting team’s laptop.
          • While presenting, each team shared its laptop display via MS Teams, unmuted its laptop microphone, and set its laptop volume to the maximum value. This avoided audio feedback with the expert panel laptop and enabled the expert panel members to clearly see each team’s slides and video, while also allowing the in-person attendees to see the team’s presentation.
          • When a particular team member was presenting they would move to the podium with the team’s laptop to ensure that they could be seen and heard by the expert panel.
        • During the question and answer session with the expert panel
          • After a team finished presenting, they muted their laptop’s microphone and set its volume to zero to avoid audio feedback.
          • Then, Prof. Kemp unmuted the expert panel laptop and set its volume to be loud.
          • Expert panelists raised their virtual hands to ask the team questions. Prof. Kemp then moderated the questions.
          • As an expert panelist asked questions the team would respond via the expert panel laptop setup (see image above). In practice, a single team member often stepped out towards the expert panel laptop in order to address a panelist’s question.
    • The presentations went well and the expert panelists were impressed. Prof. Kemp and many of the panelists noted that they learned from the projects and appreciated the novel approaches discovered by the teams.
    • Overall, it was an inspiring end to a historic term. The teams provided glimpses of the future through their human-centered explorations of mobile manipulation. Via problem-based learning, students gained a grounded understanding of future opportunities and present realities for robots that contribute to caregiving.

Credit

Zackory Erickson

As a graduate student, Prof. Zackory Erickson co-developed and co-taught the original version of this course with Prof. Kemp  in spring 2020 and spring 2021. The original version focused on projects using Assistive Gym, a physics simulation environment for assistive robotics. Prof. Zackory Erickson is now a tenure-track faculty member in the Robotics Institute at CMU. He will be teaching  16-887: Robotic Caregivers and Intelligent Physical Collaboration at CMU in Spring 2022.

 

Affiliations

Previous Terms

2020 Spring

2021 Spring