HoLLiEs Patisserie: Robot Decorating Cookies with Icing and Marshmallows

Making or decorating cookies is a fun task. Sharing this task with a service robot like HoLLiE in a very direct and intuitive way makes it even more fun! Control the CookieBot to add marshmallows and other toppings to your individual cookie design.

Service robots are now able to execute complex real world tasks and are, as a result, also deployed in many challenging applications. They are used in industrial scenarios, but also more and more in domestic and home environments. Furthermore, they are also deployed to communicate and give support to customers, requiring a direct interaction with humans. Because of this trend, even people without any robotics experience need to be able to interact with these new service robots. Many systems are completely autonomous and the user does not need to interact with them. For example, cleaning robots just need to perform their task while avoiding collisions with humans and obstacles by changing the direction of their motion, based on sensor data. For other tasks, the interaction with humans is a critical aspect, especially if the robot’s motions need to be adapted in order to reach custom goals.
The FZI CookieBot allows the user to decorate cookies using different tools, enabling them to control the robot’s end-effector position with precision and flexibility. To ease the interaction and make any user able to quickly understand how to control the robot and its behaviour, a nice web-based UI has been developed. This helps the user in the different phases of the interaction and allows them to achieve a better precision in control and get clear feedback about the robot behaviour. The setup includes our service robot HoLLiE, the marker based tracking system OptiTrack and a monitor to display the interactive user interface.
HoLLiE is a mobile service robot that we developed as part of the House of Living Labs (HoLL). Thanks to the two arms with 6 degrees of freedom each, it is able to reach all the relevant areas on the baking table. The right arm is equipped with an anthropomorphic hand with 9 degrees of freedom, which enables the grasping of the different tools with arbitrary shapes. For the left arm we used a two finger gripper, in order to handle the spatula that is used to move to cookies to different positions and deliver them to the user. We also used the two joints in the neck to pan and tilt the head in a more natural and human-like way.
On the right side of the table, HoLLie has a set of four different tools to decorate the cookies. This includes sugar stars, sprinkles and marshmallows, which can be dispensed on the cookies flipping the corresponding tool. These toppings have different colours, sizes and shapes, enabling the user to decorate their cookie in many possible and tasteful ways. We also added a sugar gun to make the user able to draw arbitrary letters and shapes on the cookie surface. The left side of the table is where the new cookies are stored. When a new user is available, HoLLiE uses its left arm to get a new cookie to put into the decoration area. While the right arm is busy with the decoration, the left arm is enabled to prepare the next cookie, adding a layer of sugar on top, using a sugar dispenser. Each user has a time limit of four minutes for decorating and after that the cookie will be delivered to them.
To enable the control of different baking utensils, we provided the user with a set of fake tools. Their positions are tracked all the time. Once the user grasps or releases one of the tools, the robot is triggered to execute the same action. All baking utensils are positioned on the user side and in the robots workspace. The two areas have a mutual correspondence and are separated from each other with a glass panel, which creates a separation between the two workspaces.

Each tool on the user side has a set of reflective markers that are used by the OpiTrack motion tracking system. In order to identify each tool, we used different patterns for each one. The default resting positions of the four tools are marked on the GUI. As a feedback to the user, the GUI highlights which tool has been taken and reports that the robot is on its way to grasp it.

After grasping the desired tool, the robot moves to the imitation control area and the GUI notifies the user that they can start to control the robot directly. This is enabled by the use of our ROS-based Motion Pipeline, which allows us to send Cartesian goals to the robot TCP. The tracked position of the user’s tool is transformed in the reference frame of the robot, in order to have the right position on top of the real cookie. The parameter of the Cartesian controller could be set in order to have a more reactive or smooth control, based on the desired application. The GUI displays a virtual representation of the control area, in order to give the user a better understanding of the robot workspace limits. If the user goes outside of the control boundaries, the closest valid positions is sent as target goal for the robot TCP. This allows the user to commit errors during the control, avoiding unwanted robot configurations and possible collisions. As a result, any kind of motion is safe for the robot and will not lead to any collisions or mechanical risks.

The scaling of the tracked human motions allows the user to have better precision in robot control. In this way, the control motions of the tracked baking tools are mapped into smaller motions of the robot TCP, allowing the user to have a more precise control on the cookie. They are also allowed to make small errors in the desired motion, maintaining a steady position of the robot tool.
Regarding the sugar gun, we provided an intuitive way to enabling and disabling the sugar flow. If the user places the corresponding tracked tool close to the virtual cookie displayed on the GUI, the gun is enabled. Lifting up the tool disables the sugar flow. When the gun is enabled, the GUI simulates the flow of the sugar, giving feedback to the user about the control command currently sent to the real tool. In order to allow this switch, based on a distance threshold from the monitor, we added a valve to the sugar gun, which is interfaced with ROS using Arduino.
Since the robot might be controlled by people with no robotic experience, it is important to provide constraints in position, rotation, velocity and acceleration. This is important to avoid risks and undesired collisions when the user is not fully confident with the system, while still maintaining full control over the robot during regular movements.
The kinematics of the robot could lead to problems if arbitrary positions and rotations were allowed. This could cause the robot to not being able to reach the desired position from its current configuration or forcing the robot to reconfigure its joints. However, in order to decorate the cookies with the various toppings it is necessary to control the rotation of the tools. For this reason, we developed a way to control the rotation of the robot tool, based on the orientation of the tracked object. The desired inclination angle is transferred into a rotation of the last joint of the robotic arm, avoiding useless reconfiguration of the entire kinematics.

Once the user finishes to decorate the individual cookie and wants to end the interaction, a button triggers the left robot arm to get the cookie and deliver it. The control of the two arms is completely independent, enabling the robot to prepare new cookies with the left arm while the right arm is used for decorating another one. In this way, the CookieBot can deliver cookies while one arm is still handling and placing the baking tool.
The CookieBot was decorating cookies at the Stallwächter Party 2019 in Berlin and two more events in Germany. Hundreds of people tried to decorate their own cookies with our application. All users found the system really easy and intuitive to use. Many of them had fun trying to decorate the cookie with different amounts of toppings and drawing letters and various shapes with the sugar.