Publications

With domestic technology on the rise, the quantity and complexity of smart-home devices are becoming an important interaction design challenge. We present a novel design for a home control interface in the form of a social robot, commanded via tangible icons and giving feedback through expressive gestures. We experimentally compare the robot to three common smart-home interfaces: a voice-control loudspeaker; a wall-mounted touch-screen; and a mobile application. Our findings suggest that interfaces that rate higher on flow rate lower on usability, and vice versa. Participants’ sense of control is highest using familiar interfaces, and lowest using voice control. Situation awareness is highest using the robot, and also lowest using voice control. These findings raise questions about voice control as a smart-home interface, and suggest that embodied social robots could provide for an engaging interface with high situation awareness, but also that their usability remains a considerable design challenge.

Designing and authoring gestures for socially expressive robots has been an increasingly important problem in recent years. In this demo we present a new pipeline that enables animators to create gestures for robots in a 3D animation authoring environment, without knowledge in computer programming. The pipeline consists of an exporter for a 3D animation software and an interpreter running on a System-on-Module translating the exported animation into motor control commands.

We propose an architecture that integrates Theory of Mind into a robot’s decision-making to infer a human’s intention and adapt to it. The architecture implements human-robot collaborative decision-making for a robot incorporating human variability in their emotional and intentional states. This research first implements a mechanism for stochastically estimating a human’s belief over the state of the actions that the human could possibly be executing. Then, we integrate this information into a novel stochastic human-robot shared planner that models the human’s preferred plan. Our contribution lies in the ability of our model to handle the conditions: 1) when the human’s intention is estimated incorrectly and the true intention may be unknown to the robot, and 2) when the human’s intention is estimated correctly but the human doesn’t want the robot’s assistance in the given context. A robot integrating this model into its decision-making process would better understand a human’s need for assistance and therefore adapt to behave less intrusively and more reasonably in assisting its human companion.

We present a systematic survey of computational research in human-robot interaction (HRI) over the past decade. Computational HRI is the subset of the field that is specifically concerned with the algorithms, techniques, models, and frameworks necessary to build robotic systems that engage in social interactions with humans. Within the field of robotics, HRI poses distinct computational challenges in each of the traditional core research areas: perception, manipulation, planning, task execution, navigation, and learning. These challenges are addressed by the research literature surveyed here. We surveyed twelve publication venues and include work that tackles computational HRI challenges, categorized into eight topics: (a) perceiving humans and their activities; (b) generating and understanding verbal expression; (c) generating and understanding nonverbal behaviors; (d) modeling, expressing, and understanding emotional states; (e) recognizing and conveying intentional action; (f) collaborating with humans; (g) navigating with and around humans; and (h) learning from humans in a social manner. For each topic, we suggest promising future research areas.

Human-robot collaboration will increasingly take place in human social settings, including contexts where ethical and honest behavior is paramount. How might these robots affect human honesty? In this paper, we present first evidence of how a robot’s presence affects people’s ethical behavior in a controlled field study. We observed people passing by a food plate marked as “reserved”, comparing three conditions: no observer, a human observer, and a robot observer. We found that a human observer elicits less attention than a robot, but evokes more of a socially normative presence causing people to act honestly. Conversely, we found that a robot observer elicits more attention, engagement, and a monitoring presence. But even though people were suspicious that they were being monitored, they still behaved dishonestly in the robot observer condition.

We describe the design process of  “Vyo”, a personal assistant serving as a centralized interface for smart home devices. Building on the concepts of ubiquitous and engaging computing in the domestic environment, we identified five design goals for the home robot: engaging, unobtrusive, device-like, respectful, and reassuring. These goals led our design process, which included simultaneous iterative development of the robot’s morphology, nonverbal behavior and interaction schemas. We continued with user-centered design research using puppet prototypes of the robot to assess and refine our design choices. The resulting robot, Vyo, straddles the boundary between a monitoring device and a socially expressive agent, and presents a number of novel design outcomes: The combination of TUI “phicons” with social robotics; gesture-related screen exposure; and a non-anthropomorphic monocular expressive face. We discuss how our design goals are expressed in the elements of the robot’s final design.

Wizard-of-Oz (WoZ) is a common technique enabling HRI researchers to explore aspects of interaction not yet backed by autonomous systems. A standardized, open, and flexible WoZ framework could therefore serve the community and accelerate research both for the design of robotic systems and for their evaluation.

This paper presents the definition of OpenWoZ, a Wizard-of-Oz framework for HRI, designed to be updated during operation by the researcher controlling the robot. OpenWoZ is implemented as a thin HTTP server running on the robot, and a cloud-backed multi-platform client schema. The WoZ server accepts representational state transfer (REST) requests from a number and variety of clients simultaneously. This “separation of concerns” in OpenWoZ allows addition of commands, new sequencing of behaviors, and adjustment of parameters, all during run-time.

Social presence has two opposing effects on human corruption: the collaborative and contagious nature of another person’s presence can cause people to behave in a more corrupt manner. In contrast, the monitoring nature of another person’s presence can decrease corruption. We hypothesize that a robot’s presence can provide the best of both worlds: Decreasing corruption by providing a monitoring presence, without increasing it by collusion. We describe an experimental study currently underway that examines this hypothesis, and report on initial findings from pilot runs of our experimental protocol.

Responsiveness to one’s bids for proximity in times of need is a linchpin of human interaction. As such, the ability to be perceived as responsive has design implications for socially assistive robotics. We report on a large-scale experimental laboratory study (n=102) examining robot responsiveness and its effects on human attitudes and behaviors. In one-on-one sessions, participants disclosed a personal event to a non-humanoid robot. The robot responded either responsively or unresponsively across two modalities: Simple gestures and written text. We replicated previous findings that the robot’s responsiveness increased perceptions of its appealing traits. In addition, we found that robot responsiveness increased nonverbal approach behaviors (physical proximity, leaning toward the robot, eye contact, smiling) and participants’ willingness to be accompanied by the robot during stressful events. These findings suggest that humans not only utilize responsiveness cues to ascribe social intentions to personal robots, but actually change their behavior towards responsive robots and may want to use such robots as a source of consolation.

We present the design and initial evaluation of Kip3, a social robotic device for students with ADHD that provides immediate feedback for inattention or impulsivity events. We designed a research platform comprised of a tablet-based Continuous Performance Test (CPT) that is used to assess inattention and impulsivity, and a socially expressive robotic device (Kip3) as feedback. We evaluated our platform with 10 students with ADHD in a within subject user study, and report that 9 out of 10 participants felt that Kip3 helped them regain focus, but wondered if it will be effective over time and how it will identify inattention in more complex situations outside the lab.

We propose the notion of Robotic Experience Companionship (REC): a person’s sense of sharing an experience with a robot. Does a robot’s presence and response to a situation affect a human’s understanding of the situation and of the robot, even without direct human-robot interaction? We present the first experimental assessment of REC, studying people’s experience of entertainment media as they share it with a robot. Both studies use an autonomous custom-designed desktop robot capable of performing gestures synchronized to the media. Study I (n=67), examining music listening companionship, finds that the robot’s dance-like response to music causes participants to feel that the robot is co-listening with them, and increases their liking of songs. The robot’s response also increases its perceived human character traits. We find REC to be moderated by music listening habits, such that social listeners were more affected by the robot’s response. Study II (n=91), examining video watching companionship supports these findings, demonstrating that social video viewers enjoy the experience more with the robot present, while habitually solitary viewers do not. Also in line with Study~I, the robot’s response to the a video clip causes people to attribute more positive human character traits to the robot. This has implications for robots as companions for digital media consumption, but also suggests design implications based on REC for other shared experiences with personal robots.

For social robots to respond to humans in an appropriate manner they need to use apt affect displays, revealing underlying emotional intelligence. We present an artificial emotional intelligence system for robots, with both a generative and a perceptual aspect. On the generative side, we explore the expressive capabilities of an abstract, faceless, creature-like robot, with very few degrees of freedom, lacking both facial expressions and the complex humanoid design found often in emotionally expressive robots. We validate our system in a series of experiments: in one study, we find an advantage in classification for animated vs static affect expressions and advantages in valence and arousal estimation and personal preference ratings for both animated vs static and physical vs on-screen expressions. In a second experiment, we show that our parametrically generated expression variables correlate with the intended user affect perception. On the perceptual side, we present a new corpus of sentiment-tagged social media posts for training the robot to perceive affect in natural language. In a third experiment we estimate how well the corpus generalizes to an independent data set through a cross validation using a perceptron and demonstrate that the predictive model is comparable to other sentiment-tagged corpi and classifiers. Combining the perceptual and generative systems, we show in a fourth experiment that our automatically generated affect responses cause participants to show signs of increased engagement and enjoyment compared with arbitrarily chosen comparable motion parameters.

Timing plays a role in a range of human-robot interaction scenarios, as humans are highly sensitive to timing and interaction fluency. It is central to spoken dialogue, with turn-taking, interruptions, and hesitation influencing both task efficiency and user affect. Timing is also an important factor in the interpretation and generation of gestures, gaze, facial expressions, and other nonverbal behavior. Beyond communication, temporal synchronization is functionally necessary for sharing resources and physical space, as well as coordinating multi-agent actions. Timing is thus crucial to the success of a broad spectrum of HRI applications, including but not limited to situated dialogue; collaborative manipulation; performance, musical, and entertainment robots; and expressive robot companions. Recent years have seen a growing interest in the HRI community in the various research topics related to human-robot timing. The purpose of this workshop is to explore and discuss theories, computational models, systems, empirical studies, and interdisciplinary insights related to the notion of timing, fluency, and rhythm in human-robot interaction.

Please refer to the new journal version of this paper.

Collaborative fluency is the coordinated meshing of joint activities between members of a well-synchronized team. We aim to build robotic team members that can work side-by-side humans by displaying the kind of fluency that humans are accustomed to from each other. As part of this effort, we have developed a number of metrics to evaluate the level of fluency in human-robot shared-location teamwork. In this paper we discuss issues in measuring fluency, present both subjective and objective metrics that have been used to measure fluency between a human and robot, and report on findings along the proposed metrics.

We evaluate the effects of robotic listening companionship on people’s enjoyment of music, and on their perception of the robot. We present a robotic speaker device designed for joint listening and embodied performance of the music played on it. The robot generates smoothed real-time beat-synchronized dance moves, uses nonverbal gestures for common ground, and can make and maintain eye-contact.

In an experimental between-subject study (n=67), participants listened to songs played on the speaker device, with the robot either moving in sync with the beat, moving off-beat, or not moving at all. We found that while the robot’s beat precision was not consciously detected by Ps, an on-beat robot positively affected song liking. There was no effect on overall experience enjoyment. In addition, the robot’s response caused Ps to attribute more positive human-like traits to the robot, as well as rate the robot as more similar to themselves. Notably, personal listening habits (solitary vs. social) affected agent attributions.

This work points to a larger question, namely how a robot’s perceived response to an event might affect a human’s perception of the same event.

Music listening is a central activity in human culture, and throughout history the introduction of new audio reproduction technologies have influenced the way music is consumed and perceived.

In this work, we discuss a robotic speaker, designed to behave as both a reproduction device and as a music listening companion. The robot is intended to enhance a human’s listening experience by providing social presence and embodied musical performance. In a sample application, it generates segment-specific, beat-synchronized gestures based on the song’s genre, and maintains eye-contact with the user.

We describe an experimental human-subject study (n=67), evaluating the effect of the robot’s behavior on people’s enjoyment of the songs played, as well as on their sense of the robot’s social presence and their impression of the robot as an autonomous agent.

Combining high-performance, sensor-rich mobile devices with simple, low-cost robotic platforms could accelerate the adoption of personal robotics in real-world environments. We present a case study of this “dumb robot, smart phone” paradigm: a robotic speaker dock and music listening companion. The robot is designed to enhance a human’s listening experience by providing social presence and embodied musical performance. In its initial application, it generates segment-specific, beat-synchronized gestures based on the song’s genre, and maintains eye-contact with the user. All of the robot’s computation, sensing, and high-level motion control is performed on a smartphone, with the rest of the robot’s parts handling mechanics and actuator bridging.

This paper suggests to turn to the performative arts for insights that may help the fluent coordination and joint-action timing of human-robot interaction (HRI). We argue that theater acting and musical performance robotics could serve as useful testbeds for the development and evaluation of action coordination in robotics. We also offer two insights from theater acting literature for HRI: the maintenance of continuous sub-surface processes that manifest in motor action, and an emphasis on fast, inaccurate responsiveness using partial information and priming in action selection.

Shimon is a interactive robotic marimba player, developed as part of our ongoing research in Robotic Musicianship. The robot listens to a human musician and continuously adapts its improvisation and choreography, while playing simultaneously with the human. We discuss the robot’s mechanism and motion-control, which uses physics simulation and animation principles to achieve both expressivity and safety. We then present an interactive improvisation system based on the notion of physical gestures for both musical and visual expression. The system also uses anticipatory action to enable real-time improvised synchronization with the human player.

We describe a study evaluating the effect of embodiment on one of our improvisation modules: antiphony, a call-and-response musical synchronization task. We conducted a 3×2 within-subject study manipulating the level of embodiment, and the accuracy of the robot’s response. Our findings indicate that synchronization is aided by visual contact when uncertainty is high, but that pianists can resort to internal rhythmic coordination in more predictable settings. We find that visual coordination is more effective for synchronization in slow sequences; and that occluded physical presence may be less effective than audio-only note generation.

Finally, we test the effects of visual contact and embodiment on audience appreciation. We find that visual contact in joint Jazz improvisation makes for a performance in which audiences rate the robot as playing better, more like a human, as more responsive, and as more inspired by the human. They also rate the duo as better synchronized, more coherent, communicating, and coordinated; and the human as more inspired and more responsive.

Shimon is a interactive robotic marimba player, developed as part of our ongoing research in Robotic Musicianship (RM). One of the potential benefits of RM is that it provides human players with embodied information that relates spatial movement to tone generation. This can aid in anticipation and coordination of synchronous playing.

As part of a human-robot Jazz improvisation system, we present an anticipatory system enabling beat-matched real-time synchronization. Our system enables flexible, yet coordinated call-and-response, a standard type of musical interaction. It was used in a live public human-robot joint Jazz performance.

We also describe a preliminary study evaluating the effect of embodiment on this call-and-response musical synchronization task. We conducted a 3×2 within-subject study manipulating the level of embodiment (visual co-presence, physical presence but visual occlusion, and synthesized sound), and the accuracy of the robot’s response.

Our findings indicate that synchronization is aided by visual contact when uncertainty is high, but that pianists can resort to internal rhythmic coordination in more predictable settings. We find that visual coordination is more effective for synchronization for slow sequences compared to faster sequences; and that occluded physical presence may be less effective than audio-only note generation.

Shimon is an autonomous marimba-playing robot designed to create interactions with human players that lead to novel musical outcomes. The robot combines music perception, interaction, and improvisation with the capacity to produce melodic and harmonic acoustic responses through choreographic gestures. We developed an anticipatory action framework, and a gesture-based behavior system, allowing the robot to play improvised Jazz with humans in synchrony, fluently, and without delay. In addition, we built an expressive non-humanoid head for musical social communication. This paper describes our system, used in a performance and demonstration at the CHI 2010 Media Showcase.

With the aim of attaining increased fluency and efficiency in human-robot teams, we have developed a cognitive architecture for robotic teammates based on the neuro-psychological principles of anticipation and perceptual simulation through top-down biasing. An instantiation of this architecture was implemented on a non-anthropomorphic robotic lamp, performing a repetitive human-robot collaborative task.

In a human-subject study in which the robot works on a joint task with untrained subjects, we find our approach to be significantly more efficient and fluent than in a comparable system without anticipatory perceptual simulation. We also show the robot and the human to improve their relative contribution at a similar rate, possibly playing a part in the human’s “like-me” perception of the robot.

In self-report, we find significant differences between the two conditions in the sense of team fluency, the team’s improvement over time, the robot’s contribution to the efficiency and fluency, the robot’s intelligence, and in the robot’s adaptation to the task. We also find differences in verbal attitudes towards the robot: most notably, subjects working with the anticipatory robot attribute more human qualities to the robot, such as gender and intelligence, as well as credit for success, but we also find increased self-blame and self-deprecation in these subjects’ responses.

We believe that this work lays the foundation towards modeling and evaluating artificial practice for robots working in collaboration with humans.

This paper describes a robotic puppeteering system used in a theatrical production involving one robot and two human performers on stage. We draw from acting theory and human-robot interaction to develop a hybrid-control puppeteering interface which combines reactive expressive gestures and parametric behaviors with a point-of-view eye contact module. Our design addresses two core considerations: allowing a single operator to puppeteer the robot’s full range of behaviors, and allowing for gradual replacement of human-controlled modules by autonomous subsystems.

We wrote a play specifically for a performance between two humans and one of our research robots, a robotic lamp which embodied a lead role in the play. We staged three performances with the robot as part of a local festival of new plays. Though we have yet to perform a formal statistical evaluation of the system, we interviewed the actors and director and present their feedback about working with the system.

We have developed a cognitive architecture for a robotic teammate based on the neuro-psychological principles of anticipation and perceptual simulation, with the aim of attaining increased fluency and efficiency in human-robot teams. An instantiation of this architecture was implemented on a non-anthropomorphic robotic lamp, performing in a human-robot collaborative task.

In a human-subject study in which the robot works on a joint task with untrained subjects, we find our approach to be significantly more efficient and fluent than in a comparable system without anticipatory perceptual simulation. We also show the robot and the human to be increasingly contributing at a similar rate, and we find significant differences in a number of self-report measures and verbal attitudes towards the robot. Notably, we identify increased self-deprecation in human subject responses vis-à-vis the anticipatory robot.

We have developed a cognitive architecture for robotic teammates based on the neuro-psychological principles of perceptual symbols and simulation, with the aim of attaining increased fluency in human-robot teams. An instantiation of this architecture was implemented on a robotic desk lamp, performing in a human-robot collaborative task. This paper describes initial results from a human-subject study measuring team efficiency and team fluency, in which the robot works on a joint task with untrained subjects. We find significant differences in a number of efficiency and fluency metrics, when comparing our architecture to a purely reactive robot with similar capabilities.

A crucial skill for fluent action meshing in human team activity is a learned and calculated selection of anticipatory actions. We believe that the same holds for robotic teammates, if they are to perform in a similarly fluent manner with their human counterparts. In this work, we propose an adaptive action selection mechanism for a robotic teammate, making anticipatory decisions based on the confidence of their validity and their relative risk. We predict an improvement in task efficiency and fluency compared to a purely reactive process.

We then present results from a study involving untrained human subjects working with a simulated version of a robot using our system. We show a significant improvement in best-case task efficiency when compared to a group of users working with a reactive agent, as well as a significant difference in the perceived commitment of the robot to the team and its contribution to the team’s fluency and success. By way of explanation, we propose a number of fluency metrics that differ significantly between the two study groups.

Time Bracketing is a novel technique to photographically depict a time-varying space or object in a single image without spatial distortion or fragmentation. The approach attempts to strike a balance between procedural generation and manual composition as well as between faithful depiction and digital manipulation. It also enables the clear representation of both dynamic and static scene elements. As a result, Time Bracketing puts the subject, rather than the technique, in the center of its artistic creation. This paper introduces the method, presents a custom-built authoring software for the creation of Time Bracketing images, and shows and architectural studies using the described technique and software.

While Reinforcement Learning (RL) is not traditionally designed for interactive supervisory input from a human teacher, several works in both robot and software agents have adapted it for human input by letting a human trainer control the reward signal. In this work, we experimentally examine the assumption underlying these works, namely that the human-given reward is compatible with the traditional RL reward signal. We describe an experimental platform with a simulated RL robot and present an analysis of real-time human teaching behavior found in a study in which untrained subjects taught the robot to perform a new task.

We report three main observations on how people administer feedback when teaching a robot a task through Reinforcement Learning: (a) they use the reward channel not only for feedback, but also for future-directed guidance; (b) they have a positive bias to their feedback — possibly using the signal as a motivational channel; and (c) they change their behavior as they develop a mental model of the robotic learner. In conclusion, we discuss future extensions to RL to accommodate these lessons.

Theater actors have been staging artificial intelligence for centuries. If one shares the view that intelligence manifests in behavior, one must wonder what lessons the AI community can draw from a practice that is historically concerned with the infusion of artificial behavior into such vessels as body and text. Like researchers in AI, actors construct minds by systematic investigation of intentions, actions, and motor processes with the proclaimed goal of artificially recreating human-like behavior. Therefore, acting methodology may hold valuable directives for designers of artificially intelligent systems. Indeed, a review of acting method literature reveals a number of insights that may be of interest to the AI community.

A mounting body of evidence in psychology and neuroscience points towards an embodied model of cognition, in which the mechanisms governing perception and action are strongly interconnected, and also play a central role in higher cognitive functions, traditionally modeled as amodal symbol systems.

We argue that robots designed to interact fluidly with humans must adopt a similar approach, and shed traditional distinctions between cognition, perception, and action. In particular, embodiment is crucial to fluid joint action, in which the robot’s performance must tightly integrate with that of a human counterpart, taking advantage of rapid sub-cognitive processes.

We thus propose a model for embodied robotic cognition that is built upon three propositions: (a) modal, perceptual models of knowledge; (b) integration of perception and action; (c) top-down bias in perceptual processing. We then discuss implications and derivatives of our approach.

We aim to build robots that go beyond command-and-response and can engage in fluent collaborative behavior with their human counterparts. This paper discusses one aspect of collaboration fluency: expectation management -predicting what a human collaborator will do next and how to act on that prediction. We propose a formal time-based collaborative framework that can be used to evaluate this and other aspects of collocated human-robot teamwork, and show how expectation management can enable a higher level of fluency and improved efficiency in this framework. We also present an implementation of the proposed theoretical framework in a simulated human-robot collaborative task.

In Socially Guided Machine Learning we explore the ways in which machine learning can more fully take advantage of natural human interaction. In this paper we are studying the role real-time human interaction plays in training assistive robots to perform new tasks. We describe an experimental platform, Sophie’s World, and present descriptive analysis of human teaching behavior found in a user study. We report three important observations of how people administer reward and punishment to teach a simulated robot a new task through Reinforcement Learning. People adjust their behavior as they develop a model of the learner, they use the reward channel for guidance as well as feedback, and they may also use it as a motivational channel.

Many new applications for robots require them to work alongside people as capable members of human-robot teams. These include—in the long term—robots for homes, hospitals, and offices, but already exist in more advanced settings, such as space exploration. The work reported in this paper is part of an ongoing collaboration with NASA JSC to develop Robonaut, a humanoid robot envisioned to work with human astronauts on maintenance operations for space missions. To date, work with Robonaut has mainly investigated performing a joint task with a human in which the robot is being teleoperated. However, perceptive disorientation, sensory noise, and control delays make teleoperation cognitively exhausting even for a highly skilled operator. Control delays in long range teleoperation also make shoulder-to-shoulder teamwork difficult. These issues motivate our work to make robots collaborating with people more autonomous.

Our work focuses on a scenario of a human and an autonomous humanoid robot working together shoulder-to-shoulder, sharing the workspace and the objects required to complete a task. A robotic member of such a team must be able to work towards a shared goal, and be in agreement with the human as to the sequence of actions that will be required to reach that goal, as well as dynamically adjust its plan according to the human’s actions. Human-robot collaboration of this nature is an important yet relatively unexplored kind of human-robot interaction.

This paper describes our work towards building a dynamic collaborative framework enabling such an interaction. We discuss our architecture and its implementation for controlling a humanoid robot, working on a task with a human partner. Our approach stems from Joint Intention Theory, which shows that for joint action to emerge, teammates must communicate to maintain a set of shared beliefs and to coordinate their actions towards the shared plan. In addition, they must demonstrate commitment to doing their own part, to the others doing theirs, to providing mutual support, and finally—to a mutual belief as to the state of the task.

We argue that to this end, the concept of task and action goals is central. We therefore present a goal-driven hierarchical task representation, and a resulting collaborative turn-taking system, implementing many of the above-mentioned requirements of a robotic teammate. Additionally, we show the implementation of relevant social skills supporting our collaborative framework.

Finally, we present a demonstration of our system for collaborative execution of a hierarchical object manipulation task by a robot-human team. Our humanoid robot is able to divide the task between the participants while taking into consideration the collaborator’s actions when deciding what to do next. It is capable of asking for mutual support in the cases where it is unable to perform a certain action. To facilitate this interaction, the robot actively maintains a clear and intuitive channel of communication to synchronize goals, task states, and actions, resulting in a fluid, efficient collaboration.