TDWEB - TDW Web Interface for Interaction Collection

Collaborative tasks often begin with partial task knowledge and thus incomplete plans. To complete these tasks, humans need to engage in situated communication with their partners and coordinate towards a complete plan with a common goal. Such collaboration requires participants to make educated guesses of their and their partners' missing knowledge. This seemingly effortless collaboration paradigm in a human-human team could be extremely challenging for human-AI collaboration, especially when dialogue understanding and Theory of Mind modeling are lacking.

I’m reponsible for building the simulated environment for human-robot interaction using ThreeDWorld (TDW). On the Web Page:

• Players can control the robot from the first perspective
• Click objects listed to pick the item and place in the bowl
• Chat with partner for collaborative tasks

By collect the collaboration data from the interaction experiments and conducting experiment on models trained w/wo Theory of Mind in the web simulator above, we expect that models augmenting with Theory of Mind will produce improved and more stable results than without in predicting the partner’s missing knowledge. We will also evaluate the influence of individual dialogue moves on performance. We hope that if an agent can predict what kind of knowledge their partner is missing they then can proactively communicate such information to help the partner acquire such missing knowledge towards common understanding of joint tasks.