In this article we focus on human–human multi-tasking dialogues, in which pairs of conversants, using speech, work on an ongoing task while occasionally completing real-time tasks. The ongoing task is a poker game in which conversants need to assemble a poker hand, and the real-time task is a picture game in which conversants need to find out whether they have a certain picture on their displays. We employ empirical corpus studies and machine learning experiments to understand the mechanisms that people use in managing these complex interactions. First, we examine task interruptions: switching from the ongoing task to a real-time task. We find that generally conversants tend to interrupt at a less disruptive context in the ongoing task when possible. We also find that the discourse markers oh and wait occur in initiating a task interruption twice as often as in the conversation of the ongoing task. Pitch is also found to be statistically correlated with task interruptions; in fact, the more disruptive the task interruption, the higher the pitch. Second, we examine task resumptions: returning to the ongoing task after completing an interrupting real-time task. We find that conversants might simply resume the conversation where they left off, but sometimes they repeat the last utterance or summarize the critical information that was exchanged before the interruption. Third, we apply machine learning to determine how well task interruptions can be recognized automatically and to investigate the usefulness of the cues that we find in the corpus studies. We find that discourse context, pitch, and the discourse markers oh and wait are important features to reliably recognize task interruptions; and with non-lexical features one can improve the performance of recognizing task interruptions with more than a 50% relative error reduction over a baseline. Finally, we discuss the implication of our findings for building a speech interface that supports multi-tasking dialogue.
