Evaluating multi-agent conversational interfaces in the early stages of the design process

Abstract

In this paper we describe a mixed-approach technique to understand user’s perceptions of concepts in the early stage of the design process. We designed an evaluation study to understand desirability of a multi-agent cognitive investment advisor, a Chabot. The study was threefold. First participants watched the video, then chose reaction card adjectives to report their perceptions, and lastly gave their opinions guided by questions about the multi-party dialogue. From this experiment, we gather positive and negative reactions from users that helped to shape the user experience of cognitive investment advisors.