On the Usability of Spoken Dialogue Systems

Abstract

 This work is centred on the methods and problems associated with defining and measuring the usability of Spoken Dialogue Systems (SDS). The starting point is the fact that speech based interfaces has several times during the last 20 years fallen short of the high expectations and predictions held by industry, researchers and analysts. Several studies in the literature of SDS indicate that this can be ascribed to a lack of attention from the speech technology community towards the usability of such systems. The experimental results presented in this work are based on a field trial with the OVID home banking spoken dialogue system. It comprises more than 700 transcribed dialogues by 310 users. A number of objective (performance) measures are derived from the corpus. The system’s learnability is analysed through the turn-taking strategies and it is shown that users are capable of taking the initiative at predicted points in the dialogue. It is demonstrated that more experienced users achieve a higher task completion rate and reduced duration of the dialogue due to user habituation. Subjective (user attitude) measures are analysed and shown to be reliable and valid. Factor analysis is applied to validate the data, and results comparable to those obtained by other researchers are demonstrated. A correspondence between the objective and subjective measures is established by the application of Multivariate Linear Regression (MLR) as used in the PARADISE scheme proposed by researchers from AT&T. The resulting model roughly explains 50% of the observed variance in the user satisfaction based on measures of task success and speech recognition accuracy, a result similar to those obtained at AT&T. The applied methods are discussed and evaluated critically. This work is centred on the methods and problems associated with defining and measuring the usability of Spoken Dialogue Systems (SDS). The starting point is the fact that speech based interfaces has several times during the last 20 years fallen short of the high expectations and predictions held by industry, researchers and analysts. Several studies in the literature of SDS indicate that this can be ascribed to a lack of attention from the speech technology community towards the usability of such systems. The experimental results presented in this work are based on a field trial with the OVID home banking spoken dialogue system. It comprises more than 700 transcribed dialogues by 310 users. A number of objective (performance) measures are derived from the corpus. The system’s learnability is analysed through the turn-taking strategies and it is shown that users are capable of taking the initiative at predicted points in the dialogue. It is demonstrated that more experienced users achieve a higher task completion rate and reduced duration of the dialogue due to user habituation. Subjective (user attitude) measures are analysed and shown to be reliable and valid. Factor analysis is applied to validate the data, and results comparable to those obtained by other researchers are demonstrated. A correspondence between the objective and subjective measures is established by the application of Multivariate Linear Regression (MLR) as used in the PARADISE scheme proposed by researchers from AT&T. The resulting model roughly explains 50% of the observed variance in the user satisfaction based on measures of task success and speech recognition accuracy, a result similar to those obtained at AT&T. The applied methods are discussed and evaluated critically