You are here: Home / Communications / Our Publications / Faculty Publications / Evaluating User Preferences in Machine Translation Using Conjoint Analysis

Evaluating User Preferences in Machine Translation Using Conjoint Analysis

K Kirchhoff, D Capurro, and A Turner (2012)
Proceedings of the European Association of Machine Translation, 16:119-126.

In spite of much ongoing research on machine translation evaluation there is little quantitative work that directly measures users' intuitive or emotional preferences regarding different types of machine translation errors. However, the elicitation andmodeling of user preferences is an important prerequisite for future research on user adaptation and customization of machine translation engines. In this paper we explore the use of conjoint analysis as a formal quantitative framework to gain insight into users' relative preferences for different translation error types. Using English-Spanish as the translation direction we conduct a crowd-sourced conjoint analysis study and obtain utility values for individual error types. Our results indicate that word order errors are clearly the most dispreferred error type, followed by word sense, morphological, and function word errors.