Question order bias revisited: A split-ballot experiment on satisfaction with public services among experienced and professional users

Public decision-makers often rely on user satisfaction surveys as performance indicators motivating budget and policy decisions. Yet, little is known about how question order may bias this input from public service users. Using a split-ballot experiment, we test if recent findings can be replicated in a least-likely case: experienced and professional government grant recipients in Denmark. We find that asking about overall satisfaction before any specific service ratings lowers reported user satisfaction, compared to the reverse order, while the correlations between specific ratings and overall satisfaction seem relatively stable. We also find that the question order effect outweighs that of a large-scale embezzlement scandal, which unexpectedly hit the government agency that our survey concerns during the data collection period. Our results support rising concerns that satisfaction measures are susceptible to bias, and suggest that practitioners and researchers use satisfaction surveys with particular care.