Programming languages and tools exist to enable software developers to program. How effectively they do so ultimately depends on the interaction between languages and tools and the developers who use them. As new language features and tools are designed, a fundamental and inherently empirical question raised is, do they help programmers work better. Answering such questions requires conducting studies with human participants. This tutorial will provide a broad overview of methods for evaluating programming languages and tools in studies with human participants. The tutorial will be aimed at SPLASH attendees that have never before conducted a human subjects study, helping introduce attendees to the basics of designing and con- ducting studies and methods for the analysis of data. Elements of a study design will be surveyed, including recruitment and selection of human participants, informed consent, experimental procedures, demographic measurements, group assignment, training, the selection and design of tasks, the measurement of common outcome variables, and study debriefing. Broader elements of the research process will also be surveyed, including finding and refining research questions for studies, techniques and models for analysing empirical data from human participants, and finding the right balance between quantitative and qualitative methods.

Thomas LaToza is an Assistant Professor of Computer Science at George Mason University. He has degrees in psychology and computer science from the University of Illinois and a PhD in software engineering from Carnegie Mellon University. His research is in the area of human aspects of software development, encompassing empirical and design work on environments for programming, software design, and collaboration. He has been active in bringing human subjects studies to the investigation of software development activity and the evaluation of software development tools and has conducted over 20 studies with software developers, including observational studies, surveys, interviews, field deployments, and controlled experiments. He has served on various program committees and is currently the co-chair of the Sixth Workshop on the Evaluation and Usability of Programming Languages and Tools.