Nikola Banovic

Portrait of Nikola Banovic
Photo: Joseph Xu/Michigan Engineering Communications & Marketing

Assistant Professor, Electrical Engineering & Computer Science
Director, Computational HCI Lab

University of Michigan, Ann Arbor

Bob and Betty Beyster Building, Room 2630
2260 Hayward
Ann Arbor, MI 48109, USA

Twitter: @nikola_banovic

Curriculum Vitae (CV)


My research focuses on understanding and modeling human behavior to support innovative information technology that will change how we study and design interactive user experiences. I envision modeling the human accurately across domains as a theoretical foundation for work in Human-Computer Interaction (HCI) in which computational models provide a foundation to study, describe, and understand complex human behaviors and support optimization and evaluation of user interfaces.

I work in domains of Computational Interaction, Behavior Data Analytics, and Human-Data Supported Interfaces to enable a future in which technology will automatically infer user goals, predict future user actions, describe common user behaviors, and even coach users. I create technology that automatically reasons about and acts in response to people’s behavior to help them be productive, healthy, and safe.

Before joining the University of Michigan, I received my Ph.D. degree from the Human-Computer Interaction Institute (HCII) at Carnegie Mellon University, and my B.Sc. and M.Sc. degrees from the University of Toronto.

For more information, see my Curriculum Vitae (CV).


Fall 2018

EECS 598 - Computational Modeling in Human-Computer Interaction (3 credits)
This seminar course will review current computational approaches to describe, simulate, and predict human behavior from empirical behavior traces data. It will contrast computational modeling with other methodologies to understand human behavior and compare computational modeling with existing behavior modeling methodologies in Human-Computer Interaction (HCI). Short assignments will give students exposure to some of the cutting-edge methods, while the final project will give them an opportunity to push the boundaries of computational modeling in HCI by modeling behaviors of their choice from an existing data set. Prerequisites: Programming experience in Java, Python, MATLAB or R.

PUBLICATIONS (Google Scholar)


In Only 3 Minutes: Perceived Exertion Limits of Smartwatch Use
Rushil Khurana, Nikola Banovic, and Kent Lyons
(To appear) In Proceedings of the 2018 ACM International Symposium on Wearable Computers (ISWC '18). ACM, New York, NY, USA.
Glanceability and low access time are arguably the key assets of a smartwatch. Smartwatches are designed for, and excel at micro-interactions--simple tasks that only take seconds to complete. However, if a user desires to transition to a task requiring sustained usage, we show that there are additional factors that prevent possible longer usage of the smartwatch. In this paper, we conduct a study with 18 participants to empirically demonstrate that interacting with the smartwatch on the wrist leads to fatigue after only a few minutes. In our study, users performed three tasks in two different poses while using a smartwatch. We demonstrate that only after three minutes of use, the change in perceived exertion of the user was anchored as “somewhat strong” on the Borg CR10 survey scale. These results place an upper bound for smartwatch usage that needs to be considered in application and interaction design.
Mapping Machine Learning Advances From HCI Research to Reveal Starting Places for Design Innovation
Qian Yang, Nikola Banovic, and John Zimmerman
In Proceedings of the 2018 CHI conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Paper 130, 11 pages.
HCI has become particularly interested in using machine learning (ML) to improve user experience (UX). However, some design researchers claim that there is a lack of design innovation in envisioning how ML might improve UX. We investigate this claim by analyzing 2,494 related HCI research publications. Our review confirmed a lack of research integrating UX and ML. To help span this gap, we mined our corpus to generate a topic landscape, mapping out 7 clusters of ML technical capabilities within HCI. Among them, we identified 3 under-explored clusters that design researchers can dig in and create sensitizing concepts for. To help operationalize these technical design materials, our analysis then identified value channels through which the technical capabilities can provide value for users: self, context, optimal, and utility-capability. The clusters and the value channels collectively mark starting places for envisioning new ways for ML technology to improve people’s lives.
Computational Model of Human Routine Behaviours
Nikola Banovic, Jennifer Mankoff, and Anind K. Dey
In Computational Interaction, Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes (Eds.). Oxford University Press. 22 pages.


Warming Up to Cold Start Personalization
Nikola Banovic and John Krumm
PACM Interactive, Mobile, Wearable and Ubiquitous Technology (IMWUT), Volume 1, Issue 4, Article 124 (December 2017). ACM, New York, NY, USA, 13 pages.
Smart agents face abandonment if they are unable to provide value to the users from the very first interaction. Existing smart agents take time to learn about new users before they can offer them personalized services. We present a method for learning personalization information about users quickly and without placing unnecessary hardship on them. Our method enables smart agents to pick which questions to ask the user when they first interact to maximize the agent’s overall knowledge about the user. We demonstrate our method on two publically available US census datasets containing 172 user variables from 1,799,394 training and 1,618,489 testing users. The questions selected using our method improve the agent’s accuracy when inferring information about future users, including information that they did not ask about. Our work enables smart agents that assist the user with personalized services soon after they start interacting.
Leveraging Human Routine Models to Detect and Generate Human Behaviors
Nikola Banovic, Anqi Wang, Yanfeng Jin, Christie Chang, Julian Ramos, Anind K. Dey, and Jennifer Mankoff
In Proceedings of the 2017 CHI conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6683-6694.
An ability to detect behaviors that negatively impact people’s wellbeing and show people how they can correct those behaviors could enable technology that improves people’s lives. Existing supervised machine learning approaches to detect and generate such behaviors require lengthy and expensive data labeling by domain experts. In this work, we focus on the domain of routine behaviors, where we model routines as a series of frequent actions that people perform in specific situations. We present an approach that bypasses labeling each behavior instance that a person exhibits. Instead, we weakly label instances using people’s demonstrated routine. We classify and generate new instances based on the probability that they belong to the routine model. We illustrate our approach on an example system that helps drivers become aware of and understand their aggressive driving behaviors. Our work enables technology that can trigger interventions and help people reflect on their behaviors when those behaviors are likely to negatively impact them.
Quantifying Aversion to Costly Typing Errors in Expert Mobile Text Entry
Nikola Banovic, Varun Rao, Abinaya Saravanan, Anind K. Dey, and Jennifer Mankoff
In Proceedings of the 2017 CHI conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4229-4241.
Text entry is an increasingly important activity for mobile device users. As a result, increasing text entry speed of expert typists is an important design goal for physical and soft keyboards. Mathematical models that predict text entry speed can help with keyboard design and optimization. Making typing errors when entering text is inevitable. However, current models do not consider how typists themselves reduce the risk of making typing errors (and lower error frequency) by typing more slowly. We demonstrate that users respond to costly typing errors by reducing their typing speed to minimize typing errors. We present a model that estimates the effects of risk aversion to errors on typing speed. We estimate the magnitude of this speed change, and show that disregarding the adjustments to typing speed that expert typists use to reduce typing errors leads to overly optimistic estimates of maximum errorless expert typing speeds.
Method for Understanding Complex Human Routine Behaviors from Large Behavior Logs
Nikola Banovic
In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 254-258.
The increasing ability to collect large amounts of human behavior data can inform technology that has the potential to help people improve their behaviors and thus improve the quality of their lives. To design and implement such technology requires understanding of those very behaviors that the technology is trying to diagnose and improve. However, existing methods to explore and make sense of human behaviors are not well suited to address the increasingly large amount of data collected in behavior logs. My research focuses on the domain of human routines where I model behaviors as sequences of actions people perform in specific situations. I leverage those computational models of routines together with different visualization tools to aid researchers and domain experts in exploring, making sense of, and generating new insights about human behavior in a principled way. My research informs the design of technology that helps people be productive, healthy, and safe.


Modeling and Understanding Human Routine Behavior
Nikola Banovic, Tofi Buzali, Fanny Chevalier, Jennifer Mankoff, and Anind K. Dey
In Proceedings of the 2016 CHI conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 248-260.
Human routines are blueprints of behavior, which allow people to accomplish purposeful repetitive tasks at many levels, ranging from the structure of their day to how they drive through an intersection. People express their routines through actions that they perform in the particular situations that triggered those actions. An ability to model routines and understand the situations in which they are likely to occur could allow technology to help people improve their bad habits, inexpert behavior, and other suboptimal routines. However, existing routine models do not capture the causal relationships between situations and actions that describe routines. Our main contribution is the insight that byproducts of an existing activity prediction algorithm can be used to model those causal relationships in routines. We apply this algorithm on two example datasets, and show that the modeled routines are meaningful—that they are predictive of people’s actions and that the modeled causal relationships provide insights about the routines that match findings from previous research. Our approach offers a generalizable solution to model and reason about routines.
To Replicate or Not to Replicate?
Nikola Banovic
GetMobile: Mobile Computing and Communications 19, 4 (March 2016), 23-27.
To replicate or not to replicate experiments and studies seems to be a burning question in the Human-Computer Interaction (HCI) community. This question is equally relevant to the field of mobile computing, where researchers face unique challenges when attempting to replicate studies of mobile device usage in the field. Some of those challenges are inherent in conducting such studies to understand complex socio-technical systems, which involve different technologies and diverse populations of users from different social, economic, and cultural backgrounds. However, those same challenges point to a great opportunity to expand our knowledge of how people use and interact with their mobile devices and understand how those interactions evolve over time.


Understanding the challenges of mobile phone usage data
Karen Church, Denzil Ferreira, Nikola Banovic, Kent Lyons
In Proceedings of the 17th international conference on Human-computer interaction with mobile devices and services (MobileHCI '15). ACM, New York, NY, USA, 504-514.
Driven by curiosity and our own three diverse smartphone application usage datasets, we sought to unpack the nuances of mobile device use by revisiting two recent Mobile HCI studies. Our goal was to add to our broader understanding of smartphone usage by investigating if differences in mobile device usage occurred not only across our three datasets, but also in relation to prior work. We found differences in the top-10 apps in each dataset, in the durations and types of interactions as well as in micro-usage patterns. However, it proved very challenging to attribute such differences to a specific factor or set of factors: was it the time frame in which the studies were executed? The recruitment procedure? The experimental method? Using our somewhat troubled analysis, we discuss the challenges and issues of conducting mobile research of this nature and reflect on caveats related to the replicability and generalizability of such work.


ProactiveTasks: the short of mobile device use sessions
Nikola Banovic, Christina Brant, Jennifer Mankoff, and Anind K. Dey
In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI '14). ACM, New York, NY, USA, 243-252.
Mobile devices have become powerful ultra-portable personal computers supporting not only communication but also running a variety of complex, interactive applications. Because of the unique characteristics of mobile interaction, a better understanding of the time duration and context of mobile device uses could help to improve and streamline the user experience. In this paper, we first explore the anatomy of mobile device use and propose a classification of use based on duration and interaction type: glance, review, and engage. We then focus our investigation on short review interactions and identify opportunities for streamlining these mobile device uses through proactively suggesting short tasks to the user that go beyond simple application notifications. We evaluate the concept through a user evaluation of an interactive lock screen prototype, called ProactiveTasks. We use the findings from our study to create and explore the design space for proactively presenting tasks to the users. Our findings underline the need for a more nuanced set of interactions that support short mobile device uses, in particular review sessions.
Indoor-ALPS: an adaptive indoor location prediction system
Christian Koehler, Nikola Banovic, Ian Oakley, Jennifer Mankoff, and Anind K. Dey
In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). ACM, New York, NY, USA, 171-181.
Location prediction enables us to use a person's mobility history to realize various applications such as efficient temperature control, opportunistic meeting support, and automated receptionists. Indoor location prediction is a challenging problem, particularly due to a high density of possible locations and short transition distances between these locations. In this paper we present Indoor-ALPS, an Adaptive Indoor Location Prediction System that uses temporal-spatial features to create individual daily models for the prediction of when a user will leave their current location (transition time) and the next location she will transition to. We tested Indoor-ALPS on the Augsburg Indoor Location Tracking Benchmark and compared our approach to the best performing temporal-spatial mobility prediction algorithm, Prediction by Partial Match (PPM). Our results show that Indoor-ALPS improves the temporal-spatial prediction accuracy over PPM for look-aheads up to 90 minutes by 6.2%, and for up to 30 minute look-aheads by 10.7%. These results demonstrate that Indoor-ALPS can be used to support a wide variety of indoor mobility prediction-based applications.


Uncovering information needs for independent spatial learning for users who are visually impaired
Nikola Banovic, Rachel L. Franz, Khai N. Truong, Jennifer Mankoff, and Anind K. Dey
In Proceedings of the 15th international ACM SIGACCESS conference on Computers and accessibility (ASSETS '13). ACM, New York, NY, USA, Article 24, 8 pages.
Sighted individuals often develop significant knowledge about their environment through what they can visually observe. In contrast, individuals who are visually impaired mostly acquire such knowledge about their environment through information that is explicitly related to them. This paper examines the practices that visually impaired individuals use to learn about their environments and the associated challenges. In the first of our two studies, we uncover four types of information needed to master and navigate the environment. We detail how individuals’ context impacts their ability to learn this information, and outline requirements for independent spatial learning. In a second study, we explore how individuals learn about places and activities in their environment. Our findings show that users not only learn information to satisfy their immediate needs, but also to enable future opportunities – something existing technologies do not fully support. From these findings, we discuss future research and design opportunities to assist the visually impaired in independent spatial learning.
The effect of time-based cost of error in target-directed pointing tasks
Nikola Banovic, Tovi Grossman, and George Fitzmaurice
In Proceedings of the 2013 ACM annual conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1373-1382.
One of the fundamental operations in today’s user interfaces is pointing to targets, such as menus, buttons, and text. Making an error when selecting those targets in real-life user interfaces often results in some cost to the user. However, the existing target-directed pointing models do not consider the cost of error when predicting task completion time. In this paper, we present a model based on expected value theory that predicts the impact of the error cost on the user’s completion time for target-directed pointing tasks. We then present a target-directed pointing user study, which results show that time-based costs of error significantly impact the user’s performance. Our results also show that users perform according to an expected completion time utility function and that optimal performance computed using our model gives good prediction of the observed task completion times.
Escape-Keyboard: a sight-free one-handed text entry method for mobile touch-screen devices
Nikola Banovic, Koji Yatani, and Khai N. Truong
International Journal of Mobile Human Computer Interaction (IJMHCI), Volume 5, Issue 3, (July 2013) 42-61.
Mobile text entry methods traditionally have been designed with the assumption that users can devote full visual and mental attention on the device, though this is not always possible. We present our iterative design and evaluation of Escape-Keyboard, a sight-free text entry method for mobile touch-screen devices. Escape-Keyboard allows the user to type letters with one hand by pressing the thumb on different areas of the screen and performing a flick gesture. We then examine the performance of Escape-Keyboard in a study that included 16 sessions in which participants typed in sighted and sight-free conditions. Qualitative results from this study highlight the importance of reducing the mental load with using Escape-Keyboard to improve user performance over time. We thus also explore features to mitigate this learnability issue. Finally, we investigate the upper bound on the sight-free performance with Escape-Keyboard by performing theoretical analysis of the expert peak performance.


Waken: reverse engineering usage information and interface structure from software videos
Nikola Banovic, Tovi Grossman, Justin Matejka, and George Fitzmaurice
In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12). ACM, New York, NY, USA, 83-92.
We present Waken, an application-independent system that recognizes UI components and activities from screen captured videos, without any prior knowledge of that application. Waken can identify the cursors, icons, menus, and tooltips that an application contains, and when those items are used. Waken uses frame differencing to identify occurrences of behaviors that are common across graphical user interfaces. Candidate templates are built, and then other occurrences of those templates are identified using a multi-phase algorithm. An evaluation demonstrates that the system can successfully reconstruct many aspects of a UI without any prior application-dependant knowledge. To showcase the design opportunities that are introduced by having this additional meta-data, we present the Waken Video Player, which allows users to directly interact with UI components that are displayed in the video.
Triggering triggers and burying barriers to customizing software
Nikola Banovic, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice
In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2717-2726.
General-purpose software applications are usually not tailored for a specific user with specific tasks, strategies or preferences. In order to achieve optimal performance with such applications, users typically need to transition to an alternative efficient behavior. Often, features of such alternative behaviors are not initially accessible and first need to be customized. However, few research works formally study and empirically measure what drives a user to customize. In this paper, we describe the challenges involved in empirically studying customization behaviors, and propose a methodology for formally measuring the impact of potential customization factors. We then demonstrate this methodology by studying the impact of different customization factors on customization behaviors. Our results show that increasing exposure and awareness of customization features, and adding social influence can significantly affect the user's customization behavior.
SpaceSense: representing geographical information to visually impaired people using spatial tactile feedback
Koji Yatani, Nikola Banovic, and Khai Truong
In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 415-424.
Learning an environment can be challenging for people with visual impairments. Braille maps allow their users to understand the spatial relationship between a set of places. However, physical Braille maps are often costly, may not always cover an area of interest with sufficient detail, and might not present up-to-date information. We built a handheld system for representing geographical information called SpaceSense, which includes custom spatial tactile feedback hardware-multiple vibration motors attached to different locations on a mobile touch-screen device. It offers high-level information about the distance and direction towards a destination and bookmarked places through vibrotactile feedback to help the user maintain the spatial relationships between these points. SpaceSense also adapts a summarization technique for online user reviews of public and commercial Venuees. Our user study shows that participants could build and maintain the spatial relationships between places on a map more accurately with SpaceSense compared to a system without spatial tactile feedback. They pointed specifically to having spatial tactile feedback as the contributing factor in successfully building and maintaining their mental map.


Design of unimanual multi-finger pie menu interaction
Nikola Banovic, Frank Chun Yat Li, David Dearman, Koji Yatani, and Khai N. Truong
In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '11). ACM, New York, NY, USA, 120-129.
Context menus, most commonly the right click menu, are a traditional method of interaction when using a keyboard and mouse. Context menus make a subset of commands in the application quickly available to the user. However, on tabletop touchscreen computers, context menus have all but disappeared. In this paper, we investigate how to design context menus for efficient unimanual multi-touch use. We investigate the limitations of the arm, wrist, and fingers and how it relates to human performance of multi-targets selection tasks on multi-touch surface. We show that selecting targets with multiple fingers simultaneously improves the performance of target selection compared to traditional single finger selection, but also increases errors. Informed by these results, we present our own context menu design for horizontal tabletop surfaces.