Our research on pervasive sensing focuses around the concept of Computational Ethnography which borrows the in-depth understanding of activity and people’s behavior from ethnography and anthropology, and combine it with the potential of capturing large sets of rich multimodal data from environmental and body-worn sensors and technology. Through the use of a variety of tracking technology (body-tracking, eye-tracking, social media, etc.) we study human behavior in context and the interaction of individuals with technology.

To support pervasive sensing we developed Lab-in-a-Box — a novel multimodal data collection infrastructure based on eye, body, gesture, voice and activity tracking — and we enhanced our analysis and visualization tools to better understand large amount of collected data. This includes the extension and integration of ChronoViz, our analysis tool for multimodal time-synchronized data, as well as novel web-based visualization and data analysis tools.

Our tools and techniques are used for in-depth study of activity in the wild, for instance we characterize interaction with Electronic Medical Records and understand patient-physician communication, to study sign language, to understand the ergonomics of laparoscopic and robotic surgery, to uncover multimodal cues of neurological disorders such as stroke, and to exploit social interactions online as a support for real-world intervention in the context of HIV/AIDS.


Ubiscope, the Ubiquitous Computing Microscope is a new kind of microscope for understanding people through the unobtrusive and pervasive sensing of physiological and behavioral data. As the microscope enabled uncovering new understanding and supported key scientific advances, UbiScope acts as a similar proxy for understanding and supporting human activity.

  • UbiStroke: Multimodal Computational Assessment of Stroke
  • ErgoKinect: Detection of Ergonomically Incorrect Posture during Laparoscopic Surgery
  • Effect of Packaging on Smoking Perception and Behavior: a Randomized Control Trial

Stroke-Kinect: A Sensor-Based Approach Towards Creating a Multimodal Stroke Signature
(FISP 3-G3133/2017,  UCSD Frontiers of Innovation Scholars Program)

Effect of Packaging on Smoking Perception and Behavior: a Randomized Control Trial
(NIH R01, National Cancer Institute)

SL-CN: Learning to Move and Moving to Learn
(National Science Foundation, Office Of Multidisciplinary Activities, Sept 2016 – Aug 2018)


Computational Ethnography

The goal of this project is to accelerate observational analysis by employing multi-modal pattern recognition capabilities to pre-segment and tag records and increase analysis power by collecting multimodal activity and  enable the investigation of macro-micro relationships of people’s interaction sin the wild, among themselves and with technology.

  • QUICK: Quantifying Electronic Medical Record Usability to Improve Clinical Workflow
  • Hands That Speak: Studying Complex Human Communicative Body Movements

A Multiscale Framework for Analyzing Activity Dynamics (US NSF IIS-0729013).
October 2009 – August 2011
Flight Crew Performance Data Collection and Analysis Tool (UCSD – Boeing Project Agreement 2011-012 / Boeing Project Agreement 2012)
January 2011 – September 2012
QUICK – Quantifying Electronic Medical Record Usability to Improve Clinical Workflow. (AHRQ R01 – HS 021290)
September 2012 – August 2016.


Connected and Open Research Ethics

The employment of pervasive sensing technology opened up the way for a critical consideration of research ethics. This is especially important with regards to the collection of behavioral, social, and personal data in clinical or health-related settings. To study these important ethical dilemmas, we created the CORE (Connected and Open Research Ethics) initiative and are actively investigating how to redefine research ethics.


RWJF CORE – Designing, building, and testing a Web-based prototype to foster the ethical design and review of health research
(RWJF Pilot Pioneer, November 2015 – October 2017)


Social Media as a Sensor

We live in an era where much of our lives are represented online through a variety of social media such as Facebook, Twitter, Instagram, etc. The intense communication that happens across friends and acquaintances and the networks that are created around those exchanges of textual and multimedia messages create incredible opportunities to better understand a variety of behavioral clue that might help resolve health and healthcare related problems. We are using a mixed approach based on natural language processing and network analysis to capitalize on these issues to help drive intervention, specifically in the setting of HIV and Physical Activity.

  • PIRC-Net: Characterizing HIV at-risk
  • SMART: Using Social Media and Mobile Technologies to Promote Improved Health Behaviors

PIRC-Net: PIRC-NET: Analyzing Social Media to characterize HIV at-risk populations among MSM in San Diego.
(CFAR Developmental Grant, January 2015 – March 2017)
Detecting HIV at-risk MSM in San Diego through Social Networks
(FISP 2014,  UCSD Frontiers of Innovation Scholars Program, January 2015 – October 2016)
Characterizing the Relationship of METH Use and Neurocognitive Impairment in HIV At-risk Online Networks
(TMARC Pilot PST7TP2/2016,  Translational Methamphetamine AIDS Research Center, March 2016 – February 2018)


Understanding Patient-Physician Communication

Communication in the medical office between physician and patient is challenging. This is often complicated by third party in the room such as relatives or an interpreter if the patient is not mother language and by the presence of the Electronic Health Record (EHR). We want to better understand the system defined by the medical office and the role that people, artifacts and technology play in this environment to inform the design of the next generation interactive tools and visualization.


ahrq-logo QUICK – Quantifying Electronic Medical Record Usability to Improve Clinical Workflow
(AHRQ R01 – HS 021290, September 2012 – June 2016)
va_logo DEUCE – Design and Evaluation of User Centered Electronic Health Records
(VA Merit 1 I01-HX000982-01A1) April 2015 – Ongoing


Archived Projects

Publications – Pervasive Sensing and Health

Start typing and press Enter to search