Goal

The goal of this project was to create an augmented reality interface that allowed the scientist to retrieve information using natural language, had a semantic understanding of queries, and provided contextual understanding of an experiment.

User

Research scientist. Focused on designing, executing, and analyzing results of experiments for neuoscience research.

Problem

The scientist performing experiments in the lab with the concern of contamination, which means that simple tasks (note taking and switching pages to read research protocols) require heavy context switching. In order to move between the experiment and taking notes or analyzing results, the researcher must remove their gloves and sometimes switch rooms. As a result, these tasks are often separate when the intellectual content is intricately linked. The scientist will may miss protocol steps and will skip recording important information from their experiments. In addition, the scientist cannot incorporant their notes help redesign and adapt experiemnts on-the-fly.

Solution

The choice of augmented reality was made to provide information to the scientist in a hands-free way and minimize the necessity to use additional sources of information like a lab notebook. I developed an augmented reality interface that provided hands-free access to information without disrupting the experimental work-flow. The voice-commands provided a way to retrieve information through natural language. These same commands could be used to provide answers to specific questions about an experiment. Contextual understanding of an experiment was also made possible by giving access to sample information experimental set-ups information with augmented reality. This interface enhance the scientist’s ability to conduct research by giving access to information at the bench, which was previously only available in notebooks and on computers.

Interactive lab notebook with voice commands

I created a program that enabled voice-commands to control the information displayed within the HoloLens device. The program displayed information about a hypothetical experimental sample, such as history, contents, and current protocol information. After hearing commands such as ”Show History” or ”Show Contents,” holograms containing sample information are displayed above the sample tube in the HoloLes device. Each of these holographic panels can also be removed from view by saying commands such as ”Hide Protocol.” This interaction demonstrated that sample information, normally stored in a notebook or computer, can be accessed while performing an experiment.

Interactive experimental set-up with voice commands

In order to demonstrate the ability of the interface to have semantic understanding of queries, I create an intractable holographic laboratory meant to enable the scientist to conceptualize their experiments and results. The hypothetical set-up was of a neurological experiment that used an electrode arrays to perturb neurons in the brain. Voice-commands can show and hide the electrode array on the brain using ”Show Setup” and ”Hide Setup.” In addition, the user can ask the device to ”Display Current Electrode” and ”Rotate Electrode” to highlight the current electrode and rotate the screwdriver to move the position of the electrode. This demonstrates that the user can ask in-context for the current sample, rather than having to compare sample ids on a computer.

QR-codes to display of real-time equipment information

In order to show that the interface could have contextual understanding of an experiment, I created a program that could automatically display sample information from bar-coded samples. In collaboration with BioBright’s computer vision expert, we added added computer vision capabilities to the interface that allowed detected of markers. The interface is now able to detect markers, as demonstrated by generation of holograms when a simple marker comes into view of the HoloLens. The left panel of the figure below shows the markers just before it was detected by the device. The right panel of the figure shows a small hologram that was create in response to the detection. In the future, this capability will be added to the experimental set-ups to automatically show sample information. This will enable the scientist to get sample information while performing an experiment, rather than having to break protocol and look up the information on a computer.