Telestration: How Helena Mentis Applies Design Thinking to Surgery

Telestration: How Helena Mentis Applies Design Thinking to Surgery

Telestration: How Helena Mentis Applies Design Thinking to Surgery

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Helena Mentis is the director of the Bodies in Motion Lab at University of Maryland, Baltimore County (UMBC) with research spanning human-computer interaction (HCI), computer supported cooperative work (CSCW) and medical informatics. During a recent visit to the Design Lab at UC San Diego, Mentis talked about her research on surgery in the operating room.

She examines the medical world through surgical instruments and the workflow inside the operating room. Mentis hones in on minimally invasive surgery and the reliance toward images.  She is particularly interested in how medical professionals see and share visual information in a collaborative fashion, which has grown over the past several years. She asks, “What happens if surgeons were given greater control over the image? What would happen to the workflow? Would it change anything?”

In one study at Thomas Hospital in London, surgeons were using a lot of pointing gestures to direct the operation. Confusion would arise and the surgeon would need to repeat his exact intention with others. This break in the workflow inspired Mentis’ team to ask: what if we were to build a touchless illustration system that responded to the surgeon’s gestures? Her team set out to build what she calls “telestration,” which enables surgeons to use gestures to illustrate their intentions through an interactive display.

During another operation, the surgeon encountered a soft bone and had to stop the procedure. As a result, the surgeon had take off their gloves to re-examine the tissue on the visual display. Mentis notes, “There is a tight coupling between images on display and feeling with the instrument in hand.” If the image on display could be more closely integrated with the workflow, would this save time in the operating room?After publishing her findings, people raved over how voice narration rather than gesture aided imaging and collaboration in surgery. Consequently Mentis asked, “If given the opportunity would doctors use voice or gesture?” The ensuing observations revealed that while doctors stated their preference for voice, gesture was more frequently used for shaping telestration images. While voice narration and gestures provided greater interaction with the image, surgeons actually spent more time in surgery. Mentis reasons, “There is more opportunity for collaborative discussion with the information.” Interestingly, this did add time to the overall operation, but it also yielded greater opportunities to uncover and discuss critical information.

About Helena Mentis, Ph.D.

Assistant Professor, Department of Information Systems
University of Maryland, Baltimore County

Helena Mentis, Ph.D., is an assistant professor in the Department of Information Systems at the University of Maryland, Baltimore County. Her research contributes to the areas of human-computer interaction (HCI), computer supported cooperative work (CSCW), and health informatics. She investigates how new interactive sensors can be integrated into the operating room to support medical collaboration and care. Before UMBC, she was a research fellow at Harvard Medical School, held a joint postdoctoral fellowship at Microsoft Research Cambridge and the University of Cambridge, and was an ERCIM postdoctoral scholar at Mobile Life in Sweden. She received her Ph.D. in Information Sciences and Technology from Pennsylvania State University.

Read Next

Design Lab Connect2health National Cancer Institute

UC San Diego Design Lab Joins FCC, NCI to Champion Critical Role of Broadband in Rural Cancer Care

The Federal Communications Commission’s Connect2Health Task Force (C2HFCC) announced last week that the FCC and the…

Frontier Design Prize Announces Winners at World Design Cities Conference

Shanghai, September 15, 2022 | The Frontier Design Prize announced the winners of its inaugural edition during the opening ceremony of the World Design Cities Conference (WDCC).  Mr. QU Xing, Deputy Director-General of UNESCO, Mr. GONG Zheng, Mayor of Shanghai, Ms. XU Xiaolan, Vice Minister of Industry and Information Technology (China), along with leading designers, scholars and industry leaders, attended the award ceremony. The Frontier Design Prize (FDP) is a visionary, innovative, world-class design award established with the aims of encouraging design innovation, enhancing the impact of design in driving industrial transformation, and promoting the role of design in shaping a better world. A central program of WDCC, it is undertaken by Design Innovation Institute Shanghai (DIIS) with guidance from the Shanghai Municipal Government. 

How They Got There: Janet Johnson

Graduate student Janet Johnson is currently working towards her doctorate degree in Computer Science, while also conducting HCI research in the UCSD Design Lab, primarily focusing on XR (extended reality).

So, what is Johnson’s research?  Johnson conducts HCI research, primarily focusing on XR. As Johnson describes it, “XR is an umbrella term for augmented reality, augmented virtuality, mixed reality, and virtual reality.” She says to think of it as a spectrum where one end is the real world alone, the other is complete virtual reality, and everything in between is varying mixes of the two. Johnson’s research primarily focuses on this mixed middle ground. “The majority of my research focuses on how we can use mixed reality or extended reality to help a novice…get help from an expert.” She then poses the example of both surgery and CPR. Johnson’s research explores ways for an expert to provide instructions to the novice as if though they were in the same room. Her goal is to help bridge the distance between novices and experts, both physically and skill wise, while also decreasing the amount of time a person receives aid. “By the time a medical personnel arrives at the scene, it’s already been 7 to 10 minutes, so each minute counts for the person’s life,” she explains. “You don’t have time in that 10 minutes to train the people around to be able to do CPR or any other sort of resuscitation, same with surgery.” 

As Johnson continues to conduct her research in this field, she’s excited for what the future holds for this technology and the ways she can contribute to it.  
San Diego And Tijuana Selected As 2024 World Design Capital

San Diego And Tijuana Selected As 2024 World Design Capital

US-Mexico Border Cities Win Historic Designation to Become First Binational Design Capital

SAN DIEGO-TIJUANA - Today the World Design Organization (WDO) announced the San Diego-Tijuana region has been selected as the World Design Capital (WDC) for 2024.

The selection of the San Diego-Tijuana joint bid makes it the first binational World Design Capital in WDO history. While the designation is for both cities as a united region, San Diego is now the first U.S. city ever to receive the WDC designation. Tijuana is the second city in Mexico to hold the title, following Mexico City in 2018.

"We did it!" said Don Norman, founder of UC San Diego’s Design Lab (now retired) and co-founder and Board advisor to the Design Forward Alliance (DFA). "Designers, city officials, and organizations in both the Tijuana and San Diego regions collaborated to make our binational community the World Design Capital for 2024. It shows the power of design as a way of thinking, to address important societal issues, and as a source of innovation for companies, organizations, and educational communities at all levels. We have built a permanent coalition of our communities to address civic and climate challenges, to grow our industrial sectors, and to support a strong culture of cross-border design."
Olga McConnell

Olga McConnell, Project Specialist and Executive Assistant to the Director of The Design Lab

As the Executive Assistant to the Director of The Design Lab, a project manager for the Lab’s special projects and annual events, and a lifelong learner who holds a M.A. in English Linguistics and Translation, and a M.B.A. in Business Administration and Management, Olga McConnell’s zest for knowledge is palpable. She is currently on track to complete a Project Management Certification at UC San Diego Extension at the end of 2021, and she is planning on obtaining her Project Management Professional (PMP) Certification after that. “I’m kind of addicted to getting degrees,” jokes McConnell. “I even thought the other day, maybe I’ll go to law school. And then I was like, no, enough, enough.” 

For nearly five years, McConnell was Executive Assistant to Don Norman, the Founding Director Emeritus of The Design Lab. She is now the Executive Assistant to the new Director of The Design Lab, Mai Thi Nguyen. It is Nguyen’s vision of human-technology-community interactions, along with her JEDI (justice, diversity, equity and inclusion) approach that has McConnell excited about this new chapter in the Lab’s legacy, saying, “I see how great she is as an efficient leader, so I’m really looking forward to working with her, supporting her administratively, as well as taking charge of certain projects that she has in mind.”
Surveillance Technology San Diego

San Diego council committee unanimously approves ordinances targeting surveillance technology

Photo courtesy of John Gibbins/The San Diego Union-Tribune

A City Council committee on Wednesday unanimously approved two proposed ordinances geared at governing surveillance technologies in the city, an action sparked by sustained pushback from activists and others who were surprised and upset last year when it was revealed that San Diego had quietly installed cameras on streetlights throughout the city.

Lilly Irani, an associate professor at UC San Diego (and Design Lab faculty) who specializes in the ethics of technology, called the vote “a win for better governance in the long term.”

Irani helped draft the ordinances and assisted the organized opposition dubbed the TRUST San Diego coalition, which focuses on responsible surveillance in the region. The coalition was born out of concerns about one specific technology — so-called smart streetlights — and ultimately landed a seat at the table to draft the proposals.

“Without Councilmember Monica Montgomery championing this... there would be no table,” Irani said.
Back To Top