Shoulder Movement Could Reveal What You’re Typing on Zoom Chat

• University researchers figure out a way to translate shoulder movement to keystrokes.
• In controlled environments, the accuracy of the predictions can reach up to 75%.
• The system relies on input data that are seemingly unconnected to the target information.

A team of researchers from the University of Texas figured out a way to discern what Zoom users could be typing on private chat by looking at their subtle shoulder movement. So if you like to share jokes with colleagues privately during meetings with your manager, you may want to reconsider this practice. It sounds like an incredibly preposterous capability, but this is the level where things move when AI is deployed.

Apparently, finger movement can be extrapolated to body and shoulder movement in particular, so the researchers were able to create a set of ties between the two. Given that everyone today uses high-resolution cameras, even the slightest pixel shifts on the user’s shoulders can be recorded and analyzed. Back, forth, left and right shoulder movements can lead to keystroke predictions using no other source data than the video itself.

Related: Zoom Announced the Launch of Its “Hardware as a Service” Program

The team set up a controlled environment with specific chairs, keyboards, and webcams and managed to discern the words typed by the test subjects at an accuracy of 75%. In uncontrolled settings, this would drop down to 20%, but both cases could easily reach higher levels of prediction success with machine learning training.

This is yet another impressive example of how snooping on a seemingly unrelated subject can draw data that could help someone deduce more valuable things and reconstruct information from signs hidden in plain sight. One way to protect yourself from this would be to lower the light in your room during Zoom meetings, place a dark background behind your shoulders, and wear a dark blouse.

It is important to understand that given enough time, even these precautions wouldn’t be enough to stop hackers from finding ways to discern what you’re typing. It is a law of physics that every action leaves a trace, so making the associated forensics possible is only a matter of having adequately sophisticated recording and analysis tools. This is now a matter of technological advancement, so it is really becoming possible.