Complex procedures on the International Space Station rely on the Mission Control Center (MCC) to provide guidance, e.g. when a medical scenario exceeds crew's knowledge. Mars missions will not have real-time communication links for MCC to provide such tele-guidance (worst-case round trip communication time of 40 minutes). Thus, there exists a need for solutions that can provide just-in-time training, monitoring, and autonomous guidance of complicated procedures, to make the crew independent of MCC.

We develop a system for automatically constructing computational models of a complex physical task, such as a medical procedure performed by humans, given only a handful of recorded expert demonstrations of the task. Once such a model is built, our system can finely analyze the same task being performed in live video, to provide measurements and analytics, improve efficiency, guide a crew member through the task, or provide just-in-time training. The illustration on the left shows such a system in action where a crew member performs a diagnostic procedure on a patient. Our system "watches" the procedure in real-time, and guides the crew member step-by-step, while proactively warning for any mistakes.

Links: NASA SBIR 2020-I contract summary