Posted by Zeeshan Zia, PhD
Motion capture or Mocap technology has been used by Hollywood for decades to transfer real human motion onto CG characters. Microsoft commoditized this technology in 2010 with their Kinect sensor often bundled with Xbox gaming consoles. Our founding team made fundamental contributions to the science behind capturing 3D wireframes from ordinary cameras in the 2000s and early 2010s, and won several awards for it, including a Microsoft Research Best Paper Award at the prestigious IEEE Workshop on 3D Representation and Recognition 2011 (3dRR-11). We applied this technology for precise hand tracking in Microsoft HoloLens, as well as got patents on it [1] [2].
In the industrial context motion tracking technology has been used for worker training on simple table top processes, provided by vendors such as LightGuide and Invisible AI. You can see such a demonstration in the following video.
Note how parts have to be placed at precise locations on the table top in this demonstration. The system tracks steps in the process by figuring out whether the worker placed his hand at a precise location on the surface.
Such hard constraints are acceptable for training purposes in limited situations, but most assembly processes have a more freewheeling nature, where parts are attached in the “air”, at a fast pace, often out-of-order, and sometimes simultaneously e.g. four screws picked at the same time in one hand while the other hands runs the torque gun to push them in.
Meeting takt time, i.e. the required product assembly duration that is needed to match the demand, is of paramount importance on the assembly line; and placing rigid constraints on the worker to perform a process does more harm than good.
Now contrast the assembly process above with an actual process on the line, being tracked by our Pathfinder platform.
2. Avoid hard rules: Another learning from our experiences building robust computer vision systems is to always avoid fixed constraints. We have developed novel technology for aligning a live video against the entire set demonstration (“training”) videos, in a “soft” manner i.e. within a neural network, where our system internally holds hundreds of probabilities that help it cover all possible ways in which a certain step could be performed. The core ideas have been peer-reviewed and published at the top conference in the field [4]. We summarize the concept in the following video.
3. Continuously improve models: Machine learning models are never perfect on day 1. Those who claim to have perfect models are lying to you! Our emphasis is on quick deployment that gets you 90-95% of the way in a week; which then continuously learns and improves beyond 99.9999% accuracy in the following weeks and months as it gets the chance to observe the same process further.
A key requirement to self-improving models, is to create algorithms that automatically understand the structure of a task, and can transfer labels from examples annotated earlier to the new data, to continuously re-train the models. We achieve this through unsupervised video learning, describe at length in our technical paper [5].
An additional benefit we get out of modeling manual processes with the above three insights is that we are able to incorporate corner cases in a consistent framework. This is a strategy that our team learned from shipping AI systems in a domain fraught with “long tail” problems i.e. self-driving.
To summarize, human motion tracking is a valuable tool in many domains. We were amongst the pioneers of democratizing that technology. But unfortunately, its far from sufficient when it comes to guiding assembly workers in 80% of processes, due to the large number of variations between workers. We have a superior solution that not only accomodates but thrives on these variations.
Huy takes our product roadmap and figures out what’s achievable with today’s AI advances, and what we need to invent to solve our customers’ pain points.
Huy led teams of scientists and engineers at NEC and invented technologies for driver assistance systems that are part of 500,000+ Honda cars. He is an inventor on more than 20 US patents, and his research papers are cited by every lab from MIT, Stanford, and CMU to Apple, Google, and Baidu.
Michael works closely with our marquis customers to understand their needs, and figures out the best solution that immediately adds dollars to their bottom line.
Michael has almost 20 years of enterprise sales experience. He has held sales leadership roles at pioneering AI and computer vision startups.
Andrey takes inventions and converts them into innovations. He leads our product engineering team to turn our scientific prototypes into robust customer-facing products.
Andrey co-founded and successfully exited a startup to immigrate to the US. He spent his next eight years building the core data science infrastructure for Amazon’s supply chain planning team, and the CV/ML infrastructure for Microsoft’s HoloLens organization.
Zeeshan focuses on delighting our customers by understanding their needs and defining product and strategy to meet and exceed these needs. He works closely with our sales leadership to grow our customer base.
Zeeshan was a scientific leader at Microsoft’s HoloLens division, where he invented AI technologies to assist frontline workers across industries. Earlier, he shipped IoT platforms and visual defect detection software for manufacturers at Siemens and MVTec GmbH.
