The Retrocausal Blog
Stories from the frontlines of the Smart Factory
Assembly processes involve a wide range of activities that cannot by covered by tracking motion alone. We explain why!
Learn about the five dimensions and five levels across which we evaluate the "leanness" of a factory floor.
Practical examples across manufacturers and vendors
Why machine vision for visual inspection is not enough!
Even if you catch assembly mistakes at the QC station, its already too late.
Is human motion tracking enough to build digital poka yoke systems?
Most assembly processes involve a wide range of "freewheeling steps" that cannot by detected by motion tracking alone.
Motion capture or Mocap technology has been used by Hollywood for decades to transfer real human motion onto CG characters. Microsoft commoditized this technology in 2010 with their Kinect sensor often bundled with Xbox gaming consoles. Our founding team made fundamental contributions to the science behind capturing 3D wireframes from ordinary cameras in the 2000s and early 2010s, and won several awards for it, including a Microsoft Research Best Paper Award at the prestigious IEEE Workshop on 3D Representation and Recognition 2011 (3dRR-11). We applied this technology for precise hand tracking in Microsoft HoloLens, as well as got patents on it [1] [2].

In the industrial context motion tracking technology has been used for worker training on simple table top processes, provided by vendors such as LightGuide and Invisible AI. You can see such a demonstration in the following video.
Note how parts have to be placed at precise locations on the table top in this demonstration. The system tracks steps in the process by figuring out whether the worker placed his hand at a precise location on the surface.

Such hard constraints are acceptable for training purposes in limited situations, but most assembly processes have a more freewheeling nature, where parts are attached in the "air", at a fast pace, often out-of-order, and sometimes simultaneously e.g. four screws picked at the same time in one hand while the other hands runs the torque gun to push them in.

Meeting takt time, i.e. the required product assembly duration that is needed to match the demand, is of paramount importance on the assembly line; and placing rigid constraints on the worker to perform a process does more harm than good.

Now contrast the assembly process above with an actual process on the line, being tracked by our Pathfinder platform.
Ours is a universal solution that can capture subtle details of a manual activity and is insanely simple to setup.
Even in this "table top" assembly process that takes place in a Shenzhen factory, you see parts being unwrapped above the table surface, the unit being moved freely on the table and being cleaned by blowing air on it. Notice how the worker is working naturally, how subtle several of the steps are, yet our solution is able to track them in bold on the left hand side.

We estimate that about 80% of manufacturing examples are like this where motion tracking alone is insufficient.

This is precisely our raison d etre: to provide a universal solution that can capture subtle details of a manual activity and is insanely simple to setup [3]. In fact, one of the most sophisticated automotive manufacturers on the planet evaluated the motion tracking approach against ours, and found us to be far ahead of the competition.

So how do we do it? What's special about our approach?

Our past two decades of applied research has brought us to three key technical ideas that allow us to build universal "digital poka yoke" systems. These are:

1. Context is important: Human motion alone isn't enough to understand a complex activity. The same motion cues can mean different things depending on the objects the worker is manipulating. We have invented technology that "discovers" objects that the worker interacts with (without requiring any object-level labels for setup), while interpreting the precise style of "grasping". Here's a video that showcases the concept (using a 3rd party tool for visualization only).
2. Avoid hard rules: Another learning from our experiences building robust computer vision systems is to always avoid fixed constraints. We have developed novel technology for aligning a live video against the entire set demonstration ("training") videos, in a "soft" manner i.e. within a neural network, where our system internally holds hundreds of probabilities that help it cover all possible ways in which a certain step could be performed. The core ideas have been peer-reviewed and published at the top conference in the field [4]. We summarize the concept in the following video.
3. Continuously improve models: Machine learning models are never perfect on day 1. Those who claim to have perfect models are lying to you! Our emphasis is on quick deployment that gets you 90-95% of the way in a week; which then continuously learns and improves beyond 99.9999% accuracy in the following weeks and months as it gets the chance to observe the same process further.

A key requirement to self-improving models, is to create algorithms that automatically understand the structure of a task, and can transfer labels from examples annotated earlier to the new data, to continuously re-train the models. We achieve this through unsupervised video learning, describe at length in our technical paper [5].
An additional benefit we get out of modeling manual processes with the above three insights is that we are able to incorporate corner cases in a consistent framework. This is a strategy that our team learned from shipping AI systems in a domain fraught with "long tail" problems i.e. self-driving.

To summarize, human motion tracking is a valuable tool in many domains. We were amongst the pioneers of democratizing that technology. But unfortunately, its far from sufficient when it comes to guiding assembly workers in 80% of processes, due to the large number of variations between workers. We have a superior solution that not only accomodates but thrives on these variations.
How to assess a factory in 30 seconds?
We define five levels of factory organization that roughly correspond to growth along the lean manufacturing house.
I was just counting our internal travel logs, and it turns out that over the past year and a half, right in the midst of COVID, our team has somehow been able to visit slightly more than a hundred factories. Many of them multiple times.

Several of these visits were arranged by our friends at the Center for Advanced Manufacturing in Puget Sound (CAMPS), Techstars, Plug and Play Detroit, and Unknown Workforce Technologies. Some were customer exploration opportunities, others, preparations for actual deployments.

Obviously we wanted to maintain a record of what we were learning from these visits, and we spent a lot of time discussing various formats and dimensions along which to quickly assess and summarize what we were seeing. The most valuable insights came from David Crawley, currently CEO of Ubiquity Robotics, who has spent more than a decade visiting factories and running such assessments from his career at McKinsey. Today we share those five dimensions along which we analyze a factory for its organization. You will undoubtedly find relations between our approach and ideas from the Toyota Production system including 5S and the Lean Manufacturing House.

Assess along five dimensions to distinguish between five levels of factory organization

We wanted a quick and clean way of thinking about the floor that will be scalable, and we arrived at the following five dimensions: man, machine, materials, mother nature (environment), and melioration (or continuous improvement).

We define five levels of factory organization that roughly correspond to growth along the lean manufacturing house as follows.
Only 3% of factories in the US fall in the Level 4 bucket.
Level 0 factory: A "level 0" factory looks like a badly organized local garage. There is a general disorder, where we see equipment not in a logical place. There are piles of large quantities of inventory found placed at random locations. We see workers operating in a very casual and unstructured way. When we talk to the plant manager or equivalent, it becomes immediately clear that there is no formal or informal process improvement plan in place. We roughly see 10% of factories as falling in this category.

Level 1 factory: A level 1 operation already looks significantly more standardized than a level 0 operation. All equipment is structured in a logical sequence, but not necessarily marked. We find large quantities of inventory placed at standardized locations. We can clearly see workers operating with a sense of purpose. We find that process improvement happens occasionally in a strict top-down manner, directed by the plant manager. We put approximately half of the factories we visit in this category.

Level 2 factory: We define a level 2 factory as one where there is a clear workflow for the equipment. You step on the factory floor and it's clear what's going on. Work is standardized. There is not a lot of inventory lying around everywhere. Workers seem over run with work. There is lighting in key locations, and the floor may be immaculate. Quality circles are regularly organized where groups of workers who do similar tasks meet to identify, analyze and solve work-related problems. We find that less than a quarter of factories in the US fall in this category.

Level 3 factory: A level 3 operation is ordered, tidy, and the steps are clear even on a first glance to an outsider. There are clearly market inventory locations, but there is very little inventory there. Everyone is busy but not rushed. It's obvious that the organization has an active 5S program, e.g. its clear they are tidying up daily. There is an energetic and active improvement program in which everyone is involved. We classify around 17% of factory floors we visit to fall into this bucket.

Level 4 factory: We have visited only three factories that completely fall in Level 4, and one which satisfies our criteria for Level 4 along 3 or 4 out of 5 dimensions. In these operations, it is impossible for the product not to go in the right place. The placement of equipment has gone through multiple rounds of optimization. You don't see any inventory that isn't being conveyed or worked on. The workers look like the interior of a swiss watch or like a military marching band. They will be operating with a rhythmic motion. There is no more lighting or cleanliness than is strictly necessary. There is a professionally led improvement program that is fully driven by worker input. Obviously, we were intrigued when we saw such operations and tried to learn more from plant leadership and workers. Across all three of these factories, we learnt that it took them decades of discipline, continuous improvement, transfer of knowledge from their sister factories and training in lean manufacturing to get where they are today!

Relation with the Lean Manufacturing House

If you have read this far, you must be seeing parallels to the Lean Manufacturing House, which is a more comprehensive analysis and improvement framework. Our objective here was to assess a factory floor within seconds and minutes, whereas workshopping growth along the lean manufacturing house is an activity that requires days and weeks.

Here we want to briefly point out some of these parallels. Our "level 1 factory" roughly corresponds to just the foundation of the lean manufacturing house. When applied to a production system, this refers to team stability, standardized work methods, continuously monitored strategy, etc. A Level 2 factory corresponds to the base of the lean manufacturing house, where Kaizen and reduction of MUDA are high priorities. The third level roughly indicates the two pillars of the house namely, JIT and JIDOKA, and so on.
Several Level 4 factories are in danger of slipping down this hierarchy.
Who do we work with?

So why do we assess factories along these dimensions? We find that the ideal customer for our RetroActivity platform lies between Level 2 and 3. Our objective is to accelerate their journey towards a Level 4 factory. In the lean manufacturing house, you will notice that our solutions correspond to the pillars of JIT and JIDOKA. At the same time, we find that in several verticals such as automotive, several Level 4 factories are in danger of slipping down this hierarchy. This is because changes in customer demands are forcing these factories to change processes and introduce new ones at an unprecedented pace. We are deploying with at least one such factory and hope to continue growing our Level 4 customer base.

We don't work with Level 0 and 1 factories. We find that such factories are able to capture the largest gains from non-technological initiatives, such as better lean manufacturing education.

References:

The Lean Manufacturing House (link)
Digitizing Humans
THE MISSING PIECE IN TRANSITIONING AUTOMOTIVE MANUFACTURING
Most Industry 4.0 investments have focused either on robotics or on adding sensors to "equipment", and human-powered processes have largely remained in a blind spot.
As we all know, the automotive industry is in transition. Trends such as electrification, self-driving, shared mobility and connected vehicles, powertrain evolution, global consolidation, and the competition for the Chinese market mean that this sector will look very different in 2030.

The leading players in this industry understand these challenges and have spent the past decade building the business and technology muscle to help them thrive in this new world. On the business strategy front, automakers who previously preferred proprietary technology and in-house engineering have been forming deep collaborations and alliances on autonomy, sharing the burden of investments with other companies. Leading manufacturers are embracing specialization by focusing on particular types of vehicles or on narrower value chain roles to benefit from economies of scale. They have become more zealous in managing costs, investing only in the critical capabilities required to deliver on their strategy while cutting costs in "table stakes" areas.

On the technology front, digitization is the talk of the town, and every single car maker has looked into predictive maintenance, IIoT, automated scheduling and ordering systems, visual part inspection, robots, and augmented reality. Even though projecting ROIs upfront with these projects has been problematic, and a significant fraction never leave the pilot purgatory -- in hindsight, the benefits have been clear cut.

However you look at it, these drastic shifts will cause the manufacturing floors to change, and only the most adaptable will survive. The aspect of manufacturing advancement that gets the most press is robotics and automation. Yet, as Apple [1], Boeing [2], Tesla [3] and several leading manufacturers have discovered, robots are not as suited as human workers in adapting to process changes. That's why robotics investments are best suited to processes which are already cast in stone. Of course, apart from lack of adaptability, state-of-the-art robots also lag behind humans on high dexterity manipulation tasks, which is why humans continue to be ubiquitous on sub-assembly stations.

Unfortunately, human workers come with their own weaknesses. Unlike the early days of assembly lines, factory floor work is no longer seen as a lucrative career by most, which means that retired experts are not getting replaced with younger workers. Meanwhile, cyclical shifts in demand and union pressures means that the majority of the new workforce comprises temps, who are poorly trained and make more mistakes than permanent workers.

A question that several players in discrete manufacturing broadly, and automotive manufacturing specifically have been asking recently is how to make human workers more efficient. Here we discuss several case studies around this question.

Digital Work Instructions

Contract manufacturers such as Jabil Circuit [4] have built customized mobile apps to provide "Digital Work Instructions" to help their workers avoid assembly mistakes. Think of these apps as PowerPoints which guide a worker through the steps in an assembly process. While such technologies require upfront investment in terms of redesigning the process around the technology and getting buy-in from workers, they have been shown to improve production yields by 10 percent and reduce manual assembly related quality issues by 60 percent. These tools also help provide visibility into these manual processes by dashboarding cycle times, error rates, and discussions around mistakes - thus "digitizing" the human element of the process. On the flip side, such tools reduce throughput and often annoy workers by requiring a lot of user interaction to help the system track the job status.

IIoT-enabled workbenches

There are also some new ideas being explored by companies in the "lot size one" manufacturing space. John Deere [5] has built the JDAAT system (John Deere Assembly Assist Tool) which comprises a smart panel approximately six feet long, equipped with tens of IIoT-enabled tools. The JDAAT system gives the worker a step-by-step sequence of operations on a screen, e.g. "tighten bolt no. 15 and nut no. 15 using gun no. 22." The gun sensor registers that the desired torque has been reached, and the system advances to the next instruction. John Deere products such as a Combine Harvester cost between $600,000 to $1,000,000 (including attachments) and one factory only produces eight pieces per day. The time window for harvesting is only three months per year, so farmers can't afford their harvesters to break down. Thus, ensuring the highest quality is really important. In contrast, Tesla produces one car every two minutes, and the workstations there cannot afford to wait for connected powered tools, as even they slow down tasks. That's why we expect that such solutions are likely an overkill for automotive assembly. In addition, JDAAT cannot track assembly steps that don't involve power tool usage, e.g. when a worker joins two parts together with her hands which is common in automotive sub-assembly.

Smart Benches and Augmented Reality

Whirlpool as well as manufacturers in the mass customized manufacturing sectors have been exploring overhead projector systems [6] coupled with cameras and basic machine vision technology to help independently train workers by providing them step-by-step guidance as well as live feedback. Fujitsu replaces these bulky projectors with AR technology in Microsoft's HoloLens [7] to reduce assembly time of the networking equipment from 120 minutes down to 97 minutes (19 percent increase in productivity), and reduce their installation time from 53 minutes to 31 minutes (42 percent improvement). While highly effective, a major challenge is that it takes months to develop these experiences, and installation is often cumbersome in the case of projector setup while the head mounted display is not suitable for wearing all day long.

Visual Analytics

Denso and a few other Tier 1s have deployed cameras on their assembly workstations to gain visibility into manual processes. These systems analyze human operators performing assembly tasks and allow manufacturing supervisors and industrial engineers to trace mistakes through the line as well as continuously perform time and motion studies. Again, these systems have proven ROI in facilitating Kaizen events given the ubiquitous use of human workers on Tier 1 assembly lines. On the flip side, these solutions also take four to six weeks to customize to a given workstation and cannot keep up with an overnight modification to a process. They are also limited to providing offline analytics, as opposed to live task guidance and feedback to individual workers.

Altogether, human operators will continue to play an important role on automotive factory floors, even more so due to the agile nature of the next decade of developments in this industry. Despite this, most Industry 4.0 investments have focused either on robotics or on adding sensors to "equipment", and human-powered processes have largely remained in a blind spot. Only in the last 2-3 years, have manufacturers started realizing the importance of bringing the human element into the realm of digitization. We believe that this trend will only accelerate due to the lasting changes brought by COVID-19. We see time to deployment, rapid adaptability, live as well as offline analytics abilities, and integration into existing manufacturing IT infrastructure such as Manufacturing Execution Systems (MES) as direction along which these new tools should grow in. At the same time, we'd love to hear how you are measuring and augmenting the frontline workers on your floors!
Made on
Tilda