Toyota’s US$350 Million Production Halt: Focus on OT/IT Transparency, Hiring Processes, and Evaluation of Cyber Threats Needed to Avoid the Risks of the Increasing Use of Software and Automation

On August 29, Toyota, Japan’s largest automotive manufacturer that sold a global total of 10.5 million vehicles in 2022, faced a pricey setback due to an issue during the company’s parts ordering system update. The company was forced to shut down not one, but all of the company’s 14 plants in Japan on that Tuesday, with 25 out of 28 production lines not in operation. The day’s worth of lost production is estimated to be around US$350 million. While the company has not provided any explicit details to explain the cause of the closure, with a spokesperson unable to say whether the glitch happened during a system update, the shutdown should raise a pressing question for all manufacturers that continue to heavily invest in new digital solutions: how much risk are operations exposed to?

Technology vendors are quick to highlight the positives of increased digital transformation in manufacturing, even more so with the current generative Artificial Intelligence (AI) zeitgeist. However, the issues that can arise from such implementations are often downplayed, with vendors or system integrators always assuring everyone that they have done their best to account for such potential drawbacks. Serious problems, such as those most likely faced by Toyota, are rare and, to some extent, cases of bad luck, rather than poorly managed software deployments.

As the market moves toward ever increasing automation and digital transformation, with lighthouse factories and lights-out manufacturing the goal for many manufacturers, the risks of massive financial losses from simple faults in software grow ever larger and increasingly likely.

ABI Research’s Manufacturing Execution Systems (MES): Their Future and Place In the Modern Factory report (AN-5622) identifies that an inhibitor to the growth of the MES software market is the cost of adoption. This, combined with the high opportunity cost of factory shutdowns, can serve to further dissuade manufacturers from digital transformation projects. Primary risks that can cause such factory shutdowns are described below:

• Software Malfunctions: Simple software glitches, errors, and malfunctions can be incredibly costly for manufacturers, halting production and taking hours or days to resolve, often at the cost of millions in lost revenue. Implementing more software solutions and, more importantly, integrating them creates an ever-increasing likelihood of failures.
• Skilled Labor Shortages: New automation software solutions are incredibly complex (see ABI Research’s Manufacturing Execution Systems (MES): Their Future and Place In the Modern Factory report (AN-5622) and Industrial Automation: SCADA/HMI Software report (AN-5624)), and while many come with low/no-code functionality, the software’s management, updating, and deployment requires excellent Information Technology (IT) skillsets. As manufacturers continue to deploy and integrate multiple complex automation solutions, the need for skilled workers to manage them increases. Unemployment rates in the United States remain low and access to labor is tight. This issue is felt even more keenly within the tech market, with estimated unemployment rates as low as 2%. Compromises on the quality or quantity of such labor only increases the likelihood of software malfunctions and poorly managed security. The United States is not alone in this issue, with Japan facing a similarly reduced labor force driven by an aging population and weak immigration. Therefore, manufacturers will either face higher operation risks or delays in adopting and deploying new digital solutions.
• Cyberattacks: Any digital system, by default, is vulnerable to cyberattacks, and as manufacturers increasingly connect systems to the cloud, the number of attack vectors only increases. Issues faced can include coding errors, especially found within complex software, that create exploitable vulnerabilities, third-party dependencies that promote unknown weaknesses, and poor user security training, with many incidents often caused by worker error and negligence.

Quality assurance has been an absolute must for manufacturers for years, and technology vendors need to take the same approach when it comes to delivering software solutions. Bugs and errors are often created in the early stages of development, so technology vendors must dedicate high levels of resources to ensure these issues are stamped out. Implementing new AI technologies for code checking provides an excellent extra layer of security at this stage of development. For manufacturers without deep investment in the comprehensive management of software, full transparency between Operational Technology (OT) and IT teams, and strong testing and training programs for workers, stories like Toyota’s will only be more likely.

Manufacturers need to be exceptionally proactive with their hiring processes, identifying critical talent gaps for the parts of their business that may not require staff in the short term, but will require experienced and company-embedded workers in the long term. Attracting new talent is no longer merely a question of compensation, with many new workers caring more about flexible working and meaningful employment, something that the manufacturing sector often struggles to effectively provide, especially regarding sustainability perceptions. Tesla is an excellent example of a manufacturer that has been able to attract talent workers away from big tech employers, such as Meta, Google, and Amazon, with its strong messaging around sustainability, driving change in the world, and high use of advanced technology in production. Compared to other automotive manufacturers, Tesla has been able to differentiate itself due to its need to not balance the production of Internal Combustion Engine (ICE) vehicles and Electric Vehicle (EV) models on production lines and instead invest solely in greenfield and highly automated gigafactories, creating a unique digital employment environment for workers. Mercedes has achieved smaller, but similar success with the company’s “Factory 56” in Sindelfingen, Germany (see more in ABI Insight “The Automotive Factory of the Future: Mercedes, Microsoft Cloud, and the MO360 Production Platform”).

Manufacturers need to ensure that they have adequately evaluated cyber risk and associated risk escalation framework. Companies are increasingly connecting processes to and putting data into the hands of massive cloud providers, such as Amazon Web Services (AWS), Google (Google Cloud), and Microsoft (Azure), and must have an excellent understanding on the processes, management, and protection of their data within these companies. The IT/OT gap in many organizations only drives complexity in this area, with IT teams often not effectively communicating standards and operating practices to OT teams, and visa versa.

Software providers should be implementing a range of processes to ensure all vulnerabilities and risks are assessed through structured patch management and comprehensive testing and validation processes. This is all underpinned by the all-important communication procedures with customers, who must understand the timings and necessity of updates and patches, alongside any newfound vulnerabilities. Software should have simple updating procedures to encourage prompt action, and end-of-life support should be readily available and communicated well ahead of time.

Overall, both manufacturers and technology vendors need to take off any rose-tinted spectacles that may have developed over recent years, as the push and hype for digital transformation, advanced software, and AI have grown. Effective communication and risk analysis channels must be developed, alongside a culture of transparency in the industry around issues in software deployment and interconnected solutions. While the digital ecosystem represents a leap forward in manufacturing solution offerings, the market needs to remain alert to its potential negative impacts on manufacturing activities.