Industrial firms are belatedly adopting DevOps as they get their heads round the fact that industrial code is behind 50 percent of their extremely expensive downtime incidents.

The 1st Annual State of Industrial DevOps Report, sponsored by Copia, showed that cybersecurity breaches were a major contributor to this, while more than three quarters of businesses said ad hoc fixes were a common occurrence.

The profit sapping effect of downtime was clear. The average cost of downtime identified in the survey was $4.2m an hour, with a fifth of companies estimating it at $5m to $10, and 15 percent saying it was upwards of $10m an hour.

In the real world, Toyota – the doyenne of the sort of agile and lean processes which have informed the development of DevOps – had to shut down its lines last year when an ordering system suffered an outage. There was also a cyberattack that forced a stoppage at the car giant.

And while half of all downtime featuring in the Industrial DevOps report was put down to industrial code, the figure rose to more than 60 percent for organizations with 75 or more sites. Human error was identified in a third of downtime situations, and environmental disaster figured in just a quarter.

Hardware malfunctions featured in 45 percent of outages, cybersecurity breaches accounted for 47 percent, with coding software issues playing a role in 41 percent. And that software can be spread across an incredible array of operational devices, from standard computer systems to sensors, scanners, CNCs, even robots.

Respondents spent an average of four hours per month reviewing code, but a whopping 45 hours debugging code. This is exacerbated by the number of ad hoc fixes, with untracked interventions contributing to cyber incidents or setting the scene for future downtime or complicating recovery from other events.

Almost every respondent felt their organization would benefit from industrial DevOps, of which 58 percent strongly agreed. There was a particular interest in the use of AI powered systems for predictive maintenance (48 percent), as well as cybersecurity automation tools (37 percent). And nine out of ten felt that cloud-based technologies could be used in the OT space, boosting flexibility and visibility to the workforce around the clock.

Overall, teams that had adopted industrial DevOps cited just 7 percent of their time being sucked up by unplanned work or rework, and 10 percent being taken up with remediating security issues.

In his foreword to the report, Copia CEO and founder, Adam Gluck wrote: “These practices address the ever-growing cybersecurity risk inherent as the boundaries between IT and OT continue to blur in our age of digital transformation and the amount of code on the plant floor continues to proliferate.”