The challenge of change

Managing and regulating automation device configuration code change, whether it be for a PLC, a human-machine interface (HMI) or even a configurable valve, poses an ever-increasing challenge for industrial automation engineers across the Asia-Pacific region. As the sheer number of configurable devices on an operating site increases, and site automation systems become more interlinked and network access points more ubiquitous, so too does the risk of unmanaged code change.

To date, managing code change has most often been addressed (if at all) by keeping manual records of the current code version, plus the code edit details and their authors for every change at each configurable device. Such systems are completely 'open loop' and highly reliant on good practice by site personnel. On a busy operating site, with many hundreds of configurable devices and the pressure and cost-implications of plant down time, retaining such records accurately is difficult to say the least. The practical reality is that such manual systems offer only a rudimentary safety net: access to controller code remains largely unpoliced and software changes are generally not recorded in a form that can be audited.

Code as an asset

For many plant maintenance engineers, automation code configuration is regarded as being in a separate world to the science of asset and maintenance management, the tools, techniques and strategies used to minimise maintenance, repair and operation (MRO) costs, and maximise plant up time. Conventional asset and maintenance management strategies tend to focus on physical plant: it's all about large rotating equipment, production lines, utilities and so on, rather than software and code, is it not?

Not quite. This definition is too narrow and configuration code is an essential and often under-valued part of the plant asset big picture. Companies don't often assign a dollar value to their PLC, HMI or drive code until it becomes a problem. The problem occurs when uncontrolled changes in the code cause the machine, line or even entire plant production to shut down. It's then that they learn the true value of code and that code, like any piece of physical plant, is an asset that should be factored into any asset management and maintenance strategy.

It is for this reason that code change management is one of the four operational elements of Rockwell Automation's software maintenance automation centre (RSMACC) collaborative asset management and maintenance tool.

In modern manufacturing facilities, security of code and the validation and verification of code have very real cost and plant-throughput implications. Poor or failed code can stall operations, impacting on plant up time and operational equipment effectiveness (OEE) and causing unnecessary maintenance costs. In the worst cases, it can lead to contaminated or off-spec product, or personnel and machine safety hazards. In highly-regulated industries, such as pharmaceutical, food and beverage or underground mining, the lack of an auditable code trail could also lead to costly non-compliance fines and legislative costs.

Collaborative approach

Collaborative asset management and maintenance software must drive solutions to four key plant maintenance areas:

• Change management, to manage and police device configuration code;
• Network health, monitoring and managing enterprise-wide control network assets;
• Enterprise online condition monitor, which integrates condition monitoring data to provide real-time analysis and correction of maintenance problems; and
• Automated asset management, linking automation with CMMS to automate parts of the maintenance system, reduce MRO costs and improve OEE.

In structure, RSMACC mirrors the ubiquitous plant-wide nature of contemporary industrial automation topologies and leverages the power of three core building blocks: Microsoft SQL Server 2000, Rockwell Software FactoryTalk and Rockwell Automation's Integrated Architecture.

Microsoft SQL Server 2000, an industrial-grade relational database designed to efficiently process high volumes of critical data, forms the heart of RSMACC and makes up the repositories for the rich data RSMACC stores and provides. Rockwell Software FactoryTalk, a manufacturing information integration strategy that enables manufacturing applications to integrate seamlessly across the enterprise, couples with Rockwell Automation's Integrated Architecture to provide the targeted routes to and from this database for real-time enterprise-wide data transfer. By using FactoryTalk-enabled software solutions, users can achieve instant connectivity to devices in the field, robust communications and diagnostics, and data transportation to other applications without a lot of extra work or potential errors. It is an architecture that gives users access to device data seamlessly throughout the entire plant and into the enterprise.

While the potential benefits of this software extend well beyond code change management, this is most often the entry point to RSMACC for many manufacturers, as change management is an immediate and growing problem for manufacturers all over the world.

In physical architecture, RSMACC change management is founded on two key existing elements on the customer side: up to 500 devices (PLCs, HMIs, drives and so on), plus the plant-wide communications network (Ethernet/IP, ControlNet, DeviceNet, Data Highway Plus (DH+) and so on) that interconnects these devices. Overlaid on this device/network background are RSMACC's two main physical building blocks: the RSMACC Server, the heart of the system and the home for the rich relational database that is the basis of RSMACC, plus a network of RSMACC clients, field distributed work stations from which RSMACC can be accessed.

From a software perspective, RSMACC core software provides the functionality of event Log, 'audit' and 'security server', while two additional RSMACC change management software modules, the archive and verification modules, provide the all-important change management functionality. Once the RSMACC core software, server and clients are in place, the foundations are established to grow the system into any or all of its four functional areas.

Asia-Pacific drivers

In Asia-Pacific, the key driver for change management is regulatory compliance, most particularly in the highly regulated industries of pharmaceutical and food and beverage, where compliance with regulations, such as the US Food and Drug Administration (FDA) 21 CFR Part 11 electronic record/electronic signature specifications, might be required.

The growing pressure across the Asia-Pacific region to maximise plant up time and throughput is also an important driver behind the move to more refined change management. Asia-Pacific manufacturers, like those in the rest of the world, are feeling the economic pressure. RSMACC change management helps minimise this down time.

A further challenge in Asia-Pacific is the high number of technically qualified personnel on operating sites, many potentially with access to code editing facilities. This access to code, unmanaged and untracked, creates the potential for problems on the plant floor. There is also the issue of plant automation evolution. Much of Asia-Pacific industry is now at an automation agglomeration stage. Courtesy of plant-wide networking, what were once disparate islands of automation are now integrated into a seamless whole-plant automation system. As a result, code change manifested in one area of the plant can have far-reaching, and sometimes unexpected, outcomes.

Security and audit trails

Policing and controlling access to code change is a top priority for most manufacturers. The number one source of all code problems is security breaches.

A further area of distinction is event audit functionality. Legacy change management tools attempt to achieve audit functionality via simple 'before and after' code comparisons. These after-the-fact comparative audits do not provide a truly accurate record of all events and are clearly fallible. Before-and-after comparison audits leave huge holes in the audit trail and actually give the user a false sense of security. For example, a great temporary force on a controller output would most often be overlooked by the before-and-after comparison audit.

People, procedures and places

Site network assessment is often a complex area, demanding careful review. It is not just about devices and device counts: there must be an efficient network architecture in place. Studies show that around 70% of industrial networks are poorly implemented and installed, so this is a crucial starting point.

Second, and more important, is a careful review of existing site operations and the development of the site's authentication matrix. This is all about studying the people, procedures and places that will ultimately be described in the rules defined in RSMACC change management, the security road map.

Once in place, RSMACC change management will provide a fully integrated code change management tool that is completely scalable to the site's needs, now and in the future. Importantly, once the foundation is in place, the important value benefits offered by the package's three additional operational areas (network health, enterprise online condition monitor and automated asset manager) are easily accessed.