Challenge
- Aging skid control systems faced obsolescence issues and presented risk of costly downtime
- Systems lacked visibility into operations
- System miscommunications generated critical alarms and resulted in downtime
Solutions
Results
- New control systems replace obsolete systems and pave the way for future upgrades
- New visibility into VFDs and other components helps speed up troubleshooting
- Reduced critical alarms resulting from system miscommunications that occurred several times per week
Aging equipment has the potential to manifest into a multimillion-dollar issue for biopharmaceutical companies if not addressed.
If the equipment were to experience a failure with no available spare parts, the production of high-value biological products would come to a costly halt and possibly also result in lost batches.
One biopharmaceutical giant recently needed to address obsolescence in two of its purification skids, and saw it as an opportunity to make some additional operational improvements in the process.
Staying Ahead of Obsolescence
Each production batch begins with a 1 to 4 milliliter ampule of cells that serves as the powerhouse for creating key proteins in their drugs.
The ampule undergoes a fermentation process in which it is thawed into a live state and combined with buffer media to promote cell growth and replication.
Through this process, the ampule of cells can grow to a final volume of as much as 12,000 liters.
Once the specified desired volume is reached, the batch is sent on to purification operations. Here, cellular debris, unwanted proteins and other elements are removed through a series of different processes.
The last of these processes is tangential flow filtration (TFF).
TFF is a pressure-driven process that removes the buffer species from the protein. This helps clarify and purify it into a final concentration that meets the drug’s target specifications, and then it can be sent on to filling.
The company has multiple TFF skids for different batch sizes, but it was the two midsize TFF skids that were nearing the end of their service lives.
The first skid was put into use in 1998, and the second skid was brought in shortly thereafter to meet growing product demand. Both skids had programmable logic controller (PLC), variable frequency drive (VFD) and human machine interface (HMI) components that were either going to be discontinued soon or had already been discontinued.
“The company knew these systems were getting near the end of their lives and needed to be replaced or upgraded,” said Tom Brown, project engineer, Banks Integration. “Based on the overall health of the skid system, it made sense to upgrade the control systems.”
The aging equipment represented a production liability of up to $10 million if they were to go down, given the lack of available spare parts and potential downtime.
More than that, however, they were also generating five to six critical alarms per week due to bad communications between components, resulting in unnecessary downtime.
A lack of diagnostic information made the alarms difficult to troubleshoot and diagnose, sometimes requiring the support of up to 10 employees from the quality, automation and manufacturing science groups to resolve a single alarm.
“It could be a lot of extra hours for something as simple as a loose wire,” Brown said.
Additionally, while the two skids were based on the same design, they were still their own systems. This meant that each skid had its own separate code that had to undergo separate testing for each new production run.
Migrating Control Systems
The biopharmaceutical producer worked with Banks Integration Group, a Rockwell Automation PartnerNetwork™ program Solution Partner, to design, engineer and validate the control-system upgrade process based on the PlantPAx® modern DCS platform from Rockwell Automation.
This involved upgrading the legacy skids’ PLCs and VFDs to Allen-Bradley® ControlLogix® 5570 controllers and Allen-Bradley PowerFlex® 525 VFDs.
A single-fault tolerant EtherNet/IP™ network replaced the company’s existing Data Highway Plus network for better device-level communications and real-time performance monitoring.
Additionally, the control systems were integrated into a redundant, client-server SCADA architecture to enable a greater level of access, including remote access.
The new skid systems use the PlantPAx process objects for life sciences – a library of controller code, display elements and faceplates that are customized for the life sciences industry.
These predefined elements can help company operators more quickly configure applications to speed up production deployments and shorten cycle times.
The new systems also use the FactoryTalk® Batch software from Rockwell Automation, with equipment editor and recipe editor capabilities that enable operators to create models and build individual recipes. The FactoryTalk Batch with PhaseManager™ capability simplifies phase logic programming and configuration by making it possible to create equipment phases without phase logic interface (PLI) code.
New Efficiencies
The upgraded TFF skids have enabled the company to stay ahead of equipment obsolescence and avoid potentially major downtime issues.
More than that, the upgrades have delivered added operational improvements and business benefits.
The new control systems support a “family” code approach in which the same code can be tested once and installed on both skids.
“Having two systems that share one set of code essentially cuts the validation costs for the company in half,” Brown said. “It’s no longer twice the amount of testing processes and twice the amount of paperwork.”
The new systems have also significantly cut down the critical alarms resulting from communications issues – from five or six alarms per week to zero in the first two months of operations.
For any issues that do arise, the new systems give the producer’s team greater visibility into the skids for faster resolutions.
For example, the VFDs on the old systems were not being monitored.
As a result, operators would re-start a machine rather than sorting through several dozen fault codes to pinpoint the issue. Now, fault codes from the new VFDs are reported back to the PLC and collected in a data historian.
“With the new system and the device-level ring, it can tell you exactly where a failure is occurring,” Brown said. “It probably saved the company a day’s worth of troubleshooting in just the first week of implementation.”
Additionally, the new network enables information collected at the control-level to flow across the facility.
This simplifies gaining visibility into how the system is operating, such as by providing Brown and other employees remote-access capabilities to skids even when they’re not on the plant floor.
“The company had very little visibility from the outside before,” Brown said. “If they get a call now, it takes two minutes, to remotely access the skid from office or another location. That’s highly beneficial just in terms of troubleshooting.”
The results mentioned above are specific to the biopharmaceutical producer’s use of Rockwell Automation products and services in conjunction with other products. Specific results may vary for other customers.
Allen-Bradley, ControlLogix, FactoryTalk, PartnerNetwork, PhaseManager, PlantPAx and PowerFlex are trademarks of Rockwell Automation Inc.
EtherNet/IP is a trademark of ODVA Inc.
