Famous distiller revamped its industrial network with no interruption to production to achieve a scalable and more standardized, reliable and secure system.
By Jim Montague, Executive Editor, Control Global
Lynchburg, Tennessee distiller Jack Daniel's is well-known for its commitment to traditional manufacturing processes, which includes using water from the same iron-free spring since its founding in 1866, making its own barrels and charcoal, and securing the highest-grade corn, rye and malted barley.
Many of these tried-and-true practices are supported by more modern industrial controls, networks and other components on Jack Daniel's six production lines for bottling, capping, labeling, packing and palletizing.
However, recent issues plagued the plant; notably, powering off a main control panel caused a multiple-line outage with 12 hours of ensuing downtime. This sparked the need to revamp and reorganize the network to prevent further interruptions and other potential problems.
As a result, Jack Daniel's enlisted Rockwell Automation Solution Partner Premier System Integrators, a CSIA-certified system integrator in Smyrna, Tennessee, to conduct a network assessment, develop a secure solution and effective network documentation, and help prevent any future issues.
Scan and Assess
"We used SolarWinds’ Network Topology Mapper (NTM) to scan the network, and build the architecture, where possible, by leveraging Simple Network Management Protocol (SNMP) and Internet Control Message Protocol (ICMP),” explains Larry Grate, technology director at Premier.
“To get the most value, SNMP strings were recorded from all managed switches,” Grate continues. “When we scanned the production network and pulled wires, we found a lot of Ethernet that had just grown and grown, but wasn't well organized. We also researched peer-to-peer messaging to determine the risk of failure with the existing network topology."
Premier also discovered that the network overseeing Jack Daniel's six production lines had:
- More than eight different manufacturers of managed and unmanaged switches.
- Unmanaged switches that required physically tracking the cabling or unplugging parts of the network and Internet protocol (IP) scanning using NTM or other tools.
- Existing IP address schemes that wouldn't scale well.
- Switches that weren't consistently configured for proper spanning-tree or Internet group management protocol (IGMP) management.
- A majority of unmanaged switches.
- Multiple places where loss of power or maintenance could cause a cascading line outage.
"This existing network architecture made maintenance difficult," says Grate. "Operators couldn't power-off unscheduled equipment as it was in the critical network path for running other equipment. Device failures caused multiple-line outages. Troubleshooting network issues was difficult due to a lack of consistency and switch-management functions.
“In addition, the network had been expanded with a convenient or lowest capital expenditure cost path, which increased operating costs. Existing network infrastructure wouldn't support expansion or desired enterprise connections. Programming changes on off-shifts weren't always well-documented, causing production losses on later shifts. Finally, that one specific event which caused more than 12 hours of downtime was really the last straw."
CPwE to the Rescue
To revamp and reorganize Jack Daniel's network, Premier recommended the distillery adopt Converged Plantwide Ethernet (CPwE) best practices. CPwE is a set of standards and best practices developed by Rockwell Automation and its Strategic Alliance Partner Cisco®. Network designers can use CPwE to help securely connect operations technology (OT) networks to information technology (IT) and enterprise networks, and help users advance their journeys to The Connected Enterprise.
To follow CPwE's best practices in constructing the new network, Grate reported that Jack Daniel's would need:
- A scalable IP address scheme and separate VLANs for I/O, human-machine interface (HMI), and management traffic;
- Redundant communications so no single point of failure on the network would result in loss of communication to the pallet handling area;
- A new fiber backbone;
- Fully managed Stratix switches employing the Cisco IOS operating system;
- Time synchronization via network time protocol (NTP);
- A migration of PLC-5s for pallet handling to ControlLogix control systems to allow use of device level ring (DLR) topology;
- Switches capable of network address translation (NAT) to allow migration of individual lines without requiring site-wide IP readdressing;
- FactoryTalk® AssetCentre deployment for file, configuration management and disaster recovery;
- FactoryTalk View SE with Stratix® aspect-oriented programming (AOP) implementation; and
- Configured FactoryTalk View ME for line-level alarming and monitoring at each PanelView™ graphic terminal main control panel (MCP).
"We installed 20-port Stratix 5700 switches for each MCP that have 100-MB fiber uplinks to the core and 1-GB interfaces to access switches online," notes Grate. "We also designed for security with multiple VLANs, including separately defined CIP and management VLANs; separately defined I/O and HMI VLANs on each line; dynamic host control protocol (DHCP) via the HMI VLAN for local network access on each line; and administratively disabled unused ports."
Grate says Premier's network was approved with several significant changes, including:
- Use of Stratix 5400 at each MCP to allow for gigabit access and uplink.
- The addition of Stratix wireless access points (WAP) for use with Apple iPads as clients.
- Use of Rockwell Automation Intelligent Motor Control with its Connected Components Workbench software for VFD backup.
- Additional drives that resulted in more required asset licenses.
FAT, Deployment and Benefits
Premier's factory acceptance test (FAT) at Jack Daniel's consisted of thoroughly testing all of the new networks to make sure they'd perform reliably now and in the future.
Premier installed the new network during the distiller’s usual weekend closures. "We installed the new core and virtual host server in a new, locked network room," notes Grate. "We upgraded pallet handling from PLC-5 to ControlLogix controllers, and installed the DLR with NAT to allow use of new IP addresses.
Premier upgraded additional lines the following weekend, and “then allowed operation for two weeks to gain confidence. In addition, the system integrator deployed FactoryTalk View SE for line monitoring and alarming, and used Stratix faceplates and situational awareness strategies. “We also scheduled upgrades of the remaining lines on down weekends, and deployed FactoryTalk AssetCentre,” says Grate.
As a result of all these network improvements, Grate reports that the distillery gained line separation — so powering down one line no longer affects others; better speed for troubleshooting, updating rates of data, and staying online; and a redundant structure that prevents outages in the event of hardware failure.
In addition, documentation now helps IT and OT understand how things are connected for future upgrades. With only one switch to learn instead of eight different manufacturer switches, training has become easier. Standardized IP addresses and network infrastructure ease working with OEMs or corporate engineering.
Other improvements included:
- No more random communication faults to I/O devices, which had led to line downtime.
- FactoryTalk View SE and ME faceplates boosted diagnostic speed to repair with faster location of failed devices.
- Robustness of the network for additional functions, such as OEE, historical data and remote access.
- Ability to easily troubleshoot from both local ports and the Electrical & Instrumentation office.
- One location in FactoryTalk AssetCentre for all PLC, HMI, VFD and switch configuration files.
"This really increased everyone's peace of mind because we're no longer concerned if we have current VFD parameters or PLC programs," Grate explains. "Jack Daniel's also has reporting to track changes to assets on the production floor."
In the future, Premier and Jack Daniel's plan to:
- Deploy an industrial demilitarized zone (iDMZ) between the core Stratix switches and the enterprise,
- Work with the distillery's corporate IT group to allow secure remote access to the iDMZ,
- Configure jump servers in the iDMZ to control access to the process network,
- Deploy engineering servers on the process network for remote management, and
- Deploy an automated OEE solution to improve maintenance and operations.
Learn more about industrial networking solutions and services from Rockwell Automation.
The Journal From Rockwell Automation and Our PartnerNetwork™ is published by Putman Media, Inc.