Hunting Downtime: A Submariner’s Perspective on Predictive Maintenance

Bryan P. Van Itallie, Chief Operations Officer, Grace Engineered Products

“Dive! Dive!” Two simple words, punctuated with the familiar “ah-OOG-ah” alarm, were the signal throughout the ship that we were about to embark on for another adventure under the sea. For many sailors, this quickly became routine, but in reality, nothing is routine about submerging an 18,000-ton, 560-ft. vessel like the USS Nevada (SSBN 733) with 165 people on board hundreds of feet below the water’s surface.

One small defect or fault in any of dozens of systems or one valve out of position, and our crew would find itself on a one-way trip to the bottom of the ocean. Remarkably, since the U.S. Navy’s nuclear submarine program began, only two subs have been lost at sea, the last being USS Scorpion (SSN 589) that sank in 1968, more than 50 years ago. This kind of uptime record is worth investigating. Let’s see what a nuclear submarine can teach today’s manufacturers in their quest to avoid costly downtime.

Technology Improves Maintenance Practices…

In the mid-1990s, before my career in manufacturing, I served as an officer onboard the Ohio-class nuclear submarine USS Nevada. In recent years, preventing downtime has spurred an interest in using technology to make better maintenance decisions.

At the heart of this technology are the fundamental principles of predictive maintenance: gathering data, analyzing it, predicting failures and taking proactive measures to prevent downtime. These fundamental principles are unchanged from how we operated the Nevada 20 years ago.

A crew of 165 people living in an underwater vessel for months at a time amplifies the meaning of hazardous conditions. Consider for a moment a transmission bearing failure causing a loss of propulsion, or a reactor coolant pump malfunction causing a meltdown, or a valve failure flooding the vessel with seawater. Clearly, nuclear submarine downtime costs lives, not just profits!

The U.S. Navy never has had a death onboard a U.S. submarine because of a radiation accident. With all the complexities of multiple systems crammed into a tiny space, operating in harsh environments, how have we maintained such an amazing safety record?

…and Safety Records

The two submarine tragedies in the 1960s resulted in the U.S. Navy’s SUBSAFE (Submarine Safety) program, which covers all systems exposed to sea pressure or that are critical to flooding recovery. SUBSAFE and the Navy’s Nuclear Power program are based on a foundation of quality in design, material, fabrication and testing. They span the submarine’s life, from initial design and construction, through ongoing maintenance and updates. Strict adherence to these programs has resulted in safe and successful missions for half a century.

But, for the most part, sailors don’t think about all the factors that went into the submarine’s design and construction. We are more focused on keeping it operating safely and efficiently.

Navy submarines always are in a hostile environment. In addition to facing the potential threat of attack from another nation, the ships also are completely surrounded by saltwater, and storms at sea can be extremely hazardous. Not to mention the explosive ordnance being carried onboard. When you depend on every system running smoothly to harvest your oxygen and freshwater and generally keep water out of the “people tank,” downtime must be hunted down and eliminated.

Fine-Tuning Maintenance Habits

We used a variety of maintenance practices to make sure everything worked at peak performance, including weekly and monthly preventive maintenance (PM) inspections and check-off sheets for routine items such as oil filter changes or motor-generator brush repair.

The principle here was to do maintenance at a specific interval, whether or not it really needed to be done. When set up properly, this can be an effective, though conservative (and potentially wasteful) approach to doing maintenance.

However, there was no guarantee that doing the weekly or monthly PM would prevent something from going wrong between the cycles, resulting in a system failure. So, we added another feature into our daily operations — ongoing data collection and trend analysis. This essentially is how predictive maintenance works in industry today — continuously evaluating how your equipment is working so you know when and how to maintain it optimally.

We had a team of 10 trained personnel who were responsible for operating the reactor plant, electrical and propulsion systems, 24 hours a day while we were underway. This team worked together in six-hour shifts (watches). When an issue or complex process arose, this team was there to address it, often with the help of another team of 10 people that were “off-duty.”

This team would record the readings of all of the hundreds of gauges, dials and indicators throughout the engine room at least every hour, and, in some cases, every 15 minutes. The watch team supervisors were expected to review these log readings every three hours, and the Division Officer (DO) and Engineering Officer (ENG) would review them daily. At first, this vast collection of numbers was overwhelming, but with experience, the team gained a sense of what was normal and quickly could identify readings that signaled something amiss.

A key example of this is the bearing temperatures and oil flow bubblers of the main reduction gears. The reduction gears take the power generated by steam turbines turning at thousands of RPM and use it to drive the propulsion shaft at much slower speeds, which turns the screw and propels the submarine through the water.

These gears are machined to precise measurements and carry a lot of power. They are massive and only can be replaced in an extended dry-dock period, requiring cutting open the hull of the submarine. They are lubricated and cooled in a continuous flow of purified oil. If something were to happen to contaminate or stop the oil flow, the gears could break or seize together, effectively shutting down the submarine’s propulsion system and rendering us “dead in the water.”

Dozens of bubbler sight glasses on the reduction gears visually show the oil flow and its condition. They are recorded every hour, along with bearing temperatures, to see that the reduction gears are being lubricated properly at all times. Using trend analysis to find a problem in this system before it becomes downtime could determine whether or not we make it back home.

Diligence Pays Off

What was the point of writing down and reviewing all these numbers? Multiple things came about through this practice:

1. The operator writing the numbers down has to look at the gauges every hour (or 15 minutes). This means that someone is paying attention. If the numbers are way out of their expected positions, the operator would see that something is wrong before a long period of time passes, enabling a quick response.
2. The supervisors are verifying that this is being done, but, more important, they are looking at trends. What is changing over the course of the six-hour watch period or even the past day? Is a bearing temperature rising slowly? Is something changing with the electrical system? Is pressure in the reactor coolant system changing? Is the temperature in the condenser rising? Do these changes match what is expected based on changing plant conditions?
3. The DO and ENG are looking at longer trends to see that the plant is operating normally. Additionally, regular analysis is done on the chemistry of the reactor coolant and radiation readings throughout the ship to verify that nothing unusual is happening in the reactor vessel.
4. We used this data collection as a training tool, as well. Occasionally, the ENG would replace normal log sets with artificial ones (for a drill or test) that had anomalies or other indications that something was happening, and then the watch team would be evaluated on their ability to detect the problem and respond in the correct way.

Predictive Maintenance Continues to Get Smarter

While this was a very manual process for collecting and analyzing data, it’s a rigorous and detailed system to evaluate what is going on in the entire engine room. While some of this labor-intensive version of predictive maintenance is being replaced with digital technology, the core principles remain unchanged.

Just as a case can be made for predictive maintenance on life-critical systems such as those on a nuclear submarine, modern manufacturing facilities also need this capability. For decades, companies have introduced process improvements like Lean Six Sigma and Just-in-Time manufacturing to streamline their operations, improve quality and minimize expenses, thus becoming more efficient and profitable.

But maintenance always has lagged behind operations in receiving improvements. Many major manufacturers still rely on scheduled maintenance or run-to-failure approaches to managing maintenance. This strategy can result in wasting maintenance resources doing unnecessary repairs or in significant costly downtime when unexpected failures occur.

Only recently have companies started to consider smarter ways of doing predictive maintenance. The strongest trend in this area is to use data to develop indications and predictions of failure before they occur, which is exactly what we were doing with the data logs on the submarine.

Today this is done with sensors linked to an analytics platform, which often resides in the cloud or local servers. This data then can be evaluated and used to identify changes or anomalies that could be precursors to failures. This linking of data and machines is what is being referred to as the Industrial Internet of Things (IIoT) or Industry 4.0.

Technology now has reached a point at which remote condition monitoring and analysis can be done affordably and reliably. Rather than having an army of personnel recording gauge readings manually every hour and taking the time to study the data looking for trends, now this can be done with sensors, machine learning and analytics.

More advanced sensors exist that detect changes in vibration, indicating early symptoms of a future problem. These devices also can communicate their findings directly to anyone who needs to know and can take appropriate actions. Computing power continues to grow by leaps and bounds, resulting in better and faster analysis and decision-making.

It’s been more than 20 years since I deployed on a nuclear submarine, but I imagine the U.S. Navy is taking advantage of this new technology and data to make smarter, timelier decisions in maintaining and operating their equipment. The same benefits can be applied to manufacturing plants. Predictive maintenance will bring huge dividends for everyone, even those who don’t live on a submarine.

Encompass™ Product Partner Grace Engineered Products, Inc., Davenport, Iowa, provides personnel safety products including programming interface ports for controllers, SafeSide permanent electrical safety devices and IR viewing windows.

The Journal From Rockwell Automation and Our PartnerNetwork™ is published by Putman Media, Inc.