The Problem

The controls to a 1200 amp 69kv rated SF6 Gas Circuit breaker stopped working. During a power outage resulting from a massive ground fault induced by lightning strike to an overhead line, operators were unable to close this critical breaker resulting in failure to restore power to a very profitable manufacturing plant. In a desperate move the breaker was finally manually closed by jumping out critical interlocks by supplying power directly to the close coil located inside the gas circuit breaker itself, in the outdoor substation yard.

The above scenario occurred several times in rapid succession over a month time period during a turbulent storm season. The result was close to a million dollars in lost revenue from half a day of total downtime during each occurrence, and one very upset and stressed management team with production budgets that were a wash for the month.

Some Background Information

The control relays for the breaker’s close circuit, trip circuit, and sync circuit are located in three physically different locations across a large industrial facility had have been virtually untouched since the original installation over 30 years ago with the exception of a new marshaling sync cabinet that contained interposing relays and selector switches for all facility gas circuit breakers to give operators better control and indication of all high voltage gas circuit breakers on site, about 5 in total.

The new cabinet was installed a year prior and was believed to have passed all standard electrical testing and check out associated with this type of equipment including testing of all open/close/sync circuit operations with their associated push buttons and hand switches.

Not only was the new cabinet tested, but it had also already been used successfully to both open and close this breaker several times in the past.

Clues Begun to Unravel

The breaker was able to be closed by manually applying power to the breaker close coil, so the first place to look was the circuit that applies power to this coil and determine what in this circuit was preventing this coil from becoming energized when the appropriate controls were applied.

This circuit contains thousands of feet of wire, with several dozens of interposing relays and auxiliary contacts, numerous power supplies, fuses, power management relays, and synchroscope as this was a co-generation facility that also produces electric power to the grid.

The first day of troubleshooting deemed that all relays, power supplies, and fuses were in working order and intact. Additionally, all wiring also checked out fine without any short circuits or failed insulation. All drawings and documentation that could be found, or that was known to exist, was located and gathered. All data was pulled from power management relays to look for unusual wave forms that might have been captured and to inspect the conditions of the ground fault that was initiating the trip logic of the protective relaying system. Lot’s of information, but not a lot of answers.

The Discovery

A drawing emerged that was covered in dirt and sweat with lightly penciled notes. It was a copy of the close circuit for one of the different facility gas circuit breakers located on site. Interposing relays for which, are also located in the same, newly installed marshaling cabinet. On this drawing was a set of normally open auxiliary contacts that are powered from the same DC power supply that feeds our misbehaving breaker’s close circuit. These contacts were circuited with a dated note that read “Found N.C., Corrected to N.O.” A normally open auxiliary contact had accidentally been installed normally closed and was apparently corrected to reflect the drawings.

We checked the date on this hand written notes against operational logs. It appeared that our misbehaving breaker operated just fine prior to this change, with the recent problems occurring after.

The Question

how could an auxiliary relay contact for a different breaker, located on a different rung, be preventing our breaker’s close circuit to energize properly?

Right around this same time, an additional discovery was made. A second party on the team reported locating an unintentional interlock that was created by a normally open auxiliary contact that was found to be wired in series with our misbehaving breaker’s control circuit, an auxiliary contact that was unrelated to our breaker and part of a different control scheme such that it could never be activated and would there for always hold up power. It was the same contact.

The first jump to conclusion is that it was obvious that whoever fixed the contact in the wrong position somehow unintentionally rewired the contact to be in series with the close circuit to our downtime causing breaker that could not be closed.

Obvious, right?

Not quite.

Mystery solved

Further inspection of the drawing and control scheme revealed this contact to be an Achilles heel that would only show itself once the contact position had been corrected.

We had two major errors. The contact was installed normally closed when it should have been installed normally open as designed AND it was wired in series when it should have been wired as a separate rung, also as designed.

While in it’s yet undiscovered normally closed state, the contact allowed current to flow into the unintentionally shorted control circuit to our beloved breaker and the wiring flaw in the form of an unintentional interlock was never discovered. When the state of the contact was found and corrected, it introduced the break in the close circuit that would never be able to be made from within the scheme of the breaker’s circuit.

Hindsight is Always 20/20

Since the controls to the plant breakers had been used successfully in the past from the marshaling sync cabinet, it was never assumed that it could have been wired incorrectly.

It was also discovered that the facility allowed testing of the controls to all of the breakers in the new marshaling cabinet except for one, which happened to be the tie breaker to the utility as they did not want to lose production for the test. The troublesome auxiliary contact belonged to this tie breaker, and most likely would have been discovered if the controls to the tie breaker had been allowed to be tested.

The tie breaker is located on the very edge of the facility and since it connects to the utility, the operational frequency of this breaker is dramatically less than any of the others.

It was discovered much later, that a similar problem arose in the middle of the night, in which this tie breaker could not be closed prior to any of the above mishaps. It was then when the troublesome contact that belonged to this breaker was discovered to be in the wrong state, and corrected, which introduced the unresolvable (until now) interlock into our control circuit. As this happened in the middle of the night, the finding and change was either never properly communicated, or followed up with.

Case Closed.

Enjoyed? Leave a comment.

Mysterious Failure of Gas Circuit Breaker Controls Results in a Million of Lost Production Dollars