Tale of Two Power Systems – UPS Edition

This one nearly fooled us; we recalled the “two power systems” nature of a recent site and so when a second data set came in with somewhat similar characteristics, we thought it might be more data from the same facility. But this is a completely different site, and a completely different problem!

Looking over a power monitoring data set recently; we came across a site with a dual personality. The site in question had a marginally high total harmonic distortion (THD ~ 5.5%) from 5/15/17 up until 5/24/17 (specifically, at 3:50 pm). After that, the THD trended much higher, rising as high as 12% (with a large amount of visible high frequency noise).

A “before / after” look at the voltage and current waveforms provides more evidence, with visible notching on the voltage waveform “before”; and very high levels of high frequency noise (broad spectrum “after”.

Before 5/24/17 After 5/24/17

Individual harmonics similarly supported the findings of the THD log, with harmonics under 3% before 5/24/17, and very high harmonics across the spectrum after 5/24/17.

Before 5/24/17 After 5/24/17


Finally, the “before / after” affect was also seen in the Neutral-Ground voltage – with severely high NG voltages after 5/24/17; consisting mainly of high frequency components.

Before 5/24/17 After 5/24/17

The clue to understand this puzzle is that the RMS voltage of the device under test was very stable and well regulated (probably a UPS or power conditioner output) before the 5/24/17 date, and higher / less well regulated after the 5/24/17 date.

And here is the “moment of truth” when the voltage changes from moderately distorted / notched to severely distorted with high frequency noise.

Moment of Truth

So what’s the scoop? We’re not on site, but here’s our bet – that there is a UPS supplying 480Y/277 VAC power to the load, but is itself being fed 480 VAC Delta (ungrounded and/or no neutral). During Inverter operation, the UPS works fairly well (although we bet the notching and higher THD are not normal for this device). But when switched to Bypass, the load loses the neutral reference, and is picking up noise from the UPS rectifier and/or inverter circuitry.

The trip report notes “No power problem suspected, multiple tube failures, want to eliminate power as an issue.” They probably assume UPS installed = no power issues (and they are probably not experiencing sags / swells / etc. on other facility loads). But if they are operating a system requiring 480Y/277 VAC from 480 VAC Delta, and relying on the UPS to provide a neutral connection point, they are probably having some serious grounding, reference, and noise issues!

UPS Overload and Bypass: CT Scanner Load

A quick consulting project came over the transom this week. A 150 KVA UPS, protecting a CT scanner, was occasionally overloading and transferring to bypass.

UPS Bypass 02

Here, the transition to Bypass is evident by the step change in voltage from a rock solid 480 VAC (UPS Inverter) to a very high 515 VAC (Bypass)

UPS Bypass 01

Drilling in a bit more, we see the CT Scanner switch on (point “A”) with a maximum current of 245 Amps and a resultant collapse of the UPS output, a short period where the CT current drops and the UPS output stabilizes, then a transition to Bypass (point “B”). Note the increase in voltage while operating on Bypass.

At the end of the CT scan (point “C”) the voltage rises due to impedance. And the UPS stays in Bypass for an extended period (point “D”) needing to be manually reset.

UPS Bypass 03

A close-up of the “start of scan” waveform shows the nature of the inrush current (higher for just once cycle) – although the UPS voltage drops more than usual, it does not really fold or collapse.

Nothing really unusual here – some finger pointing at impedance (not really an issue, the voltage drop on the unregulated bypass was just 2.7% at full load) and voltage distortion (under 3% voltage distortion on the UPS input) – neither of which is a problem. The UPS got sized in based on power monitoring, which apparently did not capture peak load condition.

I suggested that the higher voltage on Bypass (515 VAC = 7% higher than nominal) would mean lower observed current, although that did not factor into the calculations (they were monitoring further upstream on a 208 VAC source). The UPS vendor is going to see if they can tweak the protection circuitry a bit to be able to survive and supply this short overload without a bypass transition.

Support Your Independent Power Quality Consultant (or Pay a Lot More)

Came across this one on a Linked In power quality board.

*************************************************
Skiers blowing up UV light bulbs

I am looking for a small three phase voltage regulator/ power conditioner to fix utility voltage fluctuation problem that is blowing expensive UV lamps in the water purification system in a small town. You see, the water filtration facility ( or shack ) is connected to the grid at the end of a long utility line; and it’s closest neighbor is a ski hill. When the sky lifts; or the pumps for the snow making machines start/ stop, the 600V gets high voltage spikes of up to 640V. This is blowing up the UV bulbs.

I need to replace the existing auto transformer with a 600V – 3 wire input, and a 277V – 3 wire output power conditioner/ voltage regulator. Probably will need something in the 10-15kVA range.

Open Delta Autotransformer

Any ideas for good quality US or Canadian manufacturer?

*************************************************

Spoiler Alert: odds are pretty good that this site does not need a “power conditioner/ voltage regulator”. The culprit here is almost certainly the transformer used to convert 600 VAC (ph-ph) to 277 VAC (ph-n) for the lighting circuits. They are using autotransformers hooked up in an Open Delta configuration – no doubt something like this:

Open Delta Wiring

Problems:

  1. The voltages from X1 / X2 / X3 to neutral are not going to be balanced or particularly stable – they will definitely change with load. X3-N will in fact be 347 VAC. The other phases will be all over the place, and will fluctuate with load.
  2. The voltages from X1-X3 and X2-X3 will be fairly stable (direct magnetic coupling) but X1-X2 (the open delta phase) will fluctuate.

I’d probably recommend measuring the X1 / X2 / X3 voltages (Ph-Ph, Ph-N, and Ph-G) and when they come up unbalanced, I’d know where the problem was.

Now a dozen or so power quality experts (with a product to sell and a commission to accrue) had lots of suggestions for power conditioning or voltage regulation solutions. These might fix the problem, but at a much higher cost / complexity.

The best solution is mostly likely a simple Delta-Wye isolation transformer, 600 VAC primary, 480Y/277 VAC secondary.

Delta Wye Wiring

The problem is – nobody gets much of a commission on a 15 KVA isolation transformer (maybe a $1500 box plus installation, breakers, etc.) so who wants to recommend THAT?

Your friendly neighborhood independent power quality consultant, that’s who . . .

Reading the Tea Leaves

Sometimes when reviewing power monitoring data, key information is left out of the problem statement. But the astute power quality engineer can “read the tea leaves” and pick up information about the installation, equipment, and technical issues.

A set of data from a Magnetic Resonance Imaging system was presented for analysis with the following notes:

System has had several intermittent issue that have caused system to be down and functional upon arrival. System issues have been in RF section. No issues have been reported since system since installation of power recorder.

RMS Voltage and Current

Chiller RMS Logs

First clues come from looking at the RMS logs of the voltage and current. The voltage is suspiciously well regulated – and is probably a UPS or power conditioner rather than a normal utility source (which will tend to fluctuate over a 24 hour period). A second clue is the small voltage increase or swell related to load switch-off – typical of an active source, not typical of a passive source.

Second, this appears to be a regularly cycling load – a pump or compressor. MRI systems typically have a chiller or cryogen cooler associated with them – so odds are good this was monitored on this load, and not on the MRI system itself.

Chiller or Cryogen Cooler Load

Chiller Cycling RMS
More evidence supporting the chiller or cryogen cooler load – a regular (practically like clock-work) cycling load, with a marginally higher operating current (~30 Amps), but a very high inrush current (~180 Amps)

Chiller Transient

Normal chiller or cryogen cooler inrush is seen here. A minor (~5%) voltage sag was captured during each inrush current, as well as minor associated transients (probably relay or contactor switch bounce)

Abnormally High Inrush Currents

Chiller Sag

In addition to the regular inrush currents associated with chiller cycling, six instances of very high inrush current were captured. These were seen both as voltage sag events as well as current triggered events. We’re concerned that this high inrush current may be causing an overcurrent condition on the UPS / power conditioner – which may be throwing a fault or error, or perhaps switching to Bypass.

Chiller Swell RMS

Looking at the RMS logs of the high current swell / voltage sag event, we see that it precedes a period of extended chiller / compressor operation. Unknown if this is normal operation for the system / device or indicates an error or fault of some sort.

Summary

Although the accompanying technical information was thin, we’ve “read the tea leaves” and provided the following analysis bullets

  • System appears to be powered by a UPS or power conditioner. Service personnel may not have known this.
  • System appears to be a chiller, compressor, or similar device (not the medical imaging system itself)
  • Occasional high current swells were seen; these may be normal or may point to system issues.
  • Voltage sags and collapse during these high current swells may indicate that the UPS or power conditioner is overloaded, and may be experiencing faults or alarms that may be impacting system operation / uptime

The Curious Case of the UPS Loading

We recently got to review input and output monitoring data from a UPS system (make and model not specified) feeding a medical imaging system. The monitoring was done as a precaution, but we noticed something unusual.

First, take a look at the RMS voltage and current logging of the UPS input and output. Phase A voltage, Phase B current shown for clarity, but all voltage and current phases are balanced and similar.

UPS Compare Input RMS

UPS Input – Normal facility RMS voltage (daily fluctuations, with occasional sags) and RMS current peaks at approximately 80 Amps.

UPS Compare Output RMS

UPS Output – Highly regulated RMS voltage (with small load related fluctuations) and RMS current peaks at approximately 170 Amps.

The discrepancy between the input current and output current is unusual. It would be typical for input current to be marginally higher than output current (due to device efficiencies) but not lower. Our guess – the UPS DC bus (and probably, the battery string) is being called on to support the peak output load.

UPS Compare Output Highest Load

UPS Output – Step change in load current and nonlinear load is typical of medical imaging system. Very small fluctuation in output voltage related to load changes, and small increase in voltage distortion related to nonlinear load current.

UPS Compare Input Highest Load

UPS Input – Even at highest levels, current is linear, UPS must have a unity power factor front end / rectifier. However, lower current level is unusual, and indicates that UPS battery is probably being called on to supply the peak medical imaging load.

There’s really no immediate problem here – the UPS is doing a great job of correcting input power issues, as well as supplying the complex loads (step change, pulsing currents, nonlinear power factor) of the medical imaging system.

However,it’s pretty clear that the UPS batteries are getting discharged during highest current imaging system operations – not really their intended purpose, which is to ride through far less frequent utility sags and outages. So it’s possible that the UPS batteries are being stressed and may degrade or fail prematurely, and need replacement. We’ve referred this to the UPS manufacturer / supplier for attention.

As a quick “in the field” test (we’re doing this analysis remotely, not on site) we might suggest disconnecting the battery string temporarily, and seeing how the UPS performs without the battery, just relying on the DC bus. We’re guessing the UPS might start to collapse or struggle to supply the medical imaging load – and may be undersized for the application without the battery string supplied.

We’ve seen situations where a UPS that has heretofore worked well for years stops working quite so well, because the batteries started to wear out, and the unit was no longer able to supply the peak loads required by the imaging system.

Calculating Voltage Imbalance

Working on a customer site with Fluke 1750 data at the moment. The customer notes “Error shows udc voltage out of tolerance between stationary and rotating portions of the gantry.” – problems with the main DC bus voltage. Made me think it might be voltage imbalance, causing high DC bus ripple.

Voltage Imbalance Chart 1

Looking at the Fluke 1750 voltage imbalance chart shows a maximum imbalance of 1.1344%.

Voltage Imbalance Chart 2

Looking at the individual RMS voltage measurements, the RMS values are Ph A = 281.34, Ph B = 281.715 and Ph C = 276.725. Voltage imbalance seemed a little high to me; so I decided to double check the Fluke 1750 voltage imbalance.

There’s a good article on calculating voltage imbalance on the ACHRNews (Air Conditioning / Heating / Refrigeration News) Website, here – Three-Phase Motor Voltage Unbalance – specifically, calculate the average of the three phases, take the maximum deviation from average (of the three phases), and divide by the average.

In this case, the average voltage is 279.927 Vrms, and the maximum deviation is Phase C, 279.927 – 276.725 = 3.202 Vrms, Voltage Imbalance = 3.202 / 279.927 = 1.144%.

Close enough. The customer requirement is 2%, so while this imbalance looks a bit large (nearly 5 Vrms between Phase C and the other phases), it seems to meet the requirements.

Losing the Neutral Conductor

This one came across my social media timeline this morning (edited a bit):


I came home on Friday, an hour before we had a birthday party planned. There was a cable company guy who came over and asked if I was the home owner. He did not explain the problem very clearly and became very frustrating but in short, he saved our house from burning down.

He shut down the Internet and told us to shut down the electricity. Apparently the neutralizing wire that runs under ground was not working causing brown outs and power shortages. The smell of electrical fire was heavy in the house.

We managed to have a great party despite the problems. The output caused a shortage in the hot tub and pool. We have no refrigeration or dishwasher along with a few other things that burned out. Last night, we found a power strip that had really burned out with burn marks on the floor. As he moved it, the same electric burn smell filled the room.

Through it all God spared us big. We are still without a refrigerator but at least the stove works.


What happened was that this home lost the neutral conductor from the utility to the service entrance. Without that neutral, there’s no return path except for the safety ground, which is often substandard or high impedance (~25 ohms). The result: Phase-Phase voltages (such as used for an electric stove, water heater, or electric dryer) are fine, but Phase-Neutral voltages can be anywhere from 0 VAC to 240 VAC.

So yes, things blow up, burn, etc. and often in a bad way (high current but not a dead short, so not enough to trip breakers). The “power strip with burn marks on the floor” is typical as internal surge suppressors / MOVs overheat, not because of short term transients, but because of prolonged, sustained AC overvoltage.

Oftentimes this sort of situation has some warning signs: lights dimming or brightening as appliances switch on and off, light bulbs failing prematurely. One online board reports:


When i turned the oven on, the fan went back to normal, the lights normal.  The 240v load
apparently balanced the system.


Sadly, a lot of electricians and utility workers are not that well versed in this sort of issue. From the same message board:


So i get on the horn with the power company.  They come out, and basically look at what i’m experiencing and the first thing the guy does is pull the meter.  Then he measures the voltages on the incoming legs.  All is equal.  Then he tells me the problem must be on the inside.  Puts the meter back in and the imbalance returns.  “yep , he says, problem is on your side”.


 

System Down or Ride Right Through?

We recently reviewed a set of power monitor data from an MR (Magnetic Resonance) site. The facility was plagued by severe voltage sags; we ended up with a rather copious collection of classic but nonetheless ugly event waveforms. And in the course of analysis, we noticed that some sags caused system shut-down, some rode right through, and some perhaps caused an error or lock-up which the customer attempted to reset by powering down the system.

Reviewing equipment response to severe events in this way can help to calibrate system sensitivity when manufacturer or factory data about sag susceptibility is not available.

Example #1: System Rides Through Voltage Sag

Sag RMS No Shut Off

Despite a fairly serious sag, no sign of direct impact on the imaging system. Current levels shift during the sag event itself, but remain at about the same level before and after the sag.

Sag Waveform No Shut OffExample #2: System Shuts Down During Voltage Sag

Sag RMS System Down

At the time of a severe voltage sag, load current drops to a lower, standby or system-off level, and remains there.

Sag Waveform System DownSag Waveform Customer Shut OffExample #3: System Shuts Down During Second Voltage Sag

Sag RMS System Down Second Sag

During these sag events, the system appears to ride through a severe voltage sag; but shuts down during a subsequent sag 15 seconds or so after the initial sag event.

Sag Waveform System Down Second SagSag Waveform System Down First Sag

Example #4: System Current Drops Following a Voltage Sag; Customer Shuts System Down

Sag RMS Customer Shut Off

Following a severe sag event, current drops partially. Suspect that one or more subsystems shut-down and resulting system alarm or errors results in customer shutting down the system, 30 seconds after the sag event. Note drop in current not directly related to a voltage event.Sag Waveform Customer Shut Off

Power Quality as a Whipping Boy

Every so often I get supporting info with a power audit that reads something like this:

“Inordinate number of issues occurring with equipment as compared to other sites and systems. Everyone is in agreement the power sags, but questions if this has an adverse affect on the equipment.”

True Confession: It gives me a little bit of pleasure to find not a single facility or utility voltage sag in the resulting data. Power quality becomes, at times, the convenient excuse for equipment problems that are actually rooted in operator error, inadequate design, improper installation, inadequate servicing, environmental conditions, etc. When people confidently sling around power quality as the one known reason for problems, it’s almost always a good sign there’s something else going on.

Parkview Mains

Facility / utility voltage. We see a local outage, a small drop in voltage during emergency power system testing, and voltage flicker. But no serious voltage sags.

In this case, associated monitoring on the output of a UPS system showed perhaps the real problem – severe voltage sags associated with load switch-on. The UPS is either undersized for the applied load or in need of adjustment or maintenance. Yet another case of the “solution” being part of the problem.

Parkview UPS

UPS Output. Minor voltage drop during equipment operation, and severe voltage sag related to switch-on / inrush, points to a UPS that is undersized or in need of adjustment or maintenance.

Parkview Inrush

Load inrush current, with visible collapse of UPS output.

 

All Better!

Recall the case back in December when we identified a power conditioner that had apparently gone bad? When Power Conditioners Go Bad (5Dec15)

We received a follow-up power audit this week from the same site. Apparently, the power conditioner or UPS has been replaced or repaired. All better! We don’t often get to see such a full life cycle (good power → bad power → good power) but when we do, it feel good to be part of the solution!

Hannibal SnapshotVoltage waveforms are sinusoidal and well balanced, under all load conditions

Hannibal RMSRMS voltages are very stable and well regulated, with small load related fluctuations

Hannibal THDVoltage THD is low throughout the monitored period, under all line and load conditions