Tale of Two Power Systems – UPS Edition

This one nearly fooled us; we recalled the “two power systems” nature of a recent site and so when a second data set came in with somewhat similar characteristics, we thought it might be more data from the same facility. But this is a completely different site, and a completely different problem!

Looking over a power monitoring data set recently; we came across a site with a dual personality. The site in question had a marginally high total harmonic distortion (THD ~ 5.5%) from 5/15/17 up until 5/24/17 (specifically, at 3:50 pm). After that, the THD trended much higher, rising as high as 12% (with a large amount of visible high frequency noise).

A “before / after” look at the voltage and current waveforms provides more evidence, with visible notching on the voltage waveform “before”; and very high levels of high frequency noise (broad spectrum “after”.

Before 5/24/17 After 5/24/17

Individual harmonics similarly supported the findings of the THD log, with harmonics under 3% before 5/24/17, and very high harmonics across the spectrum after 5/24/17.

Before 5/24/17 After 5/24/17


Finally, the “before / after” affect was also seen in the Neutral-Ground voltage – with severely high NG voltages after 5/24/17; consisting mainly of high frequency components.

Before 5/24/17 After 5/24/17

The clue to understand this puzzle is that the RMS voltage of the device under test was very stable and well regulated (probably a UPS or power conditioner output) before the 5/24/17 date, and higher / less well regulated after the 5/24/17 date.

And here is the “moment of truth” when the voltage changes from moderately distorted / notched to severely distorted with high frequency noise.

Moment of Truth

So what’s the scoop? We’re not on site, but here’s our bet – that there is a UPS supplying 480Y/277 VAC power to the load, but is itself being fed 480 VAC Delta (ungrounded and/or no neutral). During Inverter operation, the UPS works fairly well (although we bet the notching and higher THD are not normal for this device). But when switched to Bypass, the load loses the neutral reference, and is picking up noise from the UPS rectifier and/or inverter circuitry.

The trip report notes “No power problem suspected, multiple tube failures, want to eliminate power as an issue.” They probably assume UPS installed = no power issues (and they are probably not experiencing sags / swells / etc. on other facility loads). But if they are operating a system requiring 480Y/277 VAC from 480 VAC Delta, and relying on the UPS to provide a neutral connection point, they are probably having some serious grounding, reference, and noise issues!

UPS Overload and Bypass: CT Scanner Load

A quick consulting project came over the transom this week. A 150 KVA UPS, protecting a CT scanner, was occasionally overloading and transferring to bypass.

UPS Bypass 02

Here, the transition to Bypass is evident by the step change in voltage from a rock solid 480 VAC (UPS Inverter) to a very high 515 VAC (Bypass)

UPS Bypass 01

Drilling in a bit more, we see the CT Scanner switch on (point “A”) with a maximum current of 245 Amps and a resultant collapse of the UPS output, a short period where the CT current drops and the UPS output stabilizes, then a transition to Bypass (point “B”). Note the increase in voltage while operating on Bypass.

At the end of the CT scan (point “C”) the voltage rises due to impedance. And the UPS stays in Bypass for an extended period (point “D”) needing to be manually reset.

UPS Bypass 03

A close-up of the “start of scan” waveform shows the nature of the inrush current (higher for just once cycle) – although the UPS voltage drops more than usual, it does not really fold or collapse.

Nothing really unusual here – some finger pointing at impedance (not really an issue, the voltage drop on the unregulated bypass was just 2.7% at full load) and voltage distortion (under 3% voltage distortion on the UPS input) – neither of which is a problem. The UPS got sized in based on power monitoring, which apparently did not capture peak load condition.

I suggested that the higher voltage on Bypass (515 VAC = 7% higher than nominal) would mean lower observed current, although that did not factor into the calculations (they were monitoring further upstream on a 208 VAC source). The UPS vendor is going to see if they can tweak the protection circuitry a bit to be able to survive and supply this short overload without a bypass transition.

Support Your Independent Power Quality Consultant (or Pay a Lot More)

Came across this one on a Linked In power quality board.

*************************************************
Skiers blowing up UV light bulbs

I am looking for a small three phase voltage regulator/ power conditioner to fix utility voltage fluctuation problem that is blowing expensive UV lamps in the water purification system in a small town. You see, the water filtration facility ( or shack ) is connected to the grid at the end of a long utility line; and it’s closest neighbor is a ski hill. When the sky lifts; or the pumps for the snow making machines start/ stop, the 600V gets high voltage spikes of up to 640V. This is blowing up the UV bulbs.

I need to replace the existing auto transformer with a 600V – 3 wire input, and a 277V – 3 wire output power conditioner/ voltage regulator. Probably will need something in the 10-15kVA range.

Open Delta Autotransformer

Any ideas for good quality US or Canadian manufacturer?

*************************************************

Spoiler Alert: odds are pretty good that this site does not need a “power conditioner/ voltage regulator”. The culprit here is almost certainly the transformer used to convert 600 VAC (ph-ph) to 277 VAC (ph-n) for the lighting circuits. They are using autotransformers hooked up in an Open Delta configuration – no doubt something like this:

Open Delta Wiring

Problems:

  1. The voltages from X1 / X2 / X3 to neutral are not going to be balanced or particularly stable – they will definitely change with load. X3-N will in fact be 347 VAC. The other phases will be all over the place, and will fluctuate with load.
  2. The voltages from X1-X3 and X2-X3 will be fairly stable (direct magnetic coupling) but X1-X2 (the open delta phase) will fluctuate.

I’d probably recommend measuring the X1 / X2 / X3 voltages (Ph-Ph, Ph-N, and Ph-G) and when they come up unbalanced, I’d know where the problem was.

Now a dozen or so power quality experts (with a product to sell and a commission to accrue) had lots of suggestions for power conditioning or voltage regulation solutions. These might fix the problem, but at a much higher cost / complexity.

The best solution is mostly likely a simple Delta-Wye isolation transformer, 600 VAC primary, 480Y/277 VAC secondary.

Delta Wye Wiring

The problem is – nobody gets much of a commission on a 15 KVA isolation transformer (maybe a $1500 box plus installation, breakers, etc.) so who wants to recommend THAT?

Your friendly neighborhood independent power quality consultant, that’s who . . .

Reading the Tea Leaves

Sometimes when reviewing power monitoring data, key information is left out of the problem statement. But the astute power quality engineer can “read the tea leaves” and pick up information about the installation, equipment, and technical issues.

A set of data from a Magnetic Resonance Imaging system was presented for analysis with the following notes:

System has had several intermittent issue that have caused system to be down and functional upon arrival. System issues have been in RF section. No issues have been reported since system since installation of power recorder.

RMS Voltage and Current

Chiller RMS Logs

First clues come from looking at the RMS logs of the voltage and current. The voltage is suspiciously well regulated – and is probably a UPS or power conditioner rather than a normal utility source (which will tend to fluctuate over a 24 hour period). A second clue is the small voltage increase or swell related to load switch-off – typical of an active source, not typical of a passive source.

Second, this appears to be a regularly cycling load – a pump or compressor. MRI systems typically have a chiller or cryogen cooler associated with them – so odds are good this was monitored on this load, and not on the MRI system itself.

Chiller or Cryogen Cooler Load

Chiller Cycling RMS
More evidence supporting the chiller or cryogen cooler load – a regular (practically like clock-work) cycling load, with a marginally higher operating current (~30 Amps), but a very high inrush current (~180 Amps)

Chiller Transient

Normal chiller or cryogen cooler inrush is seen here. A minor (~5%) voltage sag was captured during each inrush current, as well as minor associated transients (probably relay or contactor switch bounce)

Abnormally High Inrush Currents

Chiller Sag

In addition to the regular inrush currents associated with chiller cycling, six instances of very high inrush current were captured. These were seen both as voltage sag events as well as current triggered events. We’re concerned that this high inrush current may be causing an overcurrent condition on the UPS / power conditioner – which may be throwing a fault or error, or perhaps switching to Bypass.

Chiller Swell RMS

Looking at the RMS logs of the high current swell / voltage sag event, we see that it precedes a period of extended chiller / compressor operation. Unknown if this is normal operation for the system / device or indicates an error or fault of some sort.

Summary

Although the accompanying technical information was thin, we’ve “read the tea leaves” and provided the following analysis bullets

  • System appears to be powered by a UPS or power conditioner. Service personnel may not have known this.
  • System appears to be a chiller, compressor, or similar device (not the medical imaging system itself)
  • Occasional high current swells were seen; these may be normal or may point to system issues.
  • Voltage sags and collapse during these high current swells may indicate that the UPS or power conditioner is overloaded, and may be experiencing faults or alarms that may be impacting system operation / uptime

The Curious Case of the UPS Loading

We recently got to review input and output monitoring data from a UPS system (make and model not specified) feeding a medical imaging system. The monitoring was done as a precaution, but we noticed something unusual.

First, take a look at the RMS voltage and current logging of the UPS input and output. Phase A voltage, Phase B current shown for clarity, but all voltage and current phases are balanced and similar.

UPS Compare Input RMS

UPS Input – Normal facility RMS voltage (daily fluctuations, with occasional sags) and RMS current peaks at approximately 80 Amps.

UPS Compare Output RMS

UPS Output – Highly regulated RMS voltage (with small load related fluctuations) and RMS current peaks at approximately 170 Amps.

The discrepancy between the input current and output current is unusual. It would be typical for input current to be marginally higher than output current (due to device efficiencies) but not lower. Our guess – the UPS DC bus (and probably, the battery string) is being called on to support the peak output load.

UPS Compare Output Highest Load

UPS Output – Step change in load current and nonlinear load is typical of medical imaging system. Very small fluctuation in output voltage related to load changes, and small increase in voltage distortion related to nonlinear load current.

UPS Compare Input Highest Load

UPS Input – Even at highest levels, current is linear, UPS must have a unity power factor front end / rectifier. However, lower current level is unusual, and indicates that UPS battery is probably being called on to supply the peak medical imaging load.

There’s really no immediate problem here – the UPS is doing a great job of correcting input power issues, as well as supplying the complex loads (step change, pulsing currents, nonlinear power factor) of the medical imaging system.

However,it’s pretty clear that the UPS batteries are getting discharged during highest current imaging system operations – not really their intended purpose, which is to ride through far less frequent utility sags and outages. So it’s possible that the UPS batteries are being stressed and may degrade or fail prematurely, and need replacement. We’ve referred this to the UPS manufacturer / supplier for attention.

As a quick “in the field” test (we’re doing this analysis remotely, not on site) we might suggest disconnecting the battery string temporarily, and seeing how the UPS performs without the battery, just relying on the DC bus. We’re guessing the UPS might start to collapse or struggle to supply the medical imaging load – and may be undersized for the application without the battery string supplied.

We’ve seen situations where a UPS that has heretofore worked well for years stops working quite so well, because the batteries started to wear out, and the unit was no longer able to supply the peak loads required by the imaging system.

Philips Power Solutions

I was the national Power Quality Engineer at Philips’ North American HQ circa 1989-1995 and laid the groundwork for a lot of this. So this is interesting to see.

Overall, pretty well done. The “lousy power” waveforms are a little unrealistic / conceptual (I’d have insisted it was a little more realistic) but they hit all the high points. Think they futzed up the isolation transformer based UPS technology a bit (the isolation transformer is unique, but does not provide the conversion from AC-DC which is pretty standard in all online UPS systems). Interesting to see Rx Monitoring Services reports spotlighted; a good partner choice if one is using power protection as a revenue source.

I never got a kick ass video for my power program, if I recall correctly…