Curious Voltage Regulation

Came across an unusual case of voltage regulation for a CT Scanner site recently.

Site voltage appears to be well (and artificially) regulated, with no daily voltage fluctuations typical of utility power but with small load related voltage drops. Site had no significant outages, sags, swells, or transients over two weeks of monitoring.
Typical voltage harmonics supports some sort of static, inverter based power source, with broad band harmonics. Elevated 5th, 7th, 11th and 13th harmonics are curious.
No apparent voltage regulation related to step change in load current – which is unusual for a static UPS or power conditioning device feeding a medical imaging load.
Moderate level of increased voltage distortion related to nonlinear load current – again, not typical for a medical imaging device powered directly from a properly sized UPS or power conditioner.

So, what’s going on?

My first guess was a slow, electro-mechanical voltage regulator (such as a motor driven Variac or Powerstat) and which shows up occasionally for medical imaging or other sensitive loads. However, that sort of regulator would not eliminate short utility sags or drop-outs, or affect the voltage harmonics or THD.

My next theory is that this site has a large UPS or static power conditioning device feeding an entire medical imaging department. The voltage drop and increased voltage distortion under load would be the result of voltage drops in distribution (impedance calculated to be roughly 60 mohms @ 480 VAC) – so perhaps the power conditioning device is located in the basement with normal distribution voltage drops (primarily wiring, perhaps a transformer) from there to the equipment.

No significant power issues found, just like to understand what I am seeing in the data!

The Shoemaker’s Children…

….go barefoot. An old proverb that speaks to those with particular skills and aptitude often taking care of their own stuff last. And usually the case with me!

I had a couple of basement outlets / lights go intermittent, I tracked it down to a particular outlet, and so far as a bad neutral (because the voltage tester was chirping on a “dead” outlet.

This noontime, I pulled a shelf away from the wall, pulled the outlet, and heard the familiar (and unsettling) crackle-crackle of arcing as a nearby light on the circuit blinked.

The sparky-monkey who wired the basement (no doubt the former owner) used those stupid / dangerous outlet holes to make connections (called “back-stabbing”) One neutral connection was high resistance, getting warm, and thankfully broke connection before anything caught fire.

I cut out the outlet, replaced with properly screwed down and wrapped around the terminal connections, and all better.

Best of all, since the circuit in question also powered my desktop computer, I got to listen to my UPS chirp while I made the swap (NASCAR-like) and returned the circuit to service with 12 minutes to spare (my UPS is oversized for the load, so lots of battery time).

Sneaky Voltage Swell Events

Long time, no post! A pandemic will do that to you . . . suffice it to say I am healthy, relatively happy, and staying busy with both my engineering work (mostly stable through the troubles) and my yoga studio work (very different these days but busier than ever with online classes and the need to provide appropriate technology).

Today’s engineering bon-bon involves a trio of voltage swell events. Seemingly caused by a short circuit (another facility load fault, perhaps utility lighting arrestors) that causes a 1/4 cycle drop-out on one phase and resultant voltage swell following. The swell has a serious overvoltage, likely to cause problems for many types of equipment / power supplies.

This event was captured as a minor RMS voltage swell (5.1% above nominal) – but the peak voltage is very high (540 V vs 395 V normal). Note the ~30 Amp peak current swell.
This event was barely detectable as an RMS voltage swell (4.8% above nominal) – once I recognized the nature of these events, I went looking for more and came up with just this one. Almost no resultant current swell.
This is the big smoking gun event. Even though it did not register as a sag / swell event at all (the voltage drop-out being balanced by the following swell, just 3.25% over nominal) it was captured as a current swell (the highest current event over 6 weeks of data capture) – 650 Amps waveform peak. Something within the client load saw this event as a problem and drew a slug of current.

I’m just the hired gun reviewing the data, pulling out events and issues, writing a report. So I won’t be following this to a resolution or further troubleshooting on site. But this is a good example why automated report writing is not always sufficient – the Fluke 1750 analysis tools would see these events as minor voltage swells (if that). It takes a human being with some experience (I’ve reviewed over 5000 power quality data sets since 2003) to see something unusual, scratch one’s head, and dig in a bit deeper.

Simplex Clock Correction Issues

I’m feeling a bit nostalgic this morning. Readers of a certain age will remember the classic Simplex clocks from school days. The clocks throughout the building had a special feature – a receiver and small control system that would permit the clocks to be synchronized or changed throughout the building, using a master control device. Useful to keep all the clocks at the same time, easily adjust for seasonal daylight savings time changes, and to reset the clocks in the event of power loss.

There’s a good overview of these systems here:

The Synchronous Wire system is the most popular system in the United States. Clocks are run using a power circuit that acts as its time base. The clocks receive an hourly correction which synchronizes both the minute and the second hand. Every 12 hours the clock receives a daily correction to keep the clock perfectly synchronized.

When I first started working in the medical imaging field (circa 1989), we’d run into issues with these clocks, a lot. Hospitals and health-care facilities were big users of these (there’s a clock in every patient room, hallway, and procedure room), and one particular piece of equipment (a Phillips “Classic” generator) was particularly susceptible. The generator used a motor-driven, linear variable autotransformer (think Variac or Powerstat) to adjust for line voltage changes – and the signal injected onto the mains by the clock controller (typically around 3500 Hz) would mess with the voltage regulation circuit, and the motor would “hunt” for the duration of the pulse (usually 5 or 10 seconds), The generator would be disabled or locked out while the motor was moving, and this would drive the docs and techs crazy (since it happened hourly).

Simplex Clocks

A waveform sample from a PowerLines trip report circa 2007, using Rx Monitoring Services power analyzer to capture the clock correction pulse.

 

I’ve also come across a few old power quality threads discussing these clock pulses causing standby / hybrid UPS systems (notoriously sensitive to anything that might indicate the start of an actual outage) to switch to inverter improperly.

Back in the day, the old BMI-4800 power analyzer had a “high frequency noise” detector which looked at broad spectrum harmonics or noise, and output a distinctive “picket fence” 24 hour log when these clocks were present (I still miss this diagnostic / reporting feature on modern power analyzers). I’m sure I have an example of this graph kicking around somewhere but can’t put my paws on it at the moment. I suspect any graphs I recall pre-date digital images (when I would create reports with blank boxes, and manually paste in photo-copied disturance graphs) so I’m not finding anything in trip reports or old PowerPoint presentations)

Resolving these issues? Sometimes we’d consult with the facility engineer – oftentimes these were turned up to “10” (maximum) and we could get the amplitude turned down to the point where it worked but did not cause problems. Sometimes we’d get the clocks reprogrammed to only correct 2x a day (noon and midnight) when it would be unlikely to affect the equipment. Some resourceful field techs developed a filter circuit to protect the regulation circuity; although that was sometimes not permitted (FDA requirements for x-ray equipment forbids modification or retro-fitting).

I don’t hear too much about these lately. Clocks are now often digital, controlled wirelessly or via ethernet. Switched-mode power converters have replaced old analog systems.

 

Cause and Effect: Arcing Transients and Equipment Faults

We recently reviewed some power monitor data for a client. Problem statement:

Breaker Q1 in the WCS electronic box trips on a sporadic basis. The breaker is the M4 and M5 fan motor overcurrent protection. We have replaced the breaker multiple times.

First pass, we noticed three very high current swells in the ground current data:

Ground RMS

We also saw 100s of very serious arcing voltage transients, not related to load current changes or other voltage events. The transients showed up on Phase-Neutral and Neutral-Ground, but the NG transients were much lower amplitude (secondary, not the primary issue).

Transient PhN Transient NG

Finally, we captured three current swell events that clearly show equipment faults to ground (notice the elevated ground current) immediately following voltage transients – cause and effect.

Fault1 Fault2 Fault3

Figuring out the cause of the transients is an exercise for the local service engineers or an onsite power quality engineer. But we’ve got a pretty clear linkage here between transients and equipment faults. Most of the time, power quality problems are a lot less concrete and clear.

Power Factor Correction Capacitors on a Timer

Came across an old friend this afternoon, low frequency transients related to utility or facility power factor capacitor switching, controlled via a timer (rather than sensing voltage, current, or power factor)

Minor PFC - Voltage Waveform

Here’s the voltage waveform – this seems to be a very minor transient, hardly worth noticing.

Minor PFC - Voltage and Current Waveform

Adding the current, we see a small ringing current related to the transient event. Oftentimes with more severe transients we see a large current swell.

Minor PFC - RMS Logs

The RMS voltage logs show a small but clear step increase in voltage at the time of the transient, clear sign of power factor correction capacitor switching.

Minor PFC - Table of Events

Finally, here is a table of all such transients captured. Notice how each transient occurs at 7:03 am, on different mornings. This sort of “same time every day” incident is a clear indication that the capacitor bank is on a timer control.

Not a super serious issue, this time, but interesting to come across a timer based system. They seem to be increasingly rare as more sophisticated controllers are brought online each year.

Sneaky Low Frequency Transients

Low frequency transients, sometimes called Utility Switching Transients or Power Factor Correction Capacitor Switching Transients, can be pretty hard to identify. Traditional power monitoring equipment has never done a particularly good job at spotting these – folks of a certain age will recall that the BMI-4800 power monitor would throw a frequency error (either 61.9 Hz or 64.0 Hz) if the transient caused an extra zero-crossing – sometimes that was the only way to detect the transient, and savvy engineers would use these frequency faults as a diagnostic tool.

Looking through a lot of Fluke 1750 data sets over the years (we’re looking at Site #4472 this week), we’ve gotten pretty good at pulling these transients out of the 100s or 1000s of transient events captured. Some detection tools:

  • Some transients do indeed trigger a voltage transient event, but need to be carefully reviewed because the reported magnitude is often that of the higher frequency leading edge
  • Many transients are accompanied by a rise in RMS voltage, so carefully adjusting the voltage swell threshold can often help to spot these.
  • In Wye systems, many transients cause a Neutral-Ground swell event, which can often be spotted.

In a recent data set; none of these indicators worked out. We were very fortunate that the first current event captured (with a current swell threshold set to 10 Amps, a typical threshold for our reports) was a transient event – so we happened to notice it.

Low Frequency Transient

Current Triggered Event #1

Then, identifying the duration of the current swell event (~ 17 msec, much shorter than the normal equipment loading) and the amplitude (between 15-20 Arms, normal equipment current swells were much higher) we were able to sort through 100s of current triggered events to find nine (9) low frequency transients in the data.

Low Frequency Table

Normally, we would not be so concerned about these transients, which are comparatively minor, simply looking at the voltage waveforms, with no significant overvoltage nor multiple voltage zero-crossings, However, the associated current swell (70-100 A peak) indicates something in the equipment under test is sensitive to or reacting to these transients, and drawing a slug of current. So they are worth looking into….

Low Frequency Transient 2

Current Triggered Event #646

Tale of Two Power Systems – UPS Edition

This one nearly fooled us; we recalled the “two power systems” nature of a recent site and so when a second data set came in with somewhat similar characteristics, we thought it might be more data from the same facility. But this is a completely different site, and a completely different problem!

Looking over a power monitoring data set recently; we came across a site with a dual personality. The site in question had a marginally high total harmonic distortion (THD ~ 5.5%) from 5/15/17 up until 5/24/17 (specifically, at 3:50 pm). After that, the THD trended much higher, rising as high as 12% (with a large amount of visible high frequency noise).

A “before / after” look at the voltage and current waveforms provides more evidence, with visible notching on the voltage waveform “before”; and very high levels of high frequency noise (broad spectrum “after”.

Before 5/24/17 After 5/24/17

Individual harmonics similarly supported the findings of the THD log, with harmonics under 3% before 5/24/17, and very high harmonics across the spectrum after 5/24/17.

Before 5/24/17 After 5/24/17


Finally, the “before / after” affect was also seen in the Neutral-Ground voltage – with severely high NG voltages after 5/24/17; consisting mainly of high frequency components.

Before 5/24/17 After 5/24/17

The clue to understand this puzzle is that the RMS voltage of the device under test was very stable and well regulated (probably a UPS or power conditioner output) before the 5/24/17 date, and higher / less well regulated after the 5/24/17 date.

And here is the “moment of truth” when the voltage changes from moderately distorted / notched to severely distorted with high frequency noise.

Moment of Truth

So what’s the scoop? We’re not on site, but here’s our bet – that there is a UPS supplying 480Y/277 VAC power to the load, but is itself being fed 480 VAC Delta (ungrounded and/or no neutral). During Inverter operation, the UPS works fairly well (although we bet the notching and higher THD are not normal for this device). But when switched to Bypass, the load loses the neutral reference, and is picking up noise from the UPS rectifier and/or inverter circuitry.

The trip report notes “No power problem suspected, multiple tube failures, want to eliminate power as an issue.” They probably assume UPS installed = no power issues (and they are probably not experiencing sags / swells / etc. on other facility loads). But if they are operating a system requiring 480Y/277 VAC from 480 VAC Delta, and relying on the UPS to provide a neutral connection point, they are probably having some serious grounding, reference, and noise issues!

Tale of Two Power Systems

Looking over a power monitoring dataset recently; we came across a site with a dual personality. The site in question had low total harmonic distortion (THD ~ 1.5%) from 4/1/17 up until 4/10/17 (specifically, at 2:00 pm). After that, the THD fluctuated much higher, rising as high as 5.4% (outside of manufacturer requirements for medical imaging equipment).

Tale2Power THD

A closer “before / after” look at the voltage and current waveforms provides more evidence, with visible notching on the voltage waveform “after”; monitored current showed some noise but was similarly low.

Tale2Power Waveforms Before Tale2Power Waveforms After
Before 4/10/17 After 4/10/17

Individual harmonics similarly supported the findings of the THD log, with all harmonics higher, and 5th harmonics exceeding 3%.

Tale2Power Harmonics Before Tale2Power Harmonics After
Before 4/10/17 After 4/10/17

Tale2Power NG RMS
Finally, the “before after” affect was also seen in the Neutral-Ground voltage – with noise voltages evident although the lower frequency voltages were not much higher.

Tale2Power NG Before Tale2Power NG After
Before 4/10/17 After 4/10/17

The funny thing is that the RMS voltage and the current of the device under test were not significantly different before / after the 4/10/17 date, in terms of RMS level or in terms of stability: sags, swells or fluctuations.
Tale2Power RMS

So what’s the scoop? We’re not on site, but odds are good that some facility load (we’re betting air conditioning, but could be other facility loads) got switched on at this time. Alternately, perhaps the facility transitioned to an alternative power source. But whatever the reason, this is clearly a tale of two power systems, and we’re curious about it!

UPS Overload and Bypass: CT Scanner Load

A quick consulting project came over the transom this week. A 150 KVA UPS, protecting a CT scanner, was occasionally overloading and transferring to bypass.

UPS Bypass 02

Here, the transition to Bypass is evident by the step change in voltage from a rock solid 480 VAC (UPS Inverter) to a very high 515 VAC (Bypass)

UPS Bypass 01

Drilling in a bit more, we see the CT Scanner switch on (point “A”) with a maximum current of 245 Amps and a resultant collapse of the UPS output, a short period where the CT current drops and the UPS output stabilizes, then a transition to Bypass (point “B”). Note the increase in voltage while operating on Bypass.

At the end of the CT scan (point “C”) the voltage rises due to impedance. And the UPS stays in Bypass for an extended period (point “D”) needing to be manually reset.

UPS Bypass 03

A close-up of the “start of scan” waveform shows the nature of the inrush current (higher for just once cycle) – although the UPS voltage drops more than usual, it does not really fold or collapse.

Nothing really unusual here – some finger pointing at impedance (not really an issue, the voltage drop on the unregulated bypass was just 2.7% at full load) and voltage distortion (under 3% voltage distortion on the UPS input) – neither of which is a problem. The UPS got sized in based on power monitoring, which apparently did not capture peak load condition.

I suggested that the higher voltage on Bypass (515 VAC = 7% higher than nominal) would mean lower observed current, although that did not factor into the calculations (they were monitoring further upstream on a 208 VAC source). The UPS vendor is going to see if they can tweak the protection circuitry a bit to be able to survive and supply this short overload without a bypass transition.