Reading the Tea Leaves

Sometimes when reviewing power monitoring data, key information is left out of the problem statement. But the astute power quality engineer can “read the tea leaves” and pick up information about the installation, equipment, and technical issues.

A set of data from a Magnetic Resonance Imaging system was presented for analysis with the following notes:

System has had several intermittent issue that have caused system to be down and functional upon arrival. System issues have been in RF section. No issues have been reported since system since installation of power recorder.

RMS Voltage and Current

Chiller RMS Logs

First clues come from looking at the RMS logs of the voltage and current. The voltage is suspiciously well regulated – and is probably a UPS or power conditioner rather than a normal utility source (which will tend to fluctuate over a 24 hour period). A second clue is the small voltage increase or swell related to load switch-off – typical of an active source, not typical of a passive source.

Second, this appears to be a regularly cycling load – a pump or compressor. MRI systems typically have a chiller or cryogen cooler associated with them – so odds are good this was monitored on this load, and not on the MRI system itself.

Chiller or Cryogen Cooler Load

Chiller Cycling RMS
More evidence supporting the chiller or cryogen cooler load – a regular (practically like clock-work) cycling load, with a marginally higher operating current (~30 Amps), but a very high inrush current (~180 Amps)

Chiller Transient

Normal chiller or cryogen cooler inrush is seen here. A minor (~5%) voltage sag was captured during each inrush current, as well as minor associated transients (probably relay or contactor switch bounce)

Abnormally High Inrush Currents

Chiller Sag

In addition to the regular inrush currents associated with chiller cycling, six instances of very high inrush current were captured. These were seen both as voltage sag events as well as current triggered events. We’re concerned that this high inrush current may be causing an overcurrent condition on the UPS / power conditioner – which may be throwing a fault or error, or perhaps switching to Bypass.

Chiller Swell RMS

Looking at the RMS logs of the high current swell / voltage sag event, we see that it precedes a period of extended chiller / compressor operation. Unknown if this is normal operation for the system / device or indicates an error or fault of some sort.

Summary

Although the accompanying technical information was thin, we’ve “read the tea leaves” and provided the following analysis bullets

  • System appears to be powered by a UPS or power conditioner. Service personnel may not have known this.
  • System appears to be a chiller, compressor, or similar device (not the medical imaging system itself)
  • Occasional high current swells were seen; these may be normal or may point to system issues.
  • Voltage sags and collapse during these high current swells may indicate that the UPS or power conditioner is overloaded, and may be experiencing faults or alarms that may be impacting system operation / uptime

The Curious Case of the UPS Loading

We recently got to review input and output monitoring data from a UPS system (make and model not specified) feeding a medical imaging system. The monitoring was done as a precaution, but we noticed something unusual.

First, take a look at the RMS voltage and current logging of the UPS input and output. Phase A voltage, Phase B current shown for clarity, but all voltage and current phases are balanced and similar.

UPS Compare Input RMS

UPS Input – Normal facility RMS voltage (daily fluctuations, with occasional sags) and RMS current peaks at approximately 80 Amps.

UPS Compare Output RMS

UPS Output – Highly regulated RMS voltage (with small load related fluctuations) and RMS current peaks at approximately 170 Amps.

The discrepancy between the input current and output current is unusual. It would be typical for input current to be marginally higher than output current (due to device efficiencies) but not lower. Our guess – the UPS DC bus (and probably, the battery string) is being called on to support the peak output load.

UPS Compare Output Highest Load

UPS Output – Step change in load current and nonlinear load is typical of medical imaging system. Very small fluctuation in output voltage related to load changes, and small increase in voltage distortion related to nonlinear load current.

UPS Compare Input Highest Load

UPS Input – Even at highest levels, current is linear, UPS must have a unity power factor front end / rectifier. However, lower current level is unusual, and indicates that UPS battery is probably being called on to supply the peak medical imaging load.

There’s really no immediate problem here – the UPS is doing a great job of correcting input power issues, as well as supplying the complex loads (step change, pulsing currents, nonlinear power factor) of the medical imaging system.

However,it’s pretty clear that the UPS batteries are getting discharged during highest current imaging system operations – not really their intended purpose, which is to ride through far less frequent utility sags and outages. So it’s possible that the UPS batteries are being stressed and may degrade or fail prematurely, and need replacement. We’ve referred this to the UPS manufacturer / supplier for attention.

As a quick “in the field” test (we’re doing this analysis remotely, not on site) we might suggest disconnecting the battery string temporarily, and seeing how the UPS performs without the battery, just relying on the DC bus. We’re guessing the UPS might start to collapse or struggle to supply the medical imaging load – and may be undersized for the application without the battery string supplied.

We’ve seen situations where a UPS that has heretofore worked well for years stops working quite so well, because the batteries started to wear out, and the unit was no longer able to supply the peak loads required by the imaging system.

The Guru’s Cat

When the guru sat down to worship each evening, the ashram cat would get in the way and distract the worshipers. So he ordered that the cat be tied during evening worship.

After the guru died the cat continued to be tied during evening worship. And when the cat died, another cat was brought to the ashram so that it could be duly tied during evening worship.

Centuries later learned treatises were written by the guru’s disciples on the religious and liturgical significance of tying up a cat while worship is performed.

– Anthony De Mello, The Song of the Bird

A music festival where I’ve been a volunteer for nearly 25 years is doing their annual mid-winter pre-fest sales – selling a limited number of tickets at a reduced price. It’s a good way to carry the festival organizers over the winter, and to give regular festies a price break.

What’s NOT so good is how they do it – phone only, with a limited staff processing orders manually over a three day period. I’ve seen some posts on social media:

“After 111 attempts to get through, over a span of twelve minutes, tickets have been procured!”

Mine took longer than usual — 179 calls and 48 minutes (you re-dial faster than I do) . . .

and from one of the folks on the other end of the line:

FYI y’all S & A are working fingers off to accommodate youralls calls for tix & are very grateful for your wonderful patience

See, the thing is, this could be done online through eCommerce – put a limited number of tickets for sale (so you do not oversell) and for a limited time. Yeah, there’s a service fee (but probably not all that much higher than the credit card fees) and I suspect most customers would pony up an additional $5 or $10 per ticket to cover an eCommerce solution (and not have to dial in 100+ time). You could sell out your winter pre-fest stock without having to tie up customers, your staff, etc.

But….it’s been done this way for 25+ years and will probably always be done this way. Like the Guru’s Cat, sometimes we do things out of habit or tradition or inertia without stepping back and considering other options.