[RE-wrenches] Outback Programming Ghost

[Matthew Sirum]

Greetings All,

I discovered an issue with a grid-tied 5 OutBack Radian GS8048A system
3 days ago (Friday).  I went to the client's site late in the day and
found the system had experienced an issue about 9 hours earlier.  The
system had not "sold" any energy to the grid all day and the MATE3
system display/controller home screen looked very strange.  Only some
of the keys on the MATE3 were functioning and their associated screens
also looked strange.  The Event History of the MATE3 logged an
"Inverter Firmware Revisions are Different" and "Inverter Firmware
Revisions are the same" earlier that morning.  The 2 Events were
logged about 3 minutes apart from each other.  I took digital pictures
of all the MATE3 screens that were effected and of the Event History
log.  I opened all of the AC circuits to/from the inverters, opened
all of the DC circuits to the inverters, and disconnected the Hub
network cable and Ethernet cable from the rear of the MATE3 and also
removed the SD card.  After about 10 minutes I reconnected the network
and ethernet cables, reinstalled the SD card, and restarted the
system.  The system started up just fine, but the Sun had already set
for the day so it was not "selling" any energy.  All of the MATE3
screens now looked normal.  I was surprised to find the 2 firmware
related Events had disappeared from the Events History log.  I checked
the firmware versions for all of the OutBack components in this system
and all are the latest versions.  The system settings looked to be
unaffected.  I called OutBack Tech Support to discuss and troubleshoot
further.  I was told it was a known issue related to out of date
firmware.  I double checked all of the installed firmware versions and
the OutBack tech agreed they are all up to date.  After the tech put
me "on hold" and spoke with others at OutBack he told me this issue
can be caused by the Events History not completely erasing old Events
even though the Events History only has 128 lines which write over
once they reach 128 logged Events.  This system has a persistent "IN
AC Freq Too High" error issue that populates the Event History
hundreds of time each day, so this MATE3 is a likely candidate for
this issue.  Even though the MATE3 uses flash memory its memory seems
to act somewhat like a hard disk drive that needs to be defragmented.
The tech had me erase/delate the Events History log.  I was told that
if it happens again I should unplug the network cable from the MATE3
to reset it and erase/delete the Events History log.  After this I
reviewed  some of the settings and configuration items with the tech
still on the phone to see if any had been effected.  We did find one
unexplained issue; "existing configuration" and "new configuration" in
the Configuration menu were the same.  Meaning they both showed the
same default settings/values.  Although the actual settings and values
viewed in other menus showed they were actually set as I had last
programed them.  He opened a case for further review at OutBack and I
have emailed them my pictures and notes.

Also of note; This system is connected to OutBack's OpticsRE and I
have it set to send me email notifications for Events on the system.
Between the time the issue occurred and when I reset the system
OpticsRE emailed me many messages with the Event "Configuration
changes are made locally via MATE3/AXS PORT".  No one was working on
the system or using the MATE3, so this seems strange.  All I can think
is that the system was still seeing whatever normally triggers the "IN
AC Freq Too High" errors, but they were being logged as "Configuration
changes are made locally via MATE3/AXS PORT" during this period of
time when the system/MATE3 was experiencing this issue.  I have also
put this in my notes for OutBack.

I have attached 2 pictures; 1 of what the MATE3 home screen looked
like and 1 of what the Event History looked like.  There were also a
number of other screens effected.  I am happy to share my other
pictures if anyone would like to see them.  Please feel free to
contact me.

I am interested to hear if these MATE3 screens look like what others
in this thread have seen.

Best regards,

----MATT

Matthew Sirum
P.O. Box 1227
Greenfield, MA 01302-1227  USA
phone: +1.413.773.0611
email: MatthewSirum at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Event History Log.JPG
Type: image/jpeg
Size: 490677 bytes
Desc: not available
URL: <http://lists.re-wrenches.org/pipermail/re-wrenches-re-wrenches.org/attachments/20161114/4e7587fc/attachment-0004.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Home Screen.JPG
Type: image/jpeg
Size: 476940 bytes
Desc: not available
URL: <http://lists.re-wrenches.org/pipermail/re-wrenches-re-wrenches.org/attachments/20161114/4e7587fc/attachment-0005.jpe>