Re: Various Senoko oddities

I compiled the toolset and ran the command line at that page (with sudo). It did report "Flash verified, exiting BootROM mode" after failing thef irst time. Before flashing, the board was reporting everything was okay (as it did much of the time). After, it reported the off-balance cell configuration, and some other ridiculous values (below). After a poweroff and restart, I see the same broken state. I'd already tried refitting the balance taps. Any other ideas?

Manufacturer:       Texas Inst.
Part name:          bq20z95
Firmware version:   0x150
Charge FET:         off
Discharge FET:      off
State:              battery failure
Time until full:    65535 minutes
Time until empty:   65535 minutes
Chemistry:          LION
Serial number:      0x0001
Cycle count:        6
Battery health:     100%
Charge:             100%
Max capacity:       65501 mAh
Design capacity:    5000 mAh
Temperature:        25.1 C
Voltage:            48928 mV
Current:            0 mA
Average current:    0 mA
Target voltage:     0 mV
Target current:     0 mA
Number of cells:    3 cells
Cell 1 voltage:     0 mV
Cell 2 voltage:     8037 mV
Cell 3 voltage:     8123 mV
Cell 4 voltage:     0 mV
Charge status:      0x0
    Charging allowed?   yes
    Can suspend?        no
    Can precharge?      no
    Can maintenance?    no
    Temperature limit?  no
    Temperature limit?  no
    Can fastcharge?     no
    Pulse charging?     no
    Pulse disable CHG?  no
    Cell balancing?     no
    Precharge timeout?  no
    Fastcharge timeout? no
    Overcharge OV?      no
    Overcharge OC?      no
    Overcharge?         no
    Battery empty?      no
Alarms:
    TERMINATE CHARGE ALARM
    TERMINATE DISCHARGE ALARM
    REMAINING CAPACITY ALARM
Charge state:
    Battery initialized
    Battery discharging/relaxing
    Battery fully discharged
No errors detected
No safety alerts
Safety status:
    Pack undervoltage condition
    Cell undervoltage condition
    Permanent failure condition

Re: Various Senoko oddities

I'm still running into issues as well. I tried running on the battery right after it claimed to be fully charged, and it shut off immediately; when I booted it with the mains adapter instead I saw the crazy unbalanced voltages I'd seen before. Unplugging and replugging the balancing connector fixed those readings again, so I figured I should leave it to charge awhile longer.

Several days later (today), I went to unplug it and try to boot using the battery, and Senoko was very warm--IR thermometer read just over 60 C. I unplugged and tried to boot anyway, and ended up with the same problem (and solution) as before.

I was about to try pelrun's flashing tool, but I saw manoj's latest comment and figured I should wait until hearing more.

That said, I saw in pelrun's readme to run the flash tool and "Then configure the gas gauge via the senoko command line"; how do we do that?

28 (edited by singpolyma 2015-06-06 21:39:55)

Re: Various Senoko oddities

manoj wrote:

Reflash as in the update-senoko script? I've generally had trouble with that too (as mentioned above). It succeeded once, but every other time I've tried it (including just now, most of the script proceeds, but it gets stuck in the loop waiting for the board to appear on i2c:

# Wait for the board to show up on I2C
while ! /usr/sbin/i2cget -f -y 0 0x20 0 2> /dev/null > /dev/null
do
    echo 0 > /sys/class/gpio/gpio149/value
    echo 1 > /sys/class/gpio/gpio149/value
done

When I try that i2cget command on my own I get "Error: Read failed", and when I strace that:

open("/dev/i2c/0", O_RDWR)              = -1 ENOENT (No such file or directory)
open("/dev/i2c-0", O_RDWR)              = 3
ioctl(3, 0x705, 0xbe920640)             = 0
ioctl(3, 0x706, 0x20)                   = 0
ioctl(3, 0x720, 0xbe920640)             = -1 EAGAIN (Resource temporarily unavailable)
close(3)                                = 0
write(2, "Error: Read failed\n", 19Error: Read failed
)    = 19

I am having this same problem.  When I connect to the senoko with screen, it seems to be rebooting itself constantly:

...
ChibiOS/RT Shell
ch> 
Starting Senoko (Ver 2.2, git version c601c505c84156a3cfe822317db757933e0504de)

ChibiOS/RT Shell
ch> 
Starting Senoko (Ver 2.2, git version c601c505c84156a3cfe822317db757933e0504de)

ChibiOS/RT Shell
ch> 
...

UPDATE: I tried this a second time and it behaved the same, but after I waited awhile it finally did stop rebooting and the script finished... so I guess it worked.

Re: Various Senoko oddities

myersjustinc wrote:

I was about to try pelrun's flashing tool, but I saw manoj's latest comment and figured I should wait until hearing more.

That said, I saw in pelrun's readme to run the flash tool and "Then configure the gas gauge via the senoko command line"; how do we do that?

A quick update on my previous post: It works!

The gory details follow, in case it helps anyone else:

I used pelrun's flashing tool and ran

gg capacity 3 5000

at the Senoko shell (after a quick sanity check) to configure the gas gauge (since I'm using the stock laptop battery--change these if you have a different battery, of course).

At that point, the power manager icon in xfce showed just a bare AC plug (for an "unknown" state, which I hadn't seen before), so I shut down--and then got worried because the computer wasn't responding to the power button to turn it back on. I tried power cycling Senoko (using the on/off button on that board) and had no luck, but the reset button on the main Novena board itself worked nicely.

After that, I could shut down--and power up with the power button on the front panel--as expected, and I could do so on battery power alone, which I hadn't been able to do before! I was able to write the majority of this post on battery power. (It would've been the whole thing, but I wanted to make sure it acted as expected when I plugged in the mains adapter again and when I unplugged it. All seems well.)

Interesting extra detail: The front-panel power button was not lit during my last successful power cycle, which also was new to me. It's been on for as long as I've had the Novena so far, excluding the times I've had the mains adapter unplugged. (Worked fine, just happened to have been lit the whole time.)

Re: Various Senoko oddities

Glad to hear that's resolved.

The front panel button light can be modified by poking through /sys/class/leds/novena:white:panel as root.

31 (edited by veleiro 2015-07-13 20:57:29)

Re: Various Senoko oddities

Im still having issues with my senoko after flashing the known good state.  However, my issue may actually not be the same.  My gg stats reports everything healthy, no issues, and booted up I can see that it charges correctly, I can unplug it and use battery power and everything works as it should.  UNTIL i fully shutdown the novena.  Plugged in or on battery power, I cant boot it up by the front panel.  It randomly comes on by itself.  So I cant ever shut it down because i know it will power on randomly, or will not respond to the power on button.

I know the front panel buttons work correctly because I can press the power button while in xfce and the shutdown/suspend dialog comes up.  Also the bluetooth reset button works.  It may be worth mentioning, though, that the front panel USB port isnt working correctly right now.  DMESG says "bad cable?"

gg stats:

Manufacturer:       Texas Inst.
Part name:          bq20z95
Firmware version:   0x150
Charge FET:         on
Discharge FET:      on
State:              normal discharge
Time until full:    65535 minutes
Time until empty:   84 minutes
Chemistry:          LION
Serial number:      0x0001
Cycle count:        28
Battery health:     99%
Charge:             39%
Max capacity:       54500 mWh
Design capacity:    55500 mWh
Temperature:        30.0 C
Voltage:            11173 mV
Current:            -1216 mA
Average current:    -1350 mA
Target voltage:     12600 mV
Target current:     5000 mA
Number of cells:    3 cells
Cell 1 voltage:     3718 mV
Cell 2 voltage:     3729 mV
Cell 3 voltage:     3726 mV
Cell 4 voltage:     0 mV
Charge status:      0x200
    Charging allowed?   yes
    Can suspend?        no
    Can precharge?      no
    Can maintenance?    no
    Temperature limit?  no
    Temperature limit?  no
    Can fastcharge?     yes
    Pulse charging?     no
    Pulse disable CHG?  no
    Cell balancing?     no
    Precharge timeout?  no
    Fastcharge timeout? no
    Overcharge OV?      no
    Overcharge OC?      no
    Overcharge?         no
    Battery empty?      no
Alarms:
Charge state:
    Battery initialized
    Battery discharging/relaxing
No errors detected
No safety alerts
No safety status messages

dmesg: https://paste.debian.net/hidden/885cb47f/

Re: Various Senoko oddities

The fact that the front panel button doesn't work well makes it sound like it's an assembly problem with the front panel board/cable, especially since the button on the Senoko board itself works.  These are actually part of the same net, so if the Senoko button is working but the front panel button isn't, then it's a continuity error.  The cable is, unfortunately, annoying to get to, and is located under the HDD.

Try unscrewing the front panel board and moving it around.  Do you get spurious events?  If so, you might try re-seating the cable.

There's a 200 ms debounce timer, so it needs to be held down for a while to activate.

33 (edited by veleiro 2015-07-21 12:24:58)

Re: Various Senoko oddities

I disconnected the front panel completely from the mainboard, yet the same problem is occuring.  Its very inconsistent results.  Sometimes I hold the off button and it powers off, but almost immediately powers back on.  Other times, it stays off and pressing the on button does nothing, until it randomly turns on by itself.  Right now its been a few hours and it hasnt powered on itself, though.

The battery is charging, and pressing the level button reports the correct levels, but the on/off button on the senoko battery board doesnt work as expected.

EDIT:   Played with it some more.  If I power it off with the on/off button, and power it on before it kicks on again by itself, it seems to work like it should every time.  Once it powers on by itself and I power it off again, I can no longer use the button to turn it on.  Is it possible that theres a watchdog that turns on the board, but when it does it doesnt notify the board that its actually on? So when I power off the board again, the toggle would be in the wrong state?  Just throwing out theories here.

Re: Various Senoko oddities

How full is your battery?  There could be an issue where it's not charging...

Or it could be a dodgy solder connection on the power button itself.

Do you see the power button's events printed out to the serial console if you attach to /dev/ttymxc3?

35 (edited by veleiro 2015-07-23 08:19:06)

Re: Various Senoko oddities

Battery is all the way full, or charged well enough during all previous posts.

I dont think the solder connection is bad, and I see every event printed to the serial console.

These events vary.  In a powered off state (while connected to the battery and disconnected from ac adaptor, and still with the front panel disconnected), I pressed the on button on senoko and nothing happened.  Tried the level button and nothing showed, and there wasnt any red pwr lights on the board.  When i plugged the adaptor in, it immediately booted without pressing on. 

So then everything worked as normal, but this is what the serial console was reporting when i varied types of presses of the on button.

[Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Suspending]  [Poweron Wait]  [Powered on]  [Already on]  [Poweroff Wait]  [Suspending]  [Poweroff Wait]  [Suspending]  [Poweroff Wait]  [Suspending]

It was definitely in the wrong state because everything was already powered on, and I dont know why it suddenly changed to [Powered on] [Already on].  But I got an event every time I pressed the button, so I doubt theres a bad solder connection.

After I held the off button long enough, it did power off.  But from that state I couldnt use the same button to turn it back on.  Couldnt turn it on by unplugging/plugging in the ac adaptor again either.  I know that it will eventually power on by itself again from experience though.

Can I serial into the senoko board without going through the novena, to check the state too? Also whats the prime button do?

Re: Various Senoko oddities

I've seen cases where it thinks that power is off, even though it's on.  You can verify this by using the "power" command to see what it thinks the state is.

"Poweron Wait" means that it detected the power button, but wants to see if it should debounce it.

The next message, "Suspending", means that the button was released before the timeout was hit.  If the board is powered on, this means that it will send the "Suspend" button, which is why the XFCE window pops up.  If the board is powered off, you normally won't see this message.  In any case, it means you released the button within the 200ms window, so it didn't toggle the board's state.

Eventually, you see "Poweron Wait" followed by "Powered on", and then "Already on".  This three-message queue happened because the button was pressed ("Poweron Wait"), held down for > 200 ms ("Powered on"), and then released ("Already on").

From then on, the board thought it was on, and would give you the message "Poweroff Wait", indicating it will wait three seconds to see if the board should power off.

Are you holding the power button for > 200 ms to get it to turn on?

The "Prime" button was a failed experiment in getting the gas gauge to turn on when doing a battery swap.  When you hold it down, the battery is allowed to directly power the STM32 SoC.  In theory, we could use this to power up the gas gauge, but in practice, we haven't found a way to kick it out of hibernation without turning on the charger first, which of course requires an external power supply.

Re: Various Senoko oddities

I'm having an issue where the Novena (Laptop) powers down a second after boot, despite having been plugged in the entire time.

Pressing the "Level" button on the Senoko shows a single green LED, so it's obviously not charging.

Not sure if it's the board or the power cable at this point. Going to try talking to it via serial once I compile the usbserial module and also try getting access to a multimeter tomorrow.

Re: Various Senoko oddities

The power cable is working fine (measured 17.8V). Not sure why it isn't charging, then. I'll probably try measuring the battery next or seeing if I can flash the Senoko via serial (not sure how to flash it).

Unfortunately, I left my Novena at my girlfriend's parent's house, so I won't have access to it for another three weeks.

Re: Various Senoko oddities

Welp, I retrieved the Novena, plugged it in, started booting into Gentoo, thought better of it, booted into the recovery partition and didn't have any issues. Screen-ing into the device and querying its status showed that the battery was low, but was charging fine, and now it's fully charged. I'm guessing there was just an issue with the battery getting low. I recall hearing that leaving the Novena always plugged in is not a good idea, so I'll probably refrain from doing that now.

Re: Various Senoko oddities

Manoj or Velerio,

Where you able to solve your problems with the senoko? Yesterday, i Had the same issue of there it would go between 0 mV for Cell 1 and 57 V or 8V for cells 2 and 3.

Rightnow it seems to have found my battery, but it refuses to use the battery.

This is the output of my stats:

ch> stats
Manufacturer:       Texas Inst.
Part name:          bq20z95
Firmware version:   0x150
Charge FET:         off
Discharge FET:      off
State:              normal discharge
Time until full:    65535 minutes
Time until empty:   65535 minutes
Chemistry:          LION
Serial number:      0x0001
Cycle count:        6
Battery health:     98%
Charge:             95%
Max capacity:       4955 mAh
Design capacity:    5067 mAh
Temperature:        24.8 C
Voltage:            12334 mV
Current:            0 mA
Average current:    0 mA
Target voltage:     0 mV
Target current:     0 mA
Number of cells:    3 cells
Cell 1 voltage:     4109 mV
Cell 2 voltage:     4111 mV
Cell 3 voltage:     4114 mV
Cell 4 voltage:     0 mV
Charge status:      0x0
    Charging allowed?   yes
    Can suspend?        no
    Can precharge?      no
    Can maintenance?    no
    Temperature limit?  no
    Temperature limit?  no
    Can fastcharge?     no
    Pulse charging?     no
    Pulse disable CHG?  no
    Cell balancing?     no
    Precharge timeout?  no
    Fastcharge timeout? no
    Overcharge OV?      no
    Overcharge OC?      no
    Overcharge?         no
    Battery empty?      no
Alarms:
    TERMINATE CHARGE ALARM
    TERMINATE DISCHARGE ALARM
Charge state:
    Battery initialized
    Battery discharging/relaxing
No errors detected
No safety alerts
No safety status messages
ch> 

But when I run chg, I get:

ch> chg
Charger information:
        Charge thread:    running
        Manufacturer ID:  0x0040
        Device ID:        0x0007
        Current:          1024 mA
        Voltage:          12592 mV
        Input:            3328 mA
Command usage:
  chg set [c] [v] [[i]]  Set charge current to c mA @ v mV
  chg input [i]          Set max input AC current to [i] mA
  chg pause              Pause charge control
  chg resume             Resume charge control

So to me it looks like it sees the battery and is running the charging thread, but it is not actually charging the battery or using it.

Re: Various Senoko oddities

Also, on a whim, I reinstalled firmware-senoko. I got this error:

Setting up firmware-senoko (2.3.3-r1) ...
Error: Read failed
Error: Read failed
/var/lib/dpkg/info/firmware-senoko.postinst: 14: /var/lib/dpkg/info/firmware-senoko.postinst: arithmetic expression: expecting primary: " * 256 + "
dpkg: error processing package firmware-senoko (--configure):
 subprocess installed post-installation script returned error exit status 2

Re: Various Senoko oddities

Weird voltage values like that can happen when the gas gauge has just rebooted.  It's still sampling the values, and so there's some cruft left during the averaging window.

As for the "Read failed", what it's doing is trying to probe the Senoko to see if it needs to update the package.  It does this by querying it over i2c to get the "version" number, and then performs an update if it's too low.  In your case, the firmware-senoko upgrade script is unable to communicate with the board, which results in an invalid version number, which causes the syntax error you see.

Senoko isn't communicating via i2c for some reason.  Perhaps because it thinks it's powered off.  Run the "power" command to figure out whether it thinks it's on or off.  If it thinks it's off, run "power on" to make Senoko realize it's actually on.

43 (edited by chris4795 2016-02-04 05:12:22)

Re: Various Senoko oddities

Xobs,

You were right, the power in the senoko was off. I was able to install 'firmware-senoko' and it told me it is up to date.

However, when I ran info, I got this:

ch> info
Kernel:       3.0.0dev
Compiler:     GCC 4.9.2
Architecture: ARMv7-M
Core Variant: Cortex-M3
Port Info:    Advanced kernel mode
Platform:     STM32F10x Performance Line Medium Density
Board:        Senoko STM32F1
Build time:   Aug 12 2015 - 17:13:27

The senoko.hex file said it was modified on Dec 9 2015, so I wanted to confirm I am not getting anything strange.

The battery has normal voltages, but while it is plugged in, the stats still says "Normal Discharge", but the chg command says the charge thread is running:

ch> stats
Manufacturer:       Texas Inst.
Part name:          bq20z95
Firmware version:   0x150
Charge FET:         off
Discharge FET:      off
State:              normal discharge
Time until full:    65535 minutes
Time until empty:   65535 minutes
Chemistry:          LION
Serial number:      0x0001
Cycle count:        6
Battery health:     32%
Charge:             77%
Max capacity:       1593 mAh
Design capacity:    5067 mAh
Temperature:        24.4 C
Voltage:            11909 mV
Current:            0 mA
Average current:    0 mA
Target voltage:     0 mV
Target current:     0 mA
Number of cells:    3 cells
Cell 1 voltage:     3739 mV
Cell 2 voltage:     4076 mV
Cell 3 voltage:     4094 mV
Cell 4 voltage:     0 mV
Charge status:      0x0
    Charging allowed?   yes
    Can suspend?        no
    Can precharge?      no
    Can maintenance?    no
    Temperature limit?  no
    Temperature limit?  no
    Can fastcharge?     no
    Pulse charging?     no
    Pulse disable CHG?  no
    Cell balancing?     no
    Precharge timeout?  no
    Fastcharge timeout? no
    Overcharge OV?      no
    Overcharge OC?      no
    Overcharge?         no
    Battery empty?      no
Alarms:
    TERMINATE CHARGE ALARM
    TERMINATE DISCHARGE ALARM
Charge state:
    Battery initialized
    Battery discharging/relaxing
No errors detected
No safety alerts
No safety status messages
ch> chg
Charger information:
        Charge thread:    running
        Manufacturer ID:  0x0040
        Device ID:        0x0007
        Current:          1024 mA
        Voltage:          12592 mV
        Input:            3328 mA
Command usage:
  chg set [c] [v] [[i]]  Set charge current to c mA @ v mV
  chg input [i]          Set max input AC current to [i] mA
  chg pause              Pause charge control
  chg resume             Resume charge control

Also to note: I tried "gg chg +" to force it to charge, but after doing that and running "stats", it still showed the charge FET as off.

Re: Various Senoko oddities

Your CHG and DSG FETs are both off, yet it says no errors/faults detected.  I find that, in this case, the way to get it to work is run "gg it" to "Enable ImpedenceTrack(TM)".  There's some boot logic to try and figure out when this is needed, but it can fail.

I'm still not sure what ImpedenceTrack(TM) is, nor what it means to have it disabled or enabled, nor why it's not enabled on boot.  But that should at least fix your problem.

45 (edited by chris4795 2016-03-26 04:42:54)

Re: Various Senoko oddities

Unfortunately, I tried that with no success. I am suspecting one of the chips failed.

With "gg chg +", I have been able to turn on the charge FET (according to stats), but I still get 0 mA for current.

I have access to a lab that I can change out the chips, so I am going to try to change out the ICs that are the gas gauge, ARM, battery charger, and the voltage overprotection circuit ( one at a time, I suspect it is either the gas guage or the battery charger that is acting up so I am going to try that first)

EDIT: It turns out it was an issue with the battery itself, not the board. I fixed the issue with the battery and my senoko works

Re: Various Senoko oddities

if we not use g chg++ then?

47 (edited by chris4795 2016-04-06 05:23:05)

Re: Various Senoko oddities

I am unfortunately getting a new error...

When I do stats, I get this:

stats
Manufacturer:       error 0xFE000000
Part name:          error 0xFE000000
Firmware version:   error 0xFE000000
Config version:     error 0x1000000
State:              error 0xFE000000
Time until full:    error 0xFE000000
Time until empty:   error 0xFE000000
Chemistry:          error 0xFE000000
Serial number:      error 0xFE000000
Cycle count:        error 0xFE000000
Battery health:     error 0xFE000000
Wake current:       error
Charge:             error 0xFE000000
Max capacity:       error 0xFE000000
Design capacity:    error 0xFE000000
Temperature:        2184.5 C
Voltage:            error 0xFE000000
Current:            error 0xFE000000
Average current:    error 0xFE000000
Target voltage:     error 0xFE000000
Target current:     error 0xFE000000
Number of cells:    error 0x1000000
Cell 1 voltage:     error 0xFE000000
Cell 2 voltage:     error 0xFE000000
Cell 3 voltage:     error 0xFE000000
Cell 4 voltage:     error 0xFE000000
Charge status:      0x5555
    Charging allowed?   yes
    Can suspend?        suspended
    Can precharge?      no
    Can maintenance?    yes
    Temperature limit?  no
    Temperature limit?  yes
    Can fastcharge?     no
    Pulse charging?     yes
    Pulse disable CHG?  no
Cell balancing?     in-progress
    Precharge timeout?  no
    Fastcharge timeout? yes
    Overcharge OV?      no
    Overcharge OC?      yes
    Overcharge?         no
    Battery empty?      yes
Alarms:
    TERMINATE CHARGE ALARM
    OVER TEMP ALARM
    REMAINING TIME ALARM
Charge state:
    Battery discharging/relaxing
    Battery fully discharged
STATUS ERROR CODE: 0x5
Unable to read safety alerts
Unable to read safety status

Trying to reboot, I get this:

ch> gg reboot
Rebooting the gas gauge chip... Error: FE000000

Trying the flash program from pelrun, I get this

# ./flash xobs-gg.dfi 
Flash length: 2048
Erasing rows 0-1...
Retrying erase...
Retrying erase...
(It happens a lot of times...)
Retrying erase...
Flash verify failed, try running this again.

It seems the gas gauge is in a permanent failure mode, and I have no idea how to fix it (outside of replacing it).

Are there any thoughts?

EDIT: I feel like I have a streak of once I post up my issue on this board, the answer magically comes to me. it turns out that I needed to manually push the "prime" button.

As for why that happened in the first place, the connection issues I thought I solved are not solved. So I need to fix some connections on the battery cells.

Re: Various Senoko oddities

So what I have learned:

If I leave my laptop unpowered for a long time (think days), the gas gauge comes back to life. I am able to use the battery as normal, but it only lasts for 15 minutes and then goes back into the error 0xfe000000 mode. I have to repeat that.

Re: Various Senoko oddities

That's very bizarre.

If you do manage to get it reliably running, can you try installing v2.5 firmware?  It's more conservative in its undervoltage values.  http://repo.novena.io/debian/pool/main/ … r1_all.deb

Re: Various Senoko oddities

Just to ensure: @chris4795 is your senoko one of the once I build or is it official over crwodsupply.
At 32c3 bunnie, 2 friends and I fixed a hardware error on one of those boards. It was a resistor with a wrong value (even though the printed value was correct). Might it be something like that?