• job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Topic: hibernation

I'm trying to get hibernation on my laptop to work and will post my results here.

First I had to fix my swap situation as described here:
http://www.kosagi.com/forums/viewtopic. … 1405#p1405

Then I installed the beta 3.19 kernel from the URL that xobs mentioned in the same thread.

Hibernation works mostly at the moment. At least I get back to my X sessions at the prompt I was.

Bluetooth fails though and after a second hibernate also the USB keyboard failed.

Curious for me (not knowing about inner details of hibernation) is that CPUs are stopped during freeze, then thaw and then the image gets written to swap. Also during resume it seems to do a halt/start of CPUs.

When I have more time I will share some logs and more details.

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: hibernation

Yes, the way it works is that it suspends everything, then captures the state of the system to a buffer.

Since the system is suspended, everything (even the hard disks) is turned off, so it must resume everything in order to write it to disk.  Which is why it looks like it's "booting" again.

If you're having trouble with USB and bluetooth, try installing the novena-usb-hub package that's also in that link.  It adds a rule that resets the Bluetooth port, which has improved reliability at least on my machine.

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

xobs wrote:

If you're having trouble with USB and bluetooth, try installing the novena-usb-hub package that's also in that link.

Not quite sure which link you mean. The one with the beta kernel does not seem to have any novena-usb-hub package.
Also I seem to be running v1.1-r1 which is the latest at https://github.com/xobs/novena-usb-hub

Is there another version?

  • Registered: 2014-06-11
  • Posts: 68

Re: hibernation

Job, Here is xobs latest from last night: https://nas.xobs.io/novena/20150429/

That link includes USB hub 1.2-r1

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

Thanks!

I tried it just now but the bluetooth does not work still.
Suspect it is a more fundamental problem.

This is what I see after resume from hibernate:

[  129.956225] Bluetooth: hci0 setting interface failed (32)
[  136.333203] Bluetooth: hci0 command 0x0804 tx timeout
[  333.042160] Bluetooth: hci0 link tx timeout
[  333.046490] Bluetooth: hci0 killing stalled connection xx:xx:xx:xx:xx:xx
[  335.053188] Bluetooth: hci0 command 0x041f tx timeout
[  337.053211] Bluetooth: hci0 command 0x0406 tx timeout
[  363.041625] Bluetooth: hci0 link tx timeout
[  363.045969] Bluetooth: hci0 killing stalled connection xx:xx:xx:xx:xx:xx
[  365.043187] Bluetooth: hci0 command 0x041f tx timeout
[  367.043208] Bluetooth: hci0 command 0x0406 tx timeout

The initial 'setting interface' is from the new enabling after resume I think. It then tries to kill my keyboard connection.
But then it gets stuck just as before.

Got my own kernel compile working so I will see if I can debug this. With a bluetooth keyboard it is a bit annoying ;-)

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

Ok, good news: bluetooth works now.

Turns out it really needs to be turned of before suspending.
The script in novena-usb-hub 1.2 only resets power on resume.

I now added power off before hibernation like this:

cat /lib/systemd/system-sleep/reset-bluetooth-port.sh 

#!/bin/sh
if [ "$1" = "pre" ]
then
    /usr/bin/novena-usb-hub -d u3
fi
if [ "$1" = "post" ]
then
    /usr/bin/novena-usb-hub -d u3
    /usr/bin/novena-usb-hub -e u3
fi

Then it works on resume.

Another thing I saw with the USB keyboard that fails to work after resume:
It gets picked up by the kernel before the hibernation state is loaded from swap.
Then it switches to the resume state, which I guess confuses either the keyboard or the kernel and prevents the keyboard from ever working (even after replugging). The USB mouse works fine.

A solution might be to power down all USB ports before power down like with the Bluetooth.

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: hibernation

Interesting.

Suspend/Resume is hard.

Mind if I steal that patch and incorporate it into a 1.3 version of the package?

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

xobs wrote:

Mind if I steal that patch and incorporate it into a 1.3 version of the package?

Mind if you steal it?? :-)

I would say it is my humble contribution to all your hard work.
No need to credit me for anything.

In the mean time I will find new bugs for you ;-)
After one day in hibernation this happens during resume:

[    3.533929] it6251 2-005c: System status: 0x3a
[    3.536969] dummy 2-005e: error -5 writing to lvds addr 0x5
[    3.536976] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
[    4.026880] it6251 2-005c: System status: 0x3a
[    4.031390] it6251 2-005c: Display didn't stabilize.  This may be because the LVDS port is still in powersave mode.
[    4.041963] it6251 2-005c: Will try again in 400 msecs
[    4.444802] it6251 2-005c: System status: 0x3a
[    4.453411] dummy 2-005e: error -5 writing to lvds addr 0x5
[    4.461866] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
...
[ 1365.475269] CPU1: shutdown
[ 1365.476629] CPU2: shutdown
[ 1365.477828] CPU3: shutdown
[ 1365.478421] PM: Creating hibernation image: 
[ 1365.478421] PM: Need to copy 193833 pages   
[ 1365.478443] Enabling non-boot CPUs ...
[ 1365.479413] CPU1 is up
[ 1365.480235] CPU2 is up
[ 1365.481064] CPU3 is up
[ 1365.482723] PM: noirq restore of devices complete after 1.644 msecs
[ 1365.484488] PM: early restore of devices complete after 1.569 msecs
[ 1365.685399] sd 0:0:0:0: [sda] Starting disk 
[ 1365.736255] Bluetooth: hci0 setting interface failed (32)
[ 1370.113115] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 1370.113360] ata1.00: supports DRM functions and may not be fully accessible
[ 1370.113623] ata1.00: supports DRM functions and may not be fully accessible
[ 1370.113635] ata1.00: configured for UDMA/133
[ 1370.396732] PM: restore of devices complete after 4743.483 msecs
[ 1370.623172] imx-ipuv3 2400000.ipu: DC stop timeout after 50 ms
[ 1370.744695] it6251 2-005c: System status: 0x3a
[ 1370.753543] dummy 2-005e: error -5 writing to lvds addr 0x5
[ 1370.759671] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
[ 1370.893164] imx-ipuv3 2400000.ipu: DC stop timeout after 50 ms
[ 1370.899113] vgaarb: this pci device is not a vga device
[ 1370.936488] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 1370.946317] usb 1-1.3: USB disconnect, device number 14
[ 1371.023694] fec 2188000.ethernet eth0: Freescale FEC PHY driver [Micrel KSZ9021 Gigabit PHY] (mii_bus:phy_addr=2188000.ethernet:07, irq=-1)
[ 1371.183292] it6251 2-005c: System status: 0x3a
[ 1371.187752] it6251 2-005c: Display didn't stabilize.  This may be because the LVDS port is still in powersave mode.
[ 1371.198236] it6251 2-005c: Will try again in 400 msecs
...
[ 1398.121176] dummy 2-005e: error -5 writing to lvds addr 0x5
[ 1398.126804] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
[ 1398.487535] it6251 2-005c: System status: 0x3a
[ 1398.491988] it6251 2-005c: Display didn't stabilize.  This may be because the LVDS port is still in power
save mode.
[ 1398.502470] it6251 2-005c: Too many retries, abandoning.

I snipped most normal messages.

Seems like maybe the long power down causes the display not to start?
Seems like the message "imx-ipuv3 2400000.ipu: DC stop timeout after 50 ms" is also related.

Do you know if there is a way to attempt a restart of the screen?

Another line I see is:

[    4.903672] ------------[ cut here ]------------
[    4.908459] WARNING: CPU: 3 PID: 25 at drivers/base/firmware_class.c:1126 _request_firmware+0x480/0x6c4()
[    4.918963] ---[ end trace ac0a4a1abbae6930 ]---

But that seems related to this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1133378

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

Unfortunately I now have a non working screen after a night of hibernation :-(

This is what I see during boot:

[    3.394798] i2c 2-005c: Driver it6251 requests probe deferral
[    5.757957] it6251 2-005c: System status: 0x3a
[    5.775219] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
[    6.070971] it6251 2-005c: System status: 0x3e
[    6.078947] it6251 2-005c: RPCLKCnt: 2720
[    6.086457] it6251 2-005c: Clock: 0x193
[    6.093063] it6251 2-005c: Ref Link State: 0x00

Whichever way I reboot, shutdown, plug in power the screen won't come on any more.

Could it be damaged during hibernation?
I will leave it on charging for the day and see later if I can revive it. Maybe pulling all power from the main board.
Any tips are welcome.

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: hibernation

I've tried to improve the screen-on performance, but I'm working without documentation.  I think it has to do with calibration taking place before the screen is stabilized.

Can you try 3.19-r9 from https://nas.xobs.io/novena/20150429/lin … _armhf.deb and see if that improves things?

11 Reply by job

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

This was with that version (00268).

Do you think it is the screen or the eDP board?

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: hibernation

Hibernation removes all power.

The thing is, U-Boot should power it up and calibrate the screen.  Do you get the Tux logo in U-Boot?

I modified the it6251 driver to skip calibration if it detects that the panel is already set up.  This is the case when booting or resuming from hibernation, but is not the case when resuming from sleep.

The chip is reporting it's stable.  Are you sure the brightness isn't just at 0?  If you have a console, check /sys/class/backlight/backlight/brightness

13 Reply by job

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

It is not the backlight for sure.

I installed u-boot rc17 and now I see 20x:

Display didn't stabilize.
This may be because the LVDS port is still in powersave mode.

during u-boot.

I guess there is something fundamentally wrong somewhere.
Any ideas are welcome.

14 Reply by job

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

Ok, managed to get it working again. I detached power fully and detached the LCD cable on both sides and put it all together again.
Then the issue was gone. No idea what the root cause was, but it seems fine again. It may just have been a coincidence.

One thing I did notice is that after installing the new u-boot I lost console on ttymxc1.
I recovered this through a uEnvt.txt in /boot

finalhook=setenv bootargs ${bootargs} console=ttymxc1,115200

Will test again a longer hibernation.

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: hibernation

U-Boot is a bit clever, and probes i2c to see if it finds the eDP chip.  If it doesn't find it, then it uses the serial port.

It does sound like a connection issue.  Now that the connection is fine, it uses the LCD as the console.

16 Reply by job

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

Ok, now it happened again. Exactly the same thing. After a night of hibernation, without being on the charger, the LCD does not come back.
Same as before. Of course can still be a connection issue, but might also be systematic.

I will explore some more next few days and report back.
However, if any reason for such a failure is known or likely I am interested to test and verify.

In the meantime some logs. This is for a successful resume:

[    1.951191] i2c 2-005c: Driver it6251 requests probe deferral
[    4.003890] it6251 2-005c: System status: 0x3e
[    4.011898] it6251 2-005c: RPCLKCnt: 2720
[    4.019552] it6251 2-005c: Clock: 0x193
[    4.026230] it6251 2-005c: Ref Link State: 0x00
[    4.032875] it6251 2-005c: eDP system is already stable
...
[ 1348.456706] PM: restore of devices complete after 8613.293 msecs
[ 1348.804694] it6251 2-005c: System status: 0x3a
[ 1348.819925] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
[ 1348.932925] it6251 2-005c: System status: 0x3e
[ 1348.939068] it6251 2-005c: RPCLKCnt: 3728
[ 1348.944809] it6251 2-005c: Clock: 0x193
[ 1348.949468] it6251 2-005c: Ref Link State: 0x00

This is for a failing one:

[    1.951235] i2c 2-005c: Driver it6251 requests probe deferral
[    4.012289] PM: Using 3 thread(s) for decompression.
[    4.012289] PM: Loading and decompressing image data (205830 pages)...
[    4.243955] it6251 2-005c: System status: 0x3e
[    4.251593] it6251 2-005c: RPCLKCnt: 2208
[    4.257163] it6251 2-005c: Clock: 0x193
[    4.258346] PM: Image loading progress:   0%
[    4.266202] it6251 2-005c: Ref Link State: 0x00
[    4.272587] it6251 2-005c: eDP system is already stable
...

[  148.826679] PM: restore of devices complete after 8602.114 msecs
[  149.174443] it6251 2-005c: System status: 0x3a
[  149.191095] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
[  149.627906] it6251 2-005c: System status: 0x3a
[  149.632363] it6251 2-005c: Display didn't stabilize.  This may be because the LVDS port is still in powersave mode.
[  149.643327] it6251 2-005c: Will try again in 400 msecs
[  150.053909] it6251 2-005c: System status: 0x3a
[  150.066887] dummy 2-005e: it6251.c:it6251_init:277 error -1 writing 255 to 5
[  150.529913] it6251 2-005c: System status: 0x3a
[  150.534440] it6251 2-005c: Display didn't stabilize.  This may be because the LVDS port is still in powersave mode.
...

So it seems early boot thinks all is fine with the LCD, but then after restoring from disk, the old kernel state thinks it is faulty.
The difference being that a short hibernation is able to reset it after the first fault, while a night hibernation cannot.

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: hibernation

This is good information.  Thanks for posting logs.

Do you ever not see the U-boot logo when you first power up the board?  U-boot seems to be able to very consistently calibrate the LVDS connection.

18 Reply by job

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

Ok, I've been reading the power mgmt code in the kernel and now think I understand what might go wrong.
Basically with the it6251 driver I think SIMPLE_DEV_PM_OPS should not be used due to the power switching.
It needs a separate freeze/thaw like in gpu/drm/qxl/qxl_drv.c and as described in include/linux/pm.h
Not sure if this solves it completely, but at least it should be much better.

If you want I can give it a go and send you a patch or you may want to do it yourself?

19 Reply by job

  • job
  • Member
  • Offline
  • Registered: 2015-04-16
  • Posts: 37

Re: hibernation

Happy to report that with the patch I send you everything comes back to normal after a night of hibernation!
Will keep experimenting a bit more. Now for the crypto accelerator ;-)

  • Registered: 2015-04-23
  • Posts: 39

Re: hibernation

The CAAM seems like quite an interesting piece of hardware. I'd very much like to see how it's RNG could help. Something even more interesting is that there's apparently a way to sign keys and load them for secure boot. Unfortunately there's no free tools to do this, and the official tools require requesting through Freescale, so I doubt they're free.

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: hibernation

Thanks for the patch, @job.  I'll merge that and give it a go.