• Registered: 2014-05-25
  • Posts: 12

Topic: boot time problem: stream of systemd[1]: Time has been changed

Hi All,

Slight problem. Suddenly, for no obvious reason, I have a problem at boot time ... I see the usual kernel messages and then it seems that systemd starts to continuously spam the console with a message "Time has been changed". These race past at quite some rate ... eventually becoming prefixed with "N printk messages dropped" where N is a number e.g. 12 or 13.

I cannot seem to get into the system via the keyboard to investigate/fix and since I'm travelling with just the novena I cannot try via the network.

Any suggestions?

Cheers,
Robb.
(aka roybatty)

  • Registered: 2014-05-25
  • Posts: 12

Re: boot time problem: stream of systemd[1]: Time has been changed

Happily enough I was able to recover the system ... using the builtin recovery mode / image. Duh smile.

My first couple of attempts at this didn't work, but maybe I wasn't doing it right. It seems like the user button must be held down from the very beginning i.e. while the reset is pressed.

So that is very cool! What I didn't expect (maybe it should have been obvious to me) was that /home would also be re-initialised. Oh well, I guess it's all gone now sad

So it seems like the recovery image boots OK without the clock/time issue. Either that kernel doesn't trigger the issue or the systemd ignores the error.  Hopefully the capacitor (I assume) is now getting charged.

Root cause of the issue was that I (me, not the novena) had a virus. This meant the novena was left disconnected for four days. Presumably this then emptied the battery and also the capacitor ... But it seems like this must have happened to novenas before now. So why wouldn't it boot this time around? A real hardware problem? Hardware aging? A feature in the newer software?

Meanwhile the system is busy downloading packages and, hopefully charging the capacitor at the same time.

Night!

  • Registered: 2014-05-25
  • Posts: 12

Re: boot time problem: stream of systemd[1]: Time has been changed

Aha. Replying to myself (again) - sorry. Please ignore my slights against recovery mode. I see now that this is just your typical transient boot mode providing a temporary stable system I can work from to fix stuff. Nothing is lost! Yet. Give me time smile

But what will heppen when I reboot?!? About to find out ...

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: boot time problem: stream of systemd[1]: Time has been changed

This sounds like an issue where the clock is initialized to the incorrect capacitance, and it isn't able to figure out what time it is.  I haven't figured out the exact cause yet.

However, in the chat, people have reported that one of the patches we have in 4.4 has fixed this problem for them.  I will push it to our servers soon, because it's proven so stable lately.

5 Reply by chris4795 (edited by chris4795 2016-03-29 06:59:45)

  • Registered: 2015-05-31
  • Posts: 117

Re: boot time problem: stream of systemd[1]: Time has been changed

Good Afternoon,

I am on 4.4 and the issue is still there. Whenever I attempt to reboot, this error comes up. The only way I can fix the issue is to disconnect power and the battery.

EDIT: It seems that the hardware clock was somehow reset the beginning of time. I fixed it by setting the time with ntp (install ntp) and then issing this command: 

#hwclock -w

But this seems to only be a temporary fix. Something keeps resetting it.

chris4795's Website

  • Registered: 2015-06-02
  • Posts: 4

Re: boot time problem: stream of systemd[1]: Time has been changed

I'm having the same problem. I can boot in recovery mode fine but the system is unusable booting off its SSD; this was after updating it to the latest software just now.

  • Registered: 2014-05-25
  • Posts: 12

Re: boot time problem: stream of systemd[1]: Time has been changed

Well, just to complete the discussion or feedback from my side (bit late - sorry). I ran in recovery mode for a couple of hours, just messing about, letting the capacitor charge (so I hoped) and after that time I was able to reboot in normal mode without any problem.
FYI, this issue occurred (for me) on a novena laptop running a xobs beta 4.4 kernel build/package.
IMHO, the root cause was my leaving it suspended and then unattended for three to four days.
IMHO(2), a defect in systemd means that in that state (empty capacitor / no valid time/data information) it is no longer possible to interact with the system enough to get it booted. I am calling that a defect because lack of valid time data should not prevent a system from booting. Also IMHO wink

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: boot time problem: stream of systemd[1]: Time has been changed

We released 4.4, with the hwclock fix included.  Let me know if it still happens.

  • Registered: 2015-05-31
  • Posts: 117

Re: boot time problem: stream of systemd[1]: Time has been changed

With 4.4, I do not have the time has changed issue anymore (I think part of my issue was the problem with the senoko). However, I turned it off on Sunday and back on Tuesday, and the clock said it was still Sunday (ntp was block on the Internet connection I am on).

Is this expected behavior?

chris4795's Website

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: boot time problem: stream of systemd[1]: Time has been changed

That's definitely weird.  No, that's not normal, and I'm not sure what would cause something like that.

  • Registered: 2015-05-31
  • Posts: 117

Re: boot time problem: stream of systemd[1]: Time has been changed

Hmm....I just opened it up and turned it on now to see what it said (i left it off for a few days), i got a time of almost exactly six hours off. I am suspecting it might be a configuration issue now.

chris4795's Website

  • xobs
  • Administrator
  • Offline
  • Registered: 2014-05-25
  • Posts: 442

Re: boot time problem: stream of systemd[1]: Time has been changed

Ah.  In that case, it sounds like the classic problem where your hwclock is in localtime and sysclock is UTC, or vice-versa, since presumably your timezone is UTC-6.  Though the only way that would come to light is if you're dual-booting two different systems that set their clocks independently.

  • Registered: 2015-05-31
  • Posts: 117

Re: boot time problem: stream of systemd[1]: Time has been changed

So I thought I corrected it, but it is in the wrong time again. I am wondering if in the low power state (after shutdown, but the battery is still attached), the oscillator is being set incorrectly?

chris4795's Website

  • Registered: 2016-07-22
  • Posts: 3

Re: boot time problem: stream of systemd[1]: Time has been changed

It is seems there is an issue with the oscillator. Check it properly.

  • Registered: 2015-05-31
  • Posts: 117

Re: boot time problem: stream of systemd[1]: Time has been changed

I finally figured out the issue. The battery that powers the RTC when it powered off was at 0.3 V. I replaced that battery and it appears to work!

chris4795's Website

  • From: Utrecht, Netherlands
  • Registered: 2014-06-08
  • Posts: 47

Re: boot time problem: stream of systemd[1]: Time has been changed

my novena has the same issue (ok, it is 4 years later). I did not use the novena for almost 10 months), so the battery (or is it a supercap) also measures less than 0.3v, so that might be it. I'll let it be powered up for few hours )in case it is a supercap) and check the schematics it it might be a battery. If it is a battery I will replace it soon.
The sottware was updated last time it was switched on, but that was just apt update and apt upgrade, so probably not the case.

roheve's Website