Topic: Linux 4.7.2 with Novena patchset applied

Hi,

I've been working on an overlay of Gentoo ebuilds shadowing xobs' Debian packages for the Novena, which is nearing completion now. As part of this work, I've migrated the Novena-specific kernel patches from novena-linux (currently baselined against Linux 4.4) so that they can be applied to a 4.7.2 kernel. The resulting kernel seems to work pretty well.

Most of xobs' original patches needed no, or only small context changes to get them to apply, but as noted on the project page, there were a few exceptions (for example, with the es8328 audio codec, which has changed significantly since 4.4).

The modified patchset is available on GitHub here together with instructions for application / build (no need to be on Gentoo to use this, btw).

If you get a chance to try it out, please let me know of any issues.

Best, sakaki

Re: Linux 4.7.2 with Novena patchset applied

Thanks a lot for updating the patches!

Re: Linux 4.7.2 with Novena patchset applied

I tried installing this kernel on my novena last night unsuccessfully.  Compilation went well and applying the patches did too, but upon boot the kernel reports as panic and each cpu core halts.  it tries to reboot but is unable to, so the system is halted.  Not sure what could have gone wrong. I'll try it again this evening though.

Re: Linux 4.7.2 with Novena patchset applied

Hi veleiro -
Sorry to hear you're having trouble with this.

A few quick questions:

  • Did you make sure to make dtbs and update the novena.dtb file in /boot as well as the zImage?

  • What configuration did you use with the new kernel (one from make novena_defconfig, or your own)?

  • Are the modules installed correctly in /lib/modules (you should have a /lib/modules/4.7.2 directory present)?

  • Is the underlying kernel a vanilla 4.7.2 (downloaded via https://cdn.kernel.org/pub/linux/kernel … 7.2.tar.xz or similar), or are you trying this with a more modern kernel?

  • Are you booting with an external HDMI screen attached or with the LCD screen?

  • Which configuration are you running (just the board/desktop/laptop/heirloom)?

  • Have you tried booting / are you able to boot with the earlier 4.4 kernel built yourself from xobs' novena-linux?

If you can't get it to work after another go and you have a spare microSD card handy, you could try writing one of my Gentoo images to it and see if (purely as a diagnostic) you can boot from that (as that should definitely boot OK, unless there's some odd hardware variation I don't know about (entirely possible!!)).

Best

sakaki

Re: Linux 4.7.2 with Novena patchset applied

For convenience, I've just built a novena patched (but otherwise vanilla) 4.7.2 kernel zImage, novena.dtb and module set, and uploaded it on the novena-kernel-patches release page, here.

It's only about 102MiB compressed. You can download this and try booting from it, so (doing this as root, for simplicity, and assuming your /boot is mounted):

~ # wget -c https://github.com/sakaki-/novena-kernel-patches/releases/download/v4.7.2/4.7.2-build.tar.xz
~ # tar -xJf 4.7.2-build.tar.xz
~ # cd 4.7.2-build
4.7.2-build # cp -r place_in_lib_modules/4.7.2 /lib/modules/
4.7.2-build # mv /boot/zImage{,.bak}
4.7.2-build # mv /boot/novena.dtb{,.bak}
4.7.2-build # cp place_in_boot/{zImage,novena.dtb} /boot/
4.7.2-build # sync

Then try a reboot and see what happens.

Best, sakaki

Re: Linux 4.7.2 with Novena patchset applied

Most of the questions you asked are how it is. I have the Laptop version, and i followed the instructions from the readme on your github novena patches page.  I did novena_defconfig without additional modifications, have no external display, and i was previously using 4.4 from xobs.  the novena.dtb and zImage were both compiled, the modules installed successfully, and the files copied to /boot.  This weekend, I'll try your precompiled kernel and the gentoo image, thanks a lot!

7 (edited by sakaki 2016-10-09 22:15:29)

Re: Linux 4.7.2 with Novena patchset applied

One thing I forgot to ask was whether the kernel was panicking before or after starting init (systemd, assuming your base distro is Debian).

As a further diagnostic therefore, I've uploaded a version of the novena-patched 4.7.2 kernel which includes a tiny, integral busybox-based initramfs to the novena-kernel-patches release page, here. It is only 7MiB.

You can copy the file (zImage-busybox) to your /boot directory (renaming to zImage) with the accompanying novena.dtb, and try booting from this (no need for a module set). If successful, it should quickly drop you to a busybox (ash) console root shell. If this works and lets you do things like "ls /" and mount partitions from /dev/sda (or whatever), you can be reasonably sure that the patched 4.7.2 kernel itself works with your Novena, and that any problems lie later (during systemd init, and/or module loading).

Unfortunately I don't have access to a laptop configuration machine and so have had to do most of the testing on a "just the board" system (and some further tests on a desktop), so it is quite possible there is a hardware issue with the patchset that I haven't anticipated.

Any feedback from your tests would be very much appreciated!

hth, sakaki

Re: Linux 4.7.2 with Novena patchset applied

Hi, quick update, I've managed to borrow a laptop spec Novena now, and have replicated your boot issue. Appears to be a problem with the eDP bridge; almost certainly a patch I haven't adapted correctly from xobs' 4.4. I'll have a look and hopefully fix this weekend while I still have the loan machine. FWIW, the 4.7.2 kernel will definitely boot if you disconnect your eDP adaptor board (but I am not encouraging you to part-dismantle your machine!!)
I will post again once I have got to the bottom of this, best, sakaki

Re: Linux 4.7.2 with Novena patchset applied

Hi again,

a bit of fun with kdb, but I think this should be fixed now. Turns out the problem was not with the patches per se but with some other code that had changed between 4.4 and 4.7.2, in drivers/gpu/drm/imx/imx-ldb.c -> imx_ldb_bind(...).

Specifically, the 4.4 code used to do a two-stage endpoint lookup:

        port = of_graph_get_port_by_id(child, imx_ldb->lvds_mux ? 4 : 2);
        if (port) {
            struct device_node *endpoint, *remote;

            endpoint = of_get_child_by_name(port, "endpoint");

but this changed by 4.7.2 to a single-step lookup:

        ep = of_graph_get_endpoint_by_regs(child,
                           imx_ldb->lvds_mux ? 4 : 2,
                           -1);

Unfortunately the new  of_graph_get_endpoint_by_regs(...) call kills the kernel and stops the boot (if you have an eDP daughter card fitted, so it doesn't affect "just the board" Novenas).

I've pushed a change to novena-kernel-patches to revert to the old, two-stage lookup, and the resulting kernel is booting fine on my loaner laptop-spec Novena.

Assuming you still have the 4.7.2 tarball around (simply wget it again, if not), you can do the following to test this new kernel:

~ $ cd novena-kernel-patches
novena-kernel-patches $ git pull
(to fetch the updated patch)
novena-kernel-patches $ cd..
~ $ rm -rf linux-4.7.2
~ $ tar -xJf linux-4.7.2.tar.xz
~ $ cd linux-4.7.2
linux-4.7.2 $ ../novena-kernel-patches/apply-patches.sh
( assuming that went OK, then proceed )
linux-4.7.2 $ make novena_defconfig
linux-4.7.2 $ make -j4 zImage modules dtbs
( this will take some time! )
linux-4.7.2 $ sudo make modules_install
( assuming your /boot is mounted, do: )
linux-4.7.2 $ cd /boot
boot $ sudo mv zImage{,.bak}
boot $ sudo mv novena.dtb{,.bak}
boot $ sudo cp "${HOME}/linux-4.7.2/arch/arm/boot/zImage" .
boot $ sudo cp "${HOME}/linux-4.7.2/arch/arm/boot/dts/imx6q-novena.dtb" novena.dtb
boot $ sync

Then reboot!

Please let me know if this update works for you; if so, I will update the gentoo images with the modified kernel in the next day or so.

Thanks again for reporting the issue, and apologies for the inconvenience caused ><

Best, sakaki

Re: Linux 4.7.2 with Novena patchset applied

sakaki wrote:

Unfortunately the new  of_graph_get_endpoint_by_regs(...) call kills the kernel and stops the boot (if you have an eDP daughter card fitted, so it doesn't affect "just the board" Novenas).

Should this be reported upstream, or is it specific for the Novena somehow? I don't have enough of a clue of what's going on to know, but it should be patched in mainline if it's incorrect behavio (as opposed to the Novena being out of spec somehow).

Thanks again for your awesome work... once I get back to Novena-ing, I'll be getting Gentoo up and running!

Re: Linux 4.7.2 with Novena patchset applied

dbtayl wrote:

Should this be reported upstream, or is it specific for the Novena somehow? I don't have enough of a clue of what's going on to know, but it should be patched in mainline if it's incorrect behavio (as opposed to the Novena being out of spec somehow).

Not sure - there have been a number of commits to drivers/of/base.c (where of_graph_get_endpoint_by_regs lives) since 4.7.2, some of which may potentially change the behaviour seen (for example, 34276bb). Best thing is probably to merge the patchset against 4.8.1 (using the newer, single lookup semantics) and see if the issue is already fixed. I'll do that at some point but next up is an 'install-to-sata' utility that supports LVM over LUKS ^-^

Best, sakaki

PS the 4.7.2 Novena patch for  drivers/gpu/drm/imx/imx-ldb.c (0012) will need modification/removal for 4.8.x in any case, since this now supports drm_bridges directly (dc80d70).

Re: Linux 4.7.2 with Novena patchset applied

wow, it worked sakaki!  Sorry that i havent replied much.  when i originally tried to build and install your kernel and it didnt work, i couldnt properly recover it because my other microsd card readers werent acting correctly.  I bought a new one recently and started at it again.  Thanks!

Re: Linux 4.7.2 with Novena patchset applied

I got the hardened, deblobbed 4.7.2 kernel working, thanks sakaki!  The only issue I had was that '/dev/mmcblk0' got renamed to '/dev/mmcblk1', which caused '/etc/fstab' to throw a fit until I "fixed" it by updating the entry.  Not sure what caused the change, though.

Re: Linux 4.7.2 with Novena patchset applied

By the way, I have after the upgrade there were problems with the network. However, I recently started to use proxies and VPN to access the network, but I don't even know if this could be connected. In General, if anyone has the same problem, I can say about it..

Re: Linux 4.7.2 with Novena patchset applied

Hi, Just posting to report success. 4.7.2 working fine for me on first try.
Thanks for your work on this, Sakaki.

16 (edited by dbtayl 2017-09-11 10:24:48)

Re: Linux 4.7.2 with Novena patchset applied

I messed around with applying sakaki's patches to a more recent kernel a while back- 4.12.4. Some patches appeared to be unnecessary for various reasons (already in the kernel, I just didn't care, etc.), some required some modifications to apply, and some applied cleanly. Anyway, I worked through it, rebased the patches*, and bundled them up- see below.

I haven't tested this in any way whatsoever- it may or may not even compile. Use at your own risk.

If that doesn't deter you, you can grab the patches here: https://drive.google.com/drive/folders/ … sp=sharing

There are two tarballs- one has just the final patches ("...-minimal.tgz"), and one has my working directory ("...-full.tgz"). The only thing you might get out of the latter is seeing which patches I didn't bother to include- that's in a folder creatively labeled "no". I seem to recall many of those were around the Debian build files, which I have no interest in.

There's also a README, but I don't remember if I put anything useful in there.

I'll probably put more effort into this once I actually get around to using my Novena, but maybe somebody will find this useful before then.


*in case anybody is wondering or cares, the easiest way I found to do that was to clone the Linux git repo. I then got the patches to apply, one at a time, and checked in the code after each patch. After that, you can use git to spit out new, rebased patches for each commit.

17 (edited by chris4795 2017-10-07 22:04:01)

Re: Linux 4.7.2 with Novena patchset applied

Has anyone gotten the 4.7.2 to work an encrypted root disk? I cannot do that with the current novena_defconfig, and I have a feeling there is an option I am not enabling (I am able to make a uInitrd for with this guide):

https://github.com/umiddelb/armhf/wiki/ … ARM-device

(note to disable CONFIG_DEBUG_INFO=y so the initramfs isn't huge)

and I got past the "cannot initialize device-mapper" error with this guide:

https://unix.stackexchange.com/question … ice-mapper

The latest error is that it could not find AES-XTS-128, but that was loaded as a module. I am recompiling with a suggestion form this post:

https://www.kosagi.com/forums/viewtopic … d=642#p642

I am also using the config from /proc/config.gz right now instead of the novena_defconfig one.

EDIT: I tried both, and got an error of "unknown symbol GLOBAL_OFFSET_TABLE" and still had the error of "Check that kernel supports aes-xts-plain64 cipher". I also tried pulling off the patches to get it working in the 4.4.90 kernel (as 4.4 will be supported until 2022), but I got a blank screen on it. When I have some more time, I will hook up the SPI port and see if I see more, and if I can actually compile xobs linux-novena and get it to boot with the encrypted hard drive.

18 (edited by chris4795 2017-10-09 07:35:50)

Re: Linux 4.7.2 with Novena patchset applied

At this time I have been unable to successfully compile the kernel with Full system encryption support. Right now I am trying to compile xobs' kernel from his github page. If I try to compile it with the default novena_defconfig, I still get the "Cannot initialize device-mapper" issue where it complains that dm_mod cannot be found.

I fixed it via the guide I referenced, and the error I know get is:

"dm_crypt: Unkown symbol _GLOBAL_OFFSET_TABLE_ (err 0)"

At this point I am not really sure how to proceed, any help would be appreciated, as I would like to be able to get 4.4.90 to work so we have a recent kernel here (and possibly get 4.12.4 or the current kernel working).

EDIT: I got xobs' kernel to successfully boot with encryption support! I have added the config file here:

https://github.com/chris4795/Linux_Config_Novena

The other nice thing I found out is that there is no special hook needed in the initramfs-tools with this config. I think the default kernel did not have the correct configuration needed in order to get it to work right, hence the hacks for it in "Novena-debian-support". The strange thing is if I made some of the options above modules instead of compiling directly in, they did not work. I am now wokring on the 4.7.2 kernel and getting encryption support to work. Now that I figured it out on xobs', I am a lot more confident I can get 4.7.2 to work.

EDIT 2: I now have the 4.7.2 kernel working with full system encryption support! In addition to the edtis above, I had to compile in the XTS support. I still got a couple fo errors with ipv6 and x-tables with the "Unkown symbol _GLOBAL_OFFSET_TABLE_ (err 0)" error, and that is a matter of hunting them down in the config file and changing the module to compile in. If anyone knows where they are I would appreciate it. I have added that config file here along with instructions to recreate what I did:

https://github.com/chris4795/Linux_Config_Novena