Topic: Scattered ping timings

Hi, found some weird timing issue with the onboard PHY and latest stable g3d69696

        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 7
        Transceiver: external
        Auto-negotiation: on
        Link detected: yes

The first sequence almost clearly shows a 10ms step, though further steps are less accurate:

root@novenaboard:~# ping 192.168.10.125
PING 192.168.10.125 (192.168.10.125) 56(84) bytes of data.
64 bytes from 192.168.10.125: icmp_seq=1 ttl=64 time=0.263 ms
64 bytes from 192.168.10.125: icmp_seq=2 ttl=64 time=0.851 ms
64 bytes from 192.168.10.125: icmp_seq=3 ttl=64 time=12.7 ms
64 bytes from 192.168.10.125: icmp_seq=4 ttl=64 time=20.9 ms
64 bytes from 192.168.10.125: icmp_seq=5 ttl=64 time=30.7 ms
64 bytes from 192.168.10.125: icmp_seq=6 ttl=64 time=39.0 ms
64 bytes from 192.168.10.125: icmp_seq=7 ttl=64 time=1.73 ms
64 bytes from 192.168.10.125: icmp_seq=8 ttl=64 time=17.8 ms
64 bytes from 192.168.10.125: icmp_seq=9 ttl=64 time=39.8 ms
64 bytes from 192.168.10.125: icmp_seq=10 ttl=64 time=61.9 ms
64 bytes from 192.168.10.125: icmp_seq=11 ttl=64 time=66.9 ms
64 bytes from 192.168.10.125: icmp_seq=12 ttl=64 time=65.9 ms
64 bytes from 192.168.10.125: icmp_seq=13 ttl=64 time=2.04 ms
64 bytes from 192.168.10.125: icmp_seq=14 ttl=64 time=11.0 ms
64 bytes from 192.168.10.125: icmp_seq=15 ttl=64 time=20.8 ms
64 bytes from 192.168.10.125: icmp_seq=16 ttl=64 time=29.0 ms
64 bytes from 192.168.10.125: icmp_seq=17 ttl=64 time=38.8 ms
64 bytes from 192.168.10.125: icmp_seq=18 ttl=64 time=46.9 ms
64 bytes from 192.168.10.125: icmp_seq=19 ttl=64 time=10.8 ms
64 bytes from 192.168.10.125: icmp_seq=20 ttl=64 time=4.89 ms

dmesg:
[    0.952160] Switched to clocksource mxc_timer1

Should I try with ptp as well to see or there are other possible ideas to try against? Thanks!

Re: Scattered ping timings

Ok, it seems this is how i.MX timer works. Folks who intended to see smaller timings for some reason should play with CONFIG_*HZ.

Re: Scattered ping timings

Interesting.  Perhaps that value should be fiddled with in kernel builds we make.

Re: Scattered ping timings

For what it's worth, I've run in to this behaviour on other ARM SoCs in the past - very confusing when you're not expecting it but doesn't seem to harm anything.

5 (edited by rian 2018-02-20 05:44:08)

Re: Scattered ping timings

Trying to use my novena as a temporary IP router and I've noticed that ping times on the SoC eth0 link are much slower and vary more than ping times on the AX88772B eth1 link. Both interfaces are connected to the same subnet (and same switch for that matter):

$ ip route
default via 192.168.1.254 dev eth1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.85
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.84

If ping on eth1 I get a consistent ~0.4ms ping time:

$ ping -I eth1 -c 5 192.168.1.254
PING 192.168.1.254 (192.168.1.254) from 192.168.1.84 eth1: 56(84) bytes of data.
64 bytes from 192.168.1.254: icmp_seq=1 ttl=64 time=0.502 ms
64 bytes from 192.168.1.254: icmp_seq=2 ttl=64 time=0.323 ms
64 bytes from 192.168.1.254: icmp_seq=3 ttl=64 time=0.388 ms
64 bytes from 192.168.1.254: icmp_seq=4 ttl=64 time=0.379 ms
64 bytes from 192.168.1.254: icmp_seq=5 ttl=64 time=0.352 ms

If I do the same on eth0, results are more different than expected:

$ ping -I eth0 -c 5 192.168.1.254
PING 192.168.1.254 (192.168.1.254) from 192.168.1.85 eth0: 56(84) bytes of data.
64 bytes from 192.168.1.254: icmp_seq=1 ttl=64 time=0.272 ms
64 bytes from 192.168.1.254: icmp_seq=2 ttl=64 time=10.0 ms
64 bytes from 192.168.1.254: icmp_seq=3 ttl=64 time=18.3 ms
64 bytes from 192.168.1.254: icmp_seq=4 ttl=64 time=28.5 ms
64 bytes from 192.168.1.254: icmp_seq=5 ttl=64 time=37.0 ms

Latency is very critical as this is a router. What is the recommended way to fix this?

Re: Scattered ping timings

Hello,

Can you tell me which eth{0,1} is which? I am wondering if the issue you are running into is that eth0 is connected to the Ethernet controller while eth1 is the USB connected Ethernet adaptor.

7 (edited by creardon 2019-04-05 06:03:36)

Re: Scattered ping timings

Hi all. I found the fix to this problem a while back, when looking into getting magic packet detection  working for wake-on-lan (I think I discovered that this would never work. I had a bunch of documentation about it but can't find it right now).

Anyway, the scattered ping timings (this is on the ethernet controller eth0, the usb ethernet adapter eth1 doesn't have this problem) are due to an interrupt not being serviced when the CPU is in sleep states. I think the gist of it is that it's a hardware bug in the CPU, where the interrupt gets masked and doesn't wake up the cpu, and so it only gets serviced when something else wakes it up (this would explain why the ping timings are seemingly random). There's a workaround where you can route the interrupt through a different pin, you just need to apply the changes in the kernel and compile/install. Here's what I got from rummaging around freescale mailing lists (I can try to dig up the threads if anyone is interested).

You need to add this to the FEC device tree entry in imx6q-novena.dts:

    compatible = "fsl,imx6q-fec";
    reg = <0x02188000 0x4000>;
    interrupts-extended = <&gpio1 6 IRQ_TYPE_LEVEL_HIGH>,
                  <&intc 0 119 IRQ_TYPE_LEVEL_HIGH>;   
    fsl,magic-packet;
    fsl,err006687-workaround-present;

So the entry now looks like:

&fec {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_enet_novena>;
    phy-mode = "rgmii";
    phy-reset-gpios = <&gpio3 23 GPIO_ACTIVE_HIGH>;
    rxc-skew-ps = <3000>;
    rxdv-skew-ps = <0>;
    txc-skew-ps = <3000>;
    txen-skew-ps = <0>;
    rxd0-skew-ps = <0>;
    rxd1-skew-ps = <0>;
    rxd2-skew-ps = <0>;
    rxd3-skew-ps = <0>;
    txd0-skew-ps = <3000>;
    txd1-skew-ps = <3000>;
    txd2-skew-ps = <3000>;
    txd3-skew-ps = <3000>;
    status = "okay";
    compatible = "fsl,imx6q-fec";
    reg = <0x02188000 0x4000>;
    interrupts-extended = <&gpio1 6 IRQ_TYPE_LEVEL_HIGH>,
                  <&intc 0 119 IRQ_TYPE_LEVEL_HIGH>;   
    fsl,magic-packet;
    fsl,err006687-workaround-present;   
};


And add this to pinctrl_enet_novena:

MX6QDL_PAD_GPIO_6__ENET_IRQ        0x000b1

so it looks like this:

enet {
        pinctrl_enet_novena: enetgrp-novena {
            fsl,pins = <
                MX6QDL_PAD_ENET_MDIO__ENET_MDIO       0x1b0b0
                MX6QDL_PAD_ENET_MDC__ENET_MDC         0x1b0b0
                MX6QDL_PAD_RGMII_TXC__RGMII_TXC       0x1b020
                MX6QDL_PAD_RGMII_TD0__RGMII_TD0       0x1b028
                MX6QDL_PAD_RGMII_TD1__RGMII_TD1       0x1b028
                MX6QDL_PAD_RGMII_TD2__RGMII_TD2       0x1b028
                MX6QDL_PAD_RGMII_TD3__RGMII_TD3       0x1b028
                MX6QDL_PAD_RGMII_TX_CTL__RGMII_TX_CTL 0x1b028
                MX6QDL_PAD_ENET_REF_CLK__ENET_TX_CLK  0x1b0b0
                MX6QDL_PAD_RGMII_RXC__RGMII_RXC       0x1b0b0
                MX6QDL_PAD_RGMII_RD0__RGMII_RD0       0x1b0b0
                MX6QDL_PAD_RGMII_RD1__RGMII_RD1       0x1b0b0
                MX6QDL_PAD_RGMII_RD2__RGMII_RD2       0x1b0b0
                MX6QDL_PAD_RGMII_RD3__RGMII_RD3       0x1b0b0
                MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL 0x1b0b0
                MX6QDL_PAD_GPIO_16__ENET_REF_CLK      0x4001b0a8

                /* Ethernet reset */
                MX6QDL_PAD_EIM_D23__GPIO3_IO23  0x80000000
               
                /* IRQ workaround */
                MX6QDL_PAD_GPIO_6__ENET_IRQ        0x000b1               
            >;
        };
    };


I also applied this patch I found:

From 29380905565655bb797bf670a173bddb8e641da6 Mon Sep 17 00:00:00 2001
From: Lucas Stach <l.stach@pengutronix.de>
Date: Fri, 3 Jun 2016 18:31:19 +0200
Subject: [PATCH] ARM: imx6: disable deeper idle states when FEC is active w/o
HW workaround

The i.MX6 Q/DL has an erratum (ERR006687) that prevents the FEC from
waking the CPUs when they are in wait(unclocked) state. As the hardware
workaround isn't applicable to all boards, disable the deeper idle state
when the workaround isn't present and the FEC is in use.

This allows to safely run a kernel with CPUidle enabled on all i.MX6
boards.

Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Acked-by: David S. Miller <davem@davemloft.net> (for network changes)
Signed-off-by: Shawn Guo <shawnguo@kernel.org>
---
Documentation/devicetree/bindings/net/fsl-fec.txt |  3 +++
arch/arm/mach-imx/cpuidle-imx6q.c                 | 16 +++++++++++++++
drivers/net/ethernet/freescale/fec.h              |  2 ++
drivers/net/ethernet/freescale/fec_main.c         | 12 +++++++++++
include/soc/imx/cpuidle.h                         | 25 +++++++++++++++++++++++
5 files changed, 58 insertions(+)
create mode 100644 include/soc/imx/cpuidle.h

diff --git a/Documentation/devicetree/bindings/net/fsl-fec.txt b/Documentation/devicetree/bindings/net/fsl-fec.txt
index b037a9d78d93..a1e3693cca16 100644
--- a/Documentation/devicetree/bindings/net/fsl-fec.txt
+++ b/Documentation/devicetree/bindings/net/fsl-fec.txt
@@ -27,6 +27,9 @@ Optional properties:
   number to 1.
- fsl,magic-packet : If present, indicates that the hardware supports waking
   up via magic packet.
+- fsl,err006687-workaround-present: If present indicates that the system has
+  the hardware workaround for ERR006687 applied and does not need a software
+  workaround.

Optional subnodes:
- mdio : specifies the mdio bus in the FEC, used as a container for phy nodes
diff --git a/arch/arm/mach-imx/cpuidle-imx6q.c b/arch/arm/mach-imx/cpuidle-imx6q.c
index 353bb8774112..c3cc8ca8d2ff 100644
--- a/arch/arm/mach-imx/cpuidle-imx6q.c
+++ b/arch/arm/mach-imx/cpuidle-imx6q.c
@@ -62,6 +62,22 @@ static struct cpuidle_driver imx6q_cpuidle_driver = {
    .safe_state_index = 0,
};

+/*
+ * i.MX6 Q/DL has an erratum (ERR006687) that prevents the FEC from waking the
+ * CPUs when they are in wait(unclocked) state. As the hardware workaround isn't
+ * applicable to all boards, disable the deeper idle state when the workaround
+ * isn't present and the FEC is in use.
+ */
+void imx6q_cpuidle_fec_irqs_used(void)
+{
+    imx6q_cpuidle_driver.states[1].disabled = true;
+}
+
+void imx6q_cpuidle_fec_irqs_unused(void)
+{
+    imx6q_cpuidle_driver.states[1].disabled = false;
+}
+
int __init imx6q_cpuidle_init(void)
{
    /* Set INT_MEM_CLK_LPM bit to get a reliable WAIT mode support */
diff --git a/drivers/net/ethernet/freescale/fec.h b/drivers/net/ethernet/freescale/fec.h
index f58f9ea51639..dc71a88e9c55 100644
--- a/drivers/net/ethernet/freescale/fec.h
+++ b/drivers/net/ethernet/freescale/fec.h
@@ -442,6 +442,8 @@ struct bufdesc_ex {
#define FEC_QUIRK_SINGLE_MDIO        (1 << 11)
/* Controller supports RACC register */
#define FEC_QUIRK_HAS_RACC        (1 << 12)
+/* Interrupt doesn't wake CPU from deep idle */
+#define FEC_QUIRK_ERR006687        (1 << 13)

struct bufdesc_prop {
    int qid;
diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
index ca2cccc594fd..8c2110b61684 100644
--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -60,6 +60,7 @@
#include <linux/if_vlan.h>
#include <linux/pinctrl/consumer.h>
#include <linux/prefetch.h>
+#include <soc/imx/cpuidle.h>

#include <asm/cacheflush.h>

@@ -2820,6 +2821,9 @@ fec_enet_open(struct net_device *ndev)
    if (ret)
        goto err_enet_mii_probe;

+    if (fep->quirks & FEC_QUIRK_ERR006687)
+        imx6q_cpuidle_fec_irqs_used();
+
    napi_enable(&fep->napi);
    phy_start(ndev->phydev);
    netif_tx_start_all_queues(ndev);
@@ -2855,6 +2859,9 @@ fec_enet_close(struct net_device *ndev)

    phy_disconnect(ndev->phydev);

+    if (fep->quirks & FEC_QUIRK_ERR006687)
+        imx6q_cpuidle_fec_irqs_unused();
+
    fec_enet_clk_enable(ndev, false);
    pinctrl_pm_select_sleep_state(&fep->pdev->dev);
    pm_runtime_mark_last_busy(&fep->pdev->dev);
@@ -3294,6 +3301,11 @@ fec_probe(struct platform_device *pdev)

    platform_set_drvdata(pdev, ndev);

+    if ((of_machine_is_compatible("fsl,imx6q") ||
+         of_machine_is_compatible("fsl,imx6dl")) &&
+        !of_property_read_bool(np, "fsl,err006687-workaround-present"))
+        fep->quirks |= FEC_QUIRK_ERR006687;
+
    if (of_get_property(np, "fsl,magic-packet", NULL))
        fep->wol_flag |= FEC_WOL_HAS_MAGIC_PACKET;

diff --git a/include/soc/imx/cpuidle.h b/include/soc/imx/cpuidle.h
new file mode 100644
index 000000000000..986a4823bce1
--- /dev/null
+++ b/include/soc/imx/cpuidle.h
@@ -0,0 +1,25 @@
+/*
+ * Copyright 2016 Pengutronix, <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __SOC_IMX_CPUIDLE_H__
+#define __SOC_IMX_CPUIDLE_H__
+
+#if defined(CONFIG_CPU_IDLE) && defined(CONFIG_SOC_IMX6Q)
+void imx6q_cpuidle_fec_irqs_used(void);
+void imx6q_cpuidle_fec_irqs_unused(void);
+#else
+void imx6q_cpuidle_fec_irqs_used(void) { }
+void imx6q_cpuidle_fec_irqs_unused(void) { }
+#endif
+
+#endif /* __SOC_IMX_CPUIDLE_H__ */
--
2.11.0

Re: Scattered ping timings

top was the Dayton contentions amidst the seventies and mid eighties when they Burns ensured the spot. It because a Race to twenty three was so it would just take three racks max to wrap up.