Re: [PATCH v2 16/17] driver core: Refactor fw_devlink feature

[Saravana Kannan]

On Tue, Jan 5, 2021 at 1:04 PM Michael Walle <michael@xxxxxxxx> wrote:
>
> Am 2021-01-05 20:00, schrieb Saravana Kannan:
> > On Mon, Dec 28, 2020 at 7:35 PM Michael Walle <michael@xxxxxxxx> wrote:
> >>
> >> > The current implementation of fw_devlink is very inefficient because it
> >> > tries to get away without creating fwnode links in the name of saving
> >> > memory usage. Past attempts to optimize runtime at the cost of memory
> >> > usage were blocked with request for data showing that the optimization
> >> > made significant improvement for real world scenarios.
> >> >
> >> > We have those scenarios now. There have been several reports of boot
> >> > time increase in the order of seconds in this thread [1]. Several OEMs
> >> > and SoC manufacturers have also privately reported significant
> >> > (350-400ms) increase in boot time due to all the parsing done by
> >> > fw_devlink.
> >> >
> >> > So this patch uses all the setup done by the previous patches in this
> >> > series to refactor fw_devlink to be more efficient. Most of the code has
> >> > been moved out of firmware specific (DT mostly) code into driver core.
> >> >
> >> > This brings the following benefits:
> >> > - Instead of parsing the device tree multiple times during bootup,
> >> >   fw_devlink parses each fwnode node/property only once and creates
> >> >   fwnode links. The rest of the fw_devlink code then just looks at these
> >> >   fwnode links to do rest of the work.
> >> >
> >> > - Makes it much easier to debug probe issue due to fw_devlink in the
> >> >   future. fw_devlink=on blocks the probing of devices if they depend on
> >> >   a device that hasn't been added yet. With this refactor, it'll be very
> >> >   easy to tell what that device is because we now have a reference to
> >> >   the fwnode of the device.
> >> >
> >> > - Much easier to add fw_devlink support to ACPI and other firmware
> >> >   types. A refactor to move the common bits from DT specific code to
> >> >   driver core was in my TODO list as a prerequisite to adding ACPI
> >> >   support to fw_devlink. This series gets that done.
> >> >
> >> > [1] - https://lore.kernel.org/linux-omap/ea02f57e-871d-cd16-4418-c1da4bbc4696@xxxxxx/
> >> > Signed-off-by: Saravana Kannan <saravanak@xxxxxxxxxx>
> >> > Tested-by: Laurent Pinchart <laurent.pinchart@xxxxxxxxxxxxxxxx>
> >> > Tested-by: Grygorii Strashko <grygorii.strashko@xxxxxx>
> >>
> >> git bisect show that this commit broke my board in 5.11-rc1:
> >>
> >> [    2.294375] sysfs: cannot create duplicate filename
> >> '/devices/virtual/devlink/0000:00:00.1--0000:00:00.1'
> >> [    2.303999] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
> >> 5.11.0-rc1-00016-ga0fb284b267 #267
> >> [    2.312125] Hardware name: Kontron SMARC-sAL28 (4 Lane) (DT)
> >> [    2.317804] Call trace:
> >> [    2.320253]  dump_backtrace+0x0/0x1b8
> >> [    2.323936]  show_stack+0x20/0x70
> >> [    2.327263]  dump_stack+0xd8/0x134
> >> [    2.330677]  sysfs_warn_dup+0x6c/0x88
> >> [    2.334351]  sysfs_create_dir_ns+0xe8/0x100
> >> [    2.338547]  kobject_add_internal+0x9c/0x290
> >> [    2.342833]  kobject_add+0xa0/0x108
> >> [    2.346331]  device_add+0xfc/0x798
> >> [    2.349746]  device_link_add+0x454/0x5e0
> >> [    2.353682]  fw_devlink_create_devlink+0xb8/0xc8
> >> [    2.358316]  __fw_devlink_link_to_suppliers+0x84/0x180
> >> [    2.363474]  __fw_devlink_link_to_suppliers+0x134/0x180
> >> [    2.368718]  device_add+0x778/0x798
> >> [    2.372217]  device_register+0x28/0x38
> >> [    2.375979]  __mdiobus_register+0x94/0x340
> >> [    2.380089]  of_mdiobus_register+0xb4/0x380
> >> [    2.384285]  enetc_pf_probe+0x73c/0xb10
> >> [    2.388132]  local_pci_probe+0x48/0xb8
> >> [    2.391896]  pci_device_probe+0x120/0x1c0
> >> [    2.395920]  really_probe+0xec/0x3c0
> >> [    2.399505]  driver_probe_device+0x60/0xc0
> >> [    2.403614]  device_driver_attach+0x7c/0x88
> >> [    2.407810]  __driver_attach+0x60/0xe8
> >> [    2.411570]  bus_for_each_dev+0x7c/0xd0
> >> [    2.415419]  driver_attach+0x2c/0x38
> >> [    2.419004]  bus_add_driver+0x194/0x1f8
> >> [    2.422851]  driver_register+0x6c/0x128
> >> [    2.426698]  __pci_register_driver+0x4c/0x58
> >> [    2.430983]  enetc_pf_driver_init+0x2c/0x38
> >> [    2.435181]  do_one_initcall+0x54/0x2d8
> >> [    2.439029]  kernel_init_freeable+0x1fc/0x268
> >> [    2.443403]  kernel_init+0x1c/0x120
> >> [    2.446904]  ret_from_fork+0x10/0x30
> >> [    2.450502] kobject_add_internal failed for
> >> 0000:00:00.1--0000:00:00.1 with -EEXIST, don't try to register things
> >> with the same name in the same directory.
> >>
> >> Looks like it will generate that link twice? Let me know if I can help
> >> testing.
> >>
> >> See also: https://lavalab.kontron.com/scheduler/job/3894#L831
> >
> > I'll look into this this week. Is the DT for this board in upstream?
> > If so, can you point me to the DT file(s)?
>
> arch/arm64/boot/dts/freescale/fsl-ls1028a-kontron-kbox-a-230-ls.dts
>
> > Also, can you give me the output of this?
> > find /sys/devices -type d | grep "0000:00:00.1"
>
> # uname -a
> Linux buildroot 5.11.0-rc1-next-20210104 #298 SMP PREEMPT Tue Jan 5
> 21:55:23 CET 2021 aarch64 GNU/Linux
> # find /sys/devices -type d | grep "0000:00:00.1"
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/power
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/statistics
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/power
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-6
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-6/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-4
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-4/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-7
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-2
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-2/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-5
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-0
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-0/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-3
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-1
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-7
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-7/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-5
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-5/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-3
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-3/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-6
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-1
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/tx-1/byte_queue_limits
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-4
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-2
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/net/eno1/queues/rx-0
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/statistics
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/power
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04/statistics
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04/regulator
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04/regulator/regulator.3
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04/regulator/regulator.3/power
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04/regulator/regulator.4
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04/regulator/regulator.4/power
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/mdio_bus/0000:00:00.1/0000:00:00.1:04/power
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/msi_irqs
> /sys/devices/platform/soc/1f0000000.pcie/pci0000:00/0000:00:00.1/link
> /sys/devices/virtual/devlink/5000000.iommu--0000:00:00.1


Sent a fix. Any further discussion will most likely continue in that thread.
https://lore.kernel.org/lkml/20210106232641.459081-1-saravanak@xxxxxxxxxx/T/#u

-Saravana