Re: [RFC PATCH 0/5] Add CONFIG symbol as module attribute

[Cristina-Gabriela Moraru]

2016-08-18 19:55 GMT+02:00 Luis R. Rodriguez <mcgrof@xxxxxxxxxx>:
> On Wed, Aug 17, 2016 at 09:26:58PM +0200, Cristina Moraru wrote:
>
>> This patchset implements dynamic pegging of kconfig symbol
>> into driver modinfo section
>
> First a little bit of motivation here helps, so let me try to
> help fill in some gaps. This may help explain what you have
> been working on a bit more.
>
> First, for those that were not Cc'd but curious about this work
> so far, you can read the patches here:
>
> Original cover letter:
>
> https://lkml.kernel.org/r/1471462023-119645-1-git-send-email-cristina.moraru09@xxxxxxxxx
>
> https://marc.info/?l=linux-kernel&m=147146213519750&w=2 - patch 1
> https://marc.info/?l=linux-kernel&m=147146209019744&w=2 - patch 2
> https://marc.info/?l=linux-kernel&m=147146211819747&w=2 - patch 3
> https://marc.info/?l=linux-kernel&m=147146209119745&w=2 - patch 4
> https://marc.info/?l=linux-kernel&m=147146209519746&w=2 - patch 5
>
> There are a few situations in which you may want to extract a
> subset of Kconfig CONFIG_* symbols for a system. At least when
> only considering modules:
>
>  a) When optimizing build requirements for a kernel for a system.
>     That is you boot into a distro kernel and then want to build
>     a slim kernel only with sensible kernel configuration options.
>
>  b) When you are on a distribution kernel but the distribution
>     kernel provided lacks hardware support for your device, you
>     may either want to upgrade the full kernel in which case you
>     want to do a) or -- you may want to just a backports release
>     which provides just the modules you need, you'd use it on top
>     of the distribution kernel.
>
> Are there other *uses* outside of the two above for having a direct
> deterministic mapping between module and a kconfig symbol ? What
> if we could also do this for built-in code as well ? kconfig-sat [0] is
> work that is still being ironed out, but if that was in place and we
> handed it a set of required options we want, it should in theory help
> resolve the dependencies to give us a functional .config for a kernel.
> This of course is a long term possible goal just used as an example
> -- but are there other uses?
>
I can't think of any examples. I will add this ones in the cover
letter as motivation.

> Another possible gain here is if addressing a deterministic mapping might
> help with some of the limitations observed in practice over years with
> kconfig and semantics for keconfig. I go into some of this further below.
>
> There are not many options currently available to optimize kernel builds this
> way for modules or choose only what kernel kconfig options you need.  One of
> the best known efforts is the upstream 'make localmodconfig' added by Steven
> years ago to help with a). This has its own set of limitations though. For
> backports we also need to address possible kconfig symbol name changes, and the
> fact that you may boot a box without a driver loaded and as such not be sure
> what new kernel kconfig symbol you may need.
>
> There are shared issues with both goals, addressing shared issues
> can ultimately help both. One of the shared known issues is the
> lack of a deterministic resolution of a module to a respective
> kconfig symbol. Even answering if this is possible or should be
> possible requires some work.
>
> The work Cristina has done here for this year's Google Summer of
> Code is try to help start looking into addressing the question if
> this is possible and then start looking into how we could annotate
> such a thing, and later how we'd use it.
>
> [0] https://kernelnewbies.org/KernelProjects/kconfig-sat
>
>> * updates streamline_config.pl to generate the auxiliary file
>> scripts/mod/Module.ksymb containing associations of driver file
>> names and corresponding kconfig symbols CONFIG_*
>
> Here you make use of existing heuristics -- note however that
> heuristics are needed as we have no direct mapping available
> nor do we have any rules over how to make such a direct mapping
> possible if we wanted this mapping to be deterministic.
>
>> * updates Makefiles to trigger streamline_config.pl call for
>> Module.ksymb generation and pass the determined CONFIG_* symbol
>> as compilation parameter -D in KBUILD_KSYMB variable
>
> Passing -D for define a new KBUILD_KSYMB when available would be a way to pass
> to the build system a config symbol associated with a module, its the right
> approach, however re-using streamline_config.pl seems questionable here, and it
> seems an optimization might be / should be possible. Likewise we'd need to also
> still look into and address *why* some modules don't end up with a respective
> kconfig symbol -- and if we wanted to do something about this. Given the stated
> goals more importantly would be what the costs are to have a deterministic
> mapping.
>

Yes. I've noticed. I will change it completely.

>> * adds kconfig_symb as module attribute
>
> This seems lightweight and follows the logic of the other attributes.
> If its too much, this can be a configurable feature.
>
>> * updates modpost to set KBUILD_KSYMB macro as value for
>> kbuild_symb attribute
>
> Neat.
>
>> Note: the content of the file Module.ksymb is generated for
>> all modules that have only one associate CONFIG option. All
>> others are considered to be components linked at the final
>> modules but not final modules itselves.
>
> To be clear, you only peg the CONFIG_ option if you are
> 100% certain of having a direct mapping, is that right ?
>

Yes. And there are some drivers which have duplicate values in the hash.
foo.o -> CONFIG_FOO CONFIG_FOO

Although this one has one match it's filtered as having 2 CONFIGs.

>> The result of this patchset is the following. All modules from
>> /sys expose the correct CONFIG_* symbol in the module attribute
>> kconfig_symb with some exceptions. For a total number of 58
>> modules,
>
> What kernel configuration did you use ? What architecture ?
> If you use allmodconfig, what do the numbers look like ?
>

I have a virtual machine in which I created the configuration with
make localmodconfig and installed the new kernel.
$  uname -a
Linux ubuntu 4.7.0+ #28 SMP Mon Aug 15 20:39:19 CEST 2016 x86_64
x86_64 x86_64 GNU/Linux

Should I send you the .config file ?

>> 4 of them do not:
>>
>> snd_seq_midi_event
>
> Takashi, any reason for this to not have a Kconfig symbol ?
> If we don't want one, is there perhaps any annotation we
> could use to help make a deterministic association if a
> respective kconfig symbol is needed ?
>
>> mptscsih
>
> Same thing -- for both I suppose we can infer that these
> are just helpers, and can / should be enabled if any of
> the devices are found to be present / required.
>
> I can imagine that in certain situations a helper like
> the above might really just be optional rather than required,
> but I'd hope such an optional nature should be reflected via
> Kconfig, the default setting being the recommended option to
> assume as safe then, unless input is provided to help fill
> in the gap and disable / force enable certain options.
>
>> libahci
>
> In this case libahci is spinkled over a lot of lines on the Makefile,
> its a module though, and there is no direct association of dependency
> relationship other than the Makefile association. Using a kconfig
> symbol would replace the Makefile clutter but make the assocation
> and requirement of the dependency explicit through Kconfig. Is
> that desirable or should we just make the assumption that in these
> cases we could assume one Kconfig symbol that adds a module implicitly
> is equivalent as if the module had a kconfig symbol and we had done
> "select foo" for each ? Note though that select is heavy handed [1] --
> and its over use can lead to issues. For instance its known that
> select does not check for dependencies, its a heavy handed way of
> enabling options needed by architectures. So using select here may
> not be desirable. Using "depends on" here would work but it can hide
> from the menu drivers until you have enabled its dependency.
>
> [1] https://kernelnewbies.org/KernelProjects/kconfig-sat#head-d1734174081ec7a1612fb29277bfc850d82ba31e
>
>> mptbase
>
> Same situation as  mptscsih.
>
>> After a short research:
>>
>> For mptscsih - ./drivers/message/fusion/Makefile
>> obj-$(CONFIG_FUSION_SPI)      += mptbase.o mptscsih.o mptspi.o
>> obj-$(CONFIG_FUSION_FC)               += mptbase.o mptscsih.o mptfc.o
>> obj-$(CONFIG_FUSION_SAS)      += mptbase.o mptscsih.o mptsas.o
>> As appears in the Makefile mptscsi is part of more config
>> options so it's excluded because it's not considered a module
>> by itself but, somehow gets compiled as module. Same for the
>> mptbase. I still have to understand better the Kbuild logic so
>> I cannot state that this is loose practice.
>
> Another reason for this loose use could also be that using "select"
> in Kconfig is heavy handed and can lead to recursive issues, refer
> to commit 1c199f2878f6c1b8c52125ad9805e94fe2dde472 ("kbuild: document recursive
> dependency limitation / resolution") for more information and
> a clear example of the issue. There is also a list of commits
> with possible resolutions to issues, so that can be inspected further
> to try to understand if perhaps some of the lack of kconfig symbols
> and respective deterministic mapping from module to kconfig is due
> to prior issues with select. I doubt it given that otherwise we'd
> have at least symbols for these -- but worth taking into consideration
> *if* we do then want to add symbols for them, we do not want to lead
> to recursive issues if we add these new symbols.
>
> If we add a kconfig symbol for modules lacking them, what is a
> strategical way to do so in such a way we avoid known past
> issues but also paves the way for the future ?
>
>> For libahci there are multiple CONFIG_ symbols for different
>> architectures.
>>
>> Although past discussion recommended not to use streamline_config,
>> this patchset still relies on it, for the sake of the proof of
>> concept.
>
> OK.
>
>> Although the result is close to the target it still does
>> not provide complete correctness. However it can be replaced by
>> creating another script
>
> Or C code.
>
>> which tries to determine better the
>> module <-> CONFIG associations and output them in auxiliary file
>> Module.ksymb. Maybe this way we could also determine all CONFIGs
>> for a particular driver, not only the exact one that enables it.
>>
>> The auxiliary file is necessary because the Makefile itself does
>> not have a mapping.
>
> Clarification: at least not an easy O(1) quick mapping.
>
>> The makefile includes the config file with
>> directive include which creates a series of internal variables
>> CONFIG_FOO=y/m/n. Afterwards, when recursively descending into
>> other Makefiles, lines like
>>
>> obj-$(CONFIG_FOO) = foo.o
>>
>> are resolved in place to obj-y, obj-m etc according to 'make'
>> logic and the association is lost.
>>
>> Finally, what does this patchset provide is an infrastructure
>> to dinamically peg CONFIG_* options to associate drivers using
>> the mapping from Module.ksymb file. Generation of Module.ksymb
>> can be replaced but keeping the same format permit the usage of
>> the other patches.
>
> OK this makes sense, so as it stands it seems we need a better
> way to generate the Module.ksymb and then also address those
> modules which lack a current deterministic mapping, and address
> a resolution with the community on that.
>
>> This patchset is part of a research project within
>> Google Summer of Code of porting 'make localmodconfig'
>> for backported drivers. The goal is to enable each
>> module to expose in /sys its corresponding CONFIG_* option.
>> The value of this attribute will be dynamically pegged by
>> modpost without requiring extra work from the driver developers.
>> Further, this information will be used by a hardware interogation
>> tool to extract build information about the existing devices.
>
> Good stuff!
>
>   Luis
>
>> Cristina Moraru (5):
>>   Add generation of Module.symb in streamline_config
>>   Add CONFIG symbol to module as compilation parameter
>>   Trigger Module.ksymb generation in Makefile
>>   Set KCONFIG_KSYMB as value for kconfig_ksymb module attribute
>>   Add kconf_symb as kernel module attribute
>>
>>  Makefile                             |  4 ++++
>>  include/linux/module.h               |  1 +
>>  kernel/module.c                      |  2 ++
>>  scripts/Makefile.lib                 |  9 ++++++++-
>>  scripts/kconfig/streamline_config.pl | 30 +++++++++++++++++++++++++++++-
>>  scripts/mod/modpost.c                |  7 +++++++
>>  6 files changed, 51 insertions(+), 2 deletions(-)
>>
>> --
>> 2.7.4
>>
>>