Re: due to kconfig changes kernel config file is no longer sufficient for configuring the kernel

[Masahiro Yamada]

2018-08-07 3:07 GMT+09:00 Michal SuchÃnek <msuchanek@xxxxxxx>:
> On Mon, 30 Jul 2018 17:02:42 +0900
> Masahiro Yamada <yamada.masahiro@xxxxxxxxxxxxx> wrote:
>
>> 2018-06-28 18:16 GMT+09:00 Michal SuchÃnek <msuchanek@xxxxxxx>:
>> > On Wed, 27 Jun 2018 23:07:21 +0900
>> > Masahiro Yamada <yamada.masahiro@xxxxxxxxxxxxx> wrote:
>> >
>> >> Hi.
>> >>
>> >>
>> >> 2018-06-27 21:37 GMT+09:00 Michal SuchÃnek <msuchanek@xxxxxxx>:
>> >> > Hello,
>> >> >
>> >> > in the x86 Kconfig we have this:
>> >> >
>> >> > # Select 32 or 64 bit
>> >> > config 64BIT
>> >> >         bool "64-bit kernel" if "$(ARCH)" = "x86"
>> >> >         default "$(ARCH)" != "i386"
>> >> >         ---help---
>> >> >           Say yes to build a 64-bit kernel - formerly known as
>> >> > x86_64 Say no to build a 32-bit kernel - formerly known as i386
>> >> >
>> >> > Since commit 104daea149c4 ("kconfig: reference environment
>> >> > variables directly and remove 'option env='") the value of ARCH
>> >> > is not saved in the kernel config.
>> >>
>> >> I think this commit is unrelated.  It was just a syntax change.
>> >
>> > This does not look like syntax only change to me:
>> >
>> > diff --git a/init/Kconfig b/init/Kconfig
>> > index 15aae32e0719..1217fc62ca61 100644
>> > --- a/init/Kconfig
>> > +++ b/init/Kconfig
>> > @@ -1,20 +1,12 @@
>> > -config ARCH
>> > -       string
>> > -       option env="ARCH"
>> > -
>> > -config KERNELVERSION
>> > -       string
>> > -       option env="KERNELVERSION"
>> > -
>>
>> This is just syntax change.
>>
>> 'option env=' was used to reference an environment variable.
>>
>> Now, $(ARCH), $(KERNELVERSION) are simpler forms.
>>
>>
>> >>
>> >> Unless I am missing something,
>> >> we have never saved ARCH in the .config in the past.
>> >
>> > There was a config symbol defined for it before the commit removed
>> > it.
>>
>> No.
>>
>> CONFIG symbols with'option env='
>> are not written out to the .config file.
>>
>> We have never had CONFIG_ARCH or CONFIG_KERNELVERSION.
>>
>>
>>
>>
>> >>
>> >>
>> >> > Since commit f467c5640c29 ("kconfig: only write '#
>> >> > CONFIG_FOO is not set' for visible symbols") the value of 64BIT
>> >> > is not saved if the ARCH is set i386 or x86_64 because the
>> >> > symbol is not visible.
>> >>
>> >> This is correct.
>> >>
>> >> It was discussed a few weeks ago.
>> >>
>> >> https://lkml.org/lkml/2018/6/5/847
>> >>
>> >>
>> >> > There is a number of ways to hack this particular case to work.
>> >> >
>> >> > However, there is a more general problem with this. Some config
>> >> > options may depend on the environment, may not be saved, and the
>> >> > environment is not saved either.
>> >>
>> >> Which environment variables in particular are in your mind?
>> >
>> > Any that is used in Kconfig.
>>
>> They are provided from outside of Kconfig.
>> This is the behavior we keep since a long time ago.
>>
>> ARCH is given by the environment variable or the command line.
>> KERNELVERSION is supplied by the top Makefile.
>>
>> >>
>> >> As for ARCH, you need to pass the same ARCH as you used for
>> >> building the kernel. (For native building, you do not have to pass
>> >> ARCH explicitly, though.)
>> >
>> > Except if you do pass it to make config you may need to pass it to
>> > to make later as well.
>>
>> Right.
>>
>> For exmaple 'make ARCH=arm config' will create the config suitable
>> only for ARM architecture.
>> Then, you need to do 'make ARCH=arm' to build the kernel.
>>
>> If it is tedious to give 'ARCH=arm' to every make command,
>> you can do 'export ARCH=arm' in your shell.
>>
>> Again, this is the behavior we have for a long time.
>
> No, that's not what we had. The kernel build would fail instead of
> reconfiguring the kernel for the current arch. At least it used to work
> that way at some point.

This is not true for most architectures.

It worked for i386 prior to commit f467c5640c29 ("kconfig: only write '#
CONFIG_FOO is not set' for visible symbols").


This was already discussed, and this can be solved by setting ARCH=i386.

https://lkml.org/lkml/2018/6/7/1119
https://lkml.org/lkml/2018/6/8/149


You complaint about commit 104daea149c4 ("kconfig: reference
environment variables
directly and remove 'option env='"), but it is unrelated.


>>
>>
>>
>> >>
>> >> As for CC, HOSTCC, etc.
>> >> yes, these are new 'unsaved' environments.
>> >>
>> >> CONFIG options now depend on the compiler.
>> >> This is the concept suggested by Linus Torvalds.
>> >>
>> >>
>> >> > So in the end all the infrastructure with symlinks
>> >> > from module directory pointing to the kernel source and object
>> >> > directory is useless. To interpret the config stored there you
>> >> > need the environment and that is not saved anywhere. So if you
>> >> > try to build out-of-tree module it might end up reconfiguring
>> >> > your kernel and producing useless modules.
>> >>
>> >> No. out-of-tree module building never ever re-configures the
>> >> kernel.
>> >
>> > It does implicitly because the config values depend on environment
>> > that is not saved and the values themselves are not saved either.
>> > If that happens to expose a new variable it is even explicitly
>> > reconfigured.
>>
>>
>> You should have a built kernel tree
>> before building external modules.
>>
>> The .config is already there.
>>
>> The .config works for external modules, given that
>>
>>   - ARCH is the same
>>   - the compiler is the same
>
>  - the compiler additional plugins and/or external libraries used to
>    implement advanced features are the same
>
>>
>>
>>
>> >>
>> >> out-of-tree modules are built with exactly the same configuration
>> >> as used for the kernel.
>> >
>> > It is not true. And that is the problem. You need the config file
>> > and dump of the environment passed to the make command at
>> > configuration time to get the exact same configuration. The
>> > environment is not saved anywhere, though.
>>
>>
>> Why dump of the environment?
>>
>>
>> If you are building external modules natively
>> your distribution provides /lib/modules/$(uname -r)/build,
>> which contains files enough for building external modules.
>>
>> You can pass the directory path to M=... parameter.  That's it.
>
> No, that's not it. Since passing ARCH=i386 is the de-facto standard to
> configure a 32bit kernel and the result of passing that was not saved
> you need to pass it to make as well.


If you pass ARCH= for the configuration phase,
you need to pass the same ARCH= in the build phase.




What I can suggest for you is:


$ make ARCH=i386 defconfig
$ make ARCH=i386

   OR

$ make i386_defconfig
$ make







> And you need to patch a number of
> 3rd party build scripts that build a kernel module as part of a bigger
> project.


You do not need to patch a number of scripts.

Just one liner fix.
Add 'export ARCH=i386' in the top level script.



>>
>>
>>
>> If you are cross-building external modules,
>> you also need to
>>
>>  - pass ARCH=
>>  - use the same compiler with CROSS_COMPILE=
>>
>> You should know both
>> because you have built the kernel by your self.
>
> No, I am not cross-compiling. I am building 32bit modules and because
> the kconfig system did not save the information that the kernel is
> 32bit I get 64bit modules instead.


You are semi-cross-compiling
if you build 32-bit modules on a 64-bit host machine.





>>
>> You do not need any other information, do you?
>
> I need the information what ARCH was passed to the x86 kernel at
> configuration time when building on x86. This is new.
>
> To avoid this and similar surprises in the future I suggest to flag any
> option that depends on something dynamic (compiler option, environment
> variable) so that kconfig saves it even if it would not be saved
> otherwise.

It is saved in include/config/auto.conf.cmd


> Then kconfig can do whatever seems sensible for oldconfig but
> syncconfig should not change any options based on dynamic input. It
> should verify that the option matches (ie compiler can accept -foobar
> if the option is on and defaults to cc-option -foobar) and fail if it
> does not.


Kconfig already supports this.

Try KCONFIG_NOSILENTUPDATE=1



>
>>
>> > And it went nowhere.
>> >
>> > Anyway, the observed issue with CONFIG_64BIT on x86 is the tip of a
>> > larger problem which was unnoticed for ages. The .config simply does
>> > not contain the whole kernel configuration. ie. make oldconfig (and
>> > make syncconfig) is *not* expected to just work. It used to work
>> > just by luck until f467c5640c29 ("kconfig: only write '# CONFIG_FOO
>> > is not set' for visible symbols") finally exposed the problem.
>>
>> If you want to build the kernel for an architecture
>> other than the host machine architecture, you need to pass ARCH=.
>>
>> Building the i386 kernel on a x86_64 machine, it is a _kind_ of
>> cross-compiling. So, passing ARCH=i386 is not so weird.
>>
>>
>> > So is .config supposed to contain the kernel configuration or is it
>> > just some byproduct of the kernel build which is meaningless
>> > outside of your build environment (the object tree, shell
>> > environment, etc).
>>
>> The .config is supposed to contain the kernel configuration,
>> 'ARCH' and the compiler are exceptions.
>>
>> 'ARCH' must be passed separately.
>>
>> The .config now depends on the compiler.  So, if you pass your .config
>> to somebody else, some symbols that depend on the compiler support
>> might be configured differently.
>>
>> 'make syncconfig' will notice the compiler difference,
>> and show prompts for user input as needed.
>
> It should not prompt. I want to build the kernel non-interactively. I
> do not want kernel rpm build to wait for user input or silently
> change the kernel ABI just because I installed a new gcc plugin.

'make syncconfig' shows prompt.
This is the behavior since a long time ago.
I cannot change it to meet your requirement.


If you really want to avoid prompt,
run "make olddefconfig" beforehand in your script.




> Thanks
>
> Michal
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada