Re: linker-tables v5 testing

[Luis R. Rodriguez]

On Wed, Nov 30, 2016 at 6:51 PM, Nicholas Piggin <npiggin@xxxxxxxxx> wrote:
> On Wed, 30 Nov 2016 18:38:16 +0100
> "Luis R. Rodriguez" <mcgrof@xxxxxxxxxx> wrote:
>
>> On Wed, Nov 30, 2016 at 02:09:47PM +1100, Nicholas Piggin wrote:
>> I actually no longer have a need to have module-common.lds generated given
>> I no longer am using macro wrappers or helpers. IMHO it would be still be
>> useful to have this generated but I think we can do this separately so
>> I can drop these patches and we can address this after?
>
> Okay.

OK dropped.

>> > - Breaking up the input section descriptions like that in the linker
>> >   script is not what we want AFAIKS. It forces the linker to put them
>> >   in different locations.
>>
>> What is wrong with that ? Separating linker table and section ranges is
>
> It's not that you separate those, of course you need that. It's that
> you also separate other sections from the input section descriptions:
>
> -               *(.text.hot .text .text.fixup .text.unlikely)           \
> +               *(.text.hot .text)                                      \
> +               *(SORT(.text.rng.*))                                    \
> +               *(.text.fixup .text.unlikely)                           \
>
> [snip]

And ?

>> > Broader issues:
>> >
>> > - I still think calling these "sections" and "de facto Linux sections"
>> >   etc. is a bit confusing, especially because assembler sections are
>> >   also involved. Region, array, blob, anything. Then you get "section
>> >   ranges" and "linker tables". Fundamentally all it is is a linear memory
>> >   allocator for static data which is decentralized in the source code
>> >   (as opposed to centralized which would just be `u8 mydata[size]`).
>>
>> You are right, sorry I had not updated that part of the documentation yet.
>> I had a big nose dive into the ELF specifications after I wrote that
>> documentation and as per ELF specs all we have here are sections using
>> SHT_PROGBITS (@progbits on most architectures, on ARM this is %progbits)
>> and then we have "Special sections" [0] (one is .init for example). The
>> SHT_PROGBITS is a section type which can be used on programs to specify
>> a section is custom and its use is defined by the program itself. Now,
>> some of the "Special sections" though also have SHT_PROGBITS -- and because
>> of this it implicates programs can further customize these sections. This
>> is precisely why we can customize .init further on Linux for instance.
>>
>> I think just calling them Linux sections suffices then. Thoughts?
>
> I suppose. I don't have a much better idea so I'll end the bikeshedding
> it. Oh just one last splash of paint: the other thing is "linker tables".
> If you call the general concept a linux section, then can linker tables
> be linux section tables, or linux section arrays?

I stuck with "linker tables" as iPXE was a huge source of inspiration
and that's what they called their solution. What we have is very
different and their purpose is only one option for us now. Sure..
"linker section tables" and "linker section ranges".

>> > - It would be nice to just clearly separate the memory allocator function
>> >   from the syntatic sugar on top.
>>
>> Sorry I do not follow. What exactly do you mean by this?
>>
>> >   Actually I think if the memory allocation
>> >   and access functions are clean enough, you don't need anything more.
>>
>> Sorry I still do not follow.
>>
>> >   If we have an array of pointers and size, it's trivial C code to iterate over
>> >   it. If it needs to have a set of LINKTABLE accessors over the top of it
>> >   for this use case, then that would seem to be a failure of the underlying
>> >   API, no?
>>
>> Still did not get it.
>
> Well fundamentally the linker table is just a way to declare some type of
> array that any code can add some elements to, right?

Well in particular to a special section, and we're providing standard
easy way to do this without any hacky linker script or assembly.

> The non-C aspect of it
> is this ability of producers to be decentralized.

Not sure what you mean here, we only do a little bit of linker table
magic, tweaks and then provide helpers.

> The consumer is not really different from any other array though. They just
> want to know the address and the number of elements, and that's all. You can
> provide that with two macros and don't really need macros like for each, run
> all, etc because it's just simple C iteration over an array.

I added those at the request of hpa, and one of the reasons was that
we tend to historically use these "arrays" in special ways all over
the kernel. But these arrays are very special, they are not typical
arrays. This makes emphasis on its use. They are also needed given the
special formatting we have for start / end of these in a generic form.
This should all help consolidate this and make it easier to use in a
generic form. It should also help avoid mismatch use when the intent
was a section range and we're on a linker table, or the other way
around.

>> > - Is it really important to be able to add new allocators without modifying
>> >   a central file for the linker script? Yes it's a benefit, but is it enough
>> >   to justify the complexity?
>>
>> If by allocators you mean the ability to add new entries into sections easily
>> without having to modify the linker script -- then my answer is:
>
> Yes, but after reading a little closer it may not really be a problem I
> first thought. Thanks for providing the detailed points.

OK. So just a bit of bike shedding ?

BTW the test driver lib/test_linktables/test-linktables.c seems to
fail on ppc at run time at:

rc = trigger_config_run_named(test_dev, ".text");

I'm having issues getting a ppc / ppc64 box going to easily verify and
objdump / test this further. Anyone with a ppc / ppc64 willing to
objdump the output of this driver would be appreciated... the latest
branch is:

https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20161130-linker-tables-v5

  Luis