Announce: Linux-next (Or Andrew's dream :-))

[Stephen Rothwell]

Hi all,

Andrew was looking for someone to run a linux-next tree that just
contained the subsystem git and quilt trees for 2.6.x+1 and I (in a
moment of madness) volunteered.  So, this is to announce the creating of
such a tree (it doesn't exist yet) which will require some (hopefully)
small amount of work on the part of subsystem maintainers.

Those interested in discussion about this are encouraged to join the
linux-next@xxxxxxxxxxxxxxx mailing list.

The first things I need from the subsystem maintainers (you know who you
are) are a contact address (a list address is fine) and at least one git
branch or quilt series that contains all the things you want to see go
into 2.6.26.  I am happy for there to be multiple branches/series (in
fact there probably will need to be in some cases where there are
dependencies on others work).

The tree will be based on the current version of Linus' tree but you may
specify an earlier branch point if you need to (hopefully not - but more
likely for quilters, I guess).

I hope to recreate this tree every day automatically.  In order to do
this, any tree that has a conflict will be dropped from that days tree.
The maintainer will be notified.  I hope to provide some clue as to what
the conflict is with, but probably not initially.

I will attempt to build the tree between each merge (and a failed build
will again cause the offending tree to be dropped).  These builds will be
necessarily restricted to probably one architecture/config.  I will build
the entire tree on as many architectures/configs as seem sensible and
the results of that will be available on a web page (to be announced).

This is just a start ...

Comments?

[I suspect that Andrew's dream includes actual time to dream :-)]
-- 
Cheers,
Stephen Rothwell                    sfr@xxxxxxxxxxxxxxxx
http://www.canb.auug.org.au/~sfr/

For reference - Andrew's original musings:

I perceive the following problems with kernel development:

- The 80-odd subsystem trees are developed in isolation, leading to

  - large numbers of patch-time conflicts when I merge them all

  - large numbers of compile-time problems

  - occasional runtime problems as a result of the merging

  This is largely because the subsystem tree owners aren't being
  particularly disciplined in keeping their changes contained to their area of
  responsibility.

- It's a large amount of work to merge all the trees and it is very dull. 
  This is the main reason why -mm releases are down to one every few weeks
  rather than the old two or more per week.

- Those few -mm releases have a large number of regressions

- Developers aren't particularly responsive in handling those regression
  reports.

- Mainline kernel releases also have a large number of regressions: ones
  which are leaking through the -mm process.

- Very few kernel developers actually bother testing the -mm tree. 

  - Consequently most code hits mainline without its submitter having done
    much testing of the resulting code combination.

  - Few kernel developers are testing other kernel developers' development
    code.

- Practically the only public testing which the various trees obtain is via
  the -mm process, and that process is collapsing for the above reasons.

- People who develop against a particular tree are not spending sufficient
  time reviewing and testing changes which other developers are committing to
  other trees.

- People who raise patches are preparing them against Linus's tree.  But by
  the time we get into the late -rc timeframe, Linus's tree is very different
  from the accumulated pending trees.  So many patches fail to apply, or don't
  work correctly and the originator runtime tested them against Linus's tree,
  not against the tree into which they are to be merged.

- Developers are sneaking patches into the mainline merge window at the last
  minute, before they have had a decent period of review and testing.



So what I propose is that we start a formal unified development tree (called
the "linux-next" tree).  It will probably have the following characteristics:

- Once per day it merges all the subsystem trees: 75 git trees, 5-10 quilt
  trees.

- It will need to run compilation tests after applying each tree

  It's unreasonable to do randconfig on arch/kooky-arch/ during linux-next's
  integration, but a basic x86_64 allmodconfig would be good: something which
  is well-known by the subsystem maintainers so they can check that it will
  work.

- It will send status messages to a new mailing list and also to individual
  developers when something goes wrong.

- There are several trees which are overtly cross-subsystem and which might
  not fit into this model: mainly the driver tree, the -mm tree and perhaps
  the PCI tree.

  otoh, a lot of the -mm queue is _not_ cross-subsystem: fbdev, uml, various
  new drivers and features, etc.  So I'll need to split -mm into multiple
  queues and feed many of them into linux-next.

- Advanced topic: mechanisms will be put in place to detect a patch which
  hasn't been in linux-next for "long enough".  These patches will be flagged
  for extra consideration when offered for a mainline merge.

- Note that James is already running a cut-down version of this.  See
  http://lkml.org/lkml/2008/1/17/481



Once the linux-next tree is up and running and is reliable and useful, I would
like to see the following policies implemented around it:

- If a tree doesn't automerge it is dropped and a notification is sent

- If a tree breaks the build it is dropped and a notification is sent

- If a tree's server cannot be contacted (a fairly regular event) it is
  dropped and a notification is sent

- If a tree is causing runtime problems (ie: regressions) which impact
  testing of the whole and those problems are not getting appropriately
  resolved, the tree is dropped.  (This will require manual intervention and
  visible decision-making).

- If a tree was dropped for N% of the M days before 2.6.x is released, that
  tree is ineligible for a 2.6.x+1 merge.  This is for good software
  engineering reasons, and to provide an incentive for developers to pay more
  attention to integration and testing issues.



Other:

- The -mm tree will become a much smaller patchset, based on linux-next.

- People can set up linux-next cross-compilation farms on external servers. 
  Compilation errors can be fed back as regular bug reports.



Advantages:

- Will lead to kernel developers testing (and hopefully reviewing) each
  others code more

- Will lead to kernel developers paying more attention to integration issues

- I expect that tree-owners will be a bit more careful about what they check
  into their trees, knowing that it'll go straight into linux-next.

- Less bugs will hit Linus's tree:

  - I expect we will get a lot of public testing of the linux-next tree

  - Most kernel developers will also need to test linux-next, as that is
    the tree they're integrating into.  And kernel developers are the best
    testers, bug-reporters and bug-fixers.

- It will give me far more time to do useful things like testing, bugfixing
  and reviewing.  Particularly because I don't intend to be the person who
  operates linux-next.

- Patch submitters can now prepare and test against the linux-next tree
  rather than against the relatively out-of-date mainline tree.



Disadvantages

- More work for subsystem maintainers.

  But I don't think it's wasted time - it's largely work which would have
  needed to be done later anyway.

  If the subsystem maintainer is merging non-buggy patches and is staying
  strictly within area-of-responsibility then there should be close to zero
  extra work.

- Subsystem maintainers will need new strategies for those patches which
  touch other subsystems.

  - Ask submitters to split things up better (not a bad thing)

  - Arrange for interface changes to be implemented in a
    temporarily-back-compatible manner if poss (also not a bad thing).

  - Send some patches over to other subsystem maintainers rather than
    merging them locally (also not a bad thing).

- Subsystem maintainers will discover that getting other subsystem
  maintainers to look at and merge their patches can be a right royal pita. 
  Trust me.

  It is inefficient to have 80 subsystem maintainers re-re-re-sending
  patches at each other.

  - People can copy me on them and I can do the re-re-re-send for them,
    while including the patches in the linux-next lineup.

  - Or the subsystem maintainer can set up a standalone "patches from Tom
    to Dick" tree, and include that in the linux-next lineup (staged after
    Dick's tree) until Dick wakes up and merges it.



Open issues:

  I don't want to over-design this at this stage.  This hasn't been done
  before and I'd expect the linux-next owner would be the person who
  encounters and solves most of the problems which crop up.  But I do
  anticipate a few issues:

- Tree prioritisation.  Often, two trees will collide.  Say, git-net and
  git-nfs.  If this happens, which tree will be autodropped?

  I'd suggest that we'll just need to make decisions about which tree is
  most important, and stage that tree ahead of less important ones.

- I'd expect subsystem tree maintainers to decide to start running multiple
  trees (like Jeff's "topic branches").  This way, if there are collisions or
  other failures, then not all of their work is dropped.

- Also, a subsystem tree maintainer may have some changes which are
  considered likely to collide with other people's work.  These can be placed
  in a best-effort standalone branch which is staged late in the linux-next
  merging order and if it breaks, well, no great loss.

- A consequence of all the above is that it should be easy for subsystem
  tree maintainers to add new trees to linux-next, and for them to decide
  where in the merging order those trees are staged.

  They won't want to have to send an email to another developer for this -
  each tree-owner will need the ability to add trees with no turnaround delay.

  - This could be implemented with a world-writeable control file on
    kernel.org

  - Or a world-writeable directory on kernel.org with control files which
    start with a six-digit number

  - Or some fancy web interface or command-line tool on kernel.org

  - Or whatever.

- The need to perform compilation testing at merge-time might cause overload
  at kernel.org, so an upgraded or alternative server might be needed.

  And experience tells me that we _will_ need to perform compilation
  testing.

  I'm assuming that compilation testing will be a linear search: once per
  tree.  One could do bisection searching, but a) that gets tricky when more
  than one tree has a failure and b) it probably means that the whole tree
  needs to be re-merged once the offending tree has been weeded out.

- Ideally, all the subsystem trees are standalone and can be merged by Linus
  in any order.  There will be _some_ trees which are dependent upon other
  subsystem trees, but these are special cases and can be handled by staging
  them late in the linux-next merging order, and by not merging them in
  mainline until the prerequisites are merged.

  Obviously, the safest approach would be to merge trees into mainline in
  the same order as they are merged into linux-next, but this is probably
  impractical.

Attachment: pgp00000.pgp
Description: PGP signature