Taking a break - time to look back

[Thomas Gleixner]

Folks,

I'm about to vanish for a truly needed break until Jan 7th. Time to look
back to an interesting year.

Almost exactly a year ago, all hell broke loose and quite some people were
forced to cancel their Christmas and New Year vacation and instead of
spending quality time with family and friends they tried to bring the bits
and pieces for the Meltdown and Spectre mitigations into shape.

While the Meltdown part (KPTI) was in a halfways good shape - at least in
mainline - the Spectre mitigations did not make it into mainline on time
and caused havoc in distros. The broken microcode updates and other
unpleasant issues did not help the situation either. And no, the 6 days
extra if the embargo wouldn't have ended early would not have made any
difference. It's a wonder that it held up until Jan. 3rd at all.

The reasons for this disaster have been pretty much covered in various
ways, so no point to go back to that again. Though it's worth to mention
that some of the mitigations took quite some time to materialize and the
development was not at all driven by those who are responsible for the
problem in the first place. Primary examples are KPTI support for 32bit and
STIBP which took more than 9 months to get into the mainline. KPTI for
32bit was ignored completely and STIPB only got attention due to
performance regressions, though the response was causing more work than
help.

The next round of speculation-related issues including the scary L1TF
hardware bug was a way more "pleasant" experience to work on. While for
obvious reasons the mitigation development happened behind closed doors in
a smaller group of people, we were at least able to collaborate in a way
which is somehow close to what we are used to.

There were surely a few rough edges with respect to bringing in particular
developers and information flow, but both Intel and we as a community have
learned how to deal with that and improved a lot.

As a consequence, we are going to have a well documented and formalized
process for this in the foreseeable future. There are also efforts on the
way to have non-public testing infrastructure available for future events
of this kind.

No need to speculate whether this makes sense. I'm not overly optimistic
that we have seen all of that by now and my gut feeling tells me that we
are going to be haunted by that kind of issues for a very long time. For
the very unlikely case that I'm proven wrong, then I'm surely not going to
shed a tear about the time spent on writing the documentation and getting
things prepared.

At this point I want to say BIG THANKS to everybody involved for all the
great work which was done under not so enjoyable circumstances. Both the
required secrecy and the set in stone timelines are pretty different from
our normal workflow. At the same time I want to take the opportunity and
apologize for any outburst I had. I know that I went overboard occasionally
and it's nothing I'm proud of.

Looking back, I have to say that all of this certainly had consequences
outside of that restricted setting. The coordinated release dates forced
quite some people to put a break on other tasks which were piling up
nevertheless. The review backlog was from time to time tremendous and I'm
sure that we dropped stuff on the way and that we still have things to
catch up with on all ends.

Though a lot of this pressure and fallout is home-grown and could have been
avoided at least to some extent. The underlying reasons are not specific to
the mitigation development, the circumstances just emphasized them and made
them more observable for everyone - involved or not.

 1) Lack of code quality

    This is a problem which I observe increasing over many years.

    The feature driven duct tape engineering mode is progressing
    massively. Proper root cause analysis has become the exception not the
    rule.

    In our normal kernel development it's just annoying and eats up review
    capacity unnecessarily, but in the face of a timeline or real bugs it's
    worse. Aside of wasting time for review rounds, at some point other
    people have to just drop everything else and get it fixed.

    Even if some people don't want to admit it, the increasing complexity
    of the hardware technology and as a consequence the increasing
    complexity of the kernel code base makes it mandatory to put
    correctness and maintainability first and not to fall for the
    featuritis and performance chants which are driving this
    industry. We've learned painfully what that causes in the last year.

 2) Lack of review response

    Not addressing review feedback is not a new problem, but again under
    time pressure or in the face of real bugs it becomes a real pain and
    causes extra work for others and maintainers in particular.

 3) Outright refusal

    I've seen particularly in this year quite some people who responded to
    review feedback with outright and outspoken refusal. The points they
    refuse to address are not some esoteric whims of particular
    maintainers, no it's refusal to accept that there are documented
    process and patch submission rules which apply for everyone.

    Again, not a big problem if it's related to features. If it's related
    to actual bugs or the timelined mitigation development then it causes
    extra burden for others.

In other words, if we are exposed to more half-baked patches, sloppy
addressing of review feedback or in the worst case refusal to collaborate
and then on top getting complaints about maintainers and reviewers being
bottlenecks, then this will become a real problem in the not so distant
future.

Companies have to understand, that the kernel community cannot provide
all-inclusive educational programs for their engineers. It's about time,
that the companies catch the obvious wreckage before it leaves the house
and make sure that feedback is addressed properly and in all points.

I'm neither expecting perfect patches nor is there a guarantee that even
well thought out and well written code will go into the tree undisputed.
Though reviewing and discussing something which is well done is way less
time consuming and frustrating than dealing with the above.

I know that some people will come forth immediately and educate me once
more on maintainer models and the need to bring new maintainers in fast.

I'm all for more maintainers, but it's hard to find the right people.

All good maintainers - and I've brought quite a few of them into that role
myself - had proven themselves in their contributor role before taking that
up. Rest assured that I constantly look out for these people and try to get
them on board. Picking them out is based on their technical skills but even
more so on their mindset. Unfortunately quite some of them don't want to
step into that role because they are well aware of the responsibility and
the burden which comes with it. I respect that decision and I definitely
can understand it. I was more than once on the verge of throwing in the
towel during the last year.

I'm not opposed to try new things, quite the contrary. But something which
worked out for a particular subsystem cannot be applied blindly to
everything else in the hope that it works out. That needs a lot more
thought and I'm not at all buying that tooling is a crucial part of the
solution.

Last but not least, I'm not sure whether more maintainers can solve the
pain points which bugger me most. I rather think we'd need lots of
nursemaids and teachers to address that.

Sorry for the lengthy and maybe unpleasant read, but keeping the
frustration which built up over the year to myself would just cause me
gastric ulcer and a bad mood over Christmas. So I decided to vent and share
it with all of you even at the risk that I'm barking up the wrong tree.

That said, I'm going to vanish into vacation until Jan. 7th and I'm not
going to read any (LKML) mails until then. As I predict from experience
that my (filtered) inbox will be a untameable beast by then, don't expect
me to actually go through it mail by mail. If your mail will unfortunately
end up in the 'lkml/done' folder without being read, I'm sure you'll notice
and find a way to resend it.

I'm nevertheless looking positively forward to the new challenges of 2019
and I wish you all a Merry Christmas, a Happy New Year and a refreshing
break! I wish especially for those who suffered a year ago, that they can
enjoy quality time with their families and friends!

Thanks,

	Thomas