Re: Linux 4.4.110

[Willy Tarreau]

On Fri, Jan 05, 2018 at 07:02:35PM +0100, Greg KH wrote:
> On Fri, Jan 05, 2018 at 04:55:07PM +0100, Willy Tarreau wrote:
> > On Fri, Jan 05, 2018 at 03:54:33PM +0100, Greg KH wrote:
> > > I'm announcing the release of the 4.4.110 kernel.
> > > 
> > > All users of the 4.4 kernel series must upgrade.
> > > 
> > > But be careful, there have been some reports of problems with this
> > > release during the -rc review cycle.  Hopefully all of those issues are
> > > now resolved.
> > > 
> > > So please test, as of right now, it should be "bug compatible" with the
> > > "enterprise" kernel releases with regards to the Meltdown bug and proper
> > > support on all virtual platforms (meaning there is still a vdso issue
> > > that might trip up some old binaries, again, please test!)
> > > 
> > > If anyone has any problems, please let me know.
> > 
> > FWIW I've just booted one of our LBs on it and am hammering it at full
> > load with pti enabled and will let it run for the week-end. It takes
> > 860k irq/s and about 1.7M syscalls/s. For now it works well (but slowly).
> > Hopefully if there are any rare race conditions left it has a chance to
> > trigger them.
> 
> Thanks for the testing, let me know if you see anything.

Definitely! For now zero error after almost one billion connections
and around 15 billion syscalls and 7B irqs.

> And "slowly", does that mean it is noticable?

It depends by whom :-)  We benchmarked this machine a while ago at 93k
connections per second on 4.9 on a single process and now I'm seeing
about 60k for a single process. I don't want to digress too much about
numbers now as the test conditions certainly differ a bit, I'll have
to rerun more detailed ones later. For 99.9% of the users it will not
be noticeable. Those having to fight DDoS will certainly notice it.
I'm pretty sure we'll run with pti=off at least at the beginning.

> I have some querys from the virtual
> networking people that are getting worried about all of this.  I told
> them to go test, but they were having a hard time finding a kernel to
> test with.  Hopefully we hear back from them now that these are out...

I've tested and found a 40% perf drop on networking under KVM between
pti=off and pti=on :-(

Fortunately in our case, people running in VMs are not those interested
in performance (that's commonly the case) but I expect it willy impact
some high-performance users who tune their VMs very precisely.

I'm currently testing a completely different approach for systems like
these running basically a single task. The idea is to limit rdtsc to
privileged processes only. I just discovered that my libc happily uses
it in the ld.so so that limits my capabilities for now :-)  But
implementing an emulator could solve this for non-privileged processes,
masking the lower bits and losing precision. It would not be a fix but
an acceptable mitigation solution for some environments where pti=off
is too expensive and where untrusted users are extremely rare (ie: just
the remote cron job check disk space and collecting network stats). I
already tested the variant of the spectre poc without rdtsc (using a
thread and a counter) and it definitely is not something reasonably
usable to steal reliable information anymore, I managed to get around
1/10 byte OK, but you never know which one.

For this reason, people considering pti=off as the only solution might
sometimes prefer this one as a small improvement (and it could also
stop other classes of future attacks, maybe something for KSPP later).

I'll continue to investigate and share my observations.

Have a nice week-end!
Willy