Re: [RFC PATCH 0/2] kpatch: dynamic kernel patching

[Josh Poimboeuf]

On Fri, May 02, 2014 at 10:37:53AM +0200, Jiri Kosina wrote:
> On Thu, 1 May 2014, Josh Poimboeuf wrote:
> 
> > Since Jiri posted the kGraft patches [1], I wanted to share an
> > alternative live patching solution called kpatch, which is something
> > we've been working on at Red Hat for quite a while.
> 
> Hi Josh, Seth,
> 
> thanks a lot for following up to our RFC with your submission, I am pretty 
> sure this will help to energize the discussion and will provoke ideas for 
> further improvements.

Hi Jiri!

Thanks for your comments.  I'm pretty sure we have the same basic goals
and we definitely want to find something that works for everybody.

> 
> [ ... snip ... ]
> > kpatch vs kGraft
> > ----------------
> > 
> > I think the biggest difference between kpatch and kGraft is how they
> > ensure that the patch is applied atomically and safely.
> > 
> > kpatch checks the backtraces of all tasks in stop_machine() to ensure
> > that no instances of the old function are running when the new function
> > is applied.  I think the biggest downside of this approach is that
> > stop_machine() has to idle all other CPUs during the patching process,
> > so it inserts a small amount of latency (a few ms on an idle system).
> 
> Completely agreed with your comparative analysis, thanks for a nice 
> summary.
> 
> Additional thing that I believe is important to add here: with the 
> "stop-machine / check all tasks" aproach, there might be situations where 
> you'll always fail to patch the system; if there is a long-time sleeper in 
> the patched callchain, such a single sleeper is enough to make the 
> patching of the whole system impossible.

Yeah, this is true with our current implementation.  With
stop_machine(), functions which are always on the stack of at least one
process in the system can't be patched.  In practice this means
functions like schedule(), poll(), select() and nanosleep().

Here are all the top-level sleeping functions on my system:

  $ for i in /proc/*/wchan; do cat $i; echo; done | sort |uniq |egrep -v '^0$'
  devtmpfsd
  do_sigtimedwait
  do_wait
  ep_poll
  fsnotify_mark_destroy
  futex_wait_queue_me
  hrtimer_nanosleep
  hub_thread
  inotify_read
  irq_thread
  kauditd_thread
  khugepaged
  kjournald2
  ksm_scan_thread
  kswapd
  kthreadd
  n_tty_read
  pipe_wait
  poll_schedule_timeout
  rcu_gp_kthread
  rescuer_thread
  rfcomm_run
  scsi_error_handler
  smpboot_thread_fn
  unix_stream_recvmsg
  worker_thread

In addition to these functions, their call chain path down to schedule()
also can't be patched.  But it works out to be a tiny ratio of
unpatchable-to-patchable.

I think this is a very minor limitation, but if it turned out to be a
problem, we could do something like installing an ftrace handler for the
blocking function, waking the sleeping task, and looping in the ftrace
handler until the patching process is complete.

> 
> With the lazy/gradual aproach implemented in kGraft, the whole system is 
> gradually moving towards "fully patched" state and once all the sleepers 
> blocking the process wake up, it ultimately converges to the fully patched 
> state.
> 
> Thanks,
> 
> -- 
> Jiri Kosina
> SUSE Labs
> 

-- 
Josh
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/