Re: [PATCH bpf-next] bpf: ringbuf: Support consuming BPF_MAP_TYPE_RINGBUF from prog

[Andrii Nakryiko]

On Tue, Sep 10, 2024 at 2:07 PM Daniel Xu <dxu@xxxxxxxxx> wrote:
>
> On Tue, Sep 10, 2024 at 01:41:41PM GMT, Andrii Nakryiko wrote:
> > On Tue, Sep 10, 2024 at 11:36 AM Alexei Starovoitov
> > <alexei.starovoitov@xxxxxxxxx> wrote:
> > >
> > > On Mon, Sep 9, 2024 at 5:55 PM Daniel Xu <dxu@xxxxxxxxx> wrote:
> > > >
> > > > Right now there exists prog produce / userspace consume and userspace
> > > > produce / prog consume support. But it is also useful to have prog
> > > > produce / prog consume.
> > > >
> > > > For example, we want to track the latency overhead of cpumap in
> > > > production. Since we need to store enqueue timestamps somewhere and
> > > > cpumap is MPSC, we need an MPSC data structure to shadow cpumap. BPF
> > > > ringbuf is such a data structure. Rather than reimplement (possibly
> > > > poorly) a custom ringbuffer in BPF, extend the existing interface to
> > > > allow the final quadrant of ringbuf usecases to be filled. Note we
> > > > ignore userspace to userspace use case - there is no need to involve
> > > > kernel for that.
> > > >
> > > > Signed-off-by: Daniel Xu <dxu@xxxxxxxxx>
> > > > ---
> > > >  kernel/bpf/verifier.c                         |  6 +-
> > > >  tools/testing/selftests/bpf/Makefile          |  3 +-
> > > >  .../selftests/bpf/prog_tests/ringbuf.c        | 50 +++++++++++++++
> > > >  .../bpf/progs/test_ringbuf_bpf_to_bpf.c       | 64 +++++++++++++++++++
> > > >  4 files changed, 120 insertions(+), 3 deletions(-)
> > > >  create mode 100644 tools/testing/selftests/bpf/progs/test_ringbuf_bpf_to_bpf.c
> > > >
> > > > diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> > > > index 53d0556fbbf3..56bfe559f228 100644
> > > > --- a/kernel/bpf/verifier.c
> > > > +++ b/kernel/bpf/verifier.c
> > > > @@ -9142,7 +9142,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
> > > >                     func_id != BPF_FUNC_ringbuf_query &&
> > > >                     func_id != BPF_FUNC_ringbuf_reserve_dynptr &&
> > > >                     func_id != BPF_FUNC_ringbuf_submit_dynptr &&
> > > > -                   func_id != BPF_FUNC_ringbuf_discard_dynptr)
> > > > +                   func_id != BPF_FUNC_ringbuf_discard_dynptr &&
> > > > +                   func_id != BPF_FUNC_user_ringbuf_drain)
> > > >                         goto error;
> > > >                 break;
> > > >         case BPF_MAP_TYPE_USER_RINGBUF:
> > > > @@ -9276,7 +9277,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
> > > >                         goto error;
> > > >                 break;
> > > >         case BPF_FUNC_user_ringbuf_drain:
> > > > -               if (map->map_type != BPF_MAP_TYPE_USER_RINGBUF)
> > > > +               if (map->map_type != BPF_MAP_TYPE_USER_RINGBUF &&
> > > > +                   map->map_type != BPF_MAP_TYPE_RINGBUF)
> > > >                         goto error;
> > >
> > > I think it should work.
> > >
> > > Andrii,
> > >
> > > do you see any issues with such use?
> > >
> >
> > Not from a quick glance. Both ringbufs have the same memory layout, so
> > user_ringbuf_drain() should probably work fine for normal BPF ringbuf
> > (and either way bpf_user_ringbuf_drain() has to protect from malicious
> > user space, so its code is pretty unassuming).
> >
> > We should make it very explicit, though, that the user is responsible
> > for making sure that bpf_user_ringbuf_drain() will not be called
> > simultaneously in two threads, kernel or user space.
>
> I see an atomic_try_cmpxchg() protecting the drain. So it should be
> safe, right? What are they supposed to expect?

User space can ignore rb->busy and start consuming in parallel. Ok,
given we had this atomic_try_cmpxchg() it was rather OK for user
ringbuf, but it's not so great for BPF ringbuf, way too easy to screw
up...

>
> >
> > Also, Daniel, can you please make sure that dynptr we return for each
> > sample is read-only? We shouldn't let consumer BPF program ability to
> > corrupt ringbuf record headers (accidentally or otherwise).
>
> Sure.
>
> >
> > And as a thought exercise. I wonder what would it take to have an
> > open-coded iterator returning these read-only dynptrs for each
> > consumed record? Maybe we already have all the pieces together. So
> > consider looking into that as well.
> >
> > P.S. And yeah "user_" part in helper name is kind of unfortunate given
> > it will work for both ringbufs. Can/should we add some sort of alias
> > for this helper so it can be used with both bpf_user_ringbuf_drain()
> > and bpf_ringbuf_drain() names?
>
> You mean register a new helper that shares the impl? Easy enough, but I
> thought we didn't want to add more uapi helpers.

No, not a new helper. Just an alternative name for it,
"bpf_ringbuf_drain()". Might not be worth doing, I haven't checked
what would be the best way to do this, can't tell right now.

>
> Thanks,
> Daniel