Re: [PATCH net v3] tcp: check skb is non-NULL in tcp_rto_delta_us()

[Neal Cardwell]

On Tue, Sep 10, 2024 at 3:08 PM Josh Hunt <johunt@xxxxxxxxxx> wrote:
>
> We have some machines running stock Ubuntu 20.04.6 which is their 5.4.0-174-generic
> kernel that are running ceph and recently hit a null ptr dereference in
> tcp_rearm_rto(). Initially hitting it from the TLP path, but then later we also
> saw it getting hit from the RACK case as well. Here are examples of the oops
> messages we saw in each of those cases:
...
> The NULL ptr deref is coming from tcp_rto_delta_us() attempting to pull an skb
> off the head of the retransmit queue and then dereferencing that skb to get the
> skb_mstamp_ns value via tcp_skb_timestamp_us(skb).
>
> The crash is the same one that was reported a # of years ago here:
> https://lore.kernel.org/netdev/86c0f836-9a7c-438b-d81a-839be45f1f58@xxxxxxxxx/T/#t
>
> and the kernel we're running has the fix which was added to resolve this issue.
>
> Unfortunately we've been unsuccessful so far in reproducing this problem in the
> lab and do not have the luxury of pushing out a new kernel to try and test if
> newer kernels resolve this issue at the moment. I realize this is a report
> against both an Ubuntu kernel and also an older 5.4 kernel. I have reported this
> issue to Ubuntu here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2077657
> however I feel like since this issue has possibly cropped up again it makes
> sense to build in some protection in this path (even on the latest kernel
> versions) since the code in question just blindly assumes there's a valid skb
> without testing if it's NULL b/f it looks at the timestamp.
>
> Given we have seen crashes in this path before and now this case it seems like
> we should protect ourselves for when packets_out accounting is incorrect.
> While we should fix that root cause we should also just make sure the skb
> is not NULL before dereferencing it. Also add a warn once here to capture
> some information if/when the problem case is hit again.
>
> Fixes: e1a10ef7fa87 ("tcp: introduce tcp_rto_delta_us() helper for xmit timer fix")
> Signed-off-by: Josh Hunt <johunt@xxxxxxxxxx>
> ---
>
> v3: Added fixes tag and more packet accounting info as requested by Neal Cardwell.
>     Also updated rto calcs to reflect original code.
> v2: Removed cover letter and added context from original cover letter to
>     commit msg.
>
>  include/net/tcp.h | 21 +++++++++++++++++++--
>  1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/include/net/tcp.h b/include/net/tcp.h
> index 2aac11e7e1cc..196c148fce8a 100644
> --- a/include/net/tcp.h
> +++ b/include/net/tcp.h
> @@ -2434,9 +2434,26 @@ static inline s64 tcp_rto_delta_us(const struct sock *sk)
>  {
>         const struct sk_buff *skb = tcp_rtx_queue_head(sk);
>         u32 rto = inet_csk(sk)->icsk_rto;
> -       u64 rto_time_stamp_us = tcp_skb_timestamp_us(skb) + jiffies_to_usecs(rto);
>
> -       return rto_time_stamp_us - tcp_sk(sk)->tcp_mstamp;
> +       if (likely(skb)) {
> +               u64 rto_time_stamp_us = tcp_skb_timestamp_us(skb) + jiffies_to_usecs(rto);
> +
> +               return rto_time_stamp_us - tcp_sk(sk)->tcp_mstamp;
> +       } else {
> +               WARN_ONCE(1,
> +                       "rtx queue emtpy: "
> +                       "out:%u sacked:%u lost:%u retrans:%u "
> +                       "tlp_high_seq:%u sk_state:%u ca_state:%u "
> +                       "advmss:%u mss_cache:%u pmtu:%u\n",
> +                       tcp_sk(sk)->packets_out, tcp_sk(sk)->sacked_out,
> +                       tcp_sk(sk)->lost_out, tcp_sk(sk)->retrans_out,
> +                       tcp_sk(sk)->tlp_high_seq, sk->sk_state,
> +                       inet_csk(sk)->icsk_ca_state,
> +                       tcp_sk(sk)->advmss, tcp_sk(sk)->mss_cache,
> +                       inet_csk(sk)->icsk_pmtu_cookie);
> +               return jiffies_to_usecs(rto);
> +       }
> +
>  }

Looks good to me. Thanks, Josh!

Acked-by: Neal Cardwell <ncardwell@xxxxxxxxxx>

neal