Re: 5.5 XFS getdents regression?

From: Bhaskar Chowdhury
Date: Tue Mar 10 2020 - 07:12:26 EST


On 08:45 Tue 10 Mar 2020, Rantala, Tommi T. (Nokia - FI/Espoo) wrote:

Okay, hang on! don't you think you should query at fedora mailing list
instead here??

Because you are running fedora kernel and I believe it is patched by
their team. So, they might have much more concrete answer than to ask
the file system developer here for the outcome.

Kindly, provide the bug report to them fix your owes.

~Bhaskar




Hello,

One of my GitLab CI jobs stopped working after upgrading server 5.4.18-
100.fc30.x86_64 -> 5.5.7-100.fc30.x86_64.
(tested 5.5.8-100.fc30.x86_64 too, no change)
The server is fedora30 with XFS rootfs.
The problem reproduces always, and takes only couple minutes to run.

The CI job fails in the beginning when doing "git clean" in docker
container, and failing to rmdir some directory:
"warning: failed to remove .vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket: Directory not empty"

Quick google search finds some other people reporting similar problems
with 5.5.0:
https://gitlab.com/gitlab-org/gitlab-runner/issues/3185


Collected some data with strace, and it seems that getdents is not
returning all entries:

5.4 getdents64() returns 52+50+1+0 entries => all files in directory are deleted and rmdir() is OK

5.5 getdents64() returns 52+50+0+0 entries
=> rmdir() fails with ENOTEMPTY


Working 5.4 strace:
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 52 entries */, 2048) = 2024 <0.000020>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/cmsghdr.go") = 0 <0.000068>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/cmsghdr_bsd.go") = 0 <0.000048>
[...]
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 50 entries */, 2048) = 2048 <0.000023>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/sys_linux_386.s") = 0 <0.000062>
[...]
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 1 entries */, 2048) = 48 <0.000017>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/zsys_solaris_amd64.go") = 0 <0.000039>
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 0 entries */, 2048) = 0 <0.000015>
10:00:12 rmdir("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket") = 0 <0.000055>


Failing 5.5 strace:
10:09:15 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 52 entries */, 2048) = 2024 <0.000031>
10:09:15 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/cmsghdr.go") = 0 <0.006174>
[...]
10:09:15 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 50 entries */, 2048) = 2048 <0.000034>
10:09:15 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/sys_linux_386.s") = 0 <0.000054>
[...]
10:09:16 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 0 entries */, 2048) = 0 <0.000020>
10:09:16 rmdir("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket") = -1 ENOTEMPTY (Directory not empty) <0.000029>


Any ideas what's going wrong here?

-Tommi

Attachment: signature.asc
Description: PGP signature