Hello,
One of my GitLab CI jobs stopped working after upgrading server 5.4.18-
100.fc30.x86_64 -> 5.5.7-100.fc30.x86_64.
(tested 5.5.8-100.fc30.x86_64 too, no change)
The server is fedora30 with XFS rootfs.
The problem reproduces always, and takes only couple minutes to run.
The CI job fails in the beginning when doing "git clean" in docker
container, and failing to rmdir some directory:
"warning: failed to remove .vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket: Directory not empty"
Quick google search finds some other people reporting similar problems
with 5.5.0:
https://gitlab.com/gitlab-org/gitlab-runner/issues/3185
Collected some data with strace, and it seems that getdents is not
returning all entries:
5.4 getdents64() returns 52+50+1+0 entries => all files in directory are deleted and rmdir() is OK
5.5 getdents64() returns 52+50+0+0 entries
=> rmdir() fails with ENOTEMPTY
Working 5.4 strace:
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 52 entries */, 2048) = 2024 <0.000020>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/cmsghdr.go") = 0 <0.000068>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/cmsghdr_bsd.go") = 0 <0.000048>
[...]
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 50 entries */, 2048) = 2048 <0.000023>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/sys_linux_386.s") = 0 <0.000062>
[...]
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 1 entries */, 2048) = 48 <0.000017>
10:00:12 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/zsys_solaris_amd64.go") = 0 <0.000039>
10:00:12 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 0 entries */, 2048) = 0 <0.000015>
10:00:12 rmdir("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket") = 0 <0.000055>
Failing 5.5 strace:
10:09:15 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 52 entries */, 2048) = 2024 <0.000031>
10:09:15 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/cmsghdr.go") = 0 <0.006174>
[...]
10:09:15 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 50 entries */, 2048) = 2048 <0.000034>
10:09:15 unlink("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket/sys_linux_386.s") = 0 <0.000054>
[...]
10:09:16 getdents64(10<
/builds/xyz/.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
/internal/socket>, /* 0 entries */, 2048) = 0 <0.000020>
10:09:16 rmdir("
.vendor/pkg/mod/golang.org/x/net@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/intern
al/socket") = -1 ENOTEMPTY (Directory not empty) <0.000029>
Any ideas what's going wrong here?
-Tommi
Attachment:
signature.asc
Description: PGP signature