netfs_read_gaps(): aliased sink folio makes the read destination unreadable, SIGBUS on cifs with signing
From: Frank Sorenson
Date: Tue Sep 08 2026 - 19:25:03 EST
Hi David,
While testing some cifs patches, I found that xfstests generic/759
fails on cifs, but only when the mount uses 'sign':
mount -t cifs //server/share /mnt -o ...,sign
fsx dies with SIGBUS in domapread(), and dmesg shows repeated:
CIFS: VFS: \\server\share SMB signature verification returned error = -13
But without 'sign', or with 'signloosely' (which sets server->ignoresignature),
the test passes.
I've tracked this down to netfs_read_gaps(), where it builds a read
destination in which several bvec entries alias one page. Data written
through that iterator can't be read back through it, but cifs does
exactly that; to verify the SMB3 response signature, it rehashes the
destination iterator after the socket copy, so the check fails, the
read is abandoned, the folio never becomes uptodate, and a mapped read
of it gives SIGBUS.
Reproduced on the latest cifs-next (7.2+); the relevant code is
unchanged since v6.8.
the construction:
fs/netfs/buffered_read.c, netfs_read_gaps():
sink = folio_alloc(GFP_KERNEL, 0);
...
if (from > 0) {
bvec_set_folio(&bvec[i++], folio, from, 0);
off = from;
}
while (off < to) {
part = min_t(size_t, to - off, PAGE_SIZE);
bvec_set_folio(&bvec[i++], sink, part, 0);
off += part;
}
if (to < flen)
bvec_set_folio(&bvec[i++], folio, flen - to, to);
iov_iter_bvec(&rreq->buffer.iter, ITER_DEST, bvec, i, rreq->len);
The sink is a single order-0 folio, and every PAGE_SIZE chunk of the
dirty region points at it at offset 0. nr_bvec = flen / PAGE_SIZE + 2
shows multiple sink entries are expected, so with a large folio and
dirty_len > PAGE_SIZE the destination iterator contains N aliases of the
same page.
For pure discard that is fine, and the file data itself is always
correct, which is why this is invisible without signing. However,
the iterator no longer returns what was previously written; each sink
chunk overwrites the previous one, so a replay of the iterator yields
the last chunk's bytes wherever the earlier chunks were written.
why it matters for cifs with signing:
__cifs_calc_signature() computes the response CMAC over rqst->rq_iov[]
followed by rqst->rq_iter, and for a read response rq_iter *is* the
destination that cifs_read_iter_from_socket() just copied into. There's
no separate source buffer to hash; the payload was copied straight
into the page cache.
cifs hashes the aliased iterator, gets bytes that were never on the
wire, and smb2_verify_signature() returns -EACCES. netfs retries it
(identically), marks it I-BAD three times, and ABNDNs it. So the
folio never gets marked uptodate, and filemap_fault() returns
VM_FAULT_SIGBUS.
I used the attached bpftrace sigcheck.bt to capture the byte stream fed
to AES-CMAC by hooking aes_cmac_update() under smb2_verify_signature(),
and compared it against the PDU as it actually arrived (from pcap), and
CMAC-verified using the smb2sigcheck.py script (also attached).
1. The wire is clean; the failing responses verify correctly against
the session signing key when the signature is recomputed from the
pcap:
mid=31388 READ pdu=8272 OK wire=... calc=... doff=80 dlen=8192
So the bytes arrived intact and cifs's own key is right.
2. The kernel hashed something else. Diffing the kernel's CMAC input
against the wire PDU for one failing read (8192-byte read into an
8192-byte folio):
1675 of 8272 bytes differ, first at offset 952 (file data +872)
region: data +872 .. +4096 (3224 bytes)
== wire data +4968 (shift +4096, one PAGE_SIZE)
Every differing run in the region is byte-identical to the pcap
contents 4096 bytes further along.
That is precisely the aliased-sink signature: dirty_offset = 872,
dirty_len = 7320, to = 8192 = flen, giving
bvec[0] = folio, 872 bytes @ 0
bvec[1] = sink, 4096 bytes @ 0
bvec[2] = sink, 3224 bytes @ 0
The second sink chunk overwrites the first, so replaying the iterator
reads bvec[1]'s slot back as the bytes that were written by bvec[2] --
i.e. wire content shifted by +4096. A Python model of that bvec list
reproduces the diff exactly, including where the coincidental matches
fall.
3. All three attempts hashed byte-identical contents, confirming this
is the buffer layout, not a concurrent modification of the folio.
to replicate:
start bpftrace:
# BPFTRACE_MAX_STRLEN=8192 BPFTRACE_PERF_RB_PAGES=1024 \
bpftrace sigcheck.bt > /tmp/sig.out
start packet capture:
# tshark -F pcap -w trace.pcap -i lo tcp port 445
mount:
# mount //server/share /mnt/test -overs=3.1.1,sign
run generic/759
(or just the fsx command which crashes:
# /root/xfstests-dev/ltp/fsx -N 10000 -o 128000 -l 500000 -h -d /mnt/test/junk
stop pcap & bpftrace
run the checker script on the bpftrace output and pcap:
# ./smb2sigcheck.py --kernel sig.out trace.pcap
I won't pretend to know how to fix it, but I'd be happy to test any fixes.
Frank
--
Frank Sorenson
sorenson@xxxxxxxxxx
Principal Software Maintenance Engineer, filesystems
Red Hat
#!/usr/bin/env bpftrace
/*
* Capture the complete byte stream fed to AES-CMAC during SMB2 response
* signature verification, so it can be diffed against the wire.
*
* cifs_sig_step() -> aes_cmac_update() is out-of-line and exported, so every
* chunk hashed by __cifs_calc_signature() shows up here:
* n=0 rq_iov[0]: 64-byte sync hdr (Signature already zeroed) +
* 16-byte smb2_read_rsp
* n=1.. rq_iter: the destination page-cache folio segments
* Concatenated, those are byte-for-byte what the wire PDU should be.
*
* Run as:
* BPFTRACE_MAX_STRLEN=8192 BPFTRACE_PERF_RB_PAGES=1024 \
* bpftrace sigcheck.bt > /var/tmp/sig.out
*
* Then:
* smb2sigcheck.py -k <key> --kernel /var/tmp/sig.out --mid <N> <pcap>
*
* sync hdr: Command @12, MessageId @24, SessionId @40, Signature @48
* read_rsp: DataLength @68
*
* If fentry on the module function fails to load, swap to:
* kprobe:smb2_verify_signature { @h[tid] = *(uint64 *)(*(uint64 *)arg0); }
* (rq_iov is the first member of struct smb_rqst, iov_base the first of kvec).
*/
BEGIN { @needkey = 1; }
fentry:cifs:smb2_verify_signature
{
@h[tid] = (uint64)args.rqst->rq_iov[0].iov_base;
@n[tid] = 0;
}
/* Gated on being inside verification so the key can be tied to a SessionId */
fentry:aes_cmac_preparekey /@needkey && @h[tid] != 0/
{
printf("SIGNKEY sid=0x%016llx key=%rh\n",
*(uint64 *)((uint8 *)@h[tid] + 40), buf(args.in_key, 16));
@needkey = 0;
}
fentry:aes_cmac_update /@h[tid] != 0/
{
$h = (uint8 *)@h[tid];
/* SMB2_READ only, and only reads small enough to dump in full */
if (*(uint16 *)($h + 12) == 8 && *(uint32 *)($h + 68) <= 8192) {
printf("C mid=%llu n=%llu len=%llu %rh\n",
*(uint64 *)($h + 24), @n[tid], (uint64)args.data_len,
buf(args.data, args.data_len));
@n[tid]++;
}
}
fexit:cifs:smb2_verify_signature
{
$h = (uint8 *)@h[tid];
if (*(uint16 *)($h + 12) == 8) {
/* Signature@48 now holds the locally computed value, not the server's */
printf("R mid=%llu sid=0x%016llx dlen=%u iter=%llu rc=%d calc=%rh\n",
*(uint64 *)($h + 24),
*(uint64 *)($h + 40),
*(uint32 *)($h + 68),
(uint64)args.rqst->rq_iter.count,
retval,
buf($h + 48, 16));
}
delete(@h[tid]);
delete(@n[tid]);
}
#!/usr/bin/python3
"""
Independently verify SMB2 signatures straight off the wire.
Reads a classic pcap (tcpdump -w), reassembles the server->client TCP streams,
walks the NBSS/SMB2 framing, and recomputes AES-128-CMAC over each PDU with the
Signature field zeroed -- exactly what smb3_calc_signature() does.
wire signature VALID + kernel rc=-13 -> the kernel hashed something other
than what arrived; the destination
page-cache folios diverged
wire signature INVALID -> the bytes were already wrong before
cifs saw them
With --kernel, the signing key and the set of MessageIds to examine are both
taken from the sigcheck.bt log (its SIGNKEY line and every rc != 0 response).
tcpdump -i lo -s0 -w /var/tmp/445.pcap 'tcp port 445'
./smb2sigcheck.py --kernel /var/tmp/sig.out /var/tmp/445.pcap
./smb2sigcheck.py -k <32-hex-digits> --cmd 8 --bad /var/tmp/445.pcap
"""
import argparse
import re
import struct
import sys
from cryptography.hazmat.primitives.ciphers import algorithms
from cryptography.hazmat.primitives.cmac import CMAC
SMB2_READ = 0x08
SMB2_FLAGS_SERVER_TO_REDIR = 0x01
SMB2_FLAGS_SIGNED = 0x08
PAGE_SIZE = 4096
CMDS = {0: "NEGOTIATE", 1: "SESSION_SETUP", 2: "LOGOFF", 3: "TREE_CONNECT",
4: "TREE_DISCONNECT", 5: "CREATE", 6: "CLOSE", 7: "FLUSH", 8: "READ",
9: "WRITE", 10: "LOCK", 11: "IOCTL", 12: "CANCEL", 13: "ECHO",
14: "QUERY_DIRECTORY", 15: "CHANGE_NOTIFY", 16: "QUERY_INFO",
17: "SET_INFO", 18: "OPLOCK_BREAK"}
def read_pcap(path):
"""Yield raw link-layer frames from a classic pcap file."""
with open(path, "rb") as f:
hdr = f.read(24)
if len(hdr) < 24:
sys.exit("%s: too short to be a pcap" % path)
magic = struct.unpack("<I", hdr[:4])[0]
if magic in (0xa1b2c3d4, 0xa1b23c4d):
end = "<"
elif magic in (0xd4c3b2a1, 0x4d3cb2a1):
end = ">"
elif hdr[:4] == b"\x0a\x0d\x0d\x0a":
sys.exit("%s is pcapng; recapture with tcpdump -w (classic pcap)" % path)
else:
sys.exit("%s: unrecognized magic 0x%08x" % (path, magic))
linktype = struct.unpack(end + "I", hdr[20:24])[0]
while True:
ph = f.read(16)
if len(ph) < 16:
return
incl = struct.unpack(end + "I", ph[8:12])[0]
data = f.read(incl)
if len(data) < incl:
return
yield linktype, data
def strip_link(linktype, frame):
"""Return (ethertype, payload) after removing the link-layer header."""
if linktype == 1: # EN10MB
if len(frame) < 14:
return None, None
et = struct.unpack("!H", frame[12:14])[0]
off = 14
while et in (0x8100, 0x88a8): # VLAN
et = struct.unpack("!H", frame[off + 2:off + 4])[0]
off += 4
return et, frame[off:]
if linktype == 0: # NULL / loopback
af = struct.unpack("=I", frame[:4])[0]
return {2: 0x0800, 10: 0x86dd, 24: 0x86dd, 28: 0x86dd, 30: 0x86dd}.get(af), frame[4:]
if linktype == 113: # LINUX_SLL
return struct.unpack("!H", frame[14:16])[0], frame[16:]
if linktype == 276: # LINUX_SLL2
return struct.unpack("!H", frame[0:2])[0], frame[20:]
return None, None
def tcp_segments(path):
"""Yield (flowkey, seq, payload) for every TCP segment carrying data."""
for linktype, frame in read_pcap(path):
et, pkt = strip_link(linktype, frame)
if et == 0x0800:
if len(pkt) < 20:
continue
ihl = (pkt[0] & 0x0f) * 4
if pkt[9] != 6:
continue
total = struct.unpack("!H", pkt[2:4])[0]
src, dst = pkt[12:16], pkt[16:20]
seg = pkt[ihl:total] if total else pkt[ihl:]
elif et == 0x86dd:
if len(pkt) < 40 or pkt[6] != 6: # no ext-header walking
continue
plen = struct.unpack("!H", pkt[4:6])[0]
src, dst = pkt[8:24], pkt[24:40]
seg = pkt[40:40 + plen]
else:
continue
if len(seg) < 20:
continue
sport, dport = struct.unpack("!HH", seg[:4])
seq = struct.unpack("!I", seg[4:8])[0]
doff = (seg[12] >> 4) * 4
payload = seg[doff:]
if payload:
yield (src, sport, dst, dport), seq, payload
def reassemble(path, port):
"""Return {flowkey: bytes} for streams originating at `port`."""
chunks = {}
base = {}
for key, seq, payload in tcp_segments(path):
if key[1] != port:
continue
if key not in base:
base[key] = seq
chunks[key] = {}
off = (seq - base[key]) & 0xffffffff
if off > 0x80000000: # pre-base retransmit
continue
prev = chunks[key].get(off)
if prev is None or len(payload) > len(prev):
chunks[key][off] = payload
streams = {}
for key, parts in chunks.items():
buf = bytearray()
for off in sorted(parts):
if off > len(buf):
break # hole: stop here
buf[off:off + len(parts[off])] = parts[off]
streams[key] = bytes(buf)
return streams
def pdus(stream):
"""Yield each SMB2 PDU's signed byte range out of an NBSS stream."""
pos = 0
n = len(stream)
while pos + 4 <= n:
if stream[pos] != 0:
return # not an NBSS session msg
rlen = int.from_bytes(stream[pos + 1:pos + 4], "big")
if pos + 4 + rlen > n:
return
rec = stream[pos + 4:pos + 4 + rlen]
pos += 4 + rlen
off = 0
while off + 64 <= len(rec):
if rec[off:off + 4] != b"\xfeSMB":
break
nxt = struct.unpack("<I", rec[off + 20:off + 24])[0]
end = off + nxt if nxt else len(rec)
if end > len(rec) or end <= off:
break
yield rec[off:end]
if not nxt:
break
off = end
def unhex(s):
"""bpftrace %rh is space-separated hex; also tolerate \\xNN and colons."""
return bytes.fromhex(re.sub(r"\\x|[:\s]", "", s))
def load_kernel(path):
"""Parse sigcheck.bt output.
Returns (signing_key, {mid: {"stream": bytes, "calc": bytes, "rc": int}}).
Either element may be None/empty if the run was cut short.
"""
key = None
per_mid = {}
with open(path) as f:
for line in f:
fields = line.split()
if not fields:
continue
if fields[0] == "SIGNKEY":
if key is None:
key = unhex(line.partition("key=")[2])
continue
if fields[0] not in ("C", "R") or len(fields) < 4:
continue
kv = dict(x.split("=", 1) for x in fields[1:] if "=" in x)
if "mid" not in kv:
continue
mid = int(kv["mid"])
ent = per_mid.setdefault(mid, {"chunks": {}, "calc": None, "rc": None})
if fields[0] == "C":
n, ln = int(kv["n"]), int(kv["len"])
blob = unhex(" ".join(fields[4:]))
if len(blob) != ln:
print("warning: mid=%d chunk %d truncated (%d of %d bytes) "
"-- raise BPFTRACE_MAX_STRLEN" % (mid, n, len(blob), ln),
file=sys.stderr)
ent["chunks"][n] = blob
else:
ent["calc"] = unhex(line.partition("calc=")[2])
ent["rc"] = int(kv["rc"])
out = {}
for mid, ent in per_mid.items():
out[mid] = {"stream": b"".join(ent["chunks"][n]
for n in sorted(ent["chunks"])),
"calc": ent["calc"], "rc": ent["rc"]}
return key, out
def diff(wire, kern, gap=64):
"""Report where the kernel's CMAC input diverged from the wire PDU."""
if len(wire) != len(kern):
print("LENGTH MISMATCH: wire=%d kernel=%d" % (len(wire), len(kern)))
n = min(len(wire), len(kern))
bad = [i for i in range(n) if wire[i] != kern[i]]
if not bad:
print("kernel CMAC input is byte-identical to the wire PDU")
return
print("%d of %d bytes differ, first at offset %d (%s)" % (
len(bad), n, bad[0],
"sync header" if bad[0] < 64 else
"read_rsp body" if bad[0] < 80 else
"file data +%d" % (bad[0] - 80)))
# Coalesce into regions, tolerating short stretches of coincidental
# agreement, then see whether each region's bytes are simply the wire's
# bytes from somewhere else -- the signature of an aliased destination.
regions = []
start = prev = bad[0]
for i in bad[1:] + [None]:
if i is not None and i - prev <= gap:
prev = i
continue
regions.append((start, prev + 1))
if i is None:
break
start = prev = i
for a, b in regions:
print(" region: data +%d .. +%d (%d bytes)" % (a - 80, b - 80, b - a))
hits = []
at = wire.find(kern[a:b])
while at != -1 and len(hits) < 8:
if at != a:
hits.append(at - a)
at = wire.find(kern[a:b], at + 1)
for s in hits:
print(" == wire data +%d (shift %+d%s)" % (
a - 80 + s, s, ", one PAGE_SIZE" if abs(s) == PAGE_SIZE else ""))
if hits == [PAGE_SIZE]:
print(" -> aliased sink: netfs_read_gaps() dirty_offset=%d "
"dirty_len=%d" % (a - 80, PAGE_SIZE + b - a))
elif not hits:
print(" (content not found elsewhere in the PDU)")
lo = max(0, (bad[0] & ~0xf) - 16)
for off in range(lo, min(n, lo + 64), 16):
w, k = wire[off:off + 16], kern[off:off + 16]
print(" %06x wire %s\n %06x kern %s%s" % (
off, w.hex(" "), off, k.hex(" "), "" if w == k else " <<<"))
def check(pdu, key):
"""Return (ok, wire_sig, calc_sig)."""
wire = pdu[48:64]
c = CMAC(algorithms.AES(key))
c.update(pdu[:48] + b"\x00" * 16 + pdu[64:])
calc = c.finalize()[:16]
return wire == calc, wire, calc
def main():
ap = argparse.ArgumentParser()
ap.add_argument("pcap")
ap.add_argument("-k", "--key",
help="32 hex digits; default: the SIGNKEY line of --kernel")
ap.add_argument("-p", "--port", type=int, default=445)
ap.add_argument("--mid", type=int, action="append",
help="only this MessageId (repeatable); default with "
"--kernel: every mid the kernel failed to verify")
ap.add_argument("--cmd", type=int, help="only this SMB2 command (8 = READ)")
ap.add_argument("--bad", action="store_true", help="only signature failures")
ap.add_argument("--dump", metavar="FILE",
help="write each selected READ payload here (.<mid> is "
"appended when more than one is selected)")
ap.add_argument("--kernel", metavar="FILE",
help="sigcheck.bt output; diff its CMAC input against the "
"wire PDU")
args = ap.parse_args()
kern_key, kern = (None, {})
if args.kernel:
kern_key, kern = load_kernel(args.kernel)
key = unhex(args.key) if args.key else kern_key
if key is None:
sys.exit("no signing key: pass -k, or --kernel with a SIGNKEY line")
if len(key) != 16:
sys.exit("signing key must be 16 bytes (32 hex digits)")
mids = set(args.mid) if args.mid else None
if mids is None and kern:
mids = {m for m, e in kern.items() if e["rc"]}
if not mids:
sys.exit("%s logged no verification failures; pass --mid to pick "
"specific MessageIds" % args.kernel)
print("kernel failed to verify %d response(s): %s\n" % (
len(mids), " ".join(str(m) for m in sorted(mids))), file=sys.stderr)
total = bad = 0
for stream in reassemble(args.pcap, args.port).values():
for pdu in pdus(stream):
cmd = struct.unpack("<H", pdu[12:14])[0]
flags = struct.unpack("<I", pdu[16:20])[0]
mid = struct.unpack("<Q", pdu[24:32])[0]
if not flags & SMB2_FLAGS_SIGNED:
continue
if mids is not None and mid not in mids:
continue
if args.cmd is not None and cmd != args.cmd:
continue
ok, wire, calc = check(pdu, key)
total += 1
if not ok:
bad += 1
if args.bad and ok:
continue
extra = ""
if cmd == SMB2_READ and flags & SMB2_FLAGS_SERVER_TO_REDIR and len(pdu) >= 80:
doff = pdu[66]
dlen = struct.unpack("<I", pdu[68:72])[0]
data = pdu[doff:doff + dlen]
extra = " doff=%d dlen=%d got=%d head=%s" % (
doff, dlen, len(data), data[:16].hex())
if args.dump:
path = args.dump
if mids is None or len(mids) > 1:
path += ".%d" % mid
with open(path, "wb") as f:
f.write(data)
extra += " -> %s" % path
print("mid=%-8d %-16s pdu=%-7d %s wire=%s calc=%s%s" % (
mid, CMDS.get(cmd, "0x%x" % cmd), len(pdu),
"OK " if ok else "BAD ", wire.hex(), calc.hex(), extra))
ent = kern.get(mid)
if ent and ent["stream"]:
c = CMAC(algorithms.AES(key))
c.update(ent["stream"])
ksig = c.finalize()[:16]
print("\nkernel: rc=%s hashed=%d bytes, cmac=%s" % (
ent["rc"], len(ent["stream"]), ksig.hex()))
if ent["calc"] and ksig != ent["calc"]:
print(" (does not match the kernel's own calc=%s -- the "
"chunk log is incomplete)" % ent["calc"].hex())
diff(pdu[:48] + b"\x00" * 16 + pdu[64:], ent["stream"])
print()
elif ent:
print("\nkernel: rc=%s but no chunks logged (dlen above the "
"sigcheck.bt dump gate?)\n" % ent["rc"])
print("\n%d signed PDUs checked, %d bad" % (total, bad), file=sys.stderr)
if __name__ == "__main__":
main()