Redesign sidecar using byte-exact guest filenames without a sidecar index - #256
Merged
Merged
Conversation
![cubic-dev-ai[bot]](https://avatars.mcszy.xyz/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
henrybear327
force-pushed
the
sysroot/casefold-redesign
branch
5 times, most recently
from
August 2, 2026 20:18
6fc98a0 to
479545c
Compare
jserv
reviewed
Aug 3, 2026
jserv
left a comment
jserv
left a comment
Contributor
There was a problem hiding this comment.
Correctness pass over the casefold redesign. Seven findings inline; each is on a line this PR introduces or newly exercises. No blockers to the direction, which is a clear win over the sidecar.
| */ | ||
| if (is_link && (attr_buf.returned.commonattr & ATTR_CMN_OBJTYPE)) | ||
| *is_link = attr_buf.obj_type == VLNK; | ||
| if (!strcmp(stored, leaf)) |
Contributor
There was a problem hiding this comment.
stored points into the getattrlistat reply via attr_dataoffset and goes straight into strcmp with no bound on attr_length and no guaranteed NUL. APFS behaves, but --sysroot can point at an SMB or NFS mount that need not. A name that fills the buffer with no terminator over-reads adjacent stack. Validate name_ref offset+length against the returned buffer and require a NUL before comparing.
|
|
||
| if (!casefold_active() || !proc_sysroot_snapshot(sr, sizeof(sr))) | ||
| return false; | ||
| if (fcntl(host_dirfd, F_GETPATH, dirpath) < 0) |
Contributor
There was a problem hiding this comment.
Returning true when F_GETPATH fails means an fd whose sysroot membership cannot be established has its directory entries decoded as guest names. A host-backed directory that cannot be named back but happens to hold a foreign file literally named .ef=61 then shows the guest a phantom 'a' that no open can resolve. This decode question should fail closed the other way: return false unless the directory is proven inside the sysroot.
sys_path_has_symlink() charged a MAXSYMLINKS budget per path component, but the walk never follows a link: a symlink component is ELOOP on the spot, so nothing accumulates against the budget. All the counter could reject was a link-free path over 40 components deep, which Linux resolves, since it caps links followed, not components walked (path_resolution(7)). The absolute arm was worse: it scans the host path, charging the sysroot's own depth to the guest's budget, so a 38-component guest path failed behind a sysroot six directories down. Drop the counter; the loop is bounded by the path string. The counter that does bound link traversal, in path_openat2_crosses_mount(), stays. Add a Path Resolution section to docs/internals.md, which had nothing on how a guest path becomes a host path, leaving the two budgets looking interchangeable. The openat2 ABI shim and the link-budget constant the tests need live once in tests/linux-openat2.h, shared by the fidelity test and the new walk lane rather than hand-copied into each. Both new test lanes were observed red first, all reporting ELOOP: a 44-component link-free openat2 case, and two host-staged sysroot chains at 44 and 38 components. The sysroot lane keeps a symlink component, ELOOP throughout, so the fix cannot be read as disabling the check.
Linux resolves '..' against the process root and clamps it there: "/.."
names "/" (path_resolution(7)). The sysroot is just a host directory,
so prefixing it onto a guest path let the host kernel climb out of it:
"/.." reached the sysroot's parent, where the containment guard
reported ELOOP for a path Linux resolves, or resolution fell through to
the host's own root.
clamp_dotdot_at_guest_root() rewrites only the '..' that would climb
above the guest root, copying everything else (interior '..' included)
byte for byte. The host must still walk the interior ones: their
existence and type are the answer Linux owes ("/absent/../b" is ENOENT,
"/file/../b" is ENOTDIR), and a '..' past a symlink must keep the link
visible to the no-symlinks precheck. Classification still reads the
fully collapsed spelling, so "/usr/../home/x" is judged as "/home/x".
Both resolvers seed the host path through one helper.
A path reconstructed from a dirfd can climb out too. The relative
recheck reports when it clamped and returns the absolute host spelling
it resolved; the caller opens that rather than walking the guest's
"../x" from the descriptor, which POSIX has the kernel ignore for an
absolute path. A climbed resolution is the path that actually gets
opened, so it also honors the caller's create-parents flag, where the
probe-only case must not mkdir anything. Sidecar lookup is skipped for
climbed paths, and sidecar creates decline them for the same reason:
the sidecar's own walk from the descriptor has no clamp.
The no-symlinks precheck walks a relative path over the descriptor's
real host fd, so the clamp has to live in that walk: the descriptor's
lexical guest depth seeds a counter and a '..' at depth zero stays
put. Unclamped, the walk stepped onto the sysroot's host parent and
answered with the host's entries, macOS's own /tmp symlink among them.
Observed red first under --sysroot: "/.." and lstat("/..") reporting
ELOOP, the probe for a host sibling beside the sysroot reporting ELOOP
where Linux owes ENOENT, the three descriptor-relative openat cases, a
create through "../" into an unmaterialized /var/tmp failing ENOENT,
and four descriptor-relative openat2 RESOLVE_NO_SYMLINKS cases
reporting ELOOP or ENOENT. The interior-symlink openat2 lane is a
regression guard, not an observed fix: it pins the byte-for-byte copy
so a later collapse of interior '..' cannot hide a link from the walk.
A Linux guest names files by exact bytes; the default APFS volume matches names case- and normalization-blind, so "Foo" and "foo" cannot coexist in one directory. Introduce the representation that lets them: a name the volume cannot store as itself is stored escaped, and the escape is a pure function of the name, so nothing has to be persisted in order to reverse it. Escaping keys on the name alone: any uppercase ASCII letter, any byte above 0x7F, or a name already shaped like an escape. That is deliberately conservative, because the volume's matching is far more aggressive than case plus NFD (sharp s folds onto "ss", final sigma folds by position, compatibility mappings apply) and nothing short of full Unicode tables predicts it. What the rule leaves literal is lowercase ASCII, a fixed point of every transformation the volume applies, so two literal names cannot collide however the folding table changes. Keying on the name rather than on directory contents also means colliding creates never contend. The payload has two tiers because the per-name limit is 255 UTF-16 code units, not bytes: hex up to a 125-byte guest name (readable back with xxd -r -p), and above that twelve bits per symbol from 4096 CJK Unified Ideographs at U+4E00, a block that neither normalizes nor folds. The symbol width is a single constant from which the alphabet size and the per-name symbol count both derive, so a packing change moves every consumer together, and a _Static_assert holds both tiers to the unit limit. The codec is a leaf translation unit so the unit test links exactly the code under test; probe-volume-naming regenerates the measurements docs/filenames.md tabulates. tests/casefold-vectors.h freezes the on-disk spellings byte-for-byte in both directions, making a format change a deliberate migration rather than a silent test update.
Resolve a guest path to its host spelling one component at a time, applying the escape where a name cannot be stored as itself. This is the half of the model that has to touch the filesystem, kept apart from the codec so the codec stays a leaf translation unit. Each component is decided by asking the volume for the name as stored and comparing bytes, the only way to tell "exists as spelled" from "exists under a spelling that folded onto it": a plain stat reports success for both, while Linux resolution is byte-exact and owes ENOENT for the second. The two answers stay distinct all the way out: folded is not absent, because a caller deciding whether the sysroot has a claim on the path needs the difference. The whole path is probed prefix by prefix, since the volume validates only the last component and a wrong-case parent folds away silently. The walk is seeded from a descriptor and a prefix, so one implementation serves an absolute path measured from the sysroot and a relative one measured from a dirfd. Along the main path nothing is opened (each probe is a getattrlistat), so the walk holds no descriptors and has no cleanup path. A volume that cannot report a stored spelling falls back to reading the directory, finished with an fstatat because a name missing from the listing may still be present under a spelling that folded onto it. The probe takes the volume's reply on its own terms rather than the kernel's word: getattrlist(2) documents that a truncated reply can reference data beyond the buffer, and --sysroot accepts network mounts that need not stay inside even that contract. The name payload is located through a validator that bounds the reference and its length against min(claimed length, buffer size), rejects the signed offset pointing backwards, and requires a terminator inside the referenced bytes; a malformed reply takes the same fallback as a withheld name. The unit test feeds the validator forged replies for each malformed shape. The unit test drives staged literal names, staged escapes, symlinks, and a dangling link; its length arm grows a path a component at a time and requires a clean ENAMETOOLONG boundary, because a host path is roughly twice its guest path while macOS allows a quarter the length Linux does, and a truncated path names a different file.
Guest paths were resolved twice: proc_resolve_sysroot_path_flags concatenated the sysroot prefix and probed existence, then path_translate_at ran a second, independent walk that read the sidecar index and could override the first answer. The two disagreed about when a path falls through to the host. Fold them into one: the case-exact walk decides each component's stored spelling and reports whether the path resolved, so its verdict is the existence answer and its output is the host path. A relative name resolves through the same walk seeded from the descriptor it is measured from, and the openat2 RESOLVE_NO_SYMLINKS and RESOLVE_NO_XDEV walkers, which spelled components the guest's way and missed every escape, resolve the same way. Absence is not one answer but three, and only the first may look at the host. A path nothing claims falls through. A path whose slot is held by a differently-spelled sibling is absent to a byte-exact reader, but the sysroot holds something there, so the caller's own syscall reports ENOENT; treating it as unclaimed would send a wrong-case lookup out to an unrelated host file. A path stopped at a component that is not a directory is the same: resolution fails there (path_resolution(7)), so the walk records it and the sysroot spelling is returned, or "file/tail" would be answered by whatever host path shares those bytes, ENOENT where Linux owes ENOTDIR. Creates ask where an absent leaf would go, so the containment flag ladder tests create before nofollow (a renameat destination carries both), and the walk reports the parent separately from the leaf, replacing an access(2) probe that a folding volume answered wrongly in both directions. Directory entries and the cwd decode on the way back out, so getcwd never leaks an on-disk spelling. The dirent decode is scoped the way the resolvers are: an entry decodes only when the directory it was read from lies under the sysroot, answered once per read from the descriptor's own host path. The process-wide fold switch alone cannot make that call: a directory reached through the host fallback lists names elfuse never wrote, and decoding an escape-shaped literal there reports a name the directory does not contain and that no open resolves, while hiding the entry's real name behind it. A directory whose host path cannot be read at all fails closed for the same reason: decoding is a claim elfuse wrote the name, and it needs the directory proven inside the sysroot. With the mapping derived from the name, the sidecar has nothing left to do. Its five private syscall handlers return to the ordinary translation, and stop bypassing the /proc, FUSE, and /dev/shm handling; the index, its parsers, the process mutex, the fcntl lock, the rollback snapshots, and both caches are removed. Two elfuse processes sharing a sysroot need no coordination, because there is no shared mutable state left. The internals module table and the usage notes described that machinery, so both are rewritten rather than left naming a deleted file; the notes also record that a sysroot written by the old encoding has to be recreated, the one guest-visible consequence of this commit. Smaller corrections ride on the single walk: a trailing separator keeps Linux's ENOTDIR by asserting directory-ness on the host path; a sysroot mounted at "/" no longer turns a leaf's parent into an empty string; and a path unlinked by a peer between probe and realpath reports ENOENT rather than a containment veto's ELOOP, since neither the path nor a vanished sysroot leaves anything reachable once it stops resolving. The contract lands in two lanes, name-unique and name-relative, which also run against the qemu reference kernel, where a real Linux kernel measures what they assert. Neither can run in the elfuse lane, which has no sysroot, so the matrix gains ELFUSE_SKIP: the inverse of QEMU_SKIP, with one membership helper and a check-format gate rejecting a label that names no registered test or sits in both lists.
The naming design rests on claims a green build does not exercise; give each its own lane. Edge shapes: test-sysroot-root drives the degenerate one-character prefix over the read-only macOS root; test-nosysroot-literal-names pins that without a sysroot nothing decodes and an escape-shaped host file means itself; test-sysroot-outside-names pins the other half of that contract, that with a sysroot configured a host directory reached through the fallback still lists, opens, and creates by its own bytes, with an in-sysroot control name proving the volume folded during the run; test-sysroot-chdir requires getcwd and /proc/self/cwd to report spellings the guest can hand back to chdir. Names: test-sysroot-name-i18n pins the pairs the volume considers equal that the guest must see as two files (sharp s, positional sigma, ligatures, normalization families, no-case scripts, invalid UTF-8), and in csapfs mode pins the two documented divergences of case-sensitive APFS instead of skipping. test-sysroot-name-length pins that a guest keeps all 255 bytes Linux allows across both tier boundaries of the escape, and that 256 is refused. Lengths are two separate budgets, and the second one is the host's: test-sysroot-pathmax pins ENAMETOOLONG where macOS's 1024-byte PATH_MAX undercuts the guest's 4096, since a host path runs to roughly twice its guest path and a truncated one names a different file. It stays out of the qemu matrix because a real Linux kernel has no such ceiling and correctly builds every path the lane expects refused. Reading back what an older build wrote: test-sysroot-corpus stages a tree of on-disk spellings byte-for-byte from tests/casefold-vectors.h and opens each strictly by guest name, which is the direction an existing sysroot exercises after an upgrade. Staging happens on the host, so nothing it asserts derives from the codec under test. Host-staged names: a sysroot is an ordinary directory anything may write into, so test-sysroot-name-staged fixes the meaning of names elfuse never produces: a well-formed escape decodes no matter who wrote it, anything merely resembling one means itself, and when both spellings of one name are staged the literal wins the lookup while the listing reports both. Concurrency: nothing serializes name creation, because the on-disk spelling is a function of the name alone, the load-bearing claim. test-sysroot-name-race forks children that create colliding members and race O_EXCL for a single kernel-picked winner, and runs in check ten times over, because one round is cheap and a single round can miss the window. test-sysroot-name-soak churns creates, renames, unlinks, and listing scans from threads and forked children against a deadline; it is registered as a manual target and kept out of check for its runtime. Neither proves the absence of a race, and both headers say so. The name lanes also run against the qemu reference kernel, where a real kernel confirms Linux keeps each pair apart, and are skipped in the elfuse lane for want of a sysroot.
The representation half of docs/filenames.md landed with the codec. Fill in the resolution half: how a path is resolved a component at a time, what each answer from the volume means, and the two properties that fall out of deciding a name's spelling from the name alone: that a guest name is reachable through exactly one on-disk entry, and that nothing in the subsystem takes a lock, since each operation is one atomic syscall and no second object records what a name means. A section on symlink targets records the one thing the model cannot represent: a link stores the bytes the guest gave it, because readlink has to return them, so a target naming something stored escaped cannot be followed by the host. The pure-function claim is narrowed to what is true: two rows of the resolution table do consult the directory, both only for a fold-stable name whose escape an outsider staged. The dynamic-linker walkthrough still described a sysroot-first openat rather than the one forward resolver every path-taking handler now uses, so it is rewritten to point at path_translate_at, and its limitations section now names this document instead of claiming nothing is tracked: what the sysroot volume cannot represent is a real limitation of that path, recorded here.
A relative symlink target records the bytes the guest wrote: readlink(2) reports the stored target, so translating one on the way in would surface in every readback. But those bytes name a guest path while the disk holds host spellings, so handing them to the host kernel looks somewhere else: a guest could not follow its own symlink when a target component was stored escaped, and an absolute target resolved from the host root instead of the sysroot. Follow targets in the guest namespace instead. The walk stops at a link it must pass through and reports where; the resolver reads the target, joins a relative one to the link's directory or lets an absolute one replace the path, appends the remainder, and resolves that as an ordinary guest path. The loop is iterative, since a chain may run to MAXSYMLINKS and each step needs a whole path buffer. Detecting the link is free: ATTR_CMN_OBJTYPE rides along on the probe the walk already makes. A target may carry '..', which the walk spells through on the precondition that absolute paths arrive collapsed, so the resolver collapses the spliced path after every splice, clamping at the guest root (path_resolution(7)). The pop is exact rather than merely lexical here: every prefix component it removes was proven a non-link by the walk that just ran. The link budget counts links actually crossed, the accounting path_openat2_crosses_mount already uses: a 40-link chain resolves and the 41st link reports ELOOP. The 40- and 41-link lane cases are regression guards, recorded green at introduction, pinning that boundary against off-by-one drift. POSIX fixes which components are followed: every intermediate one, and the last only without nofollow. All four resolvers learn the rule: lookup, create, the descriptor-relative translation, and the RESOLVE_NO_SYMLINKS precheck, for which a stopped-at-a-link verdict is the ELOOP it exists to report. The precheck asks the walk directly, because splicing a link into its target hides it from any later scan; the descriptor-relative translation reuses the host path its containment check already resolved, keeping one flag mapping rather than a second copy that can drift. An absolute target resolves against the sysroot but does not inherit the host fallback a typed path gets: anything able to write a symlink into the tree could otherwise hand the guest a file from outside it. On disk an absolute target is stored relative to the link's own directory, so a native follow stays inside the sysroot and the tree survives being moved; readlink consequently reports the rewritten spelling, the one visible divergence, since nothing on disk tells a rewritten target from a relative one the guest wrote. The splice shares the existing truncation-checked concatenation, and MAXSYMLINKS moves to path.h so the path layer and the resolvers cannot disagree about when a chain has run too long.
A watch is backed by kqueue on a host fd, but the path it is added under is a guest path: opened raw, an absolute watch lands in the host's namespace and reports ENOENT for a directory the guest can chdir into. Route it through path_translate_at like every other path-taking syscall, refusing FUSE and synthetic /proc objects with ENOSYS since kqueue cannot observe them. Those two are the whole refusal. The open path's path_might_use_open_intercept is not reused for it: that predicate is a prefilter, true for every name beginning "/dev" (four bytes, so "/development" too), for the sysfs CPU tree, and for /etc/passwd whenever the sysroot carries no copy. Each of those has a real host vnode behind it (a /dev/shm leaf is redirected to one by the translation itself), and Linux grants a watch on all of them, so gating on it would answer ENOSYS where a watch is owed. A lane pins the /dev/shm leaf against that. That redirect makes the leaf a real host file, so the watch is opened nofollow. A guest may write a symlink into the /dev/shm backing directory, and following one would report the existence of, and every change to, whatever it names, including a host path is_guest_system_path() exists to keep the guest from addressing at all. Linux follows here, but the redirect is elfuse's own and every other consumer of a shm leaf already departs the same way. A second lane pins that beside the one granting the watch. The snapshots feeding named IN_CREATE/IN_DELETE events read raw directory entries, so on a folding sysroot an event named the stored .ef= spelling, bytes the guest never wrote and cannot stat. Decode each entry through path_translate_dirent_name, the choke point getdents64 uses; an over-long name is skipped exactly as getdents64 skips it, and any other failure keeps the previous baseline rather than diffing every decoded child as deleted. The decode is scoped per directory, and the snapshot answers the same question getdents64 asks, whether the watched directory is one the sysroot owns, from the host descriptor it already holds, so a watch on a host directory outside the sysroot carries entry names as they are stored; the lane pins one such watch beside the decoding ones. The lane's recipe asserts host-side that the fixture survives only under its escape, so a passing guest lane proves the events were decoded. Events are collected by reading a nonblocking fd: the emulation pumps its queue on read, and poll-only readiness is a pre-existing gap this change does not touch.
execveat with a dirfd-relative name opened the raw guest bytes: under a casefold sysroot a guest-created binary exists on disk only under its escaped spelling, so the open reported ENOENT for a file execve runs fine, and could have resolved a case-colliding host-literal file instead. Translate the name like every other *at handler, refusing FUSE and synthetic /proc objects, and open the translated spelling with O_CLOEXEC so a concurrent execve on another vCPU thread cannot inherit the descriptor. The identity that resolution publishes needs the reverse map: left as the resolved host path, /proc/self/exe reported the sysroot prefix and .ef= spellings, bytes the guest never wrote and cannot exec, which is how a self-re-exec stops working. sys_execve derives the guest-visible identity with path_host_to_guest while keeping the host path for the actual open; proc_readlink_self_exe replaces its bare prefix strip with the same reverse map; and /proc/self/fd/N runs its F_GETPATH result through it too. The lane's recipe asserts host-side that the staged binary sits on disk only under its escape, so the passing exec lanes prove resolution actually crossed the boundary.
The interpreter resolver kept its own three-strategy probe from the core loader: a literal sysroot concatenation checked with access(2), then /lib/<basename>, then the guest bytes. The concat probe answers with the volume's folding lookup, so a wrong-case interpreter spelling was accepted where Linux resolution is byte-exact, and a hit skipped the containment, /dev/shm, and FUSE handling every other exec path gets. Route the decision through the ordinary translation: a usable translated path wins, and the /lib/<basename> fallback still catches store-style interpreter paths. elf_resolve_interp stays with the core bootstrap, which probes its literal spellings first and already falls through to the same translation when both probes miss. Usability is judged under the rule the open then uses. A shm redirect is opened O_NOFOLLOW, so its leaf is probed without following: access(2) follows, and a symlink in the shm backing directory would answer usable for a path the open refuses with ELOOP, skipping the fallback that would have found the loader. Everywhere else the probe still follows, since an interpreter is routinely a symlink. On this tree the escaped-spelling case already reached the translator through the old code's literal fallthrough, so the two lanes are regression guards rather than observed-red fixes: one pins the /lib fallback for an absent store path, the other an interpreter staged under an escaped directory and exec'd from inside the guest. Both skip when the musl fixtures are absent.
proc_synth_ino carried a private copy of the hash; the pathname AF_UNIX socket translation is about to need the same digest for derived link names. One inline helper keeps the constants in one place, so the two users cannot drift.
A pathname socket's address is a filesystem path, but every sockaddr crossed the boundary through a raw byte copy: bind created the socket file at the host-literal path (outside the sysroot for any guest path not mirrored there), and connect, sendto, and sendmsg aimed at the same wrong namespace, so two guest processes could not rendezvous through a socket in a guest-created directory. Route pathname addresses through path_translate_at: bind uses create semantics, so colliding socket names coexist and bind, stat, connect, and unlink agree on which file a name means, with EADDRINUSE for an occupied name. getsockname, getpeername, accept, recvfrom, and recvmsg decode returned addresses through path_host_to_guest, so the guest reads back the spelling it bound. A translated host path routinely overflows the 104-byte macOS sun_path (the sysroot prefix plus an escape more than doubling a component), so over-long paths divert through a short symlink in the private absock namespace dir; bind through a dangling symlink creates the socket at the target and connect follows it (probed on macOS 15). Two details of that readback come with it. recvmsg reported the macOS address length beside the translated bytes it wrote; the two were interchangeable until translation made them differ, and a guest sizing sun_path from msg_namelen then read past the address into its own buffer. And the namespace dir is recognized from the namespace id rather than from having created it, so a forked child (a fresh process that inherits the id) undoes the shortening symlink too, matched on a whole path component so one namespace cannot claim another's by prefix. A /dev/shm leaf carries the never-follow rule that redirect exists for, and bind and connect take a sockaddr rather than a dirfd and at_flags, so the rule cannot ride on an open flag here and is checked outright. Following a guest-planted link there would bind the socket at the link's target, outside the tree, and would answer connect with ENOTSOCK for a host file that exists against ENOENT for one that does not, telling the guest whether any path exists. Both reach what is_guest_system_path() denies the guest by name. The exit paths for the shared namespace dir arm in absock_ensure_dir_locked, the one point where on-disk state first appears, rather than in sys_bind's abstract-socket branch alone. Cleanup is strictly per-process property: each process unlinks its own table-tracked sockets and the shortening links it recorded at mint time, and any participant may rmdir the directory, last one out winning. Link names are pid-scoped, so no two live processes ever share one; no sweep over the shared directory is needed, a child that mints links after the namespace's creator exited leaves nothing behind, and the ensure path re-creates a directory that a last-one-out rmdir removed while this process still held state pointing into it. Derived names carry a 64-bit FNV-1a tail: a 32-bit digest collides at the ~2^16 birthday bound, and the mismatch branch would then repoint a live link at another socket's target. The names lane asserts host-side that a socket survives only under its escape, the naming unit lane proves a 32-bit-colliding pair still derives distinct names, and the lifecycle lane covers both fork exit orders plus the namespace-outliving-its-creator order. With socket addresses decoded, every surface that hands names back to the guest now participates; docs/filenames.md names the full decode boundary in one place.
Two lanes join the suite, both of them oracles rather than expectations. check-name-caseexact re-runs the name suite on a case-sensitive APFS sparsebundle, where the volume itself enforces the byte-exact matching the tests assert, so an expectation that fails there disagrees with a real Linux filesystem whatever the folding lane says of it; this lane is what surfaced the ENOTDIR and vanished-path fixes in the resolver. test-sysroot-path-matrix closes the class the hand-written tests sampled: every late bug on this branch sat in one cell of a cross product (addressing mode x operation x path shape x name class) that no one had thought to visit. The matrix enumerates the product and holds each cell to path_resolution(7)'s oracle: an absolute, a cwd-relative, and a dirfd-relative spelling of one file must agree on result, errno, and object, with creates verified through the canonical absolute spelling. On first run it caught an absolute openat2(RESOLVE_NO_SYMLINKS) missing an in-sysroot intermediate link and a relative rename below an escaped directory failing where the absolute spelling succeeds. It runs on the folding tmpdir, again on the case-sensitive volume, and against the qemu reference kernel, where agreement is a measurement rather than an assertion. Putting the matrix on the byte-exact volume needs the oracle's closing sweep narrowed. That sweep fails the lane if any entry is stored escaped, which is the right invariant for the name tests but not for this one: an escape-shaped literal is one of the matrix's name classes, so a byte-exact volume owes that name back unchanged and the sweep read correct behavior as a violation. It now prunes the matrix subtree and checks that subtree positively instead (a name needing an escape on a folding volume has to be stored literally here), which keeps the coverage the prune would otherwise drop. The testing guide describes the lanes by family and states the shared recipe contract (a self-provisioned sysroot with a host-side on-disk assertion after the guest exits) instead of enumerating targets that drift.
henrybear327
force-pushed
the
sysroot/casefold-redesign
branch
from
August 5, 2026 07:11
479545c to
c65bf64
Compare
Contributor
|
Thank @henrybear327 for contributing! |
Collaborator
Author
|
This branch contains an additional change that I was about to split out as a separate PR for review, but since the PR is now merge I will document it down here. The first 2 commits of this PR is to fix a problem that was discovered on the main branch (the error line |
Contributor
Next time, do incremental changes for better review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The design is being refined as the testing and simplification of the code are carried out continuously
A Linux guest names files by exact bytes. The default APFS volume matches names
case- and normalization-blind, so
Fooandfoocannot coexist and a wrong-caselookup succeeds where Linux owes
ENOENT.The previous answer,
src/syscall/sidecar.c(2217 lines), stored such a nameunder an opaque random token (
.ef_plus 64 bits fromarc4random, retried onEEXIST) and recorded what the token meant in a per-directory.elfuse_case_indexfile. The mapping lived outside the name, and everythingbelow follows from that one decision.
A name was two objects, so no mutation was atomic
Creating, removing, or renaming a name had to touch the token and the index
file, which no host syscall can do together.
renameattherefore ran ahand-rolled two-phase commit: clone both directory indices as rollback
snapshots, write the index or indices to disk, call
renameat, and on failurewrite the saved snapshots back. The comment conceded the gap: "A failed
rollback write is the best-effort case." A crash between the index write and
the host rename left a mapping pointing at a token that had moved or did not
exist, and nothing repaired it on the next mount. Cross-directory rename took
two index locks in a fixed order, with a
swappedflag deciding which wentfirst, so the ordinary rename path carried deadlock-avoidance logic.
It forced serialization that the filesystem does not need
Two lock layers, because POSIX advisory locks are per-process and every vCPU
thread would share one: a process-wide
pthread_mutex_tacross all vCPUthreads, plus an
fcntlwrite lock on a.elfuse_case_index.locksentinel toserialize against other elfuse processes on the same sysroot. Two guests
creating unrelated files in one directory contended, and the process mutex
serialized name creation across every directory at once.
The index was advisory, so anything else desynchronized it
fcntllocks bind only the processes that take them. Any writer that is notelfuse (
tar,rsync,git, Finder) could add or delete entries withouttouching the index, leaving rows describing tokens that were gone and files
with no row at all. In the other direction the tokens are opaque, so a sysroot
was unreadable and unmaintainable outside elfuse, and copying one preserved
the mapping only if the index files came along intact.
Reads paid for it, and the cache was itself a correctness surface
The walker visited every parent directory of every translated path, probing
each for an index. That cost dominated startup, measured at "openat 61% of
getent's 7.5 ms warm path", so it needed a 64-slot open-addressed cachekeyed on
(dev, ino, mtime, ctime), last writer wins on collision. Orderingthen became load-bearing: the publish path had to invalidate the cache
before the rename, or a concurrent walker could cache "no index here" and
miss one another process had just published.
Leftovers
Three reserved names per directory (index,
.tmp,.lock) had to be filteredout of every listing and refused as guest paths. Each mutation parsed the whole
index into a heap row array and deep-cloned it for rollback, so a create in a
large directory cost work proportional to the directory. And the sidecar
resolved paths independently of
path_translate_at, so two walks disagreedabout when a guest path falls through to the host.
Approach
Decide a name's on-disk spelling from the name alone. Nothing is persisted, so
nothing can desynchronize, and no lock is needed because there is no second
object to keep consistent.
src/syscall/casefold.c): a name containing an uppercase ASCIIletter, a byte above
0x7F, or an escape-shaped prefix is stored as.ef=<payload>. What is left literal is lowercase ASCII, a fixed point ofevery transformation the volume applies, so two literal names cannot collide
however the folding table changes. The payload has two tiers because the
per-name limit is 255 UTF-16 code units rather than bytes: hex to 125 bytes,
then twelve bits per symbol from 4096 CJK ideographs at U+4E00, a block that
neither folds nor normalizes. Guest names keep all 255 bytes Linux allows.
src/syscall/casefold-walk.c): resolve a path one component at atime, asking the volume for each name as stored (
getattrlistat) andcomparing bytes, the only way to separate "exists as spelled" from "exists
under a spelling that folded onto it". Absence is three answers, not one:
unclaimed falls through to the host, a slot held by a differently-spelled
sibling is
ENOENT, and a non-directory component isENOTDIR(
path_resolution(7)).proc_resolve_sysroot_path_flagsandpath_translate_atfold into that single walk, which also backs the
openat2RESOLVE_NO_SYMLINKSandRESOLVE_NO_XDEVwalkers. Directory entries andthe cwd decode on the way back out.
and its five private syscall handlers, which now get the ordinary
/proc,FUSE, and
/dev/shmhandling. Colliding creates never contend, because thespelling is a function of the name. Two elfuse processes sharing a sysroot
need no coordination, because no shared mutable state remains.
Prior art
The same question ("the host filesystem cannot hold this name") has three
standard answers, and this branch moves elfuse from the worst to the best of
them.
managedmounts, itsencoding scheme, were removed in 1.7 in favor of real NTFS case sensitivity
via the
obcaseinsensitivekernel flag, with per-mountposix=[0|1].elfuse's equivalent is
--create-sysroot, which builds a case-sensitive APFSsparsebundle, and Docker Desktop and Lima sidestep it entirely with a Linux
VM over ext4. It is the cleanest answer when available, which is why the
check suite re-runs the whole name suite on that volume as its oracle. It is
not always available: the sparsebundle is opt-in and a user may point
--sysrootat any directory.encode, for characters Windows disallows, it transposes them into the Unicode
private use area by adding
0xf000: a pure per-character function with noside table. gocryptfs, EncFS, and eCryptfs are the same family, deriving the
on-disk name from the plaintext name and a key. This branch is that answer.
inside encrypted blocks. Even gocryptfs falls back to it, but only at the
boundary the derivation cannot cross: when an encrypted name would exceed
255 bytes it writes a
gocryptfs.longname.<hash>.namesupport file. That isprecisely what elfuse's second tier avoids. Packing twelve bits per CJK
symbol keeps the derived property across the full 255-byte guest range, so
there is no length at which a side file reappears.
Compatibility
A sysroot written by the old encoding must be recreated. Its
.ef_<token>entries and
.elfuse_case_indexfiles decode to nothing and surface undertheir literal host names.
docs/usage.mdsays so and gives the recipe.Tests
Two host unit lanes (codec, resolver) link exactly the code under test.
tests/casefold-vectors.hfreezes the on-disk spellings in both directions,making a format change a deliberate migration. Guest lanes cover edge shapes
(one-character sysroot, no sysroot, chdir), i18n pairs the volume folds, name
and path length budgets, host-staged spellings, a corpus read back by guest
name, and concurrency (a fork race run ten times in
check, plus a manualsoak).
Two lanes are oracles rather than expectations:
check-name-caseexactre-runs the name suite on a case-sensitive APFSsparsebundle, where the volume itself enforces what the tests assert. It is
what surfaced the
ENOTDIRand vanished-path fixes.test-sysroot-path-matrixenumerates addressing mode x operation x pathshape x name class and holds each cell to
path_resolution(7): absolute,cwd-relative, and dirfd-relative spellings of one file must agree on result,
errno, and object. On first run it caught two bugs nobody predicted. It runs
on the folding tmpdir, the byte-exact volume, and the qemu reference kernel.
The matrix gains
ELFUSE_SKIPas the inverse ofQEMU_SKIP, with.ci/check-matrix-lists.shrejecting a label naming no registered test orsitting in both lists.
Summary by cubic
Redesigns filename handling with a stateless
.ef=codec and a case‑exact path resolver, replacing the sidecar so Linux byte‑exact names work on case‑folding APFS without locks or per‑dir files. All path surfaces resolve and report guest bytes (symlink targets, exec, inotify, pathnameAF_UNIX, directory listings, cwd); temp roots (/tmp,/var/tmp,~/.ccache) are consistently redirected through the sysroot for creates and lookups...now clamps at the guest root, andopenat2(RESOLVE_NO_SYMLINKS)handles deep link‑free paths correctly.Bug Fixes
openat2(RESOLVE_NO_SYMLINKS)returnsELOOPwhen a link is present.PT_INTERPthrough the same walk with/lib/<basename>fallback;/proc/self/exeand/proc/self/fd/Nreport guest bytes; shm probes honorO_NOFOLLOW./dev/shmleaves are watched without following.ENAMETOOLONGat the host path ceiling (no truncation); trailing/keepsENOTDIR;--sysroot /parent split fixed; directory entry decoding is scoped to sysroot‑owned directories (host fallback and no‑sysroot lists stay literal);openat2(RESOLVE_NO_XDEV)re‑checks mount class after open;..is clamped at the guest root for absolute and dirfd‑relative paths;RESOLVE_NO_SYMLINKSno longer caps components, so deep link‑free paths resolve.getcwdand/proc/self/cwdreport guest bytes so the path is chdir‑able.Migration
.ef_<token>and.elfuse_case_indexfiles appear as literal host entries. See docs/filenames.md and docs/usage.md.Written for commit c65bf64. Summary will update on new commits.