273 Commits

Author SHA1 Message Date
mae
15dadee24a cmd/irdump: formatted disassembly 2026-02-08 03:09:20 -06:00
mae
58431161b5 cmd/irdump: basic disassembler 2026-02-08 03:09:19 -06:00
mae
c60762fe85 cmd/irdump: create cli 2026-02-07 20:34:23 -06:00
6ee3ed1711 internal/rosa/go: alternative bootstrap path
For targets where the bootstrap toolchain is not available.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-08 01:45:21 +09:00
2f3e323c46 internal/rosa/gnu: gcc toolchain artifact
This toolchain is hacked to pieces. It works well enough to bootstrap Go, though.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-08 01:00:15 +09:00
1fc9c3200f internal/rosa: libucontext artifact
Required by GCC on musl.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-07 22:33:12 +09:00
096a25ad3a cmd/mbf: dump IR of artifact presets
This exposes IR outside test cases, useful for verifying correctness of alternative IR emitters.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-07 17:21:43 +09:00
ffd2f979fb internal/pkg: skip duplicate early
This significantly increases IR generation performance.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-07 17:11:41 +09:00
31a8cc9b5c internal/rosa/gnu: binutils artifact
Appears to be required by GCC? It complains with stuff installed by LLVM.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-07 14:02:23 +09:00
bb3f60fc74 internal/rosa/gnu: gmp, mpfr, mpc artifacts
Required by GCC.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-07 13:55:40 +09:00
697c91e04d internal/rosa/cmake: expose earlier build script
This allows for more flexible build setups.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-07 13:23:13 +09:00
3f7b8b4332 internal/rosa/git: git clone helper
For obtaining sources of projects that stubbornly refuse to provide release tarballs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-06 21:10:59 +09:00
fa94155f42 internal/rosa/etc: resolv.conf
Required by programs that download from the internet in measured execArtifact.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-06 21:04:59 +09:00
233bd163fb internal/rosa/git: disable flaky test
This fails intermittently.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-06 20:45:52 +09:00
f9b69c94bc internal/rosa/ssl: prefix CA paths
This makes prefixes consistent with everything else since this will end up in the final Rosa OS image.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-06 20:41:58 +09:00
68aefa6d59 internal/rosa/openssl: fix paths
Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-06 19:53:51 +09:00
159fd55dbb internal/rosa/ssl: fix dependencies
These used to be provided by busybox.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 19:12:48 +09:00
ce6b3ff53b internal/rosa: unzip artifact
Because the zip format is too awful and cannot be streamed anyway, supporting it natively comes with no benefit.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 19:10:32 +09:00
30afa0e2ab internal/rosa/git: compile with http support
This should be able to fetch repositories deterministically.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 18:51:02 +09:00
9b751de078 internal/rosa/gnu: fix test suite flags
This sets the correct flag and also avoids changing ident per system.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 18:29:47 +09:00
d77ad3bb6e internal/rosa: curl artifact
Required for http support in git.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 18:15:16 +09:00
0142fc90b0 internal/rosa/make: post-configure script
Required for some projects with broken build scripts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 18:13:48 +09:00
3c9f7cfcd0 internal/rosa: libpsl artifact
Required by curl.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 18:06:33 +09:00
a3526b3ceb internal/rosa: openssl artifact
Optional for many programs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 18:03:18 +09:00
6ad21e2288 internal/rosa: register custom artifacts
This also encodes extra information for iana-etc.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 17:50:48 +09:00
27e2e3f996 internal/rosa/llvm: drop git dependency
This was added quite early and has no effect. Remove it to avoid compiling git early.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 17:44:58 +09:00
e0c720681b internal/pkg: standardise artifact IR
This should hopefully provide good separation between the artifact curing backend implementation and the (still work in progress) language. Making the IR parseable also guarantees uniqueness of the representation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-05 08:24:09 +09:00
f982b13a59 internal/pkg: improve error resolution
This was taking way too long for early failures.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-03 10:01:44 +09:00
443911ada1 internal/rosa: use stage3 mirror
These get taken down periodically and causes way too many rebuilds when they are taken down. Use mirror until a more elegant solution is available.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-02 23:43:34 +09:00
d7a3706db3 internal/rosa/x: regenerate build system
These come with 16-year-old scripts that do not understand aarch64 or really anything else relevant to Rosa OS.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-02 19:57:39 +09:00
3226dc44dc internal/rosa/gnu: libtool artifact
Required when generating autotools build systems.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-02 19:52:08 +09:00
9f98d12ad8 internal/rosa/gnu: automake artifact
This is very expensive. Avoid.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-02 18:49:18 +09:00
550e83dda9 internal/rosa/gnu: grep artifact
Some GNU software do not like the grep in toybox.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-02 18:38:01 +09:00
7877b4e627 cmd/mbf: print extra stage3 information
This includes ident of all three stages and the matching checksum if check is passing.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-02 18:33:16 +09:00
47ce6f5bd0 internal/rosa/llvm: conditionally add Rosa OS paths
This change also moves rpath flags to a more appropriate method.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-02-02 07:29:35 +09:00
48f4ccba33 internal/rosa/llvm: add rosa vendor
This cleans up checks specific to Rosa OS, and fixes stack overflow in llvm under certain conditions.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 22:47:13 +09:00
c31884bee4 internal/rosa: disable broken tests
These fail when running as users with supplementary groups, since they are unmapped in the container. This was not the case in the development container where all groups were dropped, so the failure was missed.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 14:12:40 +09:00
f8661ad479 internal/rosa/hakurei: backport test case fix
This patch will be removed in the next release.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 12:00:21 +09:00
536f0cbae6 internal/rosa/gnu: gettext 0.26 to 1.0
This now requires kernel headers for some reason.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 11:41:07 +09:00
8d872ff1cd internal/rosa: fetch from gnu mirror
GNU infrastructure is extraordinarily flaky and fetching from it killed the server too many times.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 11:26:48 +09:00
bf14a412e4 container: fix host-dependent test cases
These are not fully controlled by hakurei and may change depending on host configuration.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 10:59:56 +09:00
8b4576bc5f internal/rosa: migrate to make helper
This migrates artifacts that the helper cannot produce an identical instance of.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 08:55:33 +09:00
29ebc52e26 internal/rosa/hakurei: suffix variants
This makes log output more useful.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 05:03:15 +09:00
5f81aac0e2 internal/rosa: make helper
This change only migrates artifacts that remain unchanged under the helper, so this change should not cause any rebuilds.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 05:01:22 +09:00
47490823be internal/rosa: improve cmake interface
This should make the call site look better for new artifacts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 02:42:01 +09:00
1ac8ca7a80 internal/rosa: isolate make implementation
This will come with a helper eventually.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-31 02:33:14 +09:00
fd8b2fd522 internal/rosa: fix up dependencies
These are no longer provided by the (incomplete) toybox implementations, so they need to be specified explicitly.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-30 03:11:16 +09:00
20a8519044 internal/rosa/mksh: also build lksh
This is better suited for /system/bin/sh. Full mksh is still included, installed at /system/bin/mksh.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-30 00:54:30 +09:00
8c4fd00c50 internal/rosa/ninja: build in $TMPDIR
This used to build in /work/system/bin/ and unfortunately leaves its garbage there. This behaviour is from very early stages of this package, and was never fixed. This change updates it to use the "$(mktemp -d)" convention that every other artifact uses.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-30 00:36:43 +09:00
bc3dd6fbb0 internal/rosa: chmod via patch helper
This works around the zfs overlay mount overhead and significantly reduces I/O in general.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-30 00:29:44 +09:00
616ed29edf internal/rosa: early toybox variant
This is a variant of toybox with unfinished tools enabled, for artifacts that will end up in a dependency loop without them.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-30 00:24:14 +09:00
9d9b7294a4 internal/rosa: flags for toolchain-dependent artifact
This is much cleaner to add extra flags to.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 20:44:43 +09:00
6c1e2f10a7 internal/rosa: remove busybox artifact
This is no longer used and its implementation is unacceptably shaky.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 19:58:51 +09:00
abf96d2283 internal/rosa: replace busybox with toybox
The busybox artifact does not run on aarch64, and the workarounds required for it to compile successfully on x86_64 is unacceptably shaky. This change fully replaces it with toybox.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 19:51:18 +09:00
6c90e879da internal/rosa/llvm: enable asan
This is required by test suite of latest toybox.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 18:30:37 +09:00
d1b404dc3a internal/rosa: findutils artifact
Required by llvm test suite, compiler-rt sanitisers-related tests fail on toybox xargs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 18:24:01 +09:00
744e4e0632 internal/rosa: sed artifact
Required by various GNU programs as they are not happy with toybox sed.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 18:17:18 +09:00
85eda49b2b internal/rosa: xz artifact
Wanted to avoid this as much as possible. Unfortunately newer versions of GNU findutils only come in xz and is required for llvm compiler-rt sanitisers.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 18:06:17 +09:00
b26bc05bb0 internal/rosa: remove unused receiver
This returns the preset itself, it is up to the caller to load the underlying artifact.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 17:52:50 +09:00
2d63ea8fee internal/rosa: gzip artifact
Toybox does not implement this, and it is used by many programs, including toybox itself.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 17:46:30 +09:00
dd4326418c internal/rosa: toybox artifact
This compiles surprisingly quickly and required no workarounds, unlike busybox.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 17:44:30 +09:00
79c0106ea0 internal/rosa: replace busybox dash with mksh
Toybox does not provide a shell, mksh fills that gap.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 02:00:32 +09:00
536db533de internal/rosa: install bash as sh
This works around software relying on bashisms even when explicitly invoking sh.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 01:25:44 +09:00
07927006a8 internal/pkg: set User-Agent header
Avoid living under the default user agent and be at the mercy of some IDS.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 01:17:35 +09:00
77ea27b038 internal/rosa: mksh artifact
This provides a shell, as part of the effort to replace busybox.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-29 00:51:32 +09:00
e76bc6a13a internal/rosa: resolve preset by name
Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-28 20:57:51 +09:00
cc403c96d8 internal/rosa: remove busybox patch
This allows different versions of busybox to be attempted, to find one that works on arm.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-28 01:24:56 +09:00
66118ba941 internal/rosa: gawk artifact
Replaces broken awk in busybox.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-28 01:22:42 +09:00
823ba08dbc internal/rosa: use patch helper
This is significantly cleaner and runs somewhat faster.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-28 00:58:34 +09:00
660835151e internal/rosa: disable busybox SHA1_HWACCEL
This also pretties up the build output and sets correct linker path.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-28 00:51:09 +09:00
53e6df7e81 internal/rosa: remove uname
This does not change ident based on target.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 23:37:29 +09:00
bd80327a8f internal/rosa: add arm64 strings
This enables building on arm64.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 21:12:39 +09:00
41f9aebbb7 internal/pkg: allow multiarch
The armv8l busybox binary release needs this to run correctly.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 21:07:18 +09:00
a2a0e36802 internal/rosa: cross-platform stage3
The stage3 binary seed is arch-specific.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 20:58:31 +09:00
fbe93fc771 internal/rosa/busybox: cross-platform binary
The initial binary seed is arch-specific.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 20:54:11 +09:00
968d8dbaf1 internal/pkg: encode checksum in ident
This also rearranges the ident ir to be more predictable, and avoids an obvious and somewhat easy to get into inconsistent state.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 20:18:34 +09:00
f1758a6fa8 internal/rosa: nss artifacts
Not used by anything for now, but will be part of Rosa OS.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 08:17:58 +09:00
88aaa4497c internal/rosa/hakurei: dist tarball
The patch will be removed in the next release.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 07:34:45 +09:00
b7ea68de35 internal/rosa/hakurei: isolate hakurei helper
For creating the dist tarball, which runs in the same environment.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 07:28:13 +09:00
67e453f5c4 dist: run tests
This used to be impossible due to nix jank which has been addressed.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 07:00:39 +09:00
67092c835a internal/rosa/hakurei: v0.3.3 to v0.3.4
This now contains the sharefs program which pulls in fuse.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 05:40:53 +09:00
18918d9a0d internal/rosa: fuse artifact
Required by hakurei for sharefs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 05:34:42 +09:00
380ca4e022 internal/rosa: pytest artifact
Required by libfuse. This pulls in many dependencies.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 05:20:37 +09:00
887aef8514 internal/rosa: python pip helper
Fuse requires pytest which depends on many packages. This helper eases the pain of packaging them.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 05:14:59 +09:00
d61faa09eb release: 0.3.4
Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 03:56:06 +09:00
50153788ef internal/rosa: hakurei artifact
This does not yet have fuse from staging. Everything else works perfectly, though.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 02:24:49 +09:00
c84fe63217 internal/rosa: various X artifacts
Required by xcb which is required by hakurei.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 02:02:49 +09:00
eb67e5e0a8 internal/pkg: exclusive artifacts
This alleviates scheduler overhead when curing many artifacts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 01:23:50 +09:00
948afe33e5 internal/rosa/acl: use patch helper
This is significantly less ugly.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-27 00:30:50 +09:00
76c657177d internal/rosa: patch ignore whitespace
This makes it work better with patches emitted by git.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 21:56:36 +09:00
4356f978aa internal/rosa: kernel patching
The side effect of this is to work around zfs performance issue with chmod on overlay mount.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 21:20:52 +09:00
4f17dad645 internal/rosa: isolate patching helper
This is useful outside llvm as well.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 21:00:29 +09:00
68b7d41c65 internal/rosa: parallel autoconf tests
These take forever and run sequentially by default for some reason.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 19:52:59 +09:00
e48f303e38 internal/rosa: parallel perl tests
This is found in the github action, the test target does not appear to support parallelisation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 19:45:50 +09:00
f1fd406b82 internal/rosa: link libc ldd
Musl appears to implement this behaviour but does not install the symlink by default.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 08:00:03 +09:00
53b1de3395 internal/rosa: enable static on various artifacts
This is implicitly enabled sometimes, but better to be explicit.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 07:56:14 +09:00
92dcadbf27 internal/acl: connect getfacl stderr
This shows whatever failure is happening in the cure container.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 07:51:16 +09:00
0bd6a18326 internal/rosa: acl artifact
Required by hakurei.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 07:38:56 +09:00
67d592c337 internal/pkg: close gzip reader on success
The Close method panics otherwise.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 07:06:38 +09:00
fdc8a8419b internal/rosa: static libwayland
Required by hakurei.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 06:49:08 +09:00
122cfbf63a internal/rosa: run wayland tests
Broken test is disabled for now.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 06:39:45 +09:00
504f5d28fe internal/rosa: libseccomp artifact
Required by hakurei.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 05:28:36 +09:00
3eadd5c580 internal/rosa: gperf artifact
Required by libseccomp.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 05:25:39 +09:00
4d29333807 internal/rosa: wayland-protocols artifact
Required by hakurei.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 05:13:30 +09:00
e1533fa4c6 internal/rosa: wayland artifact
Required by hakurei.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 05:10:35 +09:00
9a74d5273d internal/rosa: libgd artifact
Required by graphviz which is required by wayland.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 04:20:11 +09:00
2abc8c454e internal/pkg: absolute hard link
This cannot be relative since the curing process is not in the temp directory.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 04:03:05 +09:00
fecb963e85 internal/rosa: libxml2 artifact
Required by wayland. Release tarball is xz only, unfortunately.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 03:47:42 +09:00
cd9da57f20 internal/rosa: libexpat artifact
Required by wayland.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 03:15:25 +09:00
c6a95f5a6a internal/rosa: meson artifact
Required by wayland and pipewire.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 03:03:21 +09:00
228489371d internal/rosa: setuptools artifact
Apparently the only way to install python stuff offline.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 02:28:47 +09:00
490471d22b cmd/mbf: verbose by default
It usually does not make sense to use this without verbose.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 02:12:56 +09:00
763d2572fe internal/rosa: pkg-config artifact
Used by hakurei and many other programs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 01:26:54 +09:00
bb1b6beb87 internal/rosa: name suffix by toolchain
This makes output more useful during bootstrap.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 00:57:03 +09:00
3224a7da63 cmd/mbf: disable threshold by default
This is not very useful.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-26 00:05:59 +09:00
8a86cf74ee internal/rosa/go: symlink executables
This avoids having to fix up $PATH for every artifact.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-25 23:59:08 +09:00
e34a59e332 internal/rosa/go: run toolchain tests
LLVM patches and a TMPDIR backed by tmpfs fixed most tests. Broken tests in older versions are disabled.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-25 21:21:53 +09:00
861801597d internal/pkg: expose response body
This uses the new measured reader provided by Cache. This should make httpArtifact zero-copy.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-25 16:10:34 +09:00
334578fdde internal/pkg: expose underlying reader
This will be fully implemented in httpArtifact in a future commit.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-25 14:48:25 +09:00
20790af71e internal/rosa: lazy initialise all artifacts
This improves performance, though not as drastically as lazy initialising llvm.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-25 01:43:18 +09:00
43b8a40fc0 internal/rosa: lazy initialise llvm
This significantly improves performance.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-25 00:29:46 +09:00
87c3059214 internal/rosa: run perl tests
A broken test with unexplainable failure is disabled.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-24 18:58:09 +09:00
6956dfc31a internal/pkg: block on implementation entry
This avoids blocking while not in Cure method of the implementation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-24 16:02:50 +09:00
d9ebaf20f8 internal/rosa: stage3 special case helper
This makes it cleaner to specify non-stage3 and stage3-exclusive dependencies.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-24 12:23:35 +09:00
acee0b3632 internal/pkg: increase output buffer size
This avoids truncating unreasonably long lines from llvm.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-24 11:45:44 +09:00
5e55a796df internal/rosa: gnu patch artifact
This is more robust than the busybox implementation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-24 11:32:27 +09:00
f6eaf76ec9 internal/rosa: patch library paths
This removes the need for reference LDFLAGS in the standard toolchain.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-24 11:22:25 +09:00
5c127a7035 internal/rosa: patch header search paths
This removes the need for reference CFLAGS in the standard toolchain.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-23 01:56:52 +09:00
8a26521f5b internal/rosa/go: run bootstrap toolchain tests
The objdump test will be re-enabled after fixing llvm search paths.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 07:05:48 +09:00
0fd4556e38 internal/rosa/llvm: fix broken test patch
Both stage1 and stage2 passes at this point.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 06:42:04 +09:00
50b82dcf82 internal/rosa/gnu: coreutils artifact
Required by llvm unit and regression tests.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 05:42:23 +09:00
20a8d30821 internal/rosa/busybox: link /usr/bin/env
This is required by many scripts which uses bash but still pretends to be portable.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 04:21:06 +09:00
cdf2e4a2fb internal/rosa: bash artifact
Required by llvm unit and regression tests.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 04:06:48 +09:00
dcb8a6ea06 internal/rosa: fix toolchain layer order
This allows extras to override toolchain artifacts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 03:37:46 +09:00
094a62ba9d internal/rosa: diffutils artifact
LLVM tests are not happy with busybox diff.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 03:00:59 +09:00
6420b6e6e8 internal/rosa: libffi artifact
Required by python during llvm test suite.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-22 02:47:47 +09:00
d7d058fdc5 internal/rosa/gnu: disable broken tests
These are documented as broken via comments yet not disabled on musl for some reason.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 23:31:45 +09:00
84795b5d9f internal/rosa/git: add dependencies
These are required outside the stage3 toolchain.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 22:14:48 +09:00
f84d30deed internal/rosa/gnu: run checks
Checks are not run for gettext for now since it contains broken tests.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 22:07:08 +09:00
77821feb8b internal/rosa: gettext artifact
Compile time dependency of git.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 21:52:50 +09:00
eb1060f395 internal/rosa: autoconf artifact
Required by git to reconfigure some options.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 21:44:49 +09:00
0e08254595 internal/rosa: m4 artifact
Autotools dependency.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 21:42:25 +09:00
349d8693bf internal/rosa: perl artifact
This runs without tests for now, will be enabled after some toolchain patches.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 21:33:12 +09:00
e88ae87e50 internal/rosa/llvm: run unit and regression tests
Two tests are marked expected to fail for Rosa OS.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 08:25:36 +09:00
7cd4aa838c internal/rosa/llvm: patch source tree
A few patches are required for disabling broken tests and changing default search paths.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 08:25:17 +09:00
641942a4e3 internal/rosa/cmake: chmod entire source tree
This works around builds that traverse out of the appended pathname.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 07:26:36 +09:00
b6a66acfe4 internal/rosa: git artifact
This is required by the clang unit and regression tests.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 07:00:14 +09:00
b72dc43bc3 internal/pkg: report dependency graph size
This is an interesting value to know when profiling.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 05:35:28 +09:00
8e59ff98b5 internal/rosa: include iana-etc
This is used by some programs and will likely end up in the Rosa OS system image anyway.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 05:28:41 +09:00
f06d7fd387 cmd/mbf: expose some artifacts for curing
This will remain until dist is successfully bootstrapped.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 05:18:08 +09:00
ba75587132 internal/pkg: allow user namespace creation
No good reason to filter this in the execArtifact container, and the extended filter breaks certain programs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-21 04:49:25 +09:00
9a06ce2db0 internal/rosa: bootstrap go toolchain
This runs without tests for now. Will be fixed in a later commit.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-20 07:14:11 +09:00
3ec15bcdf1 internal/rosa/cmake: use hardcoded build directory
This eliminates some nondeterminism. Still getting 3-stage non-determinism in runtimes and clang, though.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 22:51:34 +09:00
d933234784 internal/pkg: make checksum available to cure
This enables deduplication by value as implemented in execArtifact.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 21:29:56 +09:00
1c49c75f95 cmd/mbf: toolchain 3-stage non-determinism check
This unfortunately fails right now. Requires further investigation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 04:40:44 +09:00
6a01a55d7e internal/rosa: parallel cmake bootstrap
This takes a very long time otherwise.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 03:21:29 +09:00
b14964a66d internal/rosa: standard toolchain via 2-stage bootstrap
This implements the 2-stage bootstrap build without clumping the stages together in the cmake target.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 02:57:28 +09:00
ff98c9ded9 internal/rosa: llvm bootstrap artifacts
This bootstraps the LLVM toolchain across multiple artifacts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 02:48:44 +09:00
7f3d1d6375 internal/rosa: llvm artifact abstraction
The llvm bootstrap is multi-stage by nature, and cannot be completed in a single artifact.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 02:29:06 +09:00
3a4f20b759 internal/rosa: cmake abstraction
This is a helper for generating cure script for a cmake-based project.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 02:20:37 +09:00
21858ecfe4 internal/rosa: ninja artifact
Generated by cmake, recommended format for llvm toolchain.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 02:08:17 +09:00
574a64aa85 internal/rosa: cpython artifact
Dependency of llvm build scripts, also an optional cure dependency of ninja.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 01:58:53 +09:00
85d27229fd internal/rosa: zlib artifact
Dependency of llvm build scripts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 01:48:27 +09:00
83fb80d710 internal/rosa: cmake artifact
This is required for compiling the toolchain and many other programs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 01:36:38 +09:00
fe6dc62ebf internal/rosa: musl libc artifact
This will likely be included in Rosa OS. The installation is modified to be entirely contained in prefix.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 01:25:41 +09:00
823f9c76a7 internal/rosa: busybox from source
This will be part of the standard toolchain.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 01:12:47 +09:00
2df913999b internal/rosa: kernel headers
This is required by the toolchain and many other programs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 01:03:19 +09:00
52c959bd6a internal/rosa: minimal rsync artifact
For installing kernel headers.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 00:58:34 +09:00
d258dea0bf internal/rosa: bootstrap on gentoo stage3
This contains a fully working musl+llvm toolchain and many build systems in a pretty small package.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 00:51:49 +09:00
dc96302111 internal/rosa: GNU make artifact
This compiles GNU make from source. This is unfortunately required by many programs, but is a cure dependency only.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 00:40:25 +09:00
88e9a143d6 internal/rosa: toolchain abstraction
This provides a clean and easy to use API over toolchains. A toolchain is an opaque set of artifacts and environment fixups. Exported toolchains should be functionally indistinguishable from each other.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 00:28:16 +09:00
8d06c0235b internal/rosa: busybox binary artifact
This installs a statically linked busybox binary distribution for decompressing the gentoo stage3 tarball, since there is no native xz implementation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-19 00:07:56 +09:00
4155adc16a internal/rosa: static etc artifact
This places configuration files with hardcoded content in /etc to silence test suites expecting them to be present.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-18 23:51:45 +09:00
2a9525c77a cmd/mbf: command handling
This tool is a frontend for bootstrapping hakurei via internal/pkg. Named mbf for now for "maiden's best friend" as a tribute to the DOOM source port.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-18 22:19:19 +09:00
efc90c3221 internal/pkg: deduplicate dependency errors
This significantly simplifies error reporting for caller.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-17 14:41:00 +09:00
610ee13ab3 internal/pkg: lock on-filesystem cache
Any fine-grained file-based locking here significantly hurts performance and is not part of the use case of the package. This change guarantees exclusive access to prevent inconsistent state on the filesystem.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-16 18:12:51 +09:00
5936e6a4aa internal/pkg: parallelise scrub
This significantly improves scrubbing performance. Since the cache directory structure is friendly to simultaneous access, this is possible without synchronisation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-16 02:47:59 +09:00
3499a82785 internal/pkg: cache computed identifiers
This eliminates duplicate identifier computations. The new implementation also significantly reduces allocations while computing identifier for a large dependency tree.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-15 23:30:43 +09:00
088d35e4e6 internal/pkg: optional dependency graph size limit
This provides a quick check against cyclic dependencies without hurting cure performance.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-14 18:25:46 +09:00
1667df9c43 internal/pkg: zero atime and mtime
This is significantly more practical than keeping track of them in directory flattening format and setting this in every non-artifact implementation. Only tarArtifact can have meaningful deterministic checksums that are not zero and zeroing them still keeps autotools happy.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-13 01:30:30 +09:00
156dd767ef internal/pkg: remove typeflag promotion loop
Expanding this enables sharing of code common between types.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-13 00:33:13 +09:00
5fe166a4a7 internal/pkg: exec prefix verbose output
This proxies program output through msg with a name and fd prefix. This also avoids introducing additional information to the container via process stdout/stderr.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-12 22:15:01 +09:00
41a8d03dd2 internal/pkg: cure completion verbose messages
This reports cure completions to the user.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-12 21:56:00 +09:00
610572d0e6 internal/pkg: optionally named static file
These are generally for generating configuration files or build scripts, naming them is quite useful.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-12 04:34:50 +09:00
29951c5174 internal/pkg: caller-supplied reporting name for exec
This does not have a reasonable way of inferring the underlying name. For zero value it falls back to base of executable pathname.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-12 04:17:47 +09:00
91c3594dee internal/pkg: append user-facing name in messages
This makes verbose messages much more useful.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-12 03:53:19 +09:00
7ccc2fc5ec internal/pkg: exec with specific timeout
This change also updates the documentation of NewExec.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-11 17:46:04 +09:00
63e137856e internal/pkg: do not discard the result of compact
This result was mistakenly unused resulting in incorrect identifiers for artifacts with duplicate dependencies.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-11 04:40:24 +09:00
e1e46504a1 container/check: return error backed by string type
The struct turned out not necessary during initial implementation but was not unwrapped into its single string field. This change replaces it with the underlying string and removes the indirection.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-11 04:23:55 +09:00
ec9343ebd6 container/check: intern absolute pathnames
This improves performance in heavy users like internal/pkg.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-11 04:18:11 +09:00
423808ac76 nix: use package from module in default
This makes overriding hakurei easier. Also avoids building hakurei twice since nix does that for some reason.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-11 03:50:08 +09:00
2494ede106 container/init: configure interface lo
This enables loopback networking when owning the net namespace.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-11 03:36:20 +09:00
da3848b92f internal/pkg: compare interfaces for host net
An upcoming improvement in the container init makes the current host net check return the same result for both cases. This change

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-11 00:30:32 +09:00
34cb4ebd3b internal/pkg: pass context to file cure
This removes the left over embedded contexts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-09 05:31:38 +09:00
f712466714 internal/pkg: move dependency flooding to cache
This imposes a hard upper limit to concurrency during dependency satisfaction and moves all dependency-related code out of individual implementations of Artifact. This change also includes ctx and msg as part of Cache.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-09 05:20:34 +09:00
f2430b5f5e internal/pkg: use short wait delay
The cure is condemned at the point of cancellation and all of its state is destroyed by the deferred cleanup, so it makes little sense to wait for it much.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-08 18:58:51 +09:00
863e6f5db6 internal/pkg: use correct artifact count
This updates buffer sizes and counters to use correct total artifact count.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-08 08:45:37 +09:00
23df2ab999 internal/pkg: place ephemeral upperdir in tmp
This enables the use of directories made writable this way as scratch space.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-08 08:23:16 +09:00
7bd4d7d0e6 internal/pkg: support explicit overlay mount
This removes all but the /work/ auto overlay behaviour and enables much greater flexibility. This also renames ExecContainerPath to ExecPath so it is easier to type.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-08 07:55:09 +09:00
b3c30bcc51 internal/pkg: set container WaitDelay
This prevents a container from blocking forever after context is canceled.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-08 06:23:57 +09:00
38059db835 internal/pkg: make tar temporary directory writable
This allows it to be renamed to work directory.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-08 04:32:14 +09:00
409fd3149e internal/pkg: reserve kind range
This is useful for custom implementations of Artifact.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-08 01:21:51 +09:00
4eea136308 internal/pkg: do not connect stdin
This introduces external state when verbose.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 23:17:58 +09:00
c86ff02d8d internal/pkg: tar optional file
This allows tar to take a single-file directory Artifact as input.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 22:16:26 +09:00
e8dda70c41 internal/pkg: return reader for files
This improves efficiency for cache hits.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 21:36:47 +09:00
7ea4e8b643 internal/pkg: support tarball compressed via bzip2
Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 20:18:28 +09:00
5eefebcb48 internal/pkg: reject entry types disallowed in the cache
These are not encoded in the format, they are rejected here to serve as a check for cache since checksum is computed for every directory.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 03:22:44 +09:00
8e08e8f518 internal/pkg: automatic overlay mount on work
This directly submits the upperdir to cache. It is primarily used in bootstrapping where tools are limited and should not be used unless there is a very good reason to.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 03:14:33 +09:00
54da6ce03d internal/pkg: respect mount order for overlay temp
Setting it up after everything else prevents covering files in /tmp.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 01:03:49 +09:00
3a21ba1bca internal/pkg: implement file artifact
This is an Artifact implementing File, backed by a constant, caller-supplied byte slice.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 00:39:29 +09:00
45301559bf internal/pkg: fail on empty output directory
This works around the fact that execArtifact always creates the work directory when setting up the bind mount.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-07 00:04:32 +09:00
0df87ab111 internal/pkg: automatic overlay mount on tmp
This sets up the last Artifact to target /tmp as a writable overlay mount backed by the host side temp directory. This is useful for an Artifact containing source code to be built for another Artifact for example.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 23:45:08 +09:00
aa0a949cef internal/pkg: do not clear execute bit
Only write should be cleared here, clearing execute causes execArtifact to be unable to start anything since no Artifact is able to produce an executable file.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 22:31:44 +09:00
ce0064384d internal/pkg: automatic overlay mount on root
This makes it possible to use an Artifact as root without arranging for directory creation in the Artifact ahead of time.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 22:19:47 +09:00
53d80f4b66 internal/pkg/testdata: check network
This validates hostNet state.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 21:32:10 +09:00
156096ac98 internal/pkg: known checksum exec artifact
This optionally attaches an output checksum to an execArtifact and enables host networking for the resulting container.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 20:58:34 +09:00
ceb75538cf internal/pkg: update http checksum signature
This was using the old pre-KnownChecksum function signature. It did not affect correctness since httpArtifact performs internal validation to avoid the strict mode vfs roundtrip, but it prevented content-addressed cache hits.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 18:24:31 +09:00
0741a614ed internal/pkg: relocate testtool workaround
This can be reused in other test cases.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 18:06:56 +09:00
e7e9b4caea internal/pkg: exec nil path check during cure
This results in os.ErrInvalid instead of a panic, which hopefully improves user experience.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 17:46:12 +09:00
f6d32e482a internal/pkg: ensure parent for non-directory entries
This works around streams containing out of order entries.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 05:55:24 +09:00
79adf217f4 internal/pkg: implement exec artifact
This runs a program in a container environment. Artifacts can be made available to the container, they are cured concurrently and mounted in order.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 05:05:34 +09:00
8efffd72f4 internal/pkg: destroy temp during deferred cleanup
This avoids missing the cleanup when cure returns an error.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 04:24:54 +09:00
86ad8b72aa internal/pkg: expose cure through cure context
This allows a curing Artifact to cure Artifact it depends on.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 01:04:06 +09:00
e91049c3c5 internal/pkg: pass cure context as single value
This cleans up the function signature and makes backwards compatible API changes possible.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-06 00:56:49 +09:00
3d4d32932d internal/pkg: verify checksum after uneventful scrub
This checks that scrub did not condemn any entry without reporting it.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 22:54:16 +09:00
0ab6c13c77 internal/pkg: consistency check for on-disk cache
This change adds a method to check on-disk cache consistency and destroy inconsistent entries as they are encountered. This primarily helps verify artifact implementation correctness, but can also repair a cache that got into an inconsistent state from curing a misbehaving artifact, without having to destroy the entire cache.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 05:30:29 +09:00
834cb0d40b internal/pkg: override "." for directory checksum
This makes the checksum consistent with the final resting state of artifact directories without incurring the cost of an extra pair of chown syscalls.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 04:00:13 +09:00
7548a627e5 internal/pkg: delete stale done channels
There is no reason to keep these around.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 02:12:34 +09:00
b98d27f773 internal/pkg: expand single directory tarball
This enables much cleaner use of their output without giving up any meaningful data.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 01:43:23 +09:00
f3aa31e401 internal/pkg: temporary scratch space for cure
This allows for more flexibility during implementation. The use case that required this was for expanding single directory tarballs.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 01:39:18 +09:00
4da26681b5 internal/pkg: compute http identifier from url
The previous implementation exposes arbitrary user input to the cache as an identifier, which is highly error-prone and can cause the cache to enter an inconsistent state if the user is not careful. This change replaces the implementation to compute identifier late, using url string as params.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 00:43:21 +09:00
4897b0259e internal/pkg: improve artifact interface
This moves all cache I/O code to Cache. Artifact now only contains methods for constructing their actual contents.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-05 00:01:23 +09:00
d6e4f85864 internal/pkg: ignore typeflag 'g'
Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-04 12:46:56 +09:00
3eb927823f internal/pkg: create symlinks for files
These are much easier to handle than hard links and should be just as transparent for this use case.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-04 01:48:53 +09:00
d76b9d04b8 internal/pkg: implement tar artifact
This is useful for unpacking tarballs downloaded from the internet.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-04 01:34:30 +09:00
fa93476896 internal/pkg: override working directory perms
This must be writable to enable renaming, and the final result is conventionally read-only alongside the entire directory contents. This change overrides the permission bits as part of Store.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-04 00:55:52 +09:00
bd0ef086b1 internal/pkg: enable cache access during store
This is still not ideal as it makes entry into Store sequential. This will be improved after more usage code is written.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-04 00:39:14 +09:00
05202cf994 internal/pkg: pass context in request wrapper
This method is for the most common use case, and in actual use there will always be an associated context.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 23:53:52 +09:00
40081e7a06 internal/pkg: implement caching for directories
This works on any directories and should be robust against any bad state the artifact curing process might have failed at.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 22:54:46 +09:00
863d3dcf9f internal/pkg: wrap checksum string encoding
This wraps base64.URLEncoding.EncodeToString for cleaner call site.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 22:03:25 +09:00
8ad9909065 internal/pkg: compute identifier from deps
This provides infrastructure for computing a deterministic identifier based on current artifact kind, opaque parameters data, and optional dependency kind and identifiers.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 21:26:25 +09:00
deda16da38 internal/pkg: create work directory
This is used for artifacts that cure into directories.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 20:56:11 +09:00
55465c6e72 internal/pkg: optionally validate flat pathnames
This makes the decoder safe against untrusted input without hurting performance for a trusted stream. This should still not be called against untrusted input though.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 18:59:18 +09:00
ce249d23f1 internal/pkg: implement http artifact
This is useful for downloading source tarballs from the internet.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 15:29:58 +09:00
dd5d792d14 go: 1.25
Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-03 15:25:28 +09:00
d15d2ec2bd internal/pkg: relocate cache test helper
This is useful for other tests that need a cache instance.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-02 16:08:43 +09:00
3078c41ce7 internal/pkg: encode entry in custom format
The fact that Gob serialisation is deterministic is an implementation detail. This change replaces Gob with a simple custom format.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-02 15:39:42 +09:00
e9de5d3aca internal/pkg: implement caching for files
This change contains primitives for validating and caching single-file artifacts.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-02 12:57:19 +09:00
993afde840 dist: install sharefs
This also removes the deprecated hpkg program.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2026-01-02 00:57:51 +09:00
c9cd16fd2a cmd/sharefs: prepare directory early
This change also checks against filesystem daemon running as root early.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 23:17:02 +09:00
e42ea32dbe nix: configure sharefs via fileSystems
Turns out this did not work because in the vm test harness, virtualisation.fileSystems completely and silently overrides fileSystems, causing its contents to not even be evaluated anymore. This is not documented as far as I can tell, and is not obvious by any stretch of the imagination. The current hack is cargo culted from nix-community/impermanence and hopefully lasts until this project fully replaces nix.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 23:14:08 +09:00
e7982b4ee9 cmd/sharefs: create directory as root
This optional behaviour is required on NixOS as it is otherwise impossible to set this up: systemd.mounts breaks startup order somehow even though my unit looks identical to generated ones, fileSystems does not support any kind of initialisation or ordering other than against other mount points.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 22:14:33 +09:00
ef1ebf12d9 cmd/sharefs: handle mount -t fuse.sharefs
This should have been handled in a custom option parsing function, but that much extra complexity is unnecessary for this edge case. Honestly I do not know why libfuse does not handle this itself.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 20:49:27 +09:00
775a9f57c9 cmd/sharefs: check option parsing behaviour
This change makes it possible to check parseOpts behaviour as part of Go tests.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 17:33:12 +09:00
2f8ca83376 cmd/sharefs: containerise filesystem daemon
This replaces the forking daemonise libfuse function which prevents Go callbacks from calling into the runtime. This also enforces least privilege on the daemon process.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 10:16:35 +09:00
3d720ada92 container: optionally allow orphan
This is required for the typical daemonise use case.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 09:12:02 +09:00
2e5362e536 cmd/sharefs: opaque setup state
This allows unrestricted use of the type system and prepares setup code for cross-process initialisation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 04:14:00 +09:00
6d3bd27220 cmd/sharefs: expand fuse_main
This change should not change behaviour other than making output more consistent.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 02:30:28 +09:00
a27305cb4a cmd/sharefs: improve help message
This improves consistency with the fuse_main help message.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-27 02:20:41 +09:00
0e476c5e5b cmd/sharefs: allocate sharefs_private early
This also removes global state used by sharefs_init.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-26 08:08:41 +09:00
54712e0426 nix: set noatime on sharefs
Could improve performance, atime is not useful for this filesystem anyway.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-26 05:34:05 +09:00
b77c1ecfdb cmd/sharefs/test: check option handling
This verifies behaviour related to setuid/setgid when starting as root.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-26 05:28:45 +09:00
dce5839a79 nix: do not restart sharefs
This avoids disrupting running containers.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-26 04:12:14 +09:00
d597592e1f cmd/sharefs: rename fuse-helper to fuse-operations
This is not really just library wrapper functions, but instead implements the callbacks, so fuse-operations makes more sense.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-26 03:19:32 +09:00
056f5b12d4 cmd/sharefs: move translate_pathname body to macro wrapper
This is never called directly anywhere and it is simple enough to be included in the macro. This avoids passing the pointer around and dereferencing errno location, resulting in over 5% increase in throughput on the clang build. No change in the gcc build though.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-26 02:10:59 +09:00
da2bb546ba cmd/sharefs: remove readlink
This filesystem does not support symbolic links, so readlink is not useful, and unreachable in this case because of the check in getattr.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-25 06:00:58 +09:00
7bfbd59810 cmd/sharefs: implement shared filesystem
This is for passing files between applications, similar to android /sdcard.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-25 05:13:02 +09:00
ea815a59e8 nix: disable source fortification in devShell
This generates warnings when compiling without optimisation.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-21 02:22:28 +09:00
28a8dc67d2 internal/pipewire: raise Core::Sync timeout
Hopefully relieves spurious failures on a very overloaded system.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-19 00:49:33 +09:00
ec49c63c5f internal/pipewire: EPOLL_CTL_ADD instead of EPOLL_CTL_MOD
Implementation is no longer tied down by the limitations of SyscallConn.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-19 00:43:44 +09:00
5a50bf80ee internal/pipewire: hold socket fd directly
The interface provided by net is not used here and is a leftover from a previous implementation. This change removes it.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-19 00:28:24 +09:00
ce06b7b663 internal/pipewire: inform conn of blocking intent
The interface does not expose underlying kernel notification mechanisms. This change removes the need to poll in situations were the next call might block.

This is made cumbersome by the SyscallConn interface left over from a previous implementation, it will be replaced in a later commit as the current implementation does not make use of any net.Conn methods other than Close.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-19 00:00:33 +09:00
08bdc68f3a internal/pipewire: sendmsg/recvmsg errors are fatal
When returned wrapped as a syscall error, these are impossible to recover from, so wrap them as a fatal error.

Signed-off-by: Ophestra <cat@gensokyo.uk>
2025-12-18 23:33:12 +09:00
100 changed files with 13265 additions and 137 deletions

View File

@@ -72,6 +72,23 @@ jobs:
path: result/*
retention-days: 1
sharefs:
name: ShareFS
runs-on: nix
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Run NixOS test
run: nix build --out-link "result" --print-out-paths --print-build-logs .#checks.x86_64-linux.sharefs
- name: Upload test output
uses: actions/upload-artifact@v3
with:
name: "sharefs-vm-output"
path: result/*
retention-days: 1
hpkg:
name: Hpkg
runs-on: nix
@@ -96,6 +113,7 @@ jobs:
- race
- sandbox
- sandbox-race
- sharefs
- hpkg
runs-on: nix
steps:

1
.gitignore vendored
View File

@@ -27,6 +27,7 @@ go.work.sum
# go generate
/cmd/hakurei/LICENSE
/internal/pkg/testdata/testtool
# release
/dist/hakurei-*

76
cmd/irdump/main.go Normal file
View File

@@ -0,0 +1,76 @@
package main
import (
"errors"
"log"
"os"
"hakurei.app/command"
"hakurei.app/internal/pkg"
)
func main() {
log.SetFlags(0)
log.SetPrefix("irdump: ")
var (
flagOutput string
flagReal bool
flagHeader bool
flagForce bool
flagRaw bool
)
c := command.New(os.Stderr, log.Printf, "irdump", func(args []string) (err error) {
var input *os.File
if len(args) != 1 {
return errors.New("irdump requires 1 argument")
}
if input, err = os.Open(args[0]); err != nil {
return
}
defer input.Close()
var output *os.File
if flagOutput == "" {
output = os.Stdout
} else {
defer output.Close()
if output, err = os.Create(flagOutput); err != nil {
return
}
}
var out string
if out, err = pkg.Disassemble(input, flagReal, flagHeader, flagForce, flagRaw); err != nil {
return
}
if _, err = output.WriteString(out); err != nil {
return
}
return
}).Flag(
&flagOutput,
"o", command.StringFlag(""),
"Output file for asm (leave empty for stdout)",
).Flag(
&flagReal,
"r", command.BoolFlag(false),
"skip label generation; idents print real value",
).Flag(
&flagHeader,
"H", command.BoolFlag(false),
"display artifact headers",
).Flag(
&flagForce,
"f", command.BoolFlag(false),
"force display (skip validations)",
).Flag(
&flagRaw,
"R", command.BoolFlag(false),
"don't format output",
)
c.MustParse(os.Args[1:], func(err error) {
log.Fatal(err)
})
}

204
cmd/mbf/main.go Normal file
View File

@@ -0,0 +1,204 @@
package main
import (
"context"
"errors"
"fmt"
"log"
"os"
"os/signal"
"path/filepath"
"runtime"
"syscall"
"unique"
"hakurei.app/command"
"hakurei.app/container"
"hakurei.app/container/check"
"hakurei.app/internal/pkg"
"hakurei.app/internal/rosa"
"hakurei.app/message"
)
func main() {
container.TryArgv0(nil)
log.SetFlags(0)
log.SetPrefix("mbf: ")
msg := message.New(log.Default())
if os.Geteuid() == 0 {
log.Fatal("this program must not run as root")
}
var cache *pkg.Cache
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT, syscall.SIGTERM, syscall.SIGHUP)
defer stop()
defer func() {
if cache != nil {
cache.Close()
}
if r := recover(); r != nil {
fmt.Println(r)
log.Fatal("consider scrubbing the on-disk cache")
}
}()
var (
flagQuiet bool
flagCures int
flagBase string
flagTShift int
)
c := command.New(os.Stderr, log.Printf, "mbf", func([]string) (err error) {
msg.SwapVerbose(!flagQuiet)
var base *check.Absolute
if flagBase, err = filepath.Abs(flagBase); err != nil {
return
} else if base, err = check.NewAbs(flagBase); err != nil {
return
}
if cache, err = pkg.Open(ctx, msg, flagCures, base); err == nil {
if flagTShift < 0 {
cache.SetThreshold(0)
} else if flagTShift > 31 {
cache.SetThreshold(1 << 31)
} else {
cache.SetThreshold(1 << flagTShift)
}
}
return
}).Flag(
&flagQuiet,
"q", command.BoolFlag(false),
"Do not print cure messages",
).Flag(
&flagCures,
"cures", command.IntFlag(0),
"Maximum number of dependencies to cure at any given time",
).Flag(
&flagBase,
"d", command.StringFlag("cache"),
"Directory to store cured artifacts",
).Flag(
&flagTShift,
"tshift", command.IntFlag(-1),
"Dependency graph size exponent, to the power of 2",
)
{
var flagShifts int
c.NewCommand(
"scrub", "Examine the on-disk cache for errors",
func(args []string) error {
if len(args) > 0 {
return errors.New("scrub expects no arguments")
}
if flagShifts < 0 || flagShifts > 31 {
flagShifts = 12
}
return cache.Scrub(runtime.NumCPU() << flagShifts)
},
).Flag(
&flagShifts,
"shift", command.IntFlag(12),
"Scrub parallelism size exponent, to the power of 2",
)
}
c.NewCommand(
"stage3",
"Check for toolchain 3-stage non-determinism",
func(args []string) (err error) {
_, _, _, stage1 := (rosa.Std - 2).NewLLVM()
_, _, _, stage2 := (rosa.Std - 1).NewLLVM()
_, _, _, stage3 := rosa.Std.NewLLVM()
var (
pathname *check.Absolute
checksum [2]unique.Handle[pkg.Checksum]
)
if pathname, _, err = cache.Cure(stage1); err != nil {
return err
}
log.Println("stage1:", pathname)
if pathname, checksum[0], err = cache.Cure(stage2); err != nil {
return err
}
log.Println("stage2:", pathname)
if pathname, checksum[1], err = cache.Cure(stage3); err != nil {
return err
}
log.Println("stage3:", pathname)
if checksum[0] != checksum[1] {
err = &pkg.ChecksumMismatchError{
Got: checksum[0].Value(),
Want: checksum[1].Value(),
}
} else {
log.Println(
"stage2 is identical to stage3",
"("+pkg.Encode(checksum[0].Value())+")",
)
}
return
},
)
{
var (
flagDump string
)
c.NewCommand(
"cure",
"Cure the named artifact and show its path",
func(args []string) error {
if len(args) != 1 {
return errors.New("cure requires 1 argument")
}
if p, ok := rosa.ResolveName(args[0]); !ok {
return fmt.Errorf("unsupported artifact %q", args[0])
} else if flagDump == "" {
pathname, _, err := cache.Cure(rosa.Std.Load(p))
if err == nil {
log.Println(pathname)
}
return err
} else {
f, err := os.OpenFile(
flagDump,
os.O_WRONLY|os.O_CREATE|os.O_EXCL,
0644,
)
if err != nil {
return err
}
if err = cache.EncodeAll(f, rosa.Std.Load(p)); err != nil {
_ = f.Close()
return err
}
return f.Close()
}
},
).
Flag(
&flagDump,
"dump", command.StringFlag(""),
"Write IR to specified pathname and terminate",
)
}
c.MustParse(os.Args[1:], func(err error) {
if cache != nil {
cache.Close()
}
log.Fatal(err)
})
}

View File

@@ -0,0 +1,282 @@
#ifndef _GNU_SOURCE
#define _GNU_SOURCE /* O_DIRECT */
#endif
#include <dirent.h>
#include <errno.h>
#include <unistd.h>
/* TODO(ophestra): remove after 05ce67fea99ca09cd4b6625cff7aec9cc222dd5a reaches a release */
#include <sys/syscall.h>
#include "fuse-operations.h"
/* MUST_TRANSLATE_PATHNAME translates a userspace pathname to a relative pathname;
* the resulting address points to a constant string or part of pathname, it is never heap allocated. */
#define MUST_TRANSLATE_PATHNAME(pathname) \
do { \
if (pathname == NULL) \
return -EINVAL; \
while (*pathname == '/') \
pathname++; \
if (*pathname == '\0') \
pathname = "."; \
} while (0)
/* GET_CONTEXT_PRIV obtains fuse context and private data for the calling thread. */
#define GET_CONTEXT_PRIV(ctx, priv) \
do { \
ctx = fuse_get_context(); \
priv = ctx->private_data; \
} while (0)
/* impl_getattr modifies a struct stat from the kernel to present to userspace;
* impl_getattr returns a negative errno style error code. */
static int impl_getattr(struct fuse_context *ctx, struct stat *statbuf) {
/* allowlist of permitted types */
if (!S_ISDIR(statbuf->st_mode) && !S_ISREG(statbuf->st_mode) && !S_ISLNK(statbuf->st_mode)) {
return -ENOTRECOVERABLE; /* returning an errno causes all operations on the file to return EIO */
}
#define OVERRIDE_PERM(v) (statbuf->st_mode & ~0777) | (v & 0777)
if (S_ISDIR(statbuf->st_mode))
statbuf->st_mode = OVERRIDE_PERM(SHAREFS_PERM_DIR);
else if (S_ISREG(statbuf->st_mode))
statbuf->st_mode = OVERRIDE_PERM(SHAREFS_PERM_REG);
else
statbuf->st_mode = 0; /* should always be symlink in this case */
statbuf->st_uid = ctx->uid;
statbuf->st_gid = SHAREFS_MEDIA_RW_ID;
statbuf->st_ctim = statbuf->st_mtim;
statbuf->st_nlink = 1;
return 0;
}
/* fuse_operations implementation */
int sharefs_getattr(const char *pathname, struct stat *statbuf, struct fuse_file_info *fi) {
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
(void)fi;
if (fstatat(priv->dirfd, pathname, statbuf, AT_SYMLINK_NOFOLLOW) == -1)
return -errno;
return impl_getattr(ctx, statbuf);
}
int sharefs_readdir(const char *pathname, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi, enum fuse_readdir_flags flags) {
int fd;
DIR *dp;
struct stat st;
int ret = 0;
struct dirent *de;
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
(void)offset;
(void)fi;
if ((fd = openat(priv->dirfd, pathname, O_RDONLY | O_DIRECTORY | O_CLOEXEC)) == -1)
return -errno;
if ((dp = fdopendir(fd)) == NULL) {
close(fd);
return -errno;
}
errno = 0; /* for the next readdir call */
while ((de = readdir(dp)) != NULL) {
if (flags & FUSE_READDIR_PLUS) {
if (fstatat(dirfd(dp), de->d_name, &st, AT_SYMLINK_NOFOLLOW) == -1) {
ret = -errno;
break;
}
if ((ret = impl_getattr(ctx, &st)) < 0)
break;
errno = 0;
ret = filler(buf, de->d_name, &st, 0, FUSE_FILL_DIR_PLUS);
} else
ret = filler(buf, de->d_name, NULL, 0, 0);
if (ret != 0) {
ret = errno != 0 ? -errno : -EIO; /* filler */
break;
}
errno = 0; /* for the next readdir call */
}
if (ret == 0 && errno != 0)
ret = -errno; /* readdir */
closedir(dp);
return ret;
}
int sharefs_mkdir(const char *pathname, mode_t mode) {
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
(void)mode;
if (mkdirat(priv->dirfd, pathname, SHAREFS_PERM_DIR) == -1)
return -errno;
return 0;
}
int sharefs_unlink(const char *pathname) {
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
if (unlinkat(priv->dirfd, pathname, 0) == -1)
return -errno;
return 0;
}
int sharefs_rmdir(const char *pathname) {
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
if (unlinkat(priv->dirfd, pathname, AT_REMOVEDIR) == -1)
return -errno;
return 0;
}
int sharefs_rename(const char *oldpath, const char *newpath, unsigned int flags) {
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(oldpath);
MUST_TRANSLATE_PATHNAME(newpath);
/* TODO(ophestra): replace with wrapper after 05ce67fea99ca09cd4b6625cff7aec9cc222dd5a reaches a release */
if (syscall(__NR_renameat2, priv->dirfd, oldpath, priv->dirfd, newpath, flags) == -1)
return -errno;
return 0;
}
int sharefs_truncate(const char *pathname, off_t length, struct fuse_file_info *fi) {
int fd;
int ret;
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
(void)fi;
if ((fd = openat(priv->dirfd, pathname, O_WRONLY | O_CLOEXEC)) == -1)
return -errno;
if ((ret = ftruncate(fd, length)) == -1)
ret = -errno;
close(fd);
return ret;
}
int sharefs_utimens(const char *pathname, const struct timespec times[2], struct fuse_file_info *fi) {
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
(void)fi;
if (utimensat(priv->dirfd, pathname, times, AT_SYMLINK_NOFOLLOW) == -1)
return -errno;
return 0;
}
int sharefs_create(const char *pathname, mode_t mode, struct fuse_file_info *fi) {
int fd;
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
(void)mode;
if ((fd = openat(priv->dirfd, pathname, fi->flags & ~SHAREFS_FORBIDDEN_FLAGS, SHAREFS_PERM_REG)) == -1)
return -errno;
fi->fh = fd;
return 0;
}
int sharefs_open(const char *pathname, struct fuse_file_info *fi) {
int fd;
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
if ((fd = openat(priv->dirfd, pathname, fi->flags & ~SHAREFS_FORBIDDEN_FLAGS)) == -1)
return -errno;
fi->fh = fd;
return 0;
}
int sharefs_read(const char *pathname, char *buf, size_t count, off_t offset, struct fuse_file_info *fi) {
int ret;
(void)pathname;
if ((ret = pread(fi->fh, buf, count, offset)) == -1)
return -errno;
return ret;
}
int sharefs_write(const char *pathname, const char *buf, size_t count, off_t offset, struct fuse_file_info *fi) {
int ret;
(void)pathname;
if ((ret = pwrite(fi->fh, buf, count, offset)) == -1)
return -errno;
return ret;
}
int sharefs_statfs(const char *pathname, struct statvfs *statbuf) {
int fd;
int ret;
struct fuse_context *ctx;
struct sharefs_private *priv;
GET_CONTEXT_PRIV(ctx, priv);
MUST_TRANSLATE_PATHNAME(pathname);
if ((fd = openat(priv->dirfd, pathname, O_RDONLY | O_CLOEXEC)) == -1)
return -errno;
if ((ret = fstatvfs(fd, statbuf)) == -1)
ret = -errno;
close(fd);
return ret;
}
int sharefs_release(const char *pathname, struct fuse_file_info *fi) {
(void)pathname;
return close(fi->fh);
}
int sharefs_fsync(const char *pathname, int datasync, struct fuse_file_info *fi) {
(void)pathname;
if (datasync ? fdatasync(fi->fh) : fsync(fi->fh) == -1)
return -errno;
return 0;
}

View File

@@ -0,0 +1,34 @@
#define FUSE_USE_VERSION FUSE_MAKE_VERSION(3, 12)
#include <fuse.h>
#include <fuse_lowlevel.h> /* for fuse_cmdline_help */
#if (FUSE_VERSION < FUSE_MAKE_VERSION(3, 12))
#error This package requires libfuse >= v3.12
#endif
#define SHAREFS_MEDIA_RW_ID (1 << 10) - 1 /* owning gid presented to userspace */
#define SHAREFS_PERM_DIR 0700 /* permission bits for directories presented to userspace */
#define SHAREFS_PERM_REG 0600 /* permission bits for regular files presented to userspace */
#define SHAREFS_FORBIDDEN_FLAGS O_DIRECT /* these open flags are cleared unconditionally */
/* sharefs_private is populated by sharefs_init and contains process-wide context */
struct sharefs_private {
int dirfd; /* source dirfd opened during sharefs_init */
uintptr_t setup; /* cgo handle of opaque setup state */
};
int sharefs_getattr(const char *pathname, struct stat *statbuf, struct fuse_file_info *fi);
int sharefs_readdir(const char *pathname, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi, enum fuse_readdir_flags flags);
int sharefs_mkdir(const char *pathname, mode_t mode);
int sharefs_unlink(const char *pathname);
int sharefs_rmdir(const char *pathname);
int sharefs_rename(const char *oldpath, const char *newpath, unsigned int flags);
int sharefs_truncate(const char *pathname, off_t length, struct fuse_file_info *fi);
int sharefs_utimens(const char *pathname, const struct timespec times[2], struct fuse_file_info *fi);
int sharefs_create(const char *pathname, mode_t mode, struct fuse_file_info *fi);
int sharefs_open(const char *pathname, struct fuse_file_info *fi);
int sharefs_read(const char *pathname, char *buf, size_t count, off_t offset, struct fuse_file_info *fi);
int sharefs_write(const char *pathname, const char *buf, size_t count, off_t offset, struct fuse_file_info *fi);
int sharefs_statfs(const char *pathname, struct statvfs *statbuf);
int sharefs_release(const char *pathname, struct fuse_file_info *fi);
int sharefs_fsync(const char *pathname, int datasync, struct fuse_file_info *fi);

556
cmd/sharefs/fuse.go Normal file
View File

@@ -0,0 +1,556 @@
package main
/*
#cgo pkg-config: --static fuse3
#include "fuse-operations.h"
#include <stdlib.h>
#include <string.h>
extern void *sharefs_init(struct fuse_conn_info *conn, struct fuse_config *cfg);
extern void sharefs_destroy(void *private_data);
typedef void (*closure)();
static inline struct fuse_opt _FUSE_OPT_END() { return (struct fuse_opt)FUSE_OPT_END; };
*/
import "C"
import (
"context"
"encoding/gob"
"errors"
"fmt"
"io"
"log"
"os"
"os/exec"
"os/signal"
"path"
"runtime"
"runtime/cgo"
"strconv"
"syscall"
"unsafe"
"hakurei.app/container"
"hakurei.app/container/check"
"hakurei.app/container/std"
"hakurei.app/hst"
"hakurei.app/internal/helper/proc"
"hakurei.app/internal/info"
"hakurei.app/message"
)
type (
// closure represents a C function pointer.
closure = C.closure
// fuseArgs represents the fuse_args structure.
fuseArgs = C.struct_fuse_args
// setupState holds state used for setup. Its cgo handle is included in
// sharefs_private and considered opaque to non-setup callbacks.
setupState struct {
// Whether sharefs_init failed.
initFailed bool
// Whether to create source directory as root.
mkdir bool
// Open file descriptor to fuse.
Fuse int
// Pathname to open for dirfd.
Source *check.Absolute
// New uid and gid to set by sharefs_init when starting as root.
Setuid, Setgid int
}
)
func init() { gob.Register(new(setupState)) }
// destroySetup invalidates the setup [cgo.Handle] in a sharefs_private structure.
func destroySetup(private_data unsafe.Pointer) (ok bool) {
if private_data == nil {
return false
}
priv := (*C.struct_sharefs_private)(private_data)
if h := cgo.Handle(priv.setup); h != 0 {
priv.setup = 0
h.Delete()
ok = true
}
return
}
//export sharefs_init
func sharefs_init(_ *C.struct_fuse_conn_info, cfg *C.struct_fuse_config) unsafe.Pointer {
ctx := C.fuse_get_context()
priv := (*C.struct_sharefs_private)(ctx.private_data)
setup := cgo.Handle(priv.setup).Value().(*setupState)
if os.Geteuid() == 0 {
log.Println("filesystem daemon must not run as root")
goto fail
}
cfg.use_ino = C.true
cfg.direct_io = C.false
// getattr is context-dependent
cfg.attr_timeout = 0
cfg.entry_timeout = 0
cfg.negative_timeout = 0
// all future filesystem operations happen through this dirfd
if fd, err := syscall.Open(setup.Source.String(), syscall.O_DIRECTORY|syscall.O_RDONLY|syscall.O_CLOEXEC, 0); err != nil {
log.Printf("cannot open %q: %v", setup.Source, err)
goto fail
} else if err = syscall.Fchdir(fd); err != nil {
_ = syscall.Close(fd)
log.Printf("cannot enter %q: %s", setup.Source, err)
goto fail
} else {
priv.dirfd = C.int(fd)
}
return ctx.private_data
fail:
setup.initFailed = true
C.fuse_exit(ctx.fuse)
return nil
}
//export sharefs_destroy
func sharefs_destroy(private_data unsafe.Pointer) {
if private_data != nil {
destroySetup(private_data)
priv := (*C.struct_sharefs_private)(private_data)
if err := syscall.Close(int(priv.dirfd)); err != nil {
log.Printf("cannot close source directory: %v", err)
}
}
}
// showHelp prints the help message.
func showHelp(args *fuseArgs) {
executableName := sharefsName
if args.argc > 0 {
executableName = path.Base(C.GoString(*args.argv))
} else if name, err := os.Executable(); err == nil {
executableName = path.Base(name)
}
fmt.Printf("usage: %s [options] <mountpoint>\n\n", executableName)
fmt.Println("Filesystem options:")
fmt.Println(" -o source=/data/media source directory to be mounted")
fmt.Println(" -o setuid=1023 uid to run as when starting as root")
fmt.Println(" -o setgid=1023 gid to run as when starting as root")
fmt.Println("\nFUSE options:")
C.fuse_cmdline_help()
C.fuse_lib_help(args)
}
// parseOpts parses fuse options via fuse_opt_parse.
func parseOpts(args *fuseArgs, setup *setupState, log *log.Logger) (ok bool) {
var unsafeOpts struct {
// Pathname to writable source directory.
source *C.char
// Whether to create source directory as root.
mkdir C.int
// Decimal string representation of uid to set when running as root.
setuid *C.char
// Decimal string representation of gid to set when running as root.
setgid *C.char
// Decimal string representation of open file descriptor to read setupState from.
// This is an internal detail for containerisation and must not be specified directly.
setup *C.char
}
if C.fuse_opt_parse(args, unsafe.Pointer(&unsafeOpts), &[]C.struct_fuse_opt{
{templ: C.CString("source=%s"), offset: C.ulong(unsafe.Offsetof(unsafeOpts.source)), value: 0},
{templ: C.CString("mkdir"), offset: C.ulong(unsafe.Offsetof(unsafeOpts.mkdir)), value: 1},
{templ: C.CString("setuid=%s"), offset: C.ulong(unsafe.Offsetof(unsafeOpts.setuid)), value: 0},
{templ: C.CString("setgid=%s"), offset: C.ulong(unsafe.Offsetof(unsafeOpts.setgid)), value: 0},
{templ: C.CString("setup=%s"), offset: C.ulong(unsafe.Offsetof(unsafeOpts.setup)), value: 0},
C._FUSE_OPT_END(),
}[0], nil) == -1 {
return false
}
if unsafeOpts.source != nil {
defer C.free(unsafe.Pointer(unsafeOpts.source))
}
if unsafeOpts.setuid != nil {
defer C.free(unsafe.Pointer(unsafeOpts.setuid))
}
if unsafeOpts.setgid != nil {
defer C.free(unsafe.Pointer(unsafeOpts.setgid))
}
if unsafeOpts.setup != nil {
defer C.free(unsafe.Pointer(unsafeOpts.setup))
if v, err := strconv.Atoi(C.GoString(unsafeOpts.setup)); err != nil || v < 3 {
log.Println("invalid value for option setup")
return false
} else {
r := os.NewFile(uintptr(v), "setup")
defer func() {
if err = r.Close(); err != nil {
log.Println(err)
}
}()
if err = gob.NewDecoder(r).Decode(setup); err != nil {
log.Println(err)
return false
}
}
if setup.Fuse < 3 {
log.Println("invalid file descriptor", setup.Fuse)
return false
}
return true
}
if unsafeOpts.source == nil {
showHelp(args)
return false
} else if a, err := check.NewAbs(C.GoString(unsafeOpts.source)); err != nil {
log.Println(err)
return false
} else {
setup.Source = a
}
setup.mkdir = unsafeOpts.mkdir != 0
if unsafeOpts.setuid == nil {
setup.Setuid = -1
} else if v, err := strconv.Atoi(C.GoString(unsafeOpts.setuid)); err != nil || v <= 0 {
log.Println("invalid value for option setuid")
return false
} else {
setup.Setuid = v
}
if unsafeOpts.setgid == nil {
setup.Setgid = -1
} else if v, err := strconv.Atoi(C.GoString(unsafeOpts.setgid)); err != nil || v <= 0 {
log.Println("invalid value for option setgid")
return false
} else {
setup.Setgid = v
}
return true
}
// copyArgs returns a heap allocated copy of an argument slice in fuse_args representation.
func copyArgs(s ...string) fuseArgs {
if len(s) == 0 {
return fuseArgs{argc: 0, argv: nil, allocated: 0}
}
args := unsafe.Slice((**C.char)(C.malloc(C.size_t(uintptr(len(s))*unsafe.Sizeof(s[0])))), len(s))
for i, arg := range s {
args[i] = C.CString(arg)
}
return fuseArgs{argc: C.int(len(s)), argv: &args[0], allocated: 1}
}
// freeArgs frees the contents of argument list.
func freeArgs(args *fuseArgs) { C.fuse_opt_free_args(args) }
// unsafeAddArgument adds an argument to fuseArgs via fuse_opt_add_arg.
// The last byte of arg must be 0.
func unsafeAddArgument(args *fuseArgs, arg string) {
C.fuse_opt_add_arg(args, (*C.char)(unsafe.Pointer(unsafe.StringData(arg))))
}
func _main(s ...string) (exitCode int) {
msg := message.New(log.Default())
container.TryArgv0(msg)
runtime.LockOSThread()
// don't mask creation mode, kernel already did that
syscall.Umask(0)
var pinner runtime.Pinner
defer pinner.Unpin()
args := copyArgs(s...)
defer freeArgs(&args)
// this causes the kernel to enforce access control based on
// struct stat populated by sharefs_getattr
unsafeAddArgument(&args, "-odefault_permissions\x00")
var priv C.struct_sharefs_private
pinner.Pin(&priv)
var setup setupState
priv.setup = C.uintptr_t(cgo.NewHandle(&setup))
defer destroySetup(unsafe.Pointer(&priv))
var opts C.struct_fuse_cmdline_opts
if C.fuse_parse_cmdline(&args, &opts) != 0 {
return 1
}
if opts.mountpoint != nil {
defer C.free(unsafe.Pointer(opts.mountpoint))
}
if opts.show_version != 0 {
fmt.Println("hakurei version", info.Version())
fmt.Println("FUSE library version", C.GoString(C.fuse_pkgversion()))
C.fuse_lowlevel_version()
return 0
}
if opts.show_help != 0 {
showHelp(&args)
return 0
} else if opts.mountpoint == nil {
log.Println("no mountpoint specified")
return 2
} else {
// hack to keep fuse_parse_cmdline happy in the container
mountpoint := C.GoString(opts.mountpoint)
pathnameArg := -1
for i, arg := range s {
if arg == mountpoint {
pathnameArg = i
break
}
}
if pathnameArg < 0 {
log.Println("mountpoint must be absolute")
return 2
}
s[pathnameArg] = container.Nonexistent
}
if !parseOpts(&args, &setup, msg.GetLogger()) {
return 1
}
asRoot := os.Geteuid() == 0
if asRoot {
if setup.Setuid <= 0 || setup.Setgid <= 0 {
log.Println("setuid and setgid must not be 0")
return 1
}
if setup.Fuse >= 3 {
log.Println("filesystem daemon must not run as root")
return 1
}
if setup.mkdir {
if err := os.MkdirAll(setup.Source.String(), 0700); err != nil {
if !errors.Is(err, os.ErrExist) {
log.Println(err)
return 1
}
// skip setup for existing source directory
} else if err = os.Chown(setup.Source.String(), setup.Setuid, setup.Setgid); err != nil {
log.Println(err)
return 1
}
}
} else if setup.Fuse < 3 && (setup.Setuid > 0 || setup.Setgid > 0) {
log.Println("setuid and setgid has no effect when not starting as root")
return 1
} else if setup.mkdir {
log.Println("mkdir has no effect when not starting as root")
return 1
}
op := C.struct_fuse_operations{
init: closure(C.sharefs_init),
destroy: closure(C.sharefs_destroy),
// implemented in fuse-helper.c
getattr: closure(C.sharefs_getattr),
readdir: closure(C.sharefs_readdir),
mkdir: closure(C.sharefs_mkdir),
unlink: closure(C.sharefs_unlink),
rmdir: closure(C.sharefs_rmdir),
rename: closure(C.sharefs_rename),
truncate: closure(C.sharefs_truncate),
utimens: closure(C.sharefs_utimens),
create: closure(C.sharefs_create),
open: closure(C.sharefs_open),
read: closure(C.sharefs_read),
write: closure(C.sharefs_write),
statfs: closure(C.sharefs_statfs),
release: closure(C.sharefs_release),
fsync: closure(C.sharefs_fsync),
}
fuse := C.fuse_new_fn(&args, &op, C.size_t(unsafe.Sizeof(op)), unsafe.Pointer(&priv))
if fuse == nil {
return 3
}
defer C.fuse_destroy(fuse)
se := C.fuse_get_session(fuse)
if setup.Fuse < 3 {
// unconfined, set up mount point and container
if C.fuse_mount(fuse, opts.mountpoint) != 0 {
return 4
}
// unmounted by initial process
defer func() {
if exitCode == 5 {
C.fuse_unmount(fuse)
}
}()
if asRoot {
if err := syscall.Setresgid(setup.Setgid, setup.Setgid, setup.Setgid); err != nil {
log.Printf("cannot set gid: %v", err)
return 5
}
if err := syscall.Setgroups(nil); err != nil {
log.Printf("cannot set supplementary groups: %v", err)
return 5
}
if err := syscall.Setresuid(setup.Setuid, setup.Setuid, setup.Setuid); err != nil {
log.Printf("cannot set uid: %v", err)
return 5
}
}
msg.SwapVerbose(opts.debug != 0)
ctx := context.Background()
if opts.foreground != 0 {
c, cancel := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
defer cancel()
ctx = c
}
z := container.New(ctx, msg)
z.AllowOrphan = opts.foreground == 0
z.Env = os.Environ()
// keep fuse_parse_cmdline happy in the container
z.Tmpfs(check.MustAbs(container.Nonexistent), 1<<10, 0755)
if a, err := check.NewAbs(container.MustExecutable(msg)); err != nil {
log.Println(err)
return 5
} else {
z.Path = a
}
z.Args = s
z.ForwardCancel = true
z.SeccompPresets |= std.PresetStrict
z.ParentPerm = 0700
z.Bind(setup.Source, setup.Source, std.BindWritable)
if !z.AllowOrphan {
z.WaitDelay = hst.WaitDelayMax
z.Stdin, z.Stdout, z.Stderr = os.Stdin, os.Stdout, os.Stderr
}
z.Bind(z.Path, z.Path, 0)
setup.Fuse = int(proc.ExtraFileSlice(&z.ExtraFiles, os.NewFile(uintptr(C.fuse_session_fd(se)), "fuse")))
var setupWriter io.WriteCloser
if fd, w, err := container.Setup(&z.ExtraFiles); err != nil {
log.Println(err)
return 5
} else {
z.Args = append(z.Args, "-osetup="+strconv.Itoa(fd))
setupWriter = w
}
if err := z.Start(); err != nil {
if m, ok := message.GetMessage(err); ok {
log.Println(m)
} else {
log.Println(err)
}
return 5
}
if err := z.Serve(); err != nil {
if m, ok := message.GetMessage(err); ok {
log.Println(m)
} else {
log.Println(err)
}
return 5
}
if err := gob.NewEncoder(setupWriter).Encode(&setup); err != nil {
log.Println(err)
return 5
} else if err = setupWriter.Close(); err != nil {
log.Println(err)
}
if !z.AllowOrphan {
if err := z.Wait(); err != nil {
var exitError *exec.ExitError
if !errors.As(err, &exitError) || exitError == nil {
log.Println(err)
return 5
}
switch code := exitError.ExitCode(); syscall.Signal(code & 0x7f) {
case syscall.SIGINT:
case syscall.SIGTERM:
default:
return code
}
}
}
return 0
} else { // confined
C.free(unsafe.Pointer(opts.mountpoint))
// must be heap allocated
opts.mountpoint = C.CString("/dev/fd/" + strconv.Itoa(setup.Fuse))
if err := os.Chdir("/"); err != nil {
log.Println(err)
}
}
if C.fuse_mount(fuse, opts.mountpoint) != 0 {
return 4
}
defer C.fuse_unmount(fuse)
if C.fuse_set_signal_handlers(se) != 0 {
return 6
}
defer C.fuse_remove_signal_handlers(se)
if opts.singlethread != 0 {
if C.fuse_loop(fuse) != 0 {
return 8
}
} else {
loopConfig := C.fuse_loop_cfg_create()
if loopConfig == nil {
return 7
}
defer C.fuse_loop_cfg_destroy(loopConfig)
C.fuse_loop_cfg_set_clone_fd(loopConfig, C.uint(opts.clone_fd))
C.fuse_loop_cfg_set_idle_threads(loopConfig, opts.max_idle_threads)
C.fuse_loop_cfg_set_max_threads(loopConfig, opts.max_threads)
if C.fuse_loop_mt(fuse, loopConfig) != 0 {
return 8
}
}
if setup.initFailed {
return 1
}
return 0
}

113
cmd/sharefs/fuse_test.go Normal file
View File

@@ -0,0 +1,113 @@
package main
import (
"bytes"
"log"
"reflect"
"testing"
"hakurei.app/container/check"
)
func TestParseOpts(t *testing.T) {
t.Parallel()
testCases := []struct {
name string
args []string
want setupState
wantLog string
wantOk bool
}{
{"zero length", []string{}, setupState{}, "", false},
{"not absolute", []string{"sharefs",
"-o", "source=nonexistent",
"-o", "setuid=1023",
"-o", "setgid=1023",
}, setupState{}, "sharefs: path \"nonexistent\" is not absolute\n", false},
{"not specified", []string{"sharefs",
"-o", "setuid=1023",
"-o", "setgid=1023",
}, setupState{}, "", false},
{"invalid setuid", []string{"sharefs",
"-o", "source=/proc/nonexistent",
"-o", "setuid=ff",
"-o", "setgid=1023",
}, setupState{
Source: check.MustAbs("/proc/nonexistent"),
}, "sharefs: invalid value for option setuid\n", false},
{"invalid setgid", []string{"sharefs",
"-o", "source=/proc/nonexistent",
"-o", "setuid=1023",
"-o", "setgid=ff",
}, setupState{
Source: check.MustAbs("/proc/nonexistent"),
Setuid: 1023,
}, "sharefs: invalid value for option setgid\n", false},
{"simple", []string{"sharefs",
"-o", "source=/proc/nonexistent",
}, setupState{
Source: check.MustAbs("/proc/nonexistent"),
Setuid: -1,
Setgid: -1,
}, "", true},
{"root", []string{"sharefs",
"-o", "source=/proc/nonexistent",
"-o", "setuid=1023",
"-o", "setgid=1023",
}, setupState{
Source: check.MustAbs("/proc/nonexistent"),
Setuid: 1023,
Setgid: 1023,
}, "", true},
{"setuid", []string{"sharefs",
"-o", "source=/proc/nonexistent",
"-o", "setuid=1023",
}, setupState{
Source: check.MustAbs("/proc/nonexistent"),
Setuid: 1023,
Setgid: -1,
}, "", true},
{"setgid", []string{"sharefs",
"-o", "source=/proc/nonexistent",
"-o", "setgid=1023",
}, setupState{
Source: check.MustAbs("/proc/nonexistent"),
Setuid: -1,
Setgid: 1023,
}, "", true},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var (
got setupState
buf bytes.Buffer
)
args := copyArgs(tc.args...)
defer freeArgs(&args)
unsafeAddArgument(&args, "-odefault_permissions\x00")
if ok := parseOpts(&args, &got, log.New(&buf, "sharefs: ", 0)); ok != tc.wantOk {
t.Errorf("parseOpts: ok = %v, want %v", ok, tc.wantOk)
}
if !reflect.DeepEqual(&got, &tc.want) {
t.Errorf("parseOpts: setup = %#v, want %#v", got, tc.want)
}
if buf.String() != tc.wantLog {
t.Errorf("parseOpts: log =\n%s\nwant\n%s", buf.String(), tc.wantLog)
}
})
}
}

31
cmd/sharefs/main.go Normal file
View File

@@ -0,0 +1,31 @@
package main
import (
"log"
"os"
"slices"
)
// sharefsName is the prefix used by log.std in the sharefs process.
const sharefsName = "sharefs"
// handleMountArgs returns an alternative, libfuse-compatible args slice for
// args passed by mount -t fuse.sharefs [options] sharefs <mountpoint>.
//
// In this case, args always has a length of 5 with index 0 being what comes
// after "fuse." in the filesystem type, 1 is the uninterpreted string passed
// to mount (sharefsName is used as the magic string to enable this hack),
// 2 is passed through to libfuse as mountpoint, and 3 is always "-o".
func handleMountArgs(args []string) []string {
if len(args) == 5 && args[1] == sharefsName && args[3] == "-o" {
return []string{sharefsName, args[2], "-o", args[4]}
}
return slices.Clone(args)
}
func main() {
log.SetFlags(0)
log.SetPrefix(sharefsName + ": ")
os.Exit(_main(handleMountArgs(os.Args)...))
}

29
cmd/sharefs/main_test.go Normal file
View File

@@ -0,0 +1,29 @@
package main
import (
"slices"
"testing"
)
func TestHandleMountArgs(t *testing.T) {
t.Parallel()
testCases := []struct {
name string
args []string
want []string
}{
{"nil", nil, nil},
{"passthrough", []string{"sharefs", "-V"}, []string{"sharefs", "-V"}},
{"replace", []string{"/sbin/sharefs", "sharefs", "/sdcard", "-o", "rw"}, []string{"sharefs", "/sdcard", "-o", "rw"}},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := handleMountArgs(tc.args); !slices.Equal(got, tc.want) {
t.Errorf("handleMountArgs: %q, want %q", got, tc.want)
}
})
}
}

View File

@@ -0,0 +1,41 @@
{ pkgs, ... }:
{
users.users = {
alice = {
isNormalUser = true;
description = "Alice Foobar";
password = "foobar";
uid = 1000;
};
};
home-manager.users.alice.home.stateVersion = "24.11";
# Automatically login on tty1 as a normal user:
services.getty.autologinUser = "alice";
environment = {
# For benchmarking sharefs:
systemPackages = [ pkgs.fsmark ];
};
virtualisation = {
diskSize = 6 * 1024;
qemu.options = [
# Increase test performance:
"-smp 8"
];
};
environment.hakurei = rec {
enable = true;
stateDir = "/var/lib/hakurei";
sharefs.source = "${stateDir}/sdcard";
users.alice = 0;
extraHomeConfig = {
home.stateVersion = "23.05";
};
};
}

View File

@@ -0,0 +1,44 @@
{
testers,
system,
self,
}:
testers.nixosTest {
name = "sharefs";
nodes.machine =
{ options, pkgs, ... }:
let
fhs =
let
hakurei = options.environment.hakurei.package.default;
in
pkgs.buildFHSEnv {
pname = "hakurei-fhs";
inherit (hakurei) version;
targetPkgs = _: hakurei.targetPkgs;
extraOutputsToInstall = [ "dev" ];
profile = ''
export PKG_CONFIG_PATH="/usr/share/pkgconfig:$PKG_CONFIG_PATH"
'';
};
in
{
environment.systemPackages = [
# For go tests:
(pkgs.writeShellScriptBin "sharefs-workload-hakurei-tests" ''
cp -r "${self.packages.${system}.hakurei.src}" "/sdcard/hakurei" && cd "/sdcard/hakurei"
${fhs}/bin/hakurei-fhs -c 'CC="clang -O3 -Werror" go test ./...'
'')
];
imports = [
./configuration.nix
self.nixosModules.hakurei
self.inputs.home-manager.nixosModules.home-manager
];
};
testScript = builtins.readFile ./test.py;
}

60
cmd/sharefs/test/test.py Normal file
View File

@@ -0,0 +1,60 @@
start_all()
machine.wait_for_unit("multi-user.target")
# To check sharefs version:
print(machine.succeed("sharefs -V"))
# Make sure sharefs started:
machine.wait_for_unit("sdcard.mount")
machine.succeed("mkdir /mnt")
def check_bad_opts_output(opts, want, source="/etc", privileged=False):
output = machine.fail(("" if privileged else "sudo -u alice -i ") + f"sharefs -f -o source={source},{opts} /mnt 2>&1")
if output != want:
raise Exception(f"unexpected output: {output}")
# Malformed setuid/setgid representation:
check_bad_opts_output("setuid=ff", "sharefs: invalid value for option setuid\n")
check_bad_opts_output("setgid=ff", "sharefs: invalid value for option setgid\n")
# Bounds check for setuid/setgid:
check_bad_opts_output("setuid=0", "sharefs: invalid value for option setuid\n")
check_bad_opts_output("setgid=0", "sharefs: invalid value for option setgid\n")
check_bad_opts_output("setuid=-1", "sharefs: invalid value for option setuid\n")
check_bad_opts_output("setgid=-1", "sharefs: invalid value for option setgid\n")
# Non-root setuid/setgid:
check_bad_opts_output("setuid=1023", "sharefs: setuid and setgid has no effect when not starting as root\n")
check_bad_opts_output("setgid=1023", "sharefs: setuid and setgid has no effect when not starting as root\n")
check_bad_opts_output("setuid=1023,setgid=1023", "sharefs: setuid and setgid has no effect when not starting as root\n")
check_bad_opts_output("mkdir", "sharefs: mkdir has no effect when not starting as root\n")
# Starting as root without setuid/setgid:
check_bad_opts_output("allow_other", "sharefs: setuid and setgid must not be 0\n", privileged=True)
check_bad_opts_output("setuid=1023", "sharefs: setuid and setgid must not be 0\n", privileged=True)
check_bad_opts_output("setgid=1023", "sharefs: setuid and setgid must not be 0\n", privileged=True)
# Make sure nothing actually got mounted:
machine.fail("umount /mnt")
machine.succeed("rmdir /mnt")
# Unprivileged mount/unmount:
machine.succeed("sudo -u alice -i mkdir /home/alice/{sdcard,persistent}")
machine.succeed("sudo -u alice -i sharefs -o source=/home/alice/persistent /home/alice/sdcard")
machine.succeed("sudo -u alice -i touch /home/alice/sdcard/check")
machine.succeed("sudo -u alice -i umount /home/alice/sdcard")
machine.succeed("sudo -u alice -i rm /home/alice/persistent/check")
machine.succeed("sudo -u alice -i rmdir /home/alice/{sdcard,persistent}")
# Benchmark sharefs:
machine.succeed("fs_mark -v -d /sdcard/fs_mark -l /tmp/fs_log.txt")
machine.copy_from_vm("/tmp/fs_log.txt", "")
# Check permissions:
machine.succeed("sudo -u sharefs touch /var/lib/hakurei/sdcard/fs_mark/.check")
machine.succeed("sudo -u sharefs rm /var/lib/hakurei/sdcard/fs_mark/.check")
machine.succeed("sudo -u alice rm -rf /sdcard/fs_mark")
machine.fail("ls /var/lib/hakurei/sdcard/fs_mark")
# Run hakurei tests on sharefs:
machine.succeed("sudo -u alice -i sharefs-workload-hakurei-tests")

View File

@@ -14,6 +14,7 @@ const (
CAP_SYS_ADMIN = 0x15
CAP_SETPCAP = 0x8
CAP_NET_ADMIN = 0xc
CAP_DAC_OVERRIDE = 0x1
)

View File

@@ -9,46 +9,60 @@ import (
"slices"
"strings"
"syscall"
"unique"
)
// AbsoluteError is returned by [NewAbs] and holds the invalid pathname.
type AbsoluteError struct{ Pathname string }
type AbsoluteError string
func (e *AbsoluteError) Error() string { return fmt.Sprintf("path %q is not absolute", e.Pathname) }
func (e *AbsoluteError) Is(target error) bool {
var ce *AbsoluteError
func (e AbsoluteError) Error() string {
return fmt.Sprintf("path %q is not absolute", string(e))
}
func (e AbsoluteError) Is(target error) bool {
var ce AbsoluteError
if !errors.As(target, &ce) {
return errors.Is(target, syscall.EINVAL)
}
return *e == *ce
return e == ce
}
// Absolute holds a pathname checked to be absolute.
type Absolute struct{ pathname string }
type Absolute struct{ pathname unique.Handle[string] }
// ok returns whether [Absolute] is not the zero value.
func (a *Absolute) ok() bool { return a != nil && *a != (Absolute{}) }
// unsafeAbs returns [check.Absolute] on any string value.
func unsafeAbs(pathname string) *Absolute { return &Absolute{pathname} }
func unsafeAbs(pathname string) *Absolute {
return &Absolute{unique.Make(pathname)}
}
// String returns the checked pathname.
func (a *Absolute) String() string {
if a.pathname == "" {
if !a.ok() {
panic("attempted use of zero Absolute")
}
return a.pathname.Value()
}
// Handle returns the underlying [unique.Handle].
func (a *Absolute) Handle() unique.Handle[string] {
return a.pathname
}
// Is efficiently compares the underlying pathname.
func (a *Absolute) Is(v *Absolute) bool {
if a == nil && v == nil {
return true
}
return a != nil && v != nil &&
a.pathname != "" && v.pathname != "" &&
a.pathname == v.pathname
return a.ok() && v.ok() && a.pathname == v.pathname
}
// NewAbs checks pathname and returns a new [Absolute] if pathname is absolute.
func NewAbs(pathname string) (*Absolute, error) {
if !path.IsAbs(pathname) {
return nil, &AbsoluteError{pathname}
return nil, AbsoluteError(pathname)
}
return unsafeAbs(pathname), nil
}
@@ -70,35 +84,49 @@ func (a *Absolute) Append(elem ...string) *Absolute {
// Dir calls [path.Dir] with [Absolute] as its argument.
func (a *Absolute) Dir() *Absolute { return unsafeAbs(path.Dir(a.String())) }
func (a *Absolute) GobEncode() ([]byte, error) { return []byte(a.String()), nil }
// GobEncode returns the checked pathname.
func (a *Absolute) GobEncode() ([]byte, error) {
return []byte(a.String()), nil
}
// GobDecode stores data if it represents an absolute pathname.
func (a *Absolute) GobDecode(data []byte) error {
pathname := string(data)
if !path.IsAbs(pathname) {
return &AbsoluteError{pathname}
return AbsoluteError(pathname)
}
a.pathname = pathname
a.pathname = unique.Make(pathname)
return nil
}
func (a *Absolute) MarshalJSON() ([]byte, error) { return json.Marshal(a.String()) }
// MarshalJSON returns a JSON representation of the checked pathname.
func (a *Absolute) MarshalJSON() ([]byte, error) {
return json.Marshal(a.String())
}
// UnmarshalJSON stores data if it represents an absolute pathname.
func (a *Absolute) UnmarshalJSON(data []byte) error {
var pathname string
if err := json.Unmarshal(data, &pathname); err != nil {
return err
}
if !path.IsAbs(pathname) {
return &AbsoluteError{pathname}
return AbsoluteError(pathname)
}
a.pathname = pathname
a.pathname = unique.Make(pathname)
return nil
}
// SortAbs calls [slices.SortFunc] for a slice of [Absolute].
func SortAbs(x []*Absolute) {
slices.SortFunc(x, func(a, b *Absolute) int { return strings.Compare(a.String(), b.String()) })
slices.SortFunc(x, func(a, b *Absolute) int {
return strings.Compare(a.String(), b.String())
})
}
// CompactAbs calls [slices.CompactFunc] for a slice of [Absolute].
func CompactAbs(s []*Absolute) []*Absolute {
return slices.CompactFunc(s, func(a *Absolute, b *Absolute) bool { return a.String() == b.String() })
return slices.CompactFunc(s, func(a *Absolute, b *Absolute) bool {
return a.Is(b)
})
}

View File

@@ -31,8 +31,8 @@ func TestAbsoluteError(t *testing.T) {
}{
{"EINVAL", new(AbsoluteError), syscall.EINVAL, true},
{"not EINVAL", new(AbsoluteError), syscall.EBADE, false},
{"ne val", new(AbsoluteError), &AbsoluteError{Pathname: "etc"}, false},
{"equals", &AbsoluteError{Pathname: "etc"}, &AbsoluteError{Pathname: "etc"}, true},
{"ne val", new(AbsoluteError), AbsoluteError("etc"), false},
{"equals", AbsoluteError("etc"), AbsoluteError("etc"), true},
}
for _, tc := range testCases {
@@ -45,7 +45,7 @@ func TestAbsoluteError(t *testing.T) {
t.Parallel()
want := `path "etc" is not absolute`
if got := (&AbsoluteError{Pathname: "etc"}).Error(); got != want {
if got := (AbsoluteError("etc")).Error(); got != want {
t.Errorf("Error: %q, want %q", got, want)
}
})
@@ -62,8 +62,8 @@ func TestNewAbs(t *testing.T) {
wantErr error
}{
{"good", "/etc", MustAbs("/etc"), nil},
{"not absolute", "etc", nil, &AbsoluteError{Pathname: "etc"}},
{"zero", "", nil, &AbsoluteError{Pathname: ""}},
{"not absolute", "etc", nil, AbsoluteError("etc")},
{"zero", "", nil, AbsoluteError("")},
}
for _, tc := range testCases {
@@ -84,7 +84,7 @@ func TestNewAbs(t *testing.T) {
t.Parallel()
defer func() {
wantPanic := &AbsoluteError{Pathname: "etc"}
wantPanic := AbsoluteError("etc")
if r := recover(); !reflect.DeepEqual(r, wantPanic) {
t.Errorf("MustAbs: panic = %v; want %v", r, wantPanic)
@@ -175,7 +175,7 @@ func TestCodecAbsolute(t *testing.T) {
`"/etc"`, `{"val":"/etc","magic":3236757504}`},
{"not absolute", nil,
&AbsoluteError{Pathname: "etc"},
AbsoluteError("etc"),
"\t\x7f\x05\x01\x02\xff\x82\x00\x00\x00\a\xff\x80\x00\x03etc",
",\xff\x83\x03\x01\x01\x06sCheck\x01\xff\x84\x00\x01\x02\x01\bPathname\x01\xff\x80\x00\x01\x05Magic\x01\x06\x00\x00\x00\t\x7f\x05\x01\x02\xff\x82\x00\x00\x00\x0f\xff\x84\x01\x03etc\x01\xfb\x01\x81\xda\x00\x00\x00",

View File

@@ -35,6 +35,8 @@ type (
// Container represents a container environment being prepared or run.
// None of [Container] methods are safe for concurrent use.
Container struct {
// Whether the container init should stay alive after its parent terminates.
AllowOrphan bool
// Cgroup fd, nil to disable.
Cgroup *int
// ExtraFiles passed through to initial process in the container,
@@ -252,8 +254,7 @@ func (p *Container) Start() error {
}
p.cmd.Dir = fhs.Root
p.cmd.SysProcAttr = &SysProcAttr{
Setsid: !p.RetainSession,
Pdeathsig: SIGKILL,
Setsid: !p.RetainSession,
Cloneflags: CLONE_NEWUSER | CLONE_NEWPID | CLONE_NEWNS |
CLONE_NEWIPC | CLONE_NEWUTS | CLONE_NEWCGROUP,
@@ -262,12 +263,17 @@ func (p *Container) Start() error {
CAP_SYS_ADMIN,
// drop capabilities
CAP_SETPCAP,
// bring up loopback interface
CAP_NET_ADMIN,
// overlay access to upperdir and workdir
CAP_DAC_OVERRIDE,
},
UseCgroupFD: p.Cgroup != nil,
}
if !p.AllowOrphan {
p.cmd.SysProcAttr.Pdeathsig = SIGKILL
}
if p.cmd.SysProcAttr.UseCgroupFD {
p.cmd.SysProcAttr.CgroupFD = *p.Cgroup
}

View File

@@ -275,12 +275,12 @@ var containerTestCases = []struct {
),
earlyMnt(
ent("/", "/dev", "ro,nosuid,nodev,relatime", "tmpfs", "devtmpfs", ignore),
ent("/null", "/dev/null", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/zero", "/dev/zero", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/full", "/dev/full", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/random", "/dev/random", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/urandom", "/dev/urandom", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/tty", "/dev/tty", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/null", "/dev/null", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/zero", "/dev/zero", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/full", "/dev/full", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/random", "/dev/random", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/urandom", "/dev/urandom", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/tty", "/dev/tty", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/", "/dev/pts", "rw,nosuid,noexec,relatime", "devpts", "devpts", "rw,mode=620,ptmxmode=666"),
ent("/", "/dev/mqueue", "rw,nosuid,nodev,noexec,relatime", "mqueue", "mqueue", "rw"),
ent("/", "/dev/shm", "rw,nosuid,nodev,relatime", "tmpfs", "tmpfs", ignore),
@@ -293,12 +293,12 @@ var containerTestCases = []struct {
),
earlyMnt(
ent("/", "/dev", "ro,nosuid,nodev,relatime", "tmpfs", "devtmpfs", ignore),
ent("/null", "/dev/null", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/zero", "/dev/zero", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/full", "/dev/full", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/random", "/dev/random", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/urandom", "/dev/urandom", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/tty", "/dev/tty", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
ent("/null", "/dev/null", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/zero", "/dev/zero", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/full", "/dev/full", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/random", "/dev/random", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/urandom", "/dev/urandom", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/tty", "/dev/tty", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/", "/dev/pts", "rw,nosuid,noexec,relatime", "devpts", "devpts", "rw,mode=620,ptmxmode=666"),
ent("/", "/dev/shm", "rw,nosuid,nodev,relatime", "tmpfs", "tmpfs", ignore),
),
@@ -696,6 +696,9 @@ func init() {
mnt[i].VfsOptstr = strings.TrimSuffix(mnt[i].VfsOptstr, ",relatime")
mnt[i].VfsOptstr = strings.TrimSuffix(mnt[i].VfsOptstr, ",noatime")
cur.FsOptstr = strings.Replace(cur.FsOptstr, ",seclabel", "", 1)
mnt[i].FsOptstr = strings.Replace(mnt[i].FsOptstr, ",seclabel", "", 1)
if !cur.EqualWithIgnore(mnt[i], "\x00") {
fail = true
log.Printf("[FAIL] %s", cur)

View File

@@ -61,6 +61,8 @@ type syscallDispatcher interface {
mountTmpfs(fsname, target string, flags uintptr, size int, perm os.FileMode) error
// ensureFile provides ensureFile.
ensureFile(name string, perm, pperm os.FileMode) error
// mustLoopback provides mustLoopback.
mustLoopback(msg message.Msg)
// seccompLoad provides [seccomp.Load].
seccompLoad(rules []std.NativeRule, flags seccomp.ExportFlag) error
@@ -164,6 +166,7 @@ func (k direct) mountTmpfs(fsname, target string, flags uintptr, size int, perm
func (direct) ensureFile(name string, perm, pperm os.FileMode) error {
return ensureFile(name, perm, pperm)
}
func (direct) mustLoopback(msg message.Msg) { mustLoopback(msg) }
func (direct) seccompLoad(rules []std.NativeRule, flags seccomp.ExportFlag) error {
return seccomp.Load(rules, flags)

View File

@@ -465,6 +465,8 @@ func (k *kstub) ensureFile(name string, perm, pperm os.FileMode) error {
stub.CheckArg(k.Stub, "pperm", pperm, 2))
}
func (*kstub) mustLoopback(message.Msg) { /* noop */ }
func (k *kstub) seccompLoad(rules []std.NativeRule, flags seccomp.ExportFlag) error {
k.Helper()
return k.Expects("seccompLoad").Error(

View File

@@ -18,7 +18,7 @@ func messageFromError(err error) (m string, ok bool) {
if m, ok = messagePrefixP[os.PathError]("cannot ", err); ok {
return
}
if m, ok = messagePrefixP[check.AbsoluteError](zeroString, err); ok {
if m, ok = messagePrefix[check.AbsoluteError](zeroString, err); ok {
return
}
if m, ok = messagePrefix[OpRepeatError](zeroString, err); ok {

View File

@@ -37,7 +37,7 @@ func TestMessageFromError(t *testing.T) {
Err: stub.UniqueError(0xdeadbeef),
}, "cannot mount /sysroot: unique error 3735928559 injected by the test suite", true},
{"absolute", &check.AbsoluteError{Pathname: "etc/mtab"},
{"absolute", check.AbsoluteError("etc/mtab"),
`path "etc/mtab" is not absolute`, true},
{"repeat", OpRepeatError("autoetc"),

View File

@@ -26,6 +26,8 @@ var (
// AbsRunUser is [RunUser] as [check.Absolute].
AbsRunUser = unsafeAbs(RunUser)
// AbsUsr is [Usr] as [check.Absolute].
AbsUsr = unsafeAbs(Usr)
// AbsUsrBin is [UsrBin] as [check.Absolute].
AbsUsrBin = unsafeAbs(UsrBin)

View File

@@ -170,6 +170,10 @@ func initEntrypoint(k syscallDispatcher, msg message.Msg) {
offsetSetup = int(setupFd + 1)
}
if !params.HostNet {
k.mustLoopback(msg)
}
// write uid/gid map here so parent does not need to set dumpable
if err := k.setDumpable(SUID_DUMP_USER); err != nil {
k.fatalf(msg, "cannot set SUID_DUMP_USER: %v", err)

View File

@@ -312,7 +312,10 @@ func TestMountOverlayOp(t *testing.T) {
},
}},
{"ephemeral", new(Ops).OverlayEphemeral(check.MustAbs("/nix/store"), check.MustAbs("/mnt-root/nix/.ro-store")), Ops{
{"ephemeral", new(Ops).OverlayEphemeral(
check.MustAbs("/nix/store"),
check.MustAbs("/mnt-root/nix/.ro-store"),
), Ops{
&MountOverlayOp{
Target: check.MustAbs("/nix/store"),
Lower: []*check.Absolute{check.MustAbs("/mnt-root/nix/.ro-store")},
@@ -320,7 +323,10 @@ func TestMountOverlayOp(t *testing.T) {
},
}},
{"readonly", new(Ops).OverlayReadonly(check.MustAbs("/nix/store"), check.MustAbs("/mnt-root/nix/.ro-store")), Ops{
{"readonly", new(Ops).OverlayReadonly(
check.MustAbs("/nix/store"),
check.MustAbs("/mnt-root/nix/.ro-store"),
), Ops{
&MountOverlayOp{
Target: check.MustAbs("/nix/store"),
Lower: []*check.Absolute{check.MustAbs("/mnt-root/nix/.ro-store")},

View File

@@ -31,7 +31,7 @@ func (l *SymlinkOp) Valid() bool { return l != nil && l.Target != nil && l.LinkN
func (l *SymlinkOp) early(_ *setupState, k syscallDispatcher) error {
if l.Dereference {
if !path.IsAbs(l.LinkName) {
return &check.AbsoluteError{Pathname: l.LinkName}
return check.AbsoluteError(l.LinkName)
}
if name, err := k.readlink(l.LinkName); err != nil {
return err

View File

@@ -23,7 +23,7 @@ func TestSymlinkOp(t *testing.T) {
Target: check.MustAbs("/etc/mtab"),
LinkName: "etc/mtab",
Dereference: true,
}, nil, &check.AbsoluteError{Pathname: "etc/mtab"}, nil, nil},
}, nil, check.AbsoluteError("etc/mtab"), nil, nil},
{"readlink", &Params{ParentPerm: 0755}, &SymlinkOp{
Target: check.MustAbs("/etc/mtab"),

269
container/netlink.go Normal file
View File

@@ -0,0 +1,269 @@
package container
import (
"encoding/binary"
"errors"
"net"
"os"
. "syscall"
"unsafe"
"hakurei.app/container/std"
"hakurei.app/message"
)
// rtnetlink represents a NETLINK_ROUTE socket.
type rtnetlink struct {
// Sent as part of rtnetlink messages.
pid uint32
// AF_NETLINK socket.
fd int
// Whether the socket is open.
ok bool
// Message sequence number.
seq uint32
}
// open creates the underlying NETLINK_ROUTE socket.
func (s *rtnetlink) open() (err error) {
if s.ok || s.fd < 0 {
return os.ErrInvalid
}
s.pid = uint32(Getpid())
if s.fd, err = Socket(
AF_NETLINK,
SOCK_RAW|SOCK_CLOEXEC,
NETLINK_ROUTE,
); err != nil {
return os.NewSyscallError("socket", err)
} else if err = Bind(s.fd, &SockaddrNetlink{
Family: AF_NETLINK,
Pid: s.pid,
}); err != nil {
_ = s.close()
return os.NewSyscallError("bind", err)
} else {
s.ok = true
return nil
}
}
// close closes the underlying NETLINK_ROUTE socket.
func (s *rtnetlink) close() error {
if !s.ok {
return os.ErrInvalid
}
s.ok = false
err := Close(s.fd)
s.fd = -1
return err
}
// roundtrip sends a netlink message and handles the reply.
func (s *rtnetlink) roundtrip(data []byte) error {
if !s.ok {
return os.ErrInvalid
}
defer func() { s.seq++ }()
if err := Sendto(s.fd, data, 0, &SockaddrNetlink{
Family: AF_NETLINK,
}); err != nil {
return os.NewSyscallError("sendto", err)
}
buf := make([]byte, Getpagesize())
done:
for {
p := buf
if n, _, err := Recvfrom(s.fd, p, 0); err != nil {
return os.NewSyscallError("recvfrom", err)
} else if n < NLMSG_HDRLEN {
return errors.ErrUnsupported
} else {
p = p[:n]
}
if msgs, err := ParseNetlinkMessage(p); err != nil {
return err
} else {
for _, m := range msgs {
if m.Header.Seq != s.seq || m.Header.Pid != s.pid {
return errors.ErrUnsupported
}
if m.Header.Type == NLMSG_DONE {
break done
}
if m.Header.Type == NLMSG_ERROR {
if len(m.Data) >= 4 {
errno := Errno(-std.ScmpInt(binary.NativeEndian.Uint32(m.Data)))
if errno == 0 {
return nil
}
return errno
}
return errors.ErrUnsupported
}
}
}
}
return nil
}
// mustRoundtrip calls roundtrip and terminates via msg for a non-nil error.
func (s *rtnetlink) mustRoundtrip(msg message.Msg, data []byte) {
err := s.roundtrip(data)
if err == nil {
return
}
if closeErr := Close(s.fd); closeErr != nil {
msg.Verbosef("cannot close: %v", err)
}
switch err.(type) {
case *os.SyscallError:
msg.GetLogger().Fatalf("cannot %v", err)
case Errno:
msg.GetLogger().Fatalf("RTNETLINK answers: %v", err)
default:
msg.GetLogger().Fatalln("RTNETLINK answers with unexpected message")
}
}
// newaddrLo represents a RTM_NEWADDR message with two addresses.
type newaddrLo struct {
header NlMsghdr
data IfAddrmsg
r0 RtAttr
a0 [4]byte // in_addr
r1 RtAttr
a1 [4]byte // in_addr
}
// sizeofNewaddrLo is the expected size of newaddrLo.
const sizeofNewaddrLo = NLMSG_HDRLEN + SizeofIfAddrmsg + (SizeofRtAttr+4)*2
// newaddrLo returns the address of a populated newaddrLo.
func (s *rtnetlink) newaddrLo(lo int) *newaddrLo {
return &newaddrLo{NlMsghdr{
Len: sizeofNewaddrLo,
Type: RTM_NEWADDR,
Flags: NLM_F_REQUEST | NLM_F_ACK | NLM_F_CREATE | NLM_F_EXCL,
Seq: s.seq,
Pid: s.pid,
}, IfAddrmsg{
Family: AF_INET,
Prefixlen: 8,
Flags: IFA_F_PERMANENT,
Scope: RT_SCOPE_HOST,
Index: uint32(lo),
}, RtAttr{
Len: uint16(SizeofRtAttr + len(newaddrLo{}.a0)),
Type: IFA_LOCAL,
}, [4]byte{127, 0, 0, 1}, RtAttr{
Len: uint16(SizeofRtAttr + len(newaddrLo{}.a1)),
Type: IFA_ADDRESS,
}, [4]byte{127, 0, 0, 1}}
}
func (msg *newaddrLo) toWireFormat() []byte {
var buf [sizeofNewaddrLo]byte
*(*uint32)(unsafe.Pointer(&buf[0:4][0])) = msg.header.Len
*(*uint16)(unsafe.Pointer(&buf[4:6][0])) = msg.header.Type
*(*uint16)(unsafe.Pointer(&buf[6:8][0])) = msg.header.Flags
*(*uint32)(unsafe.Pointer(&buf[8:12][0])) = msg.header.Seq
*(*uint32)(unsafe.Pointer(&buf[12:16][0])) = msg.header.Pid
buf[16] = msg.data.Family
buf[17] = msg.data.Prefixlen
buf[18] = msg.data.Flags
buf[19] = msg.data.Scope
*(*uint32)(unsafe.Pointer(&buf[20:24][0])) = msg.data.Index
*(*uint16)(unsafe.Pointer(&buf[24:26][0])) = msg.r0.Len
*(*uint16)(unsafe.Pointer(&buf[26:28][0])) = msg.r0.Type
copy(buf[28:32], msg.a0[:])
*(*uint16)(unsafe.Pointer(&buf[32:34][0])) = msg.r1.Len
*(*uint16)(unsafe.Pointer(&buf[34:36][0])) = msg.r1.Type
copy(buf[36:40], msg.a1[:])
return buf[:]
}
// newlinkLo represents a RTM_NEWLINK message.
type newlinkLo struct {
header NlMsghdr
data IfInfomsg
}
// sizeofNewlinkLo is the expected size of newlinkLo.
const sizeofNewlinkLo = NLMSG_HDRLEN + SizeofIfInfomsg
// newlinkLo returns the address of a populated newlinkLo.
func (s *rtnetlink) newlinkLo(lo int) *newlinkLo {
return &newlinkLo{NlMsghdr{
Len: sizeofNewlinkLo,
Type: RTM_NEWLINK,
Flags: NLM_F_REQUEST | NLM_F_ACK,
Seq: s.seq,
Pid: s.pid,
}, IfInfomsg{
Family: AF_UNSPEC,
Index: int32(lo),
Flags: IFF_UP,
Change: IFF_UP,
}}
}
func (msg *newlinkLo) toWireFormat() []byte {
var buf [sizeofNewlinkLo]byte
*(*uint32)(unsafe.Pointer(&buf[0:4][0])) = msg.header.Len
*(*uint16)(unsafe.Pointer(&buf[4:6][0])) = msg.header.Type
*(*uint16)(unsafe.Pointer(&buf[6:8][0])) = msg.header.Flags
*(*uint32)(unsafe.Pointer(&buf[8:12][0])) = msg.header.Seq
*(*uint32)(unsafe.Pointer(&buf[12:16][0])) = msg.header.Pid
buf[16] = msg.data.Family
*(*uint16)(unsafe.Pointer(&buf[18:20][0])) = msg.data.Type
*(*int32)(unsafe.Pointer(&buf[20:24][0])) = msg.data.Index
*(*uint32)(unsafe.Pointer(&buf[24:28][0])) = msg.data.Flags
*(*uint32)(unsafe.Pointer(&buf[28:32][0])) = msg.data.Change
return buf[:]
}
// mustLoopback creates the loopback address and brings the lo interface up.
// mustLoopback calls a fatal method of the underlying [log.Logger] of m with a
// user-facing error message if RTNETLINK behaves unexpectedly.
func mustLoopback(msg message.Msg) {
log := msg.GetLogger()
var lo int
if ifi, err := net.InterfaceByName("lo"); err != nil {
log.Fatalln(err)
} else {
lo = ifi.Index
}
var s rtnetlink
if err := s.open(); err != nil {
log.Fatalln(err)
}
defer func() {
if err := s.close(); err != nil {
msg.Verbosef("cannot close netlink: %v", err)
}
}()
s.mustRoundtrip(msg, s.newaddrLo(lo).toWireFormat())
s.mustRoundtrip(msg, s.newlinkLo(lo).toWireFormat())
}

72
container/netlink_test.go Normal file
View File

@@ -0,0 +1,72 @@
package container
import (
"testing"
"unsafe"
)
func TestSizeof(t *testing.T) {
if got := unsafe.Sizeof(newaddrLo{}); got != sizeofNewaddrLo {
t.Fatalf("newaddrLo: sizeof = %#x, want %#x", got, sizeofNewaddrLo)
}
if got := unsafe.Sizeof(newlinkLo{}); got != sizeofNewlinkLo {
t.Fatalf("newlinkLo: sizeof = %#x, want %#x", got, sizeofNewlinkLo)
}
}
func TestRtnetlinkMessage(t *testing.T) {
t.Parallel()
testCases := []struct {
name string
msg interface{ toWireFormat() []byte }
want []byte
}{
{"newaddrLo", (&rtnetlink{pid: 1, seq: 0}).newaddrLo(1), []byte{
/* Len */ 0x28, 0, 0, 0,
/* Type */ 0x14, 0,
/* Flags */ 5, 6,
/* Seq */ 0, 0, 0, 0,
/* Pid */ 1, 0, 0, 0,
/* Family */ 2,
/* Prefixlen */ 8,
/* Flags */ 0x80,
/* Scope */ 0xfe,
/* Index */ 1, 0, 0, 0,
/* Len */ 8, 0,
/* Type */ 2, 0,
/* in_addr */ 127, 0, 0, 1,
/* Len */ 8, 0,
/* Type */ 1, 0,
/* in_addr */ 127, 0, 0, 1,
}},
{"newlinkLo", (&rtnetlink{pid: 1, seq: 1}).newlinkLo(1), []byte{
/* Len */ 0x20, 0, 0, 0,
/* Type */ 0x10, 0,
/* Flags */ 5, 0,
/* Seq */ 1, 0, 0, 0,
/* Pid */ 1, 0, 0, 0,
/* Family */ 0,
/* pad */ 0,
/* Type */ 0, 0,
/* Index */ 1, 0, 0, 0,
/* Flags */ 1, 0, 0, 0,
/* Change */ 1, 0, 0, 0,
}},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
if got := tc.msg.toWireFormat(); string(got) != string(tc.want) {
t.Fatalf("toWireFormat: %#v, want %#v", got, tc.want)
}
})
}
}

12
dist/install.sh vendored
View File

@@ -1,12 +1,12 @@
#!/bin/sh
cd "$(dirname -- "$0")" || exit 1
install -vDm0755 "bin/hakurei" "${HAKUREI_INSTALL_PREFIX}/usr/bin/hakurei"
install -vDm0755 "bin/hpkg" "${HAKUREI_INSTALL_PREFIX}/usr/bin/hpkg"
install -vDm0755 "bin/hakurei" "${DESTDIR}/usr/bin/hakurei"
install -vDm0755 "bin/sharefs" "${DESTDIR}/usr/bin/sharefs"
install -vDm4511 "bin/hsu" "${HAKUREI_INSTALL_PREFIX}/usr/bin/hsu"
if [ ! -f "${HAKUREI_INSTALL_PREFIX}/etc/hsurc" ]; then
install -vDm0400 "hsurc.default" "${HAKUREI_INSTALL_PREFIX}/etc/hsurc"
install -vDm4511 "bin/hsu" "${DESTDIR}/usr/bin/hsu"
if [ ! -f "${DESTDIR}/etc/hsurc" ]; then
install -vDm0400 "hsurc.default" "${DESTDIR}/etc/hsurc"
fi
install -vDm0644 "comp/_hakurei" "${HAKUREI_INSTALL_PREFIX}/usr/share/zsh/site-functions/_hakurei"
install -vDm0644 "comp/_hakurei" "${DESTDIR}/usr/share/zsh/site-functions/_hakurei"

21
dist/release.sh vendored
View File

@@ -2,19 +2,30 @@
cd "$(dirname -- "$0")/.."
VERSION="${HAKUREI_VERSION:-untagged}"
pname="hakurei-${VERSION}"
out="dist/${pname}"
out="${DESTDIR:-dist}/${pname}"
echo '# Preparing distribution files.'
mkdir -p "${out}"
cp -v "README.md" "dist/hsurc.default" "dist/install.sh" "${out}"
cp -rv "dist/comp" "${out}"
echo
echo '# Building hakurei.'
go generate ./...
go build -trimpath -v -o "${out}/bin/" -ldflags "-s -w -buildid= -extldflags '-static'
go build -trimpath -v -o "${out}/bin/" -ldflags "-s -w
-buildid= -extldflags '-static'
-X hakurei.app/internal/info.buildVersion=${VERSION}
-X hakurei.app/internal/info.hakureiPath=/usr/bin/hakurei
-X hakurei.app/internal/info.hsuPath=/usr/bin/hsu
-X main.hakureiPath=/usr/bin/hakurei" ./...
echo
rm -f "./${out}.tar.gz" && tar -C dist -czf "${out}.tar.gz" "${pname}"
rm -rf "./${out}"
(cd dist && sha512sum "${pname}.tar.gz" > "${pname}.tar.gz.sha512")
echo '# Testing hakurei.'
go test -ldflags='-buildid= -extldflags=-static' ./...
echo
echo '# Creating distribution.'
rm -f "${out}.tar.gz" && tar -C "${out}/.." -vczf "${out}.tar.gz" "${pname}"
rm -rf "${out}"
(cd "${out}/.." && sha512sum "${pname}.tar.gz" > "${pname}.tar.gz.sha512")
echo

View File

@@ -69,6 +69,8 @@
withRace = true;
};
sharefs = callPackage ./cmd/sharefs/test { inherit system self; };
hpkg = callPackage ./cmd/hpkg/test { inherit system self; };
formatting = runCommandLocal "check-formatting" { nativeBuildInputs = [ nixfmt-rfc-style ]; } ''
@@ -136,20 +138,32 @@
;
};
hsu = pkgs.callPackage ./cmd/hsu/package.nix { inherit (self.packages.${system}) hakurei; };
sharefs = pkgs.linkFarm "sharefs" {
"bin/sharefs" = "${hakurei}/libexec/sharefs";
"bin/mount.fuse.sharefs" = "${hakurei}/libexec/sharefs";
};
dist = pkgs.runCommand "${hakurei.name}-dist" { buildInputs = hakurei.targetPkgs ++ [ pkgs.pkgsStatic.musl ]; } ''
# go requires XDG_CACHE_HOME for the build cache
export XDG_CACHE_HOME="$(mktemp -d)"
dist =
pkgs.runCommand "${hakurei.name}-dist"
{
buildInputs = hakurei.targetPkgs ++ [
pkgs.pkgsStatic.musl
];
}
''
cd $(mktemp -d) \
&& cp -r ${hakurei.src}/. . \
&& chmod +w cmd && cp -r ${hsu.src}/. cmd/hsu/ \
&& chmod -R +w .
# get a different workdir as go does not like /build
cd $(mktemp -d) \
&& cp -r ${hakurei.src}/. . \
&& chmod +w cmd && cp -r ${hsu.src}/. cmd/hsu/ \
&& chmod -R +w .
export HAKUREI_VERSION="v${hakurei.version}"
CC="clang -O3 -Werror" ./dist/release.sh && mkdir $out && cp -v "dist/hakurei-$HAKUREI_VERSION.tar.gz"* $out
'';
CC="musl-clang -O3 -Werror -Qunused-arguments" \
GOCACHE="$(mktemp -d)" \
HAKUREI_TEST_SKIP_ACL=1 \
PATH="${pkgs.pkgsStatic.musl.bin}/bin:$PATH" \
DESTDIR="$out" \
HAKUREI_VERSION="v${hakurei.version}" \
./dist/release.sh
'';
}
);
@@ -160,7 +174,10 @@
pkgs = nixpkgsFor.${system};
in
{
default = pkgs.mkShell { buildInputs = hakurei.targetPkgs; };
default = pkgs.mkShell {
buildInputs = hakurei.targetPkgs;
hardeningDisable = [ "fortify" ];
};
withPackage = pkgs.mkShell { buildInputs = [ hakurei ] ++ hakurei.targetPkgs; };
vm =

2
go.mod
View File

@@ -1,3 +1,3 @@
module hakurei.app
go 1.24
go 1.25

View File

@@ -24,9 +24,8 @@ var (
)
func TestUpdate(t *testing.T) {
if os.Getenv("GO_TEST_SKIP_ACL") == "1" {
t.Log("acl test skipped")
t.SkipNow()
if os.Getenv("HAKUREI_TEST_SKIP_ACL") == "1" {
t.Skip("acl test skipped")
}
testFilePath := path.Join(t.TempDir(), testFileName)
@@ -143,6 +142,7 @@ func (c *getFAclInvocation) run(name string) error {
}
c.cmd = exec.Command("getfacl", "--omit-header", "--absolute-names", "--numeric", name)
c.cmd.Stderr = os.Stderr
scanErr := make(chan error, 1)
if p, err := c.cmd.StdoutPipe(); err != nil {
@@ -254,7 +254,7 @@ func getfacl(t *testing.T, name string) []*getFAclResp {
t.Fatalf("getfacl: error = %v", err)
}
if len(c.pe) != 0 {
t.Errorf("errors encountered parsing getfacl output\n%s", errors.Join(c.pe...).Error())
t.Errorf("errors encountered parsing getfacl output\n%s", errors.Join(c.pe...))
}
return c.val
}

View File

@@ -108,7 +108,7 @@ func TestSpPulseOp(t *testing.T) {
call("lookupEnv", stub.ExpectArgs{"PULSE_COOKIE"}, "proc/nonexistent/cookie", nil),
}, nil, nil, &hst.AppError{
Step: "locate PulseAudio cookie",
Err: &check.AbsoluteError{Pathname: "proc/nonexistent/cookie"},
Err: check.AbsoluteError("proc/nonexistent/cookie"),
}, nil, nil, nil, nil, nil},
{"cookie loadFile", func(bool, bool) outcomeOp {
@@ -272,7 +272,7 @@ func TestDiscoverPulseCookie(t *testing.T) {
call("verbose", stub.ExpectArgs{[]any{(*check.Absolute)(nil)}}, nil, nil),
}}, &hst.AppError{
Step: "locate PulseAudio cookie",
Err: &check.AbsoluteError{Pathname: "proc/nonexistent/pulse-cookie"},
Err: check.AbsoluteError("proc/nonexistent/pulse-cookie"),
}},
{"success override", fCheckPathname, stub.Expect{Calls: []stub.Call{
@@ -286,7 +286,7 @@ func TestDiscoverPulseCookie(t *testing.T) {
call("verbose", stub.ExpectArgs{[]any{(*check.Absolute)(nil)}}, nil, nil),
}}, &hst.AppError{
Step: "locate PulseAudio cookie",
Err: &check.AbsoluteError{Pathname: "proc/nonexistent/home"},
Err: check.AbsoluteError("proc/nonexistent/home"),
}},
{"home stat", fCheckPathname, stub.Expect{Calls: []stub.Call{
@@ -321,7 +321,7 @@ func TestDiscoverPulseCookie(t *testing.T) {
call("verbose", stub.ExpectArgs{[]any{(*check.Absolute)(nil)}}, nil, nil),
}}, &hst.AppError{
Step: "locate PulseAudio cookie",
Err: &check.AbsoluteError{Pathname: "proc/nonexistent/xdg"},
Err: check.AbsoluteError("proc/nonexistent/xdg"),
}},
{"xdg stat", fCheckPathname, stub.Expect{Calls: []stub.Call{

View File

@@ -514,7 +514,7 @@ var ErrNotDone = errors.New("did not receive a Core::Done event targeting previo
const (
// syncTimeout is the maximum duration [Core.Sync] is allowed to take before
// receiving [CoreDone] or failing.
syncTimeout = 5 * time.Second
syncTimeout = 10 * time.Second
)
// Sync queues a [CoreSync] message for the PipeWire server and initiates a Roundtrip.

View File

@@ -19,7 +19,6 @@ import (
"errors"
"fmt"
"io"
"net"
"os"
"path"
"runtime"
@@ -27,10 +26,16 @@ import (
"strconv"
"strings"
"syscall"
"time"
)
// Conn is a low level unix socket interface used by [Context].
type Conn interface {
// MightBlock informs the implementation that the next call to
// Recvmsg or Sendmsg might block. A zero or negative timeout
// cancels this behaviour.
MightBlock(timeout time.Duration)
// Recvmsg calls syscall.Recvmsg on the underlying socket.
Recvmsg(p, oob []byte, flags int) (n, oobn, recvflags int, err error)
@@ -138,45 +143,142 @@ func New(conn Conn, props SPADict) (*Context, error) {
return &ctx, nil
}
// A SyscallConnCloser is a [syscall.Conn] that implements [io.Closer].
type SyscallConnCloser interface {
syscall.Conn
io.Closer
// unixConn is an implementation of the [Conn] interface for connections
// to Unix domain sockets.
type unixConn struct {
fd int
// Whether creation of a new epoll instance was attempted.
epoll bool
// File descriptor referring to the new epoll instance.
// Valid if epoll is true and epollErr is nil.
epollFd int
// Error returned by syscall.EpollCreate1.
epollErr error
// Stores epoll events from the kernel.
epollBuf [32]syscall.EpollEvent
// If non-zero, next call is treated as a blocking call.
timeout time.Duration
}
// A SyscallConn is a [Conn] adapter for [syscall.Conn].
type SyscallConn struct{ SyscallConnCloser }
// Dial connects to a Unix domain socket described by name.
func Dial(name string) (Conn, error) {
if fd, err := syscall.Socket(syscall.AF_UNIX, syscall.SOCK_STREAM|syscall.SOCK_CLOEXEC|syscall.SOCK_NONBLOCK, 0); err != nil {
return nil, os.NewSyscallError("socket", err)
} else if err = syscall.Connect(fd, &syscall.SockaddrUnix{Name: name}); err != nil {
_ = syscall.Close(fd)
return nil, os.NewSyscallError("connect", err)
} else {
return &unixConn{fd: fd}, nil
}
}
// Recvmsg implements [Conn.Recvmsg] via [syscall.Conn.SyscallConn].
func (conn SyscallConn) Recvmsg(p, oob []byte, flags int) (n, oobn, recvflags int, err error) {
var rc syscall.RawConn
if rc, err = conn.SyscallConn(); err != nil {
// MightBlock informs the implementation that the next call
// might block for a non-zero timeout.
func (conn *unixConn) MightBlock(timeout time.Duration) {
if timeout < 0 {
timeout = 0
}
conn.timeout = timeout
}
// wantsEpoll is called at the beginning of any method that might use epoll.
func (conn *unixConn) wantsEpoll() error {
if !conn.epoll {
conn.epoll = true
conn.epollFd, conn.epollErr = syscall.EpollCreate1(syscall.EPOLL_CLOEXEC)
if conn.epollErr == nil {
if conn.epollErr = syscall.EpollCtl(conn.epollFd, syscall.EPOLL_CTL_ADD, conn.fd, &syscall.EpollEvent{
Events: syscall.EPOLLERR | syscall.EPOLLHUP,
Fd: int32(conn.fd),
}); conn.epollErr != nil {
_ = syscall.Close(conn.epollFd)
}
}
}
return conn.epollErr
}
// wait waits for a specific I/O event on fd. Caller must arrange for wantsEpoll
// to be called somewhere before wait is called.
func (conn *unixConn) wait(event uint32) (err error) {
if conn.timeout == 0 {
return nil
}
deadline := time.Now().Add(conn.timeout)
conn.timeout = 0
if err = syscall.EpollCtl(conn.epollFd, syscall.EPOLL_CTL_MOD, conn.fd, &syscall.EpollEvent{
Events: event | syscall.EPOLLERR | syscall.EPOLLHUP,
Fd: int32(conn.fd),
}); err != nil {
return
}
if controlErr := rc.Control(func(fd uintptr) {
n, oobn, recvflags, _, err = syscall.Recvmsg(int(fd), p, oob, flags)
}); controlErr != nil && err == nil {
err = controlErr
for timeout := deadline.Sub(time.Now()); timeout > 0; timeout = deadline.Sub(time.Now()) {
var n int
if n, err = syscall.EpollWait(conn.epollFd, conn.epollBuf[:], int(timeout/time.Millisecond)); err != nil {
return
}
switch n {
case 1: // only the socket fd is ever added
if conn.epollBuf[0].Fd != int32(conn.fd) { // unreachable
return syscall.ENOTRECOVERABLE
}
if conn.epollBuf[0].Events&event == event ||
conn.epollBuf[0].Events&syscall.EPOLLERR|syscall.EPOLLHUP != 0 {
return nil
}
err = syscall.ETIME
continue
case 0: // timeout
return syscall.ETIMEDOUT
default: // unreachable
return syscall.ENOTRECOVERABLE
}
}
return
}
// Sendmsg implements [Conn.Sendmsg] via [syscall.Conn.SyscallConn].
func (conn SyscallConn) Sendmsg(p, oob []byte, flags int) (n int, err error) {
var rc syscall.RawConn
if rc, err = conn.SyscallConn(); err != nil {
// Recvmsg calls syscall.Recvmsg on the underlying socket.
func (conn *unixConn) Recvmsg(p, oob []byte, flags int) (n, oobn, recvflags int, err error) {
if err = conn.wantsEpoll(); err != nil {
return
} else if err = conn.wait(syscall.EPOLLIN); err != nil {
return
}
if controlErr := rc.Control(func(fd uintptr) {
n, err = syscall.SendmsgN(int(fd), p, oob, nil, flags)
}); controlErr != nil && err == nil {
err = controlErr
}
n, oobn, recvflags, _, err = syscall.Recvmsg(conn.fd, p, oob, flags)
return
}
// Sendmsg calls syscall.Sendmsg on the underlying socket.
func (conn *unixConn) Sendmsg(p, oob []byte, flags int) (n int, err error) {
if err = conn.wantsEpoll(); err != nil {
return
} else if err = conn.wait(syscall.EPOLLOUT); err != nil {
return
}
n, err = syscall.SendmsgN(conn.fd, p, oob, nil, flags)
return
}
// Close closes the underlying socket and the epoll fd if populated.
func (conn *unixConn) Close() (err error) {
if conn.epoll && conn.epollErr == nil {
conn.epollErr = syscall.Close(conn.epollFd)
}
if err = syscall.Close(conn.fd); err != nil {
return
}
return conn.epollErr
}
// MustNew calls [New](conn, props) and panics on error.
// It is intended for use in tests with hard-coded strings.
func MustNew(conn Conn, props SPADict) *Context {
@@ -310,7 +412,7 @@ func (ctx *Context) recvmsg(remaining []byte) (payload []byte, err error) {
}
if err != syscall.EAGAIN && err != syscall.EWOULDBLOCK {
ctx.closeReceivedFiles()
return nil, os.NewSyscallError("recvmsg", err)
return nil, &ProxyFatalError{Err: os.NewSyscallError("recvmsg", err), ProxyErrs: ctx.cloneAsProxyErrors()}
}
}
@@ -347,7 +449,7 @@ func (ctx *Context) sendmsg(p []byte, fds ...int) error {
}
if err != nil && err != syscall.EAGAIN && err != syscall.EWOULDBLOCK {
return os.NewSyscallError("sendmsg", err)
return &ProxyFatalError{Err: os.NewSyscallError("sendmsg", err), ProxyErrs: ctx.cloneAsProxyErrors()}
}
return err
}
@@ -598,8 +700,15 @@ func (ctx *Context) Roundtrip() (err error) {
return
}
const (
// roundtripTimeout is the maximum duration socket operations during
// Context.roundtrip is allowed to block for.
roundtripTimeout = 5 * time.Second
)
// roundtrip implements the Roundtrip method without checking proxyErrors.
func (ctx *Context) roundtrip() (err error) {
ctx.conn.MightBlock(roundtripTimeout)
if err = ctx.sendmsg(ctx.buf, ctx.pendingFiles...); err != nil {
return
}
@@ -633,6 +742,7 @@ func (ctx *Context) roundtrip() (err error) {
}()
var remaining []byte
ctx.conn.MightBlock(roundtripTimeout)
for {
remaining, err = ctx.consume(remaining)
if err == nil {
@@ -857,14 +967,14 @@ const Remote = "PIPEWIRE_REMOTE"
const DEFAULT_SYSTEM_RUNTIME_DIR = "/run/pipewire"
// connectName connects to a PipeWire remote by name and returns the [net.UnixConn].
func connectName(name string, manager bool) (conn *net.UnixConn, err error) {
// connectName connects to a PipeWire remote by name and returns the resulting [Conn].
func connectName(name string, manager bool) (conn Conn, err error) {
if manager && !strings.HasSuffix(name, "-manager") {
return connectName(name+"-manager", false)
}
if path.IsAbs(name) || (len(name) > 0 && name[0] == '@') {
return net.DialUnix("unix", nil, &net.UnixAddr{Name: name, Net: "unix"})
return Dial(name)
} else {
runtimeDir, ok := os.LookupEnv("PIPEWIRE_RUNTIME_DIR")
if !ok || !path.IsAbs(runtimeDir) {
@@ -879,7 +989,7 @@ func connectName(name string, manager bool) (conn *net.UnixConn, err error) {
if !ok || !path.IsAbs(runtimeDir) {
runtimeDir = DEFAULT_SYSTEM_RUNTIME_DIR
}
return net.DialUnix("unix", nil, &net.UnixAddr{Name: path.Join(runtimeDir, name), Net: "unix"})
return Dial(path.Join(runtimeDir, name))
}
}
@@ -897,12 +1007,11 @@ func ConnectName(name string, manager bool, props SPADict) (ctx *Context, err er
}
}
var conn *net.UnixConn
var conn Conn
if conn, err = connectName(name, manager); err != nil {
return
}
if ctx, err = New(SyscallConn{conn}, props); err != nil {
if ctx, err = New(conn, props); err != nil {
ctx = nil
_ = conn.Close()
}

View File

@@ -6,6 +6,7 @@ import (
"strconv"
. "syscall"
"testing"
"time"
"hakurei.app/container/stub"
"hakurei.app/internal/pipewire"
@@ -715,6 +716,18 @@ type stubUnixConn struct {
current int
}
func (conn *stubUnixConn) MightBlock(timeout time.Duration) {
if timeout != 5*time.Second {
panic("unexpected timeout " + timeout.String())
}
if conn.current == 0 ||
(conn.samples[conn.current-1].nr == SYS_RECVMSG && conn.samples[conn.current-1].errno == EAGAIN && conn.samples[conn.current].nr == SYS_SENDMSG) ||
(conn.samples[conn.current-1].nr == SYS_SENDMSG && conn.samples[conn.current].nr == SYS_RECVMSG) {
return
}
panic("unexpected blocking hint before sample " + strconv.Itoa(conn.current))
}
// nextSample returns the current sample and increments the counter.
func (conn *stubUnixConn) nextSample(nr uintptr) (sample *stubUnixConnSample, wantOOB []byte, err error) {
sample = &conn.samples[conn.current]

216
internal/pkg/asm.go Normal file
View File

@@ -0,0 +1,216 @@
package pkg
import (
"encoding/binary"
"fmt"
"io"
"strconv"
"strings"
)
type asmOutLine struct {
pos int
word int
kindData int64
valueData []byte
indent int
kind string
value string
}
var spacingLine = asmOutLine{
pos: -1,
kindData: -1,
valueData: nil,
indent: 0,
kind: "",
value: "",
}
func Disassemble(r io.Reader, real bool, showHeader bool, force bool, raw bool) (s string, err error) {
var lines []asmOutLine
sb := new(strings.Builder)
header := true
pos := new(int)
for err == nil {
if header {
var kind uint64
var size uint64
var bsize []byte
p := *pos
if _, kind, err = nextUint64(r, pos); err != nil {
break
}
if bsize, size, err = nextUint64(r, pos); err != nil {
break
}
if showHeader {
lines = append(lines, asmOutLine{p, 8, int64(kind), bsize, 0, "head " + intToKind(kind), ""})
}
for i := 0; uint64(i) < size; i++ {
var did Checksum
var dkind uint64
p := *pos
if _, dkind, err = nextUint64(r, pos); err != nil {
break
}
if _, did, err = nextIdent(r, pos); err != nil {
break
}
if showHeader {
lines = append(lines, asmOutLine{p, 8, int64(dkind), nil, 1, intToKind(dkind), Encode(did)})
}
}
header = false
}
var k uint32
p := *pos
if _, k, err = nextUint32(r, pos); err != nil {
break
}
kind := IRValueKind(k)
switch kind {
case IRKindEnd:
var a uint32
var ba []byte
if ba, a, err = nextUint32(r, pos); err != nil {
break
}
if a&1 != 0 {
var sum Checksum
if _, sum, err = nextIdent(r, pos); err != nil {
break
}
lines = append(lines, asmOutLine{p, 4, int64(kind), ba, 1, "end ", Encode(sum)})
} else {
lines = append(lines, asmOutLine{p, 4, int64(kind), []byte{0, 0, 0, 0}, 1, "end ", ""})
}
lines = append(lines, spacingLine)
header = true
continue
case IRKindIdent:
var a []byte
// discard ancillary
if a, _, err = nextUint32(r, pos); err != nil {
break
}
var sum Checksum
if _, sum, err = nextIdent(r, pos); err != nil {
break
}
lines = append(lines, asmOutLine{p, 4, int64(kind), a, 1, "id ", Encode(sum)})
continue
case IRKindUint32:
var i uint32
var bi []byte
if bi, i, err = nextUint32(r, pos); err != nil {
break
}
lines = append(lines, asmOutLine{p, 4, int64(kind), bi, 1, "int ", strconv.FormatUint(uint64(i), 10)})
case IRKindString:
var l uint32
var bl []byte
if bl, l, err = nextUint32(r, pos); err != nil {
break
}
s := make([]byte, l+(wordSize-(l)%wordSize)%wordSize)
var n int
if n, err = r.Read(s); err != nil {
break
}
*pos = *pos + n
lines = append(lines, asmOutLine{p, 4, int64(kind), bl, 1, "str ", strconv.Quote(string(s[:l]))})
continue
default:
var bi []byte
if bi, _, err = nextUint32(r, pos); err != nil {
break
}
lines = append(lines, asmOutLine{p, 4, int64(kind), bi, 1, "????", ""})
}
}
if err != io.EOF {
return
}
err = nil
for _, line := range lines {
if raw {
if line.pos != -1 {
sb.WriteString(fmt.Sprintf("%s\t%s\n", line.kind, line.value))
}
} else {
if line.pos == -1 {
sb.WriteString("\n")
} else if line.word == 4 {
sb.WriteString(fmt.Sprintf("%06x: %04x %04x%s %s %s\n", line.pos, binary.LittleEndian.AppendUint32(nil, uint32(line.kindData)), line.valueData, headerSpacing(showHeader), line.kind, line.value))
} else {
kind := binary.LittleEndian.AppendUint64(nil, uint64(line.kindData))
value := line.valueData
if len(value) == 8 {
sb.WriteString(fmt.Sprintf("%06x: %04x %04x %04x %04x %s %s\n", line.pos, kind[:4], kind[4:], value[:4], value[4:], line.kind, line.value))
} else {
sb.WriteString(fmt.Sprintf("%06x: %04x %04x %s %s\n", line.pos, kind[:4], kind[4:], line.kind, line.value))
}
}
}
}
return sb.String(), err
}
func nextUint32(r io.Reader, pos *int) ([]byte, uint32, error) {
i := make([]byte, 4)
_, err := r.Read(i)
if err != nil {
return i, 0, err
}
p := *pos + 4
*pos = p
return i, binary.LittleEndian.Uint32(i), nil
}
func nextUint64(r io.Reader, pos *int) ([]byte, uint64, error) {
i := make([]byte, 8)
_, err := r.Read(i)
if err != nil {
return i, 0, err
}
p := *pos + 8
*pos = p
return i, binary.LittleEndian.Uint64(i), nil
}
func nextIdent(r io.Reader, pos *int) ([]byte, Checksum, error) {
i := make([]byte, 48)
if _, err := r.Read(i); err != nil {
return i, Checksum{}, err
}
p := *pos + 48
*pos = p
return i, Checksum(i), nil
}
func intToKind(i uint64) string {
switch Kind(i) {
case KindHTTPGet:
return "http"
case KindTar:
return "tar "
case KindExec:
return "exec"
case KindExecNet:
return "exen"
case KindFile:
return "file"
default:
return fmt.Sprintf("$%d ", i-KindCustomOffset)
}
}
func headerSpacing(showHeader bool) string {
if showHeader {
return " "
}
return ""
}

203
internal/pkg/dir.go Normal file
View File

@@ -0,0 +1,203 @@
package pkg
import (
"crypto/sha512"
"encoding/binary"
"errors"
"io"
"io/fs"
"math"
"os"
"path/filepath"
"syscall"
"hakurei.app/container/check"
)
// FlatEntry is a directory entry to be encoded for [Flatten].
type FlatEntry struct {
Mode fs.FileMode // file mode bits
Path string // pathname of the file
Data []byte // file content or symlink destination
}
/*
| mode uint32 | path_sz uint32 |
| data_sz uint64 |
| path string |
| data []byte |
*/
// Encode encodes the entry for transmission or hashing.
func (ent *FlatEntry) Encode(w io.Writer) (n int, err error) {
pPathSize := alignSize(len(ent.Path))
if pPathSize > math.MaxUint32 {
return 0, syscall.E2BIG
}
pDataSize := alignSize(len(ent.Data))
payload := make([]byte, wordSize*2+pPathSize+pDataSize)
binary.LittleEndian.PutUint32(payload, uint32(ent.Mode))
binary.LittleEndian.PutUint32(payload[wordSize/2:], uint32(len(ent.Path)))
binary.LittleEndian.PutUint64(payload[wordSize:], uint64(len(ent.Data)))
copy(payload[wordSize*2:], ent.Path)
copy(payload[wordSize*2+pPathSize:], ent.Data)
return w.Write(payload)
}
// ErrInsecurePath is returned by [FlatEntry.Decode] if validation is requested
// and a nonlocal path is encountered in the stream.
var ErrInsecurePath = errors.New("insecure file path")
// Decode decodes the entry from its representation produced by Encode.
func (ent *FlatEntry) Decode(r io.Reader, validate bool) (n int, err error) {
var nr int
header := make([]byte, wordSize*2)
nr, err = r.Read(header)
n += nr
if err != nil {
if errors.Is(err, io.EOF) && n != 0 {
err = io.ErrUnexpectedEOF
}
return
}
ent.Mode = fs.FileMode(binary.LittleEndian.Uint32(header))
pathSize := int(binary.LittleEndian.Uint32(header[wordSize/2:]))
pPathSize := alignSize(pathSize)
dataSize := int(binary.LittleEndian.Uint64(header[wordSize:]))
pDataSize := alignSize(dataSize)
buf := make([]byte, pPathSize+pDataSize)
nr, err = r.Read(buf)
n += nr
if err != nil {
if errors.Is(err, io.EOF) {
if nr != len(buf) {
err = io.ErrUnexpectedEOF
return
}
} else {
return
}
}
ent.Path = string(buf[:pathSize])
if ent.Mode.IsDir() {
ent.Data = nil
} else {
ent.Data = buf[pPathSize : pPathSize+dataSize]
}
if validate && !filepath.IsLocal(ent.Path) {
err = ErrInsecurePath
}
return
}
// DirScanner provides an efficient interface for reading a stream of encoded
// [FlatEntry]. Successive calls to the Scan method will step through the
// entries in the stream.
type DirScanner struct {
// Underlying reader to scan [FlatEntry] representations from.
r io.Reader
// First non-EOF I/O error, returned by the Err method.
err error
// Entry to store results in. Its address is returned by the Entry method
// and is updated on every call to Scan.
ent FlatEntry
// Validate pathnames during decoding.
validate bool
}
// NewDirScanner returns the address of a new instance of [DirScanner] reading
// from r. The caller must no longer read from r after this function returns.
func NewDirScanner(r io.Reader, validate bool) *DirScanner {
return &DirScanner{r: r, validate: validate}
}
// Err returns the first non-EOF I/O error.
func (s *DirScanner) Err() error {
if errors.Is(s.err, io.EOF) {
return nil
}
return s.err
}
// Entry returns the address to the [FlatEntry] value storing the last result.
func (s *DirScanner) Entry() *FlatEntry { return &s.ent }
// Scan advances to the next [FlatEntry].
func (s *DirScanner) Scan() bool {
if s.err != nil {
return false
}
var n int
n, s.err = s.ent.Decode(s.r, s.validate)
if errors.Is(s.err, io.EOF) {
return n != 0
}
return s.err == nil
}
// Flatten writes a deterministic representation of the contents of fsys to w.
// The resulting data can be hashed to produce a deterministic checksum for the
// directory.
func Flatten(fsys fs.FS, root string, w io.Writer) (n int, err error) {
var nr int
err = fs.WalkDir(fsys, root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
var fi fs.FileInfo
fi, err = d.Info()
if err != nil {
return err
}
ent := FlatEntry{
Path: path,
Mode: fi.Mode(),
}
if ent.Mode.IsRegular() {
if ent.Data, err = fs.ReadFile(fsys, path); err != nil {
return err
}
} else if ent.Mode&fs.ModeSymlink != 0 {
var newpath string
if newpath, err = fs.ReadLink(fsys, path); err != nil {
return err
}
ent.Data = []byte(newpath)
} else if !ent.Mode.IsDir() {
return InvalidFileModeError(ent.Mode)
}
nr, err = ent.Encode(w)
n += nr
return err
})
return
}
// HashFS returns a checksum produced by hashing the result of [Flatten].
func HashFS(buf *Checksum, fsys fs.FS, root string) error {
h := sha512.New384()
if _, err := Flatten(fsys, root, h); err != nil {
return err
}
h.Sum(buf[:0])
return nil
}
// HashDir returns a checksum produced by hashing the result of [Flatten].
func HashDir(buf *Checksum, pathname *check.Absolute) error {
return HashFS(buf, os.DirFS(pathname.String()), ".")
}

570
internal/pkg/dir_test.go Normal file
View File

@@ -0,0 +1,570 @@
package pkg_test
import (
"bytes"
"io/fs"
"reflect"
"testing"
"testing/fstest"
"hakurei.app/internal/pkg"
)
func TestFlatten(t *testing.T) {
t.Parallel()
testCases := []struct {
name string
fsys fs.FS
entries []pkg.FlatEntry
sum pkg.Checksum
err error
}{
{"bad type", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"invalid": {Mode: fs.ModeCharDevice | 0400},
}, nil, pkg.Checksum{}, pkg.InvalidFileModeError(
fs.ModeCharDevice | 0400,
)},
{"empty", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"identifier": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("E4vEZKhCcL2gPZ2Tt59FS3lDng-d_2SKa2i5G_RbDfwGn6EemptFaGLPUDiOa94C"), nil},
{"sample cache file", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX": {Mode: 0400, Data: []byte{0}},
"checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq": {Mode: 0400, Data: []byte{0, 0, 0, 0, 0xad, 0xb, 0, 4, 0xfe, 0xfe, 0, 0, 0xfe, 0xca, 0, 0}},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX")},
"identifier/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq")},
"identifier/cafebabecafebabecafebabecafebabecafebabecafebabecafebabecafebabe": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq")},
"identifier/deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq")},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: 0400, Path: "checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq", Data: []byte{0, 0, 0, 0, 0xad, 0xb, 0, 4, 0xfe, 0xfe, 0, 0, 0xfe, 0xca, 0, 0}},
{Mode: 0400, Path: "checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX", Data: []byte{0}},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq", Data: []byte("../checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/cafebabecafebabecafebabecafebabecafebabecafebabecafebabecafebabe", Data: []byte("../checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef", Data: []byte("../checksum/0bSFPu5Tnd-2Jj0Mv6co23PW2t3BmHc7eLFj9TgY3eIBg8zislo7xZYNBqovVLcq")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX", Data: []byte("../checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX")},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("St9rlE-mGZ5gXwiv_hzQ_B8bZP-UUvSNmf4nHUZzCMOumb6hKnheZSe0dmnuc4Q2"), nil},
{"sample http get cure", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/fLYGIMHgN1louE-JzITJZJo2SDniPu-IHBXubtvQWFO-hXnDVKNuscV7-zlyr5fU": {Mode: 0400, Data: []byte("\x7f\xe1\x69\xa2\xdd\x63\x96\x26\x83\x79\x61\x8b\xf0\x3f\xd5\x16\x9a\x39\x3a\xdb\xcf\xb1\xbc\x8d\x33\xff\x75\xee\x62\x56\xa9\xf0\x27\xac\x13\x94\x69")},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/oM-2pUlk-mOxK1t3aMWZer69UdOQlAXiAgMrpZ1476VoOqpYVP1aGFS9_HYy-D8_": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/fLYGIMHgN1louE-JzITJZJo2SDniPu-IHBXubtvQWFO-hXnDVKNuscV7-zlyr5fU")},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: 0400, Path: "checksum/fLYGIMHgN1louE-JzITJZJo2SDniPu-IHBXubtvQWFO-hXnDVKNuscV7-zlyr5fU", Data: []byte("\x7f\xe1\x69\xa2\xdd\x63\x96\x26\x83\x79\x61\x8b\xf0\x3f\xd5\x16\x9a\x39\x3a\xdb\xcf\xb1\xbc\x8d\x33\xff\x75\xee\x62\x56\xa9\xf0\x27\xac\x13\x94\x69")},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/oM-2pUlk-mOxK1t3aMWZer69UdOQlAXiAgMrpZ1476VoOqpYVP1aGFS9_HYy-D8_", Data: []byte("../checksum/fLYGIMHgN1louE-JzITJZJo2SDniPu-IHBXubtvQWFO-hXnDVKNuscV7-zlyr5fU")},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("L_0RFHpr9JUS4Zp14rz2dESSRvfLzpvqsLhR1-YjQt8hYlmEdVl7vI3_-v8UNPKs"), nil},
{"sample directory step simple", fstest.MapFS{
".": {Mode: fs.ModeDir | 0500},
"check": {Mode: 0400, Data: []byte{0, 0}},
"lib": {Mode: fs.ModeDir | 0700},
"lib/libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
"lib/pkgconfig": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0500, Path: "."},
{Mode: 0400, Path: "check", Data: []byte{0, 0}},
{Mode: fs.ModeDir | 0700, Path: "lib"},
{Mode: fs.ModeSymlink | 0777, Path: "lib/libedac.so", Data: []byte("/proc/nonexistent/libedac.so")},
{Mode: fs.ModeDir | 0700, Path: "lib/pkgconfig"},
}, pkg.MustDecode("qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b"), nil},
{"sample directory step garbage", fstest.MapFS{
".": {Mode: fs.ModeDir | 0500},
"lib": {Mode: fs.ModeDir | 0500},
"lib/check": {Mode: 0400, Data: []byte{}},
"lib/pkgconfig": {Mode: fs.ModeDir | 0500},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0500, Path: "."},
{Mode: fs.ModeDir | 0500, Path: "lib"},
{Mode: 0400, Path: "lib/check", Data: []byte{}},
{Mode: fs.ModeDir | 0500, Path: "lib/pkgconfig"},
}, pkg.MustDecode("CUx-3hSbTWPsbMfDhgalG4Ni_GmR9TnVX8F99tY_P5GtkYvczg9RrF5zO0jX9XYT"), nil},
{"sample directory", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b": {Mode: fs.ModeDir | 0500},
"checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/check": {Mode: 0400, Data: []byte{0, 0}},
"checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/lib": {Mode: fs.ModeDir | 0700},
"checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/lib/pkgconfig": {Mode: fs.ModeDir | 0700},
"checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/lib/libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/HnySzeLQvSBZuTUcvfmLEX_OmH4yJWWH788NxuLuv7kVn8_uPM6Ks4rqFWM2NZJY": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b")},
"identifier/Zx5ZG9BAwegNT3zQwCySuI2ktCXxNgxirkGLFjW4FW06PtojYVaCdtEw8yuntPLa": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b")},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b"},
{Mode: 0400, Path: "checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/check", Data: []byte{0, 0}},
{Mode: fs.ModeDir | 0700, Path: "checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/lib"},
{Mode: fs.ModeSymlink | 0777, Path: "checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/lib/libedac.so", Data: []byte("/proc/nonexistent/libedac.so")},
{Mode: fs.ModeDir | 0700, Path: "checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b/lib/pkgconfig"},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/HnySzeLQvSBZuTUcvfmLEX_OmH4yJWWH788NxuLuv7kVn8_uPM6Ks4rqFWM2NZJY", Data: []byte("../checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/Zx5ZG9BAwegNT3zQwCySuI2ktCXxNgxirkGLFjW4FW06PtojYVaCdtEw8yuntPLa", Data: []byte("../checksum/qRN6in76LndiiOZJheHkwyW8UT1N5-f-bXvHfDvwrMw2fSkOoZdh8pWE1qhLk65b")},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("WVpvsVqVKg9Nsh744x57h51AuWUoUR2nnh8Md-EYBQpk6ziyTuUn6PLtF2e0Eu_d"), nil},
{"sample tar step unpack", fstest.MapFS{
".": {Mode: fs.ModeDir | 0500},
"checksum": {Mode: fs.ModeDir | 0500},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP": {Mode: fs.ModeDir | 0500},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/check": {Mode: 0400, Data: []byte{0, 0}},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib": {Mode: fs.ModeDir | 0500},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/pkgconfig": {Mode: fs.ModeDir | 0500},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
"identifier": {Mode: fs.ModeDir | 0500},
"identifier/HnySzeLQvSBZuTUcvfmLEX_OmH4yJWWH788NxuLuv7kVn8_uPM6Ks4rqFWM2NZJY": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
"identifier/Zx5ZG9BAwegNT3zQwCySuI2ktCXxNgxirkGLFjW4FW06PtojYVaCdtEw8yuntPLa": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
"work": {Mode: fs.ModeDir | 0500},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0500, Path: "."},
{Mode: fs.ModeDir | 0500, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP"},
{Mode: 0400, Path: "checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/check", Data: []byte{0, 0}},
{Mode: fs.ModeDir | 0500, Path: "checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib"},
{Mode: fs.ModeSymlink | 0777, Path: "checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/libedac.so", Data: []byte("/proc/nonexistent/libedac.so")},
{Mode: fs.ModeDir | 0500, Path: "checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/pkgconfig"},
{Mode: fs.ModeDir | 0500, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/HnySzeLQvSBZuTUcvfmLEX_OmH4yJWWH788NxuLuv7kVn8_uPM6Ks4rqFWM2NZJY", Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/Zx5ZG9BAwegNT3zQwCySuI2ktCXxNgxirkGLFjW4FW06PtojYVaCdtEw8yuntPLa", Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
{Mode: fs.ModeDir | 0500, Path: "work"},
}, pkg.MustDecode("cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM"), nil},
{"sample tar", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM": {Mode: fs.ModeDir | 0500},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum": {Mode: fs.ModeDir | 0500},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP": {Mode: fs.ModeDir | 0500},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/check": {Mode: 0400, Data: []byte{0, 0}},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib": {Mode: fs.ModeDir | 0500},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/pkgconfig": {Mode: fs.ModeDir | 0500},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/identifier": {Mode: fs.ModeDir | 0500},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/identifier/HnySzeLQvSBZuTUcvfmLEX_OmH4yJWWH788NxuLuv7kVn8_uPM6Ks4rqFWM2NZJY": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/identifier/Zx5ZG9BAwegNT3zQwCySuI2ktCXxNgxirkGLFjW4FW06PtojYVaCdtEw8yuntPLa": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
"checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/work": {Mode: fs.ModeDir | 0500},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/W5S65DEhawz_WKaok5NjUKLmnD9dNl5RPauNJjcOVcB3VM4eGhSaLGmXbL8vZpiw": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM")},
"identifier/rg7F1D5hwv6o4xctjD5zDq4i5MD0mArTsUIWfhUbik8xC6Bsyt3mjXXOm3goojTz": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM"},
{Mode: fs.ModeDir | 0500, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP"},
{Mode: 0400, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/check", Data: []byte{0, 0}},
{Mode: fs.ModeDir | 0500, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib"},
{Mode: fs.ModeSymlink | 0777, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/libedac.so", Data: []byte("/proc/nonexistent/libedac.so")},
{Mode: fs.ModeDir | 0500, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/pkgconfig"},
{Mode: fs.ModeDir | 0500, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/identifier/HnySzeLQvSBZuTUcvfmLEX_OmH4yJWWH788NxuLuv7kVn8_uPM6Ks4rqFWM2NZJY", Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
{Mode: fs.ModeSymlink | 0777, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/identifier/Zx5ZG9BAwegNT3zQwCySuI2ktCXxNgxirkGLFjW4FW06PtojYVaCdtEw8yuntPLa", Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
{Mode: fs.ModeDir | 0500, Path: "checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM/work"},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/W5S65DEhawz_WKaok5NjUKLmnD9dNl5RPauNJjcOVcB3VM4eGhSaLGmXbL8vZpiw", Data: []byte("../checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/rg7F1D5hwv6o4xctjD5zDq4i5MD0mArTsUIWfhUbik8xC6Bsyt3mjXXOm3goojTz", Data: []byte("../checksum/cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("NQTlc466JmSVLIyWklm_u8_g95jEEb98PxJU-kjwxLpfdjwMWJq0G8ze9R4Vo1Vu"), nil},
{"sample tar expand step unpack", fstest.MapFS{
".": {Mode: fs.ModeDir | 0500},
"libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0500, Path: "."},
{Mode: fs.ModeSymlink | 0777, Path: "libedac.so", Data: []byte("/proc/nonexistent/libedac.so")},
}, pkg.MustDecode("CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN"), nil},
{"sample tar expand", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN": {Mode: fs.ModeDir | 0500},
"checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN/libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/W5S65DEhawz_WKaok5NjUKLmnD9dNl5RPauNJjcOVcB3VM4eGhSaLGmXbL8vZpiw": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN")},
"identifier/_v1blm2h-_KA-dVaawdpLas6MjHc6rbhhFS8JWwx8iJxZGUu8EBbRrhr5AaZ9PJL": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN"},
{Mode: fs.ModeSymlink | 0777, Path: "checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN/libedac.so", Data: []byte("/proc/nonexistent/libedac.so")},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/W5S65DEhawz_WKaok5NjUKLmnD9dNl5RPauNJjcOVcB3VM4eGhSaLGmXbL8vZpiw", Data: []byte("../checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/_v1blm2h-_KA-dVaawdpLas6MjHc6rbhhFS8JWwx8iJxZGUu8EBbRrhr5AaZ9PJL", Data: []byte("../checksum/CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("hSoSSgCYTNonX3Q8FjvjD1fBl-E-BQyA6OTXro2OadXqbST4tZ-akGXszdeqphRe"), nil},
{"testtool", fstest.MapFS{
".": {Mode: fs.ModeDir | 0500},
"check": {Mode: 0400, Data: []byte{0}},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0500, Path: "."},
{Mode: 0400, Path: "check", Data: []byte{0}},
}, pkg.MustDecode("GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9"), nil},
{"sample exec container", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9": {Mode: fs.ModeDir | 0500},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check": {Mode: 0400, Data: []byte{0}},
"checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU": {Mode: fs.ModeDir | 0500},
"checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb": {Mode: 0400, Data: []byte{}},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/_gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb")},
"identifier/dztPS6jRjiZtCF4_p8AzfnxGp6obkhrgFVsxdodbKWUoAEVtDz3MykepJB4kI_ks": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
"identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9"},
{Mode: 0400, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check", Data: []byte{0}},
{Mode: fs.ModeDir | 0500, Path: "checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU"},
{Mode: 0400, Path: "checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb", Data: []byte{}},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/_gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", Data: []byte("../checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/dztPS6jRjiZtCF4_p8AzfnxGp6obkhrgFVsxdodbKWUoAEVtDz3MykepJB4kI_ks", Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK", Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("Q5DluWQCAeohLoiGRImurwFp3vdz9IfQCoj7Fuhh73s4KQPRHpEQEnHTdNHmB8Fx"), nil},
{"testtool net", fstest.MapFS{
".": {Mode: fs.ModeDir | 0500},
"check": {Mode: 0400, Data: []byte("net")},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0500, Path: "."},
{Mode: 0400, Path: "check", Data: []byte("net")},
}, pkg.MustDecode("a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W"), nil},
{"sample exec net container", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU": {Mode: fs.ModeDir | 0500},
"checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb": {Mode: 0400, Data: []byte{}},
"checksum/a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W": {Mode: fs.ModeDir | 0500},
"checksum/a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W/check": {Mode: 0400, Data: []byte("net")},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/G8qPxD9puvvoOVV7lrT80eyDeIl3G_CCFoKw12c8mCjMdG1zF7NEPkwYpNubClK3": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W")},
"identifier/_gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb")},
"identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU"},
{Mode: 0400, Path: "checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb", Data: []byte{}},
{Mode: fs.ModeDir | 0500, Path: "checksum/a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W"},
{Mode: 0400, Path: "checksum/a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W/check", Data: []byte("net")},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/G8qPxD9puvvoOVV7lrT80eyDeIl3G_CCFoKw12c8mCjMdG1zF7NEPkwYpNubClK3", Data: []byte("../checksum/a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/_gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", Data: []byte("../checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK", Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("bPYvvqxpfV7xcC1EptqyKNK1klLJgYHMDkzBcoOyK6j_Aj5hb0mXNPwTwPSK5F6Z"), nil},
{"sample exec container overlay root", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9": {Mode: fs.ModeDir | 0500},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check": {Mode: 0400, Data: []byte{0}},
"checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU": {Mode: fs.ModeDir | 0500},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/RdMA-mubnrHuu3Ky1wWyxauSYCO0ZH_zCPUj3uDHqkfwv5sGcByoF_g5PjlGiClb": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
"identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9"},
{Mode: 0400, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check", Data: []byte{0}},
{Mode: fs.ModeDir | 0500, Path: "checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU"},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/RdMA-mubnrHuu3Ky1wWyxauSYCO0ZH_zCPUj3uDHqkfwv5sGcByoF_g5PjlGiClb", Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK", Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("PO2DSSCa4yoSgEYRcCSZfQfwow1yRigL3Ry-hI0RDI4aGuFBha-EfXeSJnG_5_Rl"), nil},
{"sample exec container overlay work", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9": {Mode: fs.ModeDir | 0500},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check": {Mode: 0400, Data: []byte{0}},
"checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU": {Mode: fs.ModeDir | 0500},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/5hlaukCirnXE4W_RSLJFOZN47Z5RiHnacXzdFp_70cLgiJUGR6cSb_HaFftkzi0-": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
"identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9"},
{Mode: 0400, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check", Data: []byte{0}},
{Mode: fs.ModeDir | 0500, Path: "checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU"},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/5hlaukCirnXE4W_RSLJFOZN47Z5RiHnacXzdFp_70cLgiJUGR6cSb_HaFftkzi0-", Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK", Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("iaRt6l_Wm2n-h5UsDewZxQkCmjZjyL8r7wv32QT2kyV55-Lx09Dq4gfg9BiwPnKs"), nil},
{"sample exec container multiple layers", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9": {Mode: fs.ModeDir | 0500},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check": {Mode: 0400, Data: []byte{0}},
"checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU": {Mode: fs.ModeDir | 0500},
"checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb": {Mode: 0400, Data: []byte{}},
"checksum/nY_CUdiaUM1OL4cPr5TS92FCJ3rCRV7Hm5oVTzAvMXwC03_QnTRfQ5PPs7mOU9fK": {Mode: fs.ModeDir | 0500},
"checksum/nY_CUdiaUM1OL4cPr5TS92FCJ3rCRV7Hm5oVTzAvMXwC03_QnTRfQ5PPs7mOU9fK/check": {Mode: 0400, Data: []byte("layers")},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/_gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb")},
"identifier/B-kc5iJMx8GtlCua4dz6BiJHnDAOUfPjgpbKq4e-QEn0_CZkSYs3fOA1ve06qMs2": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/nY_CUdiaUM1OL4cPr5TS92FCJ3rCRV7Hm5oVTzAvMXwC03_QnTRfQ5PPs7mOU9fK")},
"identifier/p1t_drXr34i-jZNuxDMLaMOdL6tZvQqhavNafGynGqxOZoXAUTSn7kqNh3Ovv3DT": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
"identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9"},
{Mode: 0400, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check", Data: []byte{0}},
{Mode: fs.ModeDir | 0500, Path: "checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU"},
{Mode: 0400, Path: "checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb", Data: []byte{}},
{Mode: fs.ModeDir | 0500, Path: "checksum/nY_CUdiaUM1OL4cPr5TS92FCJ3rCRV7Hm5oVTzAvMXwC03_QnTRfQ5PPs7mOU9fK"},
{Mode: 0400, Path: "checksum/nY_CUdiaUM1OL4cPr5TS92FCJ3rCRV7Hm5oVTzAvMXwC03_QnTRfQ5PPs7mOU9fK/check", Data: []byte("layers")},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/B-kc5iJMx8GtlCua4dz6BiJHnDAOUfPjgpbKq4e-QEn0_CZkSYs3fOA1ve06qMs2", Data: []byte("../checksum/nY_CUdiaUM1OL4cPr5TS92FCJ3rCRV7Hm5oVTzAvMXwC03_QnTRfQ5PPs7mOU9fK")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/_gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", Data: []byte("../checksum/OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/p1t_drXr34i-jZNuxDMLaMOdL6tZvQqhavNafGynGqxOZoXAUTSn7kqNh3Ovv3DT", Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK", Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("O2YzyR7IUGU5J2CADy0hUZ3A5NkP_Vwzs4UadEdn2oMZZVWRtH0xZGJ3HXiimTnZ"), nil},
{"sample exec container layer promotion", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9": {Mode: fs.ModeDir | 0500},
"checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check": {Mode: 0400, Data: []byte{0}},
"checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU": {Mode: fs.ModeDir | 0500},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/kvJIqZo5DKFOxC2ZQ-8_nPaQzEAz9cIm3p6guO-uLqm-xaiPu7oRkSnsu411jd_U": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
"identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
"identifier/xXTIYcXmgJWNLC91c417RRrNM9cjELwEZHpGvf8Fk_GNP5agRJp_SicD0w9aMeLJ": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
"temp": {Mode: fs.ModeDir | 0700},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: fs.ModeDir | 0500, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9"},
{Mode: 0400, Path: "checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9/check", Data: []byte{0}},
{Mode: fs.ModeDir | 0500, Path: "checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU"},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/kvJIqZo5DKFOxC2ZQ-8_nPaQzEAz9cIm3p6guO-uLqm-xaiPu7oRkSnsu411jd_U", Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/vjz1MHPcGBKV7sjcs8jQP3cqxJ1hgPTiQBMCEHP9BGXjGxd-tJmEmXKaStObo5gK", Data: []byte("../checksum/MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU")},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/xXTIYcXmgJWNLC91c417RRrNM9cjELwEZHpGvf8Fk_GNP5agRJp_SicD0w9aMeLJ", Data: []byte("../checksum/GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9")},
{Mode: fs.ModeDir | 0700, Path: "temp"},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("3EaW6WibLi9gl03_UieiFPaFcPy5p4x3JPxrnLJxGaTI-bh3HU9DK9IMx7c3rrNm"), nil},
{"sample file short", fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX": {Mode: 0400, Data: []byte{0}},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/3376ALA7hIUm2LbzH2fDvRezgzod1eTK_G6XjyOgbM2u-6swvkFaF0BOwSl_juBi": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX")},
"work": {Mode: fs.ModeDir | 0700},
}, []pkg.FlatEntry{
{Mode: fs.ModeDir | 0700, Path: "."},
{Mode: fs.ModeDir | 0700, Path: "checksum"},
{Mode: 0400, Path: "checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX", Data: []byte{0}},
{Mode: fs.ModeDir | 0700, Path: "identifier"},
{Mode: fs.ModeSymlink | 0777, Path: "identifier/3376ALA7hIUm2LbzH2fDvRezgzod1eTK_G6XjyOgbM2u-6swvkFaF0BOwSl_juBi", Data: []byte("../checksum/vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX")},
{Mode: fs.ModeDir | 0700, Path: "work"},
}, pkg.MustDecode("iR6H5OIsyOW4EwEgtm9rGzGF6DVtyHLySEtwnFE8bnus9VJcoCbR4JIek7Lw-vwT"), nil},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
t.Run("roundtrip", func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
if _, err := pkg.Flatten(
tc.fsys,
".",
&buf,
); !reflect.DeepEqual(err, tc.err) {
t.Fatalf("Flatten: error = %v, want %v", err, tc.err)
} else if tc.err != nil {
return
}
s := pkg.NewDirScanner(bytes.NewReader(buf.Bytes()), true)
var got []pkg.FlatEntry
for s.Scan() {
got = append(got, *s.Entry())
}
if err := s.Err(); err != nil {
t.Fatalf("Err: error = %v", err)
}
if !reflect.DeepEqual(got, tc.entries) {
t.Fatalf("Scan: %#v, want %#v", got, tc.entries)
}
})
if tc.err != nil {
return
}
t.Run("hash", func(t *testing.T) {
t.Parallel()
var got pkg.Checksum
if err := pkg.HashFS(&got, tc.fsys, "."); err != nil {
t.Fatalf("HashFS: error = %v", err)
} else if got != tc.sum {
t.Fatalf("HashFS: %v", &pkg.ChecksumMismatchError{
Got: got,
Want: tc.sum,
})
}
})
})
}
}

499
internal/pkg/exec.go Normal file
View File

@@ -0,0 +1,499 @@
package pkg
import (
"bufio"
"context"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path"
"slices"
"strconv"
"syscall"
"time"
"unique"
"hakurei.app/container"
"hakurei.app/container/check"
"hakurei.app/container/fhs"
"hakurei.app/container/seccomp"
"hakurei.app/container/std"
"hakurei.app/message"
)
// AbsWork is the container pathname [CureContext.GetWorkDir] is mounted on.
var AbsWork = fhs.AbsRoot.Append("work/")
// ExecPath is a slice of [Artifact] and the [check.Absolute] pathname to make
// it available at under in the container.
type ExecPath struct {
// Pathname in the container mount namespace.
P *check.Absolute
// Artifacts to mount on the pathname, must contain at least one [Artifact].
// If there are multiple entries or W is true, P is set up as an overlay
// mount, and entries of A must not implement [FileArtifact].
A []Artifact
// Whether to make the mount point writable via the temp directory.
W bool
}
// layers returns pathnames collected from A deduplicated by checksum.
func (p *ExecPath) layers(f *FContext) []*check.Absolute {
msg := f.GetMessage()
layers := make([]*check.Absolute, 0, len(p.A))
checksums := make(map[unique.Handle[Checksum]]struct{}, len(p.A))
for i := range p.A {
d := p.A[len(p.A)-1-i]
pathname, checksum := f.GetArtifact(d)
if _, ok := checksums[checksum]; ok {
if msg.IsVerbose() {
msg.Verbosef(
"promoted layer %d as %s",
len(p.A)-1-i, reportName(d, f.cache.Ident(d)),
)
}
continue
}
checksums[checksum] = struct{}{}
layers = append(layers, pathname)
}
slices.Reverse(layers)
return layers
}
// Path returns a populated [ExecPath].
func Path(pathname *check.Absolute, writable bool, a ...Artifact) ExecPath {
return ExecPath{pathname, a, writable}
}
// MustPath is like [Path], but takes a string pathname via [check.MustAbs].
func MustPath(pathname string, writable bool, a ...Artifact) ExecPath {
return ExecPath{check.MustAbs(pathname), a, writable}
}
const (
// ExecTimeoutDefault replaces out of range [NewExec] timeout values.
ExecTimeoutDefault = 15 * time.Minute
// ExecTimeoutMax is the arbitrary upper bound of [NewExec] timeout.
ExecTimeoutMax = 48 * time.Hour
)
// An execArtifact is an [Artifact] that produces output by running a program
// part of another [Artifact] in a [container] to produce its output.
//
// Methods of execArtifact does not modify any struct field or underlying arrays
// referred to by slices.
type execArtifact struct {
// Caller-supplied user-facing reporting name, guaranteed to be nonzero
// during initialisation.
name string
// Caller-supplied inner mount points.
paths []ExecPath
// Passed through to [container.Params].
dir *check.Absolute
// Passed through to [container.Params].
env []string
// Passed through to [container.Params].
path *check.Absolute
// Passed through to [container.Params].
args []string
// Duration the initial process is allowed to run. The zero value is
// equivalent to [ExecTimeoutDefault].
timeout time.Duration
// Caller-supplied exclusivity value, returned as is by IsExclusive.
exclusive bool
}
var _ fmt.Stringer = new(execArtifact)
// execNetArtifact is like execArtifact but implements [KnownChecksum] and has
// its resulting container keep the host net namespace.
type execNetArtifact struct {
checksum Checksum
execArtifact
}
var _ KnownChecksum = new(execNetArtifact)
// Checksum returns the caller-supplied checksum.
func (a *execNetArtifact) Checksum() Checksum { return a.checksum }
// Kind returns the hardcoded [Kind] constant.
func (*execNetArtifact) Kind() Kind { return KindExecNet }
// Cure cures the [Artifact] in the container described by the caller. The
// container retains host networking.
func (a *execNetArtifact) Cure(f *FContext) error {
return a.cure(f, true)
}
// NewExec returns a new [Artifact] that executes the program path in a
// container with specified paths bind mounted read-only in order. A private
// instance of /proc and /dev is made available to the container.
//
// The working and temporary directories are both created and mounted writable
// on [AbsWork] and [fhs.AbsTmp] respectively. If one or more paths target
// [AbsWork], the final entry is set up as a writable overlay mount on /work for
// which the upperdir is the host side work directory. In this configuration,
// the W field is ignored, and the program must avoid causing whiteout files to
// be created. Cure fails if upperdir ends up with entries other than directory,
// regular or symlink.
//
// If checksum is non-nil, the resulting [Artifact] implements [KnownChecksum]
// and its container runs in the host net namespace.
//
// The container is allowed to run for the specified duration before the initial
// process and all processes originating from it is terminated. A zero or
// negative timeout value is equivalent tp [ExecTimeoutDefault], a timeout value
// greater than [ExecTimeoutMax] is equivalent to [ExecTimeoutMax].
//
// The user-facing name and exclusivity value are not accessible from the
// container and does not affect curing outcome. Because of this, it is omitted
// from parameter data for computing identifier.
func NewExec(
name string,
checksum *Checksum,
timeout time.Duration,
exclusive bool,
dir *check.Absolute,
env []string,
pathname *check.Absolute,
args []string,
paths ...ExecPath,
) Artifact {
if name == "" {
name = "exec-" + path.Base(pathname.String())
}
if timeout <= 0 {
timeout = ExecTimeoutDefault
}
if timeout > ExecTimeoutMax {
timeout = ExecTimeoutMax
}
a := execArtifact{name, paths, dir, env, pathname, args, timeout, exclusive}
if checksum == nil {
return &a
}
return &execNetArtifact{*checksum, a}
}
// Kind returns the hardcoded [Kind] constant.
func (*execArtifact) Kind() Kind { return KindExec }
// Params writes paths, executable pathname and args.
func (a *execArtifact) Params(ctx *IContext) {
ctx.WriteString(a.name)
ctx.WriteUint32(uint32(len(a.paths)))
for _, p := range a.paths {
if p.P != nil {
ctx.WriteString(p.P.String())
} else {
ctx.WriteString("invalid P\x00")
}
ctx.WriteUint32(uint32(len(p.A)))
for _, d := range p.A {
ctx.WriteIdent(d)
}
if p.W {
ctx.WriteUint32(1)
} else {
ctx.WriteUint32(0)
}
}
ctx.WriteString(a.dir.String())
ctx.WriteUint32(uint32(len(a.env)))
for _, e := range a.env {
ctx.WriteString(e)
}
ctx.WriteString(a.path.String())
ctx.WriteUint32(uint32(len(a.args)))
for _, arg := range a.args {
ctx.WriteString(arg)
}
ctx.WriteUint32(uint32(a.timeout & 0xffffffff))
ctx.WriteUint32(uint32(a.timeout >> 32))
if a.exclusive {
ctx.WriteUint32(1)
} else {
ctx.WriteUint32(0)
}
}
// readExecArtifact interprets IR values and returns the address of execArtifact
// or execNetArtifact.
func readExecArtifact(r *IRReader, net bool) Artifact {
r.DiscardAll()
name := r.ReadString()
sz := r.ReadUint32()
if sz > irMaxDeps {
panic(ErrIRDepend)
}
paths := make([]ExecPath, sz)
for i := range paths {
paths[i].P = check.MustAbs(r.ReadString())
sz = r.ReadUint32()
if sz > irMaxDeps {
panic(ErrIRDepend)
}
paths[i].A = make([]Artifact, sz)
for j := range paths[i].A {
paths[i].A[j] = r.ReadIdent()
}
paths[i].W = r.ReadUint32() != 0
}
dir := check.MustAbs(r.ReadString())
sz = r.ReadUint32()
if sz > irMaxValues {
panic(ErrIRValues)
}
env := make([]string, sz)
for i := range env {
env[i] = r.ReadString()
}
pathname := check.MustAbs(r.ReadString())
sz = r.ReadUint32()
if sz > irMaxValues {
panic(ErrIRValues)
}
args := make([]string, sz)
for i := range args {
args[i] = r.ReadString()
}
timeout := time.Duration(r.ReadUint32())
timeout |= time.Duration(r.ReadUint32()) << 32
exclusive := r.ReadUint32() != 0
checksum, ok := r.Finalise()
var checksumP *Checksum
if net {
if !ok {
panic(ErrExpectedChecksum)
}
checksumVal := checksum.Value()
checksumP = &checksumVal
} else {
if ok {
panic(ErrUnexpectedChecksum)
}
}
return NewExec(
name, checksumP, timeout, exclusive, dir, env, pathname, args, paths...,
)
}
func init() {
register(KindExec,
func(r *IRReader) Artifact { return readExecArtifact(r, false) })
register(KindExecNet,
func(r *IRReader) Artifact { return readExecArtifact(r, true) })
}
// Dependencies returns a slice of all artifacts collected from caller-supplied
// [ExecPath].
func (a *execArtifact) Dependencies() []Artifact {
artifacts := make([][]Artifact, 0, len(a.paths))
for _, p := range a.paths {
artifacts = append(artifacts, p.A)
}
return slices.Concat(artifacts...)
}
// IsExclusive returns the caller-supplied exclusivity value.
func (a *execArtifact) IsExclusive() bool { return a.exclusive }
// String returns the caller-supplied reporting name.
func (a *execArtifact) String() string { return a.name }
// Cure cures the [Artifact] in the container described by the caller.
func (a *execArtifact) Cure(f *FContext) (err error) {
return a.cure(f, false)
}
const (
// execWaitDelay is passed through to [container.Params].
execWaitDelay = time.Nanosecond
)
// scanVerbose prefixes program output for a verbose [message.Msg].
func scanVerbose(
msg message.Msg,
done chan<- struct{},
prefix string,
r io.Reader,
) {
defer close(done)
s := bufio.NewScanner(r)
s.Buffer(
make([]byte, bufio.MaxScanTokenSize),
bufio.MaxScanTokenSize<<12,
)
for s.Scan() {
msg.Verbose(prefix, s.Text())
}
if err := s.Err(); err != nil && !errors.Is(err, os.ErrClosed) {
msg.Verbose("*"+prefix, err)
}
}
// cure is like Cure but allows optional host net namespace. This is used for
// the [KnownChecksum] variant where networking is allowed.
func (a *execArtifact) cure(f *FContext, hostNet bool) (err error) {
overlayWorkIndex := -1
for i, p := range a.paths {
if p.P == nil || len(p.A) == 0 {
return os.ErrInvalid
}
if p.P.Is(AbsWork) {
overlayWorkIndex = i
}
}
var artifactCount int
for _, p := range a.paths {
artifactCount += len(p.A)
}
ctx, cancel := context.WithTimeout(f.Unwrap(), a.timeout)
defer cancel()
z := container.New(ctx, f.GetMessage())
z.WaitDelay = execWaitDelay
z.SeccompPresets |= std.PresetStrict & ^std.PresetDenyNS
z.SeccompFlags |= seccomp.AllowMultiarch
z.ParentPerm = 0700
z.HostNet = hostNet
z.Hostname = "cure"
if z.HostNet {
z.Hostname = "cure-net"
}
z.Uid, z.Gid = (1<<10)-1, (1<<10)-1
if msg := f.GetMessage(); msg.IsVerbose() {
var stdout, stderr io.ReadCloser
if stdout, err = z.StdoutPipe(); err != nil {
return
}
if stderr, err = z.StderrPipe(); err != nil {
_ = stdout.Close()
return
}
defer func() {
if err != nil && !errors.As(err, new(*exec.ExitError)) {
_ = stdout.Close()
_ = stderr.Close()
}
}()
stdoutDone, stderrDone := make(chan struct{}), make(chan struct{})
go scanVerbose(msg, stdoutDone, "("+a.name+":1)", stdout)
go scanVerbose(msg, stderrDone, "("+a.name+":2)", stderr)
defer func() { <-stdoutDone; <-stderrDone }()
}
z.Dir, z.Env, z.Path, z.Args = a.dir, a.env, a.path, a.args
z.Grow(len(a.paths) + 4)
temp, work := f.GetTempDir(), f.GetWorkDir()
for i, b := range a.paths {
if i == overlayWorkIndex {
if err = os.MkdirAll(work.String(), 0700); err != nil {
return
}
tempWork := temp.Append(".work")
if err = os.MkdirAll(tempWork.String(), 0700); err != nil {
return
}
z.Overlay(
AbsWork,
work,
tempWork,
b.layers(f)...,
)
continue
}
if a.paths[i].W {
tempUpper, tempWork := temp.Append(
".upper", strconv.Itoa(i),
), temp.Append(
".work", strconv.Itoa(i),
)
if err = os.MkdirAll(tempUpper.String(), 0700); err != nil {
return
}
if err = os.MkdirAll(tempWork.String(), 0700); err != nil {
return
}
z.Overlay(b.P, tempUpper, tempWork, b.layers(f)...)
} else if len(b.A) == 1 {
pathname, _ := f.GetArtifact(b.A[0])
z.Bind(pathname, b.P, 0)
} else {
z.OverlayReadonly(b.P, b.layers(f)...)
}
}
if overlayWorkIndex < 0 {
z.Bind(
work,
AbsWork,
std.BindWritable|std.BindEnsure,
)
}
z.Bind(
f.GetTempDir(),
fhs.AbsTmp,
std.BindWritable|std.BindEnsure,
)
z.Proc(fhs.AbsProc).Dev(fhs.AbsDev, true)
if err = z.Start(); err != nil {
return
}
if err = z.Serve(); err != nil {
return
}
if err = z.Wait(); err != nil {
return
}
// do not allow empty directories to succeed
for {
err = syscall.Rmdir(work.String())
if err != syscall.EINTR {
break
}
}
if err != nil && errors.Is(err, syscall.ENOTEMPTY) {
err = nil
}
return
}

339
internal/pkg/exec_test.go Normal file
View File

@@ -0,0 +1,339 @@
package pkg_test
//go:generate env CGO_ENABLED=0 go build -tags testtool -o testdata/testtool ./testdata
import (
_ "embed"
"encoding/gob"
"errors"
"net"
"os"
"os/exec"
"slices"
"testing"
"unique"
"hakurei.app/container/check"
"hakurei.app/container/stub"
"hakurei.app/hst"
"hakurei.app/internal/pkg"
)
// testtoolBin is the container test tool binary made available to the
// execArtifact for testing its curing environment.
//
//go:embed testdata/testtool
var testtoolBin []byte
func TestExec(t *testing.T) {
t.Parallel()
wantChecksumOffline := pkg.MustDecode(
"GPa4aBakdSJd7Tz7LYj_VJFoojzyZinmVcG3k6M5xI6CZ821J5sXLhLDDuS47gi9",
)
checkWithCache(t, []cacheTestCase{
{"offline", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
c.SetStrict(true)
testtool, testtoolDestroy := newTesttool()
cureMany(t, c, []cureStep{
{"container", pkg.NewExec(
"exec-offline", nil, 0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool"},
pkg.MustPath("/file", false, newStubFile(
pkg.KindHTTPGet,
pkg.ID{0xfe, 0},
nil,
nil, nil,
)),
pkg.MustPath("/.hakurei", false, &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}),
pkg.MustPath("/opt", false, testtool),
), ignorePathname, wantChecksumOffline, nil},
{"error passthrough", pkg.NewExec(
"", nil, 0, true,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool"},
pkg.MustPath("/proc/nonexistent", false, &stubArtifact{
kind: pkg.KindTar,
params: []byte("doomed artifact"),
cure: func(t *pkg.TContext) error {
return stub.UniqueError(0xcafe)
},
}),
), nil, pkg.Checksum{}, &pkg.DependencyCureError{
{
Ident: unique.Make(pkg.ID(pkg.MustDecode(
"Sowo6oZRmG6xVtUaxB6bDWZhVsqAJsIJWUp0OPKlE103cY0lodx7dem8J-qQF0Z1",
))),
Err: stub.UniqueError(0xcafe),
},
}},
{"invalid paths", pkg.NewExec(
"", nil, 0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool"},
pkg.ExecPath{},
), nil, pkg.Checksum{}, os.ErrInvalid},
})
// check init failure passthrough
var exitError *exec.ExitError
if _, _, err := c.Cure(pkg.NewExec(
"", nil, 0, false,
pkg.AbsWork,
nil,
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool"},
)); !errors.As(err, &exitError) ||
exitError.ExitCode() != hst.ExitFailure {
t.Fatalf("Cure: error = %v, want init exit status 1", err)
}
testtoolDestroy(t, base, c)
}, pkg.MustDecode("Q5DluWQCAeohLoiGRImurwFp3vdz9IfQCoj7Fuhh73s4KQPRHpEQEnHTdNHmB8Fx")},
{"net", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
c.SetStrict(true)
testtool, testtoolDestroy := newTesttool()
wantChecksum := pkg.MustDecode(
"a1F_i9PVQI4qMcoHgTQkORuyWLkC1GLIxOhDt2JpU1NGAxWc5VJzdlfRK-PYBh3W",
)
cureMany(t, c, []cureStep{
{"container", pkg.NewExec(
"exec-net", &wantChecksum, 0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool", "net"},
pkg.MustPath("/file", false, newStubFile(
pkg.KindHTTPGet,
pkg.ID{0xfe, 0},
nil,
nil, nil,
)),
pkg.MustPath("/.hakurei", false, &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}),
pkg.MustPath("/opt", false, testtool),
), ignorePathname, wantChecksum, nil},
})
testtoolDestroy(t, base, c)
}, pkg.MustDecode("bPYvvqxpfV7xcC1EptqyKNK1klLJgYHMDkzBcoOyK6j_Aj5hb0mXNPwTwPSK5F6Z")},
{"overlay root", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
c.SetStrict(true)
testtool, testtoolDestroy := newTesttool()
cureMany(t, c, []cureStep{
{"container", pkg.NewExec(
"exec-overlay-root", nil, 0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1", "HAKUREI_ROOT=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool"},
pkg.MustPath("/", true, &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}),
pkg.MustPath("/opt", false, testtool),
), ignorePathname, wantChecksumOffline, nil},
})
testtoolDestroy(t, base, c)
}, pkg.MustDecode("PO2DSSCa4yoSgEYRcCSZfQfwow1yRigL3Ry-hI0RDI4aGuFBha-EfXeSJnG_5_Rl")},
{"overlay work", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
c.SetStrict(true)
testtool, testtoolDestroy := newTesttool()
cureMany(t, c, []cureStep{
{"container", pkg.NewExec(
"exec-overlay-work", nil, 0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1", "HAKUREI_ROOT=1"},
check.MustAbs("/work/bin/testtool"),
[]string{"testtool"},
pkg.MustPath("/", true, &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}), pkg.MustPath("/work/", false, &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}), pkg.Path(pkg.AbsWork, false /* ignored */, testtool),
), ignorePathname, wantChecksumOffline, nil},
})
testtoolDestroy(t, base, c)
}, pkg.MustDecode("iaRt6l_Wm2n-h5UsDewZxQkCmjZjyL8r7wv32QT2kyV55-Lx09Dq4gfg9BiwPnKs")},
{"multiple layers", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
c.SetStrict(true)
testtool, testtoolDestroy := newTesttool()
cureMany(t, c, []cureStep{
{"container", pkg.NewExec(
"exec-multiple-layers", nil, 0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1", "HAKUREI_ROOT=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool", "layers"},
pkg.MustPath("/", true, &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}, &stubArtifactF{
kind: pkg.KindExec,
params: []byte("test sample with dependencies"),
deps: slices.Repeat([]pkg.Artifact{newStubFile(
pkg.KindHTTPGet,
pkg.ID{0xfe, 0},
nil,
nil, nil,
), &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
// this is queued and might run instead of the other
// one so do not leave it as nil
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}}, 1<<5 /* concurrent cache hits */),
cure: func(f *pkg.FContext) error {
work := f.GetWorkDir()
if err := os.MkdirAll(work.String(), 0700); err != nil {
return err
}
return os.WriteFile(work.Append("check").String(), []byte("layers"), 0400)
},
}),
pkg.MustPath("/opt", false, testtool),
), ignorePathname, wantChecksumOffline, nil},
})
testtoolDestroy(t, base, c)
}, pkg.MustDecode("O2YzyR7IUGU5J2CADy0hUZ3A5NkP_Vwzs4UadEdn2oMZZVWRtH0xZGJ3HXiimTnZ")},
{"overlay layer promotion", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
c.SetStrict(true)
testtool, testtoolDestroy := newTesttool()
cureMany(t, c, []cureStep{
{"container", pkg.NewExec(
"exec-layer-promotion", nil, 0, true,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1", "HAKUREI_ROOT=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool", "promote"},
pkg.MustPath("/", true, &stubArtifact{
kind: pkg.KindTar,
params: []byte("another empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}, &stubArtifact{
kind: pkg.KindTar,
params: []byte("empty directory"),
cure: func(t *pkg.TContext) error {
return os.MkdirAll(t.GetWorkDir().String(), 0700)
},
}),
pkg.MustPath("/opt", false, testtool),
), ignorePathname, wantChecksumOffline, nil},
})
testtoolDestroy(t, base, c)
}, pkg.MustDecode("3EaW6WibLi9gl03_UieiFPaFcPy5p4x3JPxrnLJxGaTI-bh3HU9DK9IMx7c3rrNm")},
})
}
// newTesttool returns an [Artifact] that cures into testtoolBin. The returned
// function must be called at the end of the test but not deferred.
func newTesttool() (
testtool pkg.Artifact,
testtoolDestroy func(t *testing.T, base *check.Absolute, c *pkg.Cache),
) {
// testtoolBin is built during go:generate and is not deterministic
testtool = overrideIdent{pkg.ID{0xfe, 0xff}, &stubArtifact{
kind: pkg.KindTar,
cure: func(t *pkg.TContext) error {
work := t.GetWorkDir()
if err := os.MkdirAll(
work.Append("bin").String(),
0700,
); err != nil {
return err
}
if ift, err := net.Interfaces(); err != nil {
return err
} else {
var f *os.File
if f, err = os.Create(t.GetWorkDir().Append(
"ift",
).String()); err != nil {
return err
} else {
err = gob.NewEncoder(f).Encode(ift)
closeErr := f.Close()
if err != nil {
return err
}
if closeErr != nil {
return closeErr
}
}
}
return os.WriteFile(t.GetWorkDir().Append(
"bin",
"testtool",
).String(), testtoolBin, 0500)
},
}}
testtoolDestroy = newDestroyArtifactFunc(testtool)
return
}

81
internal/pkg/file.go Normal file
View File

@@ -0,0 +1,81 @@
package pkg
import (
"bytes"
"crypto/sha512"
"fmt"
"io"
)
// A fileArtifact is an [Artifact] that cures into data known ahead of time.
type fileArtifact []byte
var _ KnownChecksum = new(fileArtifact)
// fileArtifactNamed embeds fileArtifact alongside a caller-supplied name.
type fileArtifactNamed struct {
fileArtifact
// Caller-supplied user-facing reporting name.
name string
}
var _ fmt.Stringer = new(fileArtifactNamed)
var _ KnownChecksum = new(fileArtifactNamed)
// String returns the caller-supplied reporting name.
func (a *fileArtifactNamed) String() string { return a.name }
// Params writes the caller-supplied reporting name and the file body.
func (a *fileArtifactNamed) Params(ctx *IContext) {
ctx.WriteString(a.name)
ctx.Write(a.fileArtifact)
}
// NewFile returns a [FileArtifact] that cures into a caller-supplied byte slice.
//
// Caller must not modify data after NewFile returns.
func NewFile(name string, data []byte) FileArtifact {
f := fileArtifact(data)
if name != "" {
return &fileArtifactNamed{f, name}
}
return &f
}
// Kind returns the hardcoded [Kind] constant.
func (*fileArtifact) Kind() Kind { return KindFile }
// Params writes an empty string and the file body.
func (a *fileArtifact) Params(ctx *IContext) {
ctx.WriteString("")
ctx.Write(*a)
}
func init() {
register(KindFile, func(r *IRReader) Artifact {
name := r.ReadString()
data := r.ReadStringBytes()
if _, ok := r.Finalise(); !ok {
panic(ErrExpectedChecksum)
}
return NewFile(name, data)
})
}
// Dependencies returns a nil slice.
func (*fileArtifact) Dependencies() []Artifact { return nil }
// IsExclusive returns false: Cure returns a prepopulated buffer.
func (*fileArtifact) IsExclusive() bool { return false }
// Checksum computes and returns the checksum of caller-supplied data.
func (a *fileArtifact) Checksum() Checksum {
h := sha512.New384()
h.Write(*a)
return Checksum(h.Sum(nil))
}
// Cure returns the caller-supplied data.
func (a *fileArtifact) Cure(*RContext) (io.ReadCloser, error) {
return io.NopCloser(bytes.NewReader(*a)), nil
}

29
internal/pkg/file_test.go Normal file
View File

@@ -0,0 +1,29 @@
package pkg_test
import (
"testing"
"hakurei.app/container/check"
"hakurei.app/internal/pkg"
)
func TestFile(t *testing.T) {
t.Parallel()
checkWithCache(t, []cacheTestCase{
{"file", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
c.SetStrict(true)
cureMany(t, c, []cureStep{
{"short", pkg.NewFile("null", []byte{0}), base.Append(
"identifier",
"3376ALA7hIUm2LbzH2fDvRezgzod1eTK_G6XjyOgbM2u-6swvkFaF0BOwSl_juBi",
), pkg.MustDecode(
"vsAhtPNo4waRNOASwrQwcIPTqb3SBuJOXw2G4T1mNmVZM-wrQTRllmgXqcIIoRcX",
), nil},
})
}, pkg.MustDecode(
"iR6H5OIsyOW4EwEgtm9rGzGF6DVtyHLySEtwnFE8bnus9VJcoCbR4JIek7Lw-vwT",
)},
})
}

762
internal/pkg/ir.go Normal file
View File

@@ -0,0 +1,762 @@
package pkg
import (
"bufio"
"bytes"
"context"
"crypto/sha512"
"encoding/binary"
"errors"
"fmt"
"io"
"slices"
"strconv"
"syscall"
"unique"
"unsafe"
)
// wordSize is the boundary which binary segments are always aligned to.
const wordSize = 8
// alignSize returns the padded size for aligning sz.
func alignSize(sz int) int {
return sz + (wordSize-(sz)%wordSize)%wordSize
}
// panicToError recovers from a panic and replaces a nil error with the panicked
// error value. If the value does not implement error, it is re-panicked.
func panicToError(errP *error) {
r := recover()
if r == nil {
return
}
if err, ok := r.(error); !ok {
panic(r)
} else if *errP == nil {
*errP = err
}
}
// IContext is passed to [Artifact.Params] and provides methods for writing
// values to the IR writer. It does not expose the underlying [io.Writer].
//
// IContext is valid until [Artifact.Params] returns.
type IContext struct {
// Address of underlying [Cache], should be zeroed or made unusable after
// [Artifact.Params] returns and must not be exposed directly.
cache *Cache
// Written to by various methods, should be zeroed after [Artifact.Params]
// returns and must not be exposed directly.
w io.Writer
}
// Unwrap returns the underlying [context.Context].
func (i *IContext) Unwrap() context.Context { return i.cache.ctx }
// irZero is a zero IR word.
var irZero [wordSize]byte
// IRValueKind denotes the kind of encoded value.
type IRValueKind uint32
const (
// IRKindEnd denotes the end of the current parameters stream. The ancillary
// value is interpreted as [IREndFlag].
IRKindEnd IRValueKind = iota
// IRKindIdent denotes the identifier of a dependency [Artifact]. The
// ancillary value is reserved for future use.
IRKindIdent
// IRKindUint32 denotes an inlined uint32 value.
IRKindUint32
// IRKindString denotes a string with its true length encoded in header
// ancillary data. Its wire length is always aligned to 8 byte boundary.
IRKindString
irHeaderShift = 32
irHeaderMask = 0xffffffff
)
// String returns a user-facing name of k.
func (k IRValueKind) String() string {
switch k {
case IRKindEnd:
return "terminator"
case IRKindIdent:
return "ident"
case IRKindUint32:
return "uint32"
case IRKindString:
return "string"
default:
return "invalid kind " + strconv.Itoa(int(k))
}
}
// irValueHeader encodes [IRValueKind] and a 32-bit ancillary value.
type irValueHeader uint64
// encodeHeader returns irValueHeader encoding [IRValueKind] and ancillary data.
func (k IRValueKind) encodeHeader(v uint32) irValueHeader {
return irValueHeader(v)<<irHeaderShift | irValueHeader(k)
}
// put stores h in b[0:8].
func (h irValueHeader) put(b []byte) {
binary.LittleEndian.PutUint64(b[:], uint64(h))
}
// append appends the bytes of h to b and returns the appended slice.
func (h irValueHeader) append(b []byte) []byte {
return binary.LittleEndian.AppendUint64(b, uint64(h))
}
// IREndFlag is ancillary data encoded in the header of an [IRKindEnd] value and
// specifies the presence of optional fields in the remaining [IRKindEnd] data.
// Order of present fields is the order of their corresponding constants defined
// below.
type IREndFlag uint32
const (
// IREndKnownChecksum denotes a [KnownChecksum] artifact. For an [IRKindEnd]
// value with this flag set, the remaining data contains the [Checksum].
IREndKnownChecksum IREndFlag = 1 << iota
)
// mustWrite writes to IContext.w and panics on error. The panic is recovered
// from by the caller and used as the return value.
func (i *IContext) mustWrite(p []byte) {
if _, err := i.w.Write(p); err != nil {
panic(err)
}
}
// WriteIdent writes the identifier of [Artifact] to the IR. The behaviour of
// WriteIdent is not defined for an [Artifact] not part of the slice returned by
// [Artifact.Dependencies].
func (i *IContext) WriteIdent(a Artifact) {
buf := i.cache.getIdentBuf()
defer i.cache.putIdentBuf(buf)
IRKindIdent.encodeHeader(0).put(buf[:])
*(*ID)(buf[wordSize:]) = i.cache.Ident(a).Value()
i.mustWrite(buf[:])
}
// WriteUint32 writes a uint32 value to the IR.
func (i *IContext) WriteUint32(v uint32) {
i.mustWrite(IRKindUint32.encodeHeader(v).append(nil))
}
// irMaxStringLength is the maximum acceptable wire size of [IRKindString].
const irMaxStringLength = 1 << 20
// IRStringError is a string value too big to encode in IR.
type IRStringError string
func (IRStringError) Error() string {
return "params value too big to encode in IR"
}
// Write writes p as a string value to the IR.
func (i *IContext) Write(p []byte) {
sz := alignSize(len(p))
if len(p) > irMaxStringLength || sz > irMaxStringLength {
panic(IRStringError(p))
}
i.mustWrite(IRKindString.encodeHeader(uint32(len(p))).append(nil))
i.mustWrite(p)
psz := sz - len(p)
if psz > 0 {
i.mustWrite(irZero[:psz])
}
}
// WriteString writes s as a string value to the IR.
func (i *IContext) WriteString(s string) {
p := unsafe.Slice(unsafe.StringData(s), len(s))
i.Write(p)
}
// Encode writes a deterministic, efficient representation of a to w and returns
// the first non-nil error encountered while writing to w.
func (c *Cache) Encode(w io.Writer, a Artifact) (err error) {
deps := a.Dependencies()
idents := make([]*extIdent, len(deps))
for i, d := range deps {
dbuf, did := c.unsafeIdent(d, true)
if dbuf == nil {
dbuf = c.getIdentBuf()
binary.LittleEndian.PutUint64(dbuf[:], uint64(d.Kind()))
*(*ID)(dbuf[wordSize:]) = did.Value()
} else {
c.storeIdent(d, dbuf)
}
defer c.putIdentBuf(dbuf)
idents[i] = dbuf
}
slices.SortFunc(idents, func(a, b *extIdent) int {
return bytes.Compare(a[:], b[:])
})
idents = slices.CompactFunc(idents, func(a, b *extIdent) bool {
return *a == *b
})
// kind uint64 | deps_sz uint64
var buf [wordSize * 2]byte
binary.LittleEndian.PutUint64(buf[:], uint64(a.Kind()))
binary.LittleEndian.PutUint64(buf[wordSize:], uint64(len(idents)))
if _, err = w.Write(buf[:]); err != nil {
return
}
for _, dn := range idents {
// kind uint64 | ident ID
if _, err = w.Write(dn[:]); err != nil {
return
}
}
func() {
i := IContext{c, w}
defer panicToError(&err)
defer func() { i.cache, i.w = nil, nil }()
a.Params(&i)
}()
if err != nil {
return
}
var f IREndFlag
kcBuf := c.getIdentBuf()
sz := wordSize
if kc, ok := a.(KnownChecksum); ok {
f |= IREndKnownChecksum
*(*Checksum)(kcBuf[wordSize:]) = kc.Checksum()
sz += len(Checksum{})
}
IRKindEnd.encodeHeader(uint32(f)).put(kcBuf[:])
_, err = w.Write(kcBuf[:sz])
c.putIdentBuf(kcBuf)
return
}
// encodeAll implements EncodeAll by recursively encoding dependencies and
// performs deduplication by value via the encoded map.
func (c *Cache) encodeAll(
w io.Writer,
a Artifact,
encoded map[Artifact]struct{},
) (err error) {
if _, ok := encoded[a]; ok {
return
}
for _, d := range a.Dependencies() {
if err = c.encodeAll(w, d, encoded); err != nil {
return
}
}
encoded[a] = struct{}{}
return c.Encode(w, a)
}
// EncodeAll writes a self-describing IR stream of a to w and returns the first
// non-nil error encountered while writing to w.
//
// EncodeAll tries to avoid encoding the same [Artifact] more than once, however
// it will fail to do so if they do not compare equal by value, as that will
// require buffering and greatly reduce performance. It is therefore up to the
// caller to avoid causing dependencies to be represented in a way such that
// two equivalent artifacts do not compare equal. While an IR stream with
// repeated artifacts is valid, it is somewhat inefficient, and the reference
// [IRDecoder] implementation produces a warning for it.
//
// Note that while EncodeAll makes use of the ident free list, it does not use
// the ident cache, nor does it contribute identifiers it computes back to the
// ident cache. Because of this, multiple invocations of EncodeAll will have
// similar cost and does not amortise when combined with a call to Cure.
func (c *Cache) EncodeAll(w io.Writer, a Artifact) error {
return c.encodeAll(w, a, make(map[Artifact]struct{}))
}
// ErrRemainingIR is returned for a [IRReadFunc] that failed to call
// [IRReader.Finalise] before returning.
var ErrRemainingIR = errors.New("implementation did not consume final value")
// DanglingIdentError is an identifier in a [IRKindIdent] value that was never
// described in the IR stream before it was encountered.
type DanglingIdentError unique.Handle[ID]
func (e DanglingIdentError) Error() string {
return "artifact " + Encode(unique.Handle[ID](e).Value()) +
" was never described"
}
type (
// IRDecoder decodes [Artifact] from an IR stream. The stream is read to
// EOF and the final [Artifact] is returned. Previous artifacts may be
// looked up by their identifier.
//
// An [Artifact] may appear more than once in the same IR stream. A
// repeating [Artifact] generates a warning via [Cache] and will appear if
// verbose logging is enabled. Artifacts may only depend on artifacts
// previously described in the IR stream.
//
// Methods of IRDecoder are not safe for concurrent use.
IRDecoder struct {
// Address of underlying [Cache], must not be exposed directly.
c *Cache
// Underlying IR reader. Methods of [IRReader] must not use this as it
// bypasses ident measurement.
r io.Reader
// Artifacts already seen in the IR stream.
ident map[unique.Handle[ID]]Artifact
// Whether Decode returned, and the entire IR stream was decoded.
done, ok bool
}
// IRReader provides methods to decode the IR wire format and read values
// from the reader embedded in the underlying [IRDecoder]. It is
// deliberately impossible to obtain the [IRValueKind] of the next value,
// and callers must never recover from panics in any read method.
//
// It is the responsibility of the caller to call Finalise after all IR
// values have been read. Failure to call Finalise causes the resulting
// [Artifact] to be rejected with [ErrRemainingIR].
//
// For an [Artifact] expected to have dependencies, the caller must consume
// all dependencies by calling Next until all dependencies are depleted, or
// call DiscardAll to explicitly discard them and rely on values encoded as
// [IRKindIdent] instead. Failure to consume all unstructured dependencies
// causes the resulting [Artifact] to be rejected with [MissedDependencyError].
//
// Requesting the value of an unstructured dependency not yet described in
// the IR stream via Next, or reading an [IRKindIdent] value not part of
// unstructured dependencies via ReadIdent may cause the resulting
// [Artifact] to be rejected with [DanglingIdentError], however either
// method may return a non-nil [Artifact] implementation of unspecified
// value.
IRReader struct {
// Address of underlying [IRDecoder], should be zeroed or made unusable
// after finalisation and must not be exposed directly.
d *IRDecoder
// Common buffer for word-sized reads.
buf [wordSize]byte
// Dependencies sent before params, sorted by identifier. Resliced on
// each call to Next and checked to be depleted during Finalise.
deps []*extIdent
// Number of values already read, -1 denotes a finalised IRReader.
count int
// Header of value currently being read.
h irValueHeader
// Measured IR reader. All reads for the current [Artifact] must go
// through this to produce a correct ident.
r io.Reader
// Buffers measure writes. Flushed and returned to d during Finalise.
ibw *bufio.Writer
}
// IRReadFunc reads IR values written by [Artifact.Params] to produce an
// instance of [Artifact] identical to the one to produce these values.
IRReadFunc func(r *IRReader) Artifact
)
// kind returns the [IRValueKind] encoded in h.
func (h irValueHeader) kind() IRValueKind {
return IRValueKind(h & irHeaderMask)
}
// value returns ancillary data encoded in h.
func (h irValueHeader) value() uint32 {
return uint32(h >> irHeaderShift)
}
// irArtifact refers to artifact IR interpretation functions and must not be
// written to directly.
var irArtifact = make(map[Kind]IRReadFunc)
// InvalidKindError is an unregistered [Kind] value.
type InvalidKindError Kind
func (e InvalidKindError) Error() string {
return "invalid artifact kind " + strconv.Itoa(int(e))
}
// register records the [IRReadFunc] of an implementation of [Artifact] under
// the specified [Kind]. Expecting to be used only during initialization, it
// panics if the mapping between [Kind] and [IRReadFunc] is not a bijection.
//
// register is not safe for concurrent use. register must not be called after
// the first instance of [Cache] has been opened.
func register(k Kind, f IRReadFunc) {
if _, ok := irArtifact[k]; ok {
panic("attempting to register " + strconv.Itoa(int(k)) + " twice")
}
irArtifact[k] = f
}
// Register records the [IRReadFunc] of a custom implementation of [Artifact]
// under the specified [Kind]. Expecting to be used only during initialization,
// it panics if the mapping between [Kind] and [IRReadFunc] is not a bijection,
// or the specified [Kind] is below [KindCustomOffset].
//
// Register is not safe for concurrent use. Register must not be called after
// the first instance of [Cache] has been opened.
func Register(k Kind, f IRReadFunc) {
if k < KindCustomOffset {
panic("attempting to register within internal kind range")
}
register(k, f)
}
// NewDecoder returns a new [IRDecoder] that reads from the [io.Reader].
func (c *Cache) NewDecoder(r io.Reader) *IRDecoder {
return &IRDecoder{c, r, make(map[unique.Handle[ID]]Artifact), false, false}
}
const (
// irMaxValues is the arbitrary maximum number of values allowed to be
// written by [Artifact.Params] and subsequently read via [IRReader].
irMaxValues = 1 << 12
// irMaxDeps is the arbitrary maximum number of direct dependencies allowed
// to be returned by [Artifact.Dependencies] and subsequently decoded by
// [IRDecoder].
irMaxDeps = 1 << 10
)
var (
// ErrIRValues is returned for an [Artifact] with too many parameter values.
ErrIRValues = errors.New("artifact has too many IR parameter values")
// ErrIRDepend is returned for an [Artifact] with too many dependencies.
ErrIRDepend = errors.New("artifact has too many dependencies")
// ErrAlreadyFinalised is returned when attempting to use an [IRReader] that
// has already been finalised.
ErrAlreadyFinalised = errors.New("reader has already finalised")
)
// enterReader panics with an appropriate error for an out-of-bounds count and
// must be called at some point in any exported method.
func (ir *IRReader) enterReader(read bool) {
if ir.count < 0 {
panic(ErrAlreadyFinalised)
}
if ir.count >= irMaxValues {
panic(ErrIRValues)
}
if read {
ir.count++
}
}
// IRKindError describes an attempt to read an IR value of unexpected kind.
type IRKindError struct {
Got, Want IRValueKind
Ancillary uint32
}
func (e *IRKindError) Error() string {
return fmt.Sprintf(
"got %s IR value (%#x) instead of %s",
e.Got, e.Ancillary, e.Want,
)
}
// readFull reads until either p is filled or an error is encountered.
func (ir *IRReader) readFull(p []byte) (n int, err error) {
for n < len(p) && err == nil {
var nn int
nn, err = ir.r.Read(p[n:])
n += nn
}
return
}
// mustRead reads from the underlying measured reader and panics on error. If
// an [io.EOF] is encountered and n != len(p), the error is promoted to a
// [io.ErrUnexpectedEOF], if n == 0, [io.EOF] is kept as is, otherwise it is
// zeroed.
func (ir *IRReader) mustRead(p []byte) {
n, err := ir.readFull(p)
if err == nil {
return
}
if errors.Is(err, io.EOF) {
if n == len(p) {
return
}
err = io.ErrUnexpectedEOF
}
panic(err)
}
// mustReadHeader reads the next header via d and checks its kind.
func (ir *IRReader) mustReadHeader(k IRValueKind) {
ir.mustRead(ir.buf[:])
ir.h = irValueHeader(binary.LittleEndian.Uint64(ir.buf[:]))
if wk := ir.h.kind(); wk != k {
panic(&IRKindError{wk, k, ir.h.value()})
}
}
// putAll returns all dependency buffers to the underlying [Cache].
func (ir *IRReader) putAll() {
for _, buf := range ir.deps {
ir.d.c.putIdentBuf(buf)
}
ir.deps = nil
}
// DiscardAll discards all unstructured dependencies. This is useful to
// implementations that encode dependencies as [IRKindIdent] which are read back
// via ReadIdent.
func (ir *IRReader) DiscardAll() {
if ir.deps == nil {
panic("attempting to discard dependencies twice")
}
ir.putAll()
}
// ErrDependencyDepleted is returned when attempting to advance to the next
// unstructured dependency when there are none left.
var ErrDependencyDepleted = errors.New("reading past end of dependencies")
// Next returns the next unstructured dependency.
func (ir *IRReader) Next() Artifact {
if len(ir.deps) == 0 {
panic(ErrDependencyDepleted)
}
id := unique.Make(ID(ir.deps[0][wordSize:]))
ir.d.c.putIdentBuf(ir.deps[0])
ir.deps = ir.deps[1:]
if a, ok := ir.d.ident[id]; !ok {
ir.putAll()
panic(DanglingIdentError(id))
} else {
return a
}
}
// MissedDependencyError is the number of unstructured dependencies remaining
// in [IRReader] that was never requested or explicitly discarded before
// finalisation.
type MissedDependencyError int
func (e MissedDependencyError) Error() string {
return "missed " + strconv.Itoa(int(e)) + " unstructured dependencies"
}
var (
// ErrUnexpectedChecksum is returned by a [IRReadFunc] that does not expect
// a checksum but received one in [IRKindEnd] anyway.
ErrUnexpectedChecksum = errors.New("checksum specified on unsupported artifact")
// ErrExpectedChecksum is returned by a [IRReadFunc] that expects a checksum
// but did not receive one in [IRKindEnd].
ErrExpectedChecksum = errors.New("checksum required but not specified")
)
// Finalise reads the final [IRKindEnd] value and marks r as finalised. Methods
// of r are invalid upon entry into Finalise. If a [Checksum] is available via
// [IREndKnownChecksum], its handle is returned and the caller must store its
// value in the resulting [Artifact].
func (ir *IRReader) Finalise() (checksum unique.Handle[Checksum], ok bool) {
ir.enterReader(true)
ir.count = -1
ir.mustReadHeader(IRKindEnd)
f := IREndFlag(ir.h.value())
if f&IREndKnownChecksum != 0 {
buf := ir.d.c.getIdentBuf()
defer ir.d.c.putIdentBuf(buf)
ir.mustRead(buf[wordSize:])
checksum = unique.Make(Checksum(buf[wordSize:]))
ok = true
}
if err := ir.ibw.Flush(); err != nil {
panic(err)
}
ir.r, ir.ibw = nil, nil
if len(ir.deps) != 0 {
panic(MissedDependencyError(len(ir.deps)))
}
return
}
// ReadIdent reads the next value as [IRKindIdent].
func (ir *IRReader) ReadIdent() Artifact {
ir.enterReader(true)
ir.mustReadHeader(IRKindIdent)
buf := ir.d.c.getIdentBuf()
defer ir.d.c.putIdentBuf(buf)
ir.mustRead(buf[wordSize:])
id := unique.Make(ID(buf[wordSize:]))
if a, ok := ir.d.ident[id]; !ok {
panic(DanglingIdentError(id))
} else {
return a
}
}
// ReadUint32 reads the next value as [IRKindUint32].
func (ir *IRReader) ReadUint32() uint32 {
ir.enterReader(true)
ir.mustReadHeader(IRKindUint32)
return ir.h.value()
}
// ReadStringBytes reads the next value as [IRKindString] but returns it as a
// byte slice instead.
func (ir *IRReader) ReadStringBytes() []byte {
ir.enterReader(true)
ir.mustReadHeader(IRKindString)
sz := int(ir.h.value())
szWire := alignSize(sz)
if szWire > irMaxStringLength {
panic(IRStringError("\x00"))
}
p := make([]byte, szWire)
ir.mustRead(p)
return p[:sz]
}
// ReadString reads the next value as [IRKindString].
func (ir *IRReader) ReadString() string {
p := ir.ReadStringBytes()
return unsafe.String(unsafe.SliceData(p), len(p))
}
// decode decodes the next [Artifact] in the IR stream and returns any buffer
// originating from [Cache] before returning. decode returns [io.EOF] if and
// only if the underlying [io.Reader] is already read to EOF.
func (d *IRDecoder) decode() (a Artifact, err error) {
defer panicToError(&err)
var ir IRReader
defer func() { ir.d = nil }()
ir.d = d
h := sha512.New384()
ir.ibw = d.c.getWriter(h)
defer d.c.putWriter(ir.ibw)
ir.r = io.TeeReader(d.r, ir.ibw)
if n, _err := ir.readFull(ir.buf[:]); _err != nil {
if errors.Is(_err, io.EOF) {
if n != 0 {
_err = io.ErrUnexpectedEOF
}
}
err = _err
return
}
ak := Kind(binary.LittleEndian.Uint64(ir.buf[:]))
f, ok := irArtifact[ak]
if !ok {
err = InvalidKindError(ak)
return
}
defer ir.putAll()
ir.mustRead(ir.buf[:])
sz := binary.LittleEndian.Uint64(ir.buf[:])
if sz > irMaxDeps {
err = ErrIRDepend
return
}
ir.deps = make([]*extIdent, sz)
for i := range ir.deps {
ir.deps[i] = d.c.getIdentBuf()
}
for _, buf := range ir.deps {
ir.mustRead(buf[:])
}
a = f(&ir)
if a == nil {
err = syscall.ENOTRECOVERABLE
return
}
if ir.count != -1 {
err = ErrRemainingIR
return
}
buf := d.c.getIdentBuf()
h.Sum(buf[wordSize:wordSize])
id := unique.Make(ID(buf[wordSize:]))
d.c.putIdentBuf(buf)
if _, ok = d.ident[id]; !ok {
d.ident[id] = a
} else {
d.c.msg.Verbosef(
"artifact %s appeared more than once in IR stream",
Encode(id.Value()),
)
}
return
}
// Decode consumes the IR stream to EOF and returns the final [Artifact]. After
// Decode returns, Lookup is available and Decode must not be called again.
func (d *IRDecoder) Decode() (a Artifact, err error) {
if d.done {
panic("attempting to decode an IR stream twice")
}
defer func() { d.done = true }()
var cur Artifact
next:
a, err = d.decode()
if err == nil {
cur = a
goto next
}
if errors.Is(err, io.EOF) {
a, err = cur, nil
d.ok = true
}
return
}
// Lookup looks up an [Artifact] described by the IR stream by its identifier.
func (d *IRDecoder) Lookup(id unique.Handle[ID]) (a Artifact, ok bool) {
if !d.ok {
panic("attempting to look up artifact without full IR stream")
}
a, ok = d.ident[id]
return
}

114
internal/pkg/ir_test.go Normal file
View File

@@ -0,0 +1,114 @@
package pkg_test
import (
"bytes"
"io"
"reflect"
"testing"
"hakurei.app/container/check"
"hakurei.app/internal/pkg"
)
func TestIRRoundtrip(t *testing.T) {
t.Parallel()
testCases := []struct {
name string
a pkg.Artifact
}{
{"http get aligned", pkg.NewHTTPGet(
nil, "file:///testdata",
pkg.Checksum(bytes.Repeat([]byte{0xfd}, len(pkg.Checksum{}))),
)},
{"http get unaligned", pkg.NewHTTPGet(
nil, "https://hakurei.app",
pkg.Checksum(bytes.Repeat([]byte{0xfc}, len(pkg.Checksum{}))),
)},
{"http get tar", pkg.NewHTTPGetTar(
nil, "file:///testdata",
pkg.Checksum(bytes.Repeat([]byte{0xff}, len(pkg.Checksum{}))),
pkg.TarBzip2,
)},
{"http get tar unaligned", pkg.NewHTTPGetTar(
nil, "https://hakurei.app",
pkg.Checksum(bytes.Repeat([]byte{0xfe}, len(pkg.Checksum{}))),
pkg.TarUncompressed,
)},
{"exec offline", pkg.NewExec(
"exec-offline", nil, 0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool"},
pkg.MustPath("/file", false, pkg.NewFile("file", []byte(
"stub file",
))), pkg.MustPath("/.hakurei", false, pkg.NewHTTPGetTar(
nil, "file:///hakurei.tar",
pkg.Checksum(bytes.Repeat([]byte{0xfc}, len(pkg.Checksum{}))),
pkg.TarUncompressed,
)), pkg.MustPath("/opt", false, pkg.NewHTTPGetTar(
nil, "file:///testtool.tar.gz",
pkg.Checksum(bytes.Repeat([]byte{0xfc}, len(pkg.Checksum{}))),
pkg.TarGzip,
)),
)},
{"exec net", pkg.NewExec(
"exec-net",
(*pkg.Checksum)(bytes.Repeat([]byte{0xfc}, len(pkg.Checksum{}))),
0, false,
pkg.AbsWork,
[]string{"HAKUREI_TEST=1"},
check.MustAbs("/opt/bin/testtool"),
[]string{"testtool", "net"},
pkg.MustPath("/file", false, pkg.NewFile("file", []byte(
"stub file",
))), pkg.MustPath("/.hakurei", false, pkg.NewHTTPGetTar(
nil, "file:///hakurei.tar",
pkg.Checksum(bytes.Repeat([]byte{0xfc}, len(pkg.Checksum{}))),
pkg.TarUncompressed,
)), pkg.MustPath("/opt", false, pkg.NewHTTPGetTar(
nil, "file:///testtool.tar.gz",
pkg.Checksum(bytes.Repeat([]byte{0xfc}, len(pkg.Checksum{}))),
pkg.TarGzip,
)),
)},
{"file anonymous", pkg.NewFile("", []byte{0})},
{"file", pkg.NewFile("stub", []byte("stub"))},
}
testCasesCache := make([]cacheTestCase, len(testCases))
for i, tc := range testCases {
want := tc.a
testCasesCache[i] = cacheTestCase{tc.name, nil,
func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
r, w := io.Pipe()
done := make(chan error, 1)
go func() {
t.Helper()
done <- c.EncodeAll(w, want)
_ = w.Close()
}()
if got, err := c.NewDecoder(r).Decode(); err != nil {
t.Fatalf("Decode: error = %v", err)
} else if !reflect.DeepEqual(got, want) {
t.Fatalf("Decode: %#v, want %#v", got, want)
}
if err := <-done; err != nil {
t.Fatalf("EncodeAll: error = %v", err)
}
}, pkg.MustDecode(
"E4vEZKhCcL2gPZ2Tt59FS3lDng-d_2SKa2i5G_RbDfwGn6EemptFaGLPUDiOa94C",
),
}
}
checkWithCache(t, testCasesCache)
}

106
internal/pkg/net.go Normal file
View File

@@ -0,0 +1,106 @@
package pkg
import (
"fmt"
"io"
"net/http"
"path"
"unique"
)
// An httpArtifact is an [Artifact] backed by a [http] url string. The method is
// hardcoded as [http.MethodGet]. Request body is not allowed because it cannot
// be deterministically represented by Params.
type httpArtifact struct {
// Caller-supplied url string.
url string
// Caller-supplied checksum of the response body. This is validated when
// closing the [io.ReadCloser] returned by Cure.
checksum unique.Handle[Checksum]
// client is the address of the caller-supplied [http.Client].
client *http.Client
}
var _ KnownChecksum = new(httpArtifact)
var _ fmt.Stringer = new(httpArtifact)
// NewHTTPGet returns a new [FileArtifact] backed by the supplied client. A GET
// request is set up for url. If c is nil, [http.DefaultClient] is used instead.
func NewHTTPGet(
c *http.Client,
url string,
checksum Checksum,
) FileArtifact {
return &httpArtifact{url: url, checksum: unique.Make(checksum), client: c}
}
// Kind returns the hardcoded [Kind] constant.
func (*httpArtifact) Kind() Kind { return KindHTTPGet }
// Params writes the backing url string. Client is not represented as it does
// not affect [Cache.Cure] outcome.
func (a *httpArtifact) Params(ctx *IContext) { ctx.WriteString(a.url) }
func init() {
register(KindHTTPGet, func(r *IRReader) Artifact {
url := r.ReadString()
checksum, ok := r.Finalise()
if !ok {
panic(ErrExpectedChecksum)
}
return NewHTTPGet(nil, url, checksum.Value())
})
}
// Dependencies returns a nil slice.
func (*httpArtifact) Dependencies() []Artifact { return nil }
// IsExclusive returns false: Cure returns as soon as a response is received.
func (*httpArtifact) IsExclusive() bool { return false }
// Checksum returns the caller-supplied checksum.
func (a *httpArtifact) Checksum() Checksum { return a.checksum.Value() }
// String returns [path.Base] over the backing url.
func (a *httpArtifact) String() string { return path.Base(a.url) }
// ResponseStatusError is returned for a response returned by an [http.Client]
// with a status code other than [http.StatusOK].
type ResponseStatusError int
func (e ResponseStatusError) Error() string {
return "the requested URL returned non-OK status: " + http.StatusText(int(e))
}
// Cure sends the http request and returns the resulting response body reader
// wrapped to perform checksum validation. It is valid but not encouraged to
// close the resulting [io.ReadCloser] before it is read to EOF, as that causes
// Close to block until all remaining data is consumed and validated.
func (a *httpArtifact) Cure(r *RContext) (rc io.ReadCloser, err error) {
var req *http.Request
req, err = http.NewRequestWithContext(r.Unwrap(), http.MethodGet, a.url, nil)
if err != nil {
return
}
req.Header.Set("User-Agent", "Hakurei/1.1")
c := a.client
if c == nil {
c = http.DefaultClient
}
var resp *http.Response
if resp, err = c.Do(req); err != nil {
return
}
if resp.StatusCode != http.StatusOK {
_ = resp.Body.Close()
return nil, ResponseStatusError(resp.StatusCode)
}
rc = r.NewMeasuredReader(resp.Body, a.checksum)
return
}

161
internal/pkg/net_test.go Normal file
View File

@@ -0,0 +1,161 @@
package pkg_test
import (
"crypto/sha512"
"io"
"net/http"
"reflect"
"testing"
"testing/fstest"
"unique"
"unsafe"
"hakurei.app/container/check"
"hakurei.app/internal/pkg"
)
func TestHTTPGet(t *testing.T) {
t.Parallel()
const testdata = "\x7f\xe1\x69\xa2\xdd\x63\x96\x26\x83\x79\x61\x8b\xf0\x3f\xd5\x16\x9a\x39\x3a\xdb\xcf\xb1\xbc\x8d\x33\xff\x75\xee\x62\x56\xa9\xf0\x27\xac\x13\x94\x69"
testdataChecksum := func() unique.Handle[pkg.Checksum] {
h := sha512.New384()
h.Write([]byte(testdata))
return unique.Make(pkg.Checksum(h.Sum(nil)))
}()
var transport http.Transport
client := http.Client{Transport: &transport}
transport.RegisterProtocol("file", http.NewFileTransportFS(fstest.MapFS{
"testdata": {Data: []byte(testdata), Mode: 0400},
}))
checkWithCache(t, []cacheTestCase{
{"direct", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
var r pkg.RContext
rCacheVal := reflect.ValueOf(&r).Elem().FieldByName("cache")
reflect.NewAt(
rCacheVal.Type(),
unsafe.Pointer(rCacheVal.UnsafeAddr()),
).Elem().Set(reflect.ValueOf(c))
f := pkg.NewHTTPGet(
&client,
"file:///testdata",
testdataChecksum.Value(),
)
var got []byte
if rc, err := f.Cure(&r); err != nil {
t.Fatalf("Cure: error = %v", err)
} else if got, err = io.ReadAll(rc); err != nil {
t.Fatalf("ReadAll: error = %v", err)
} else if string(got) != testdata {
t.Fatalf("Cure: %x, want %x", got, testdata)
} else if err = rc.Close(); err != nil {
t.Fatalf("Close: error = %v", err)
}
// check direct validation
f = pkg.NewHTTPGet(
&client,
"file:///testdata",
pkg.Checksum{},
)
wantErrMismatch := &pkg.ChecksumMismatchError{
Got: testdataChecksum.Value(),
}
if rc, err := f.Cure(&r); err != nil {
t.Fatalf("Cure: error = %v", err)
} else if got, err = io.ReadAll(rc); err != nil {
t.Fatalf("ReadAll: error = %v", err)
} else if string(got) != testdata {
t.Fatalf("Cure: %x, want %x", got, testdata)
} else if err = rc.Close(); !reflect.DeepEqual(err, wantErrMismatch) {
t.Fatalf("Close: error = %#v, want %#v", err, wantErrMismatch)
}
// check fallback validation
if rc, err := f.Cure(&r); err != nil {
t.Fatalf("Cure: error = %v", err)
} else if err = rc.Close(); !reflect.DeepEqual(err, wantErrMismatch) {
t.Fatalf("Close: error = %#v, want %#v", err, wantErrMismatch)
}
// check direct response error
f = pkg.NewHTTPGet(
&client,
"file:///nonexistent",
pkg.Checksum{},
)
wantErrNotFound := pkg.ResponseStatusError(http.StatusNotFound)
if _, err := f.Cure(&r); !reflect.DeepEqual(err, wantErrNotFound) {
t.Fatalf("Cure: error = %#v, want %#v", err, wantErrNotFound)
}
}, pkg.MustDecode("E4vEZKhCcL2gPZ2Tt59FS3lDng-d_2SKa2i5G_RbDfwGn6EemptFaGLPUDiOa94C")},
{"cure", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
var r pkg.RContext
rCacheVal := reflect.ValueOf(&r).Elem().FieldByName("cache")
reflect.NewAt(
rCacheVal.Type(),
unsafe.Pointer(rCacheVal.UnsafeAddr()),
).Elem().Set(reflect.ValueOf(c))
f := pkg.NewHTTPGet(
&client,
"file:///testdata",
testdataChecksum.Value(),
)
wantPathname := base.Append(
"identifier",
"oM-2pUlk-mOxK1t3aMWZer69UdOQlAXiAgMrpZ1476VoOqpYVP1aGFS9_HYy-D8_",
)
if pathname, checksum, err := c.Cure(f); err != nil {
t.Fatalf("Cure: error = %v", err)
} else if !pathname.Is(wantPathname) {
t.Fatalf("Cure: %q, want %q", pathname, wantPathname)
} else if checksum != testdataChecksum {
t.Fatalf("Cure: %x, want %x", checksum.Value(), testdataChecksum.Value())
}
var got []byte
if rc, err := f.Cure(&r); err != nil {
t.Fatalf("Cure: error = %v", err)
} else if got, err = io.ReadAll(rc); err != nil {
t.Fatalf("ReadAll: error = %v", err)
} else if string(got) != testdata {
t.Fatalf("Cure: %x, want %x", got, testdata)
} else if err = rc.Close(); err != nil {
t.Fatalf("Close: error = %v", err)
}
// check load from cache
f = pkg.NewHTTPGet(
&client,
"file:///testdata",
testdataChecksum.Value(),
)
if rc, err := f.Cure(&r); err != nil {
t.Fatalf("Cure: error = %v", err)
} else if got, err = io.ReadAll(rc); err != nil {
t.Fatalf("ReadAll: error = %v", err)
} else if string(got) != testdata {
t.Fatalf("Cure: %x, want %x", got, testdata)
} else if err = rc.Close(); err != nil {
t.Fatalf("Close: error = %v", err)
}
// check error passthrough
f = pkg.NewHTTPGet(
&client,
"file:///nonexistent",
pkg.Checksum{},
)
wantErrNotFound := pkg.ResponseStatusError(http.StatusNotFound)
if _, _, err := c.Cure(f); !reflect.DeepEqual(err, wantErrNotFound) {
t.Fatalf("Pathname: error = %#v, want %#v", err, wantErrNotFound)
}
}, pkg.MustDecode("L_0RFHpr9JUS4Zp14rz2dESSRvfLzpvqsLhR1-YjQt8hYlmEdVl7vI3_-v8UNPKs")},
})
}

1737
internal/pkg/pkg.go Normal file

File diff suppressed because it is too large Load Diff

1233
internal/pkg/pkg_test.go Normal file

File diff suppressed because it is too large Load Diff

250
internal/pkg/tar.go Normal file
View File

@@ -0,0 +1,250 @@
package pkg
import (
"archive/tar"
"compress/bzip2"
"compress/gzip"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"os"
"hakurei.app/container/check"
)
const (
// TarUncompressed denotes an uncompressed tarball.
TarUncompressed = iota
// TarGzip denotes a tarball compressed via [gzip].
TarGzip
// TarBzip2 denotes a tarball compressed via [bzip2].
TarBzip2
)
// A tarArtifact is an [Artifact] unpacking a tarball backed by a [FileArtifact].
type tarArtifact struct {
// Caller-supplied backing tarball.
f Artifact
// Compression on top of the tarball.
compression uint32
}
// tarArtifactNamed embeds tarArtifact for a [fmt.Stringer] tarball.
type tarArtifactNamed struct {
tarArtifact
// Copied from tarArtifact.f.
name string
}
var _ fmt.Stringer = new(tarArtifactNamed)
// String returns the name of the underlying [Artifact] suffixed with unpack.
func (a *tarArtifactNamed) String() string { return a.name + "-unpack" }
// NewTar returns a new [Artifact] backed by the supplied [Artifact] and
// compression method. The source [Artifact] must be compatible with
// [TContext.Open].
func NewTar(a Artifact, compression uint32) Artifact {
ta := tarArtifact{a, compression}
if s, ok := a.(fmt.Stringer); ok {
if name := s.String(); name != "" {
return &tarArtifactNamed{ta, name}
}
}
return &ta
}
// NewHTTPGetTar is abbreviation for NewHTTPGet passed to NewTar.
func NewHTTPGetTar(
hc *http.Client,
url string,
checksum Checksum,
compression uint32,
) Artifact {
return NewTar(NewHTTPGet(hc, url, checksum), compression)
}
// Kind returns the hardcoded [Kind] constant.
func (a *tarArtifact) Kind() Kind { return KindTar }
// Params writes compression encoded in little endian.
func (a *tarArtifact) Params(ctx *IContext) { ctx.WriteUint32(a.compression) }
func init() {
register(KindTar, func(r *IRReader) Artifact {
a := NewTar(r.Next(), r.ReadUint32())
if _, ok := r.Finalise(); ok {
panic(ErrUnexpectedChecksum)
}
return a
})
}
// Dependencies returns a slice containing the backing file.
func (a *tarArtifact) Dependencies() []Artifact {
return []Artifact{a.f}
}
// IsExclusive returns false: decompressor and tar reader are fully sequential.
func (a *tarArtifact) IsExclusive() bool { return false }
// A DisallowedTypeflagError describes a disallowed typeflag encountered while
// unpacking a tarball.
type DisallowedTypeflagError byte
func (e DisallowedTypeflagError) Error() string {
return "disallowed typeflag '" + string(e) + "'"
}
// Cure cures the [Artifact], producing a directory located at work.
func (a *tarArtifact) Cure(t *TContext) (err error) {
temp := t.GetTempDir()
var tr io.ReadCloser
if tr, err = t.Open(a.f); err != nil {
return
}
defer func(f io.ReadCloser) {
if err == nil {
err = tr.Close()
}
closeErr := f.Close()
if err == nil {
err = closeErr
}
}(tr)
tr = io.NopCloser(tr)
switch a.compression {
case TarUncompressed:
break
case TarGzip:
if tr, err = gzip.NewReader(tr); err != nil {
return
}
break
case TarBzip2:
tr = io.NopCloser(bzip2.NewReader(tr))
break
default:
return os.ErrInvalid
}
type dirTargetPerm struct {
path *check.Absolute
mode fs.FileMode
}
var madeDirectories []dirTargetPerm
if err = os.MkdirAll(temp.String(), 0700); err != nil {
return
}
var header *tar.Header
r := tar.NewReader(tr)
for header, err = r.Next(); err == nil; header, err = r.Next() {
typeflag := header.Typeflag
if typeflag == 0 {
if len(header.Name) > 0 && header.Name[len(header.Name)-1] == '/' {
typeflag = tar.TypeDir
} else {
typeflag = tar.TypeReg
}
}
pathname := temp.Append(header.Name)
if typeflag >= '0' && typeflag <= '9' && typeflag != tar.TypeDir {
if err = os.MkdirAll(pathname.Dir().String(), 0700); err != nil {
return
}
}
switch typeflag {
case tar.TypeReg:
var f *os.File
if f, err = os.OpenFile(
pathname.String(),
os.O_CREATE|os.O_EXCL|os.O_WRONLY,
header.FileInfo().Mode()&0500,
); err != nil {
return
}
if _, err = io.Copy(f, r); err != nil {
_ = f.Close()
return
} else if err = f.Close(); err != nil {
return
}
break
case tar.TypeLink:
if err = os.Link(
temp.Append(header.Linkname).String(),
pathname.String(),
); err != nil {
return
}
break
case tar.TypeSymlink:
if err = os.Symlink(header.Linkname, pathname.String()); err != nil {
return
}
break
case tar.TypeDir:
madeDirectories = append(madeDirectories, dirTargetPerm{
path: pathname,
mode: header.FileInfo().Mode(),
})
if err = os.MkdirAll(pathname.String(), 0700); err != nil {
return
}
break
case tar.TypeXGlobalHeader:
continue // ignore
default:
return DisallowedTypeflagError(typeflag)
}
}
if errors.Is(err, io.EOF) {
err = nil
}
if err == nil {
for _, e := range madeDirectories {
if err = os.Chmod(e.path.String(), e.mode&0500); err != nil {
return
}
}
} else {
return
}
if err = os.Chmod(temp.String(), 0700); err != nil {
return
}
var entries []os.DirEntry
if entries, err = os.ReadDir(temp.String()); err != nil {
return
}
if len(entries) == 1 && entries[0].IsDir() {
p := temp.Append(entries[0].Name())
if err = os.Chmod(p.String(), 0700); err != nil {
return
}
err = os.Rename(p.String(), t.GetWorkDir().String())
} else {
err = os.Rename(temp.String(), t.GetWorkDir().String())
}
return
}

226
internal/pkg/tar_test.go Normal file
View File

@@ -0,0 +1,226 @@
package pkg_test
import (
"archive/tar"
"bytes"
"compress/gzip"
"crypto/sha512"
"errors"
"io/fs"
"net/http"
"os"
"testing"
"testing/fstest"
"hakurei.app/container/check"
"hakurei.app/container/stub"
"hakurei.app/internal/pkg"
)
func TestTar(t *testing.T) {
t.Parallel()
checkWithCache(t, []cacheTestCase{
{"http", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
checkTarHTTP(t, base, c, fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"checksum": {Mode: fs.ModeDir | 0700},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP": {Mode: fs.ModeDir | 0700},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/check": {Mode: 0400, Data: []byte{0, 0}},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib": {Mode: fs.ModeDir | 0700},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/pkgconfig": {Mode: fs.ModeDir | 0700},
"checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP/lib/libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
"identifier": {Mode: fs.ModeDir | 0700},
"identifier/HnySzeLQvSBZuTUcvfmLEX_OmH4yJWWH788NxuLuv7kVn8_uPM6Ks4rqFWM2NZJY": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
"identifier/Zx5ZG9BAwegNT3zQwCySuI2ktCXxNgxirkGLFjW4FW06PtojYVaCdtEw8yuntPLa": {Mode: fs.ModeSymlink | 0777, Data: []byte("../checksum/1TL00Qb8dcqayX7wTO8WNaraHvY6b-KCsctLDTrb64QBCmxj_-byK1HdIUwMaFEP")},
"work": {Mode: fs.ModeDir | 0700},
}, pkg.MustDecode(
"cTw0h3AmYe7XudSoyEMByduYXqGi-N5ZkTZ0t9K5elsu3i_jNIVF5T08KR1roBFM",
))
}, pkg.MustDecode("NQTlc466JmSVLIyWklm_u8_g95jEEb98PxJU-kjwxLpfdjwMWJq0G8ze9R4Vo1Vu")},
{"http expand", nil, func(t *testing.T, base *check.Absolute, c *pkg.Cache) {
checkTarHTTP(t, base, c, fstest.MapFS{
".": {Mode: fs.ModeDir | 0700},
"lib": {Mode: fs.ModeDir | 0700},
"lib/libedac.so": {Mode: fs.ModeSymlink | 0777, Data: []byte("/proc/nonexistent/libedac.so")},
}, pkg.MustDecode(
"CH3AiUrCCcVOjOYLaMKKK1Da78989JtfHeIsxMzWOQFiN4mrCLDYpoDxLWqJWCUN",
))
}, pkg.MustDecode("hSoSSgCYTNonX3Q8FjvjD1fBl-E-BQyA6OTXro2OadXqbST4tZ-akGXszdeqphRe")},
})
}
func checkTarHTTP(
t *testing.T,
base *check.Absolute,
c *pkg.Cache,
testdataFsys fs.FS,
wantChecksum pkg.Checksum,
) {
var testdata string
{
var buf bytes.Buffer
w := tar.NewWriter(&buf)
if err := w.AddFS(testdataFsys); err != nil {
t.Fatalf("AddFS: error = %v", err)
}
if err := w.Close(); err != nil {
t.Fatalf("Close: error = %v", err)
}
var zbuf bytes.Buffer
gw := gzip.NewWriter(&zbuf)
if _, err := gw.Write(buf.Bytes()); err != nil {
t.Fatalf("Write: error = %v", err)
}
if err := gw.Close(); err != nil {
t.Fatalf("Close: error = %v", err)
}
testdata = zbuf.String()
}
testdataChecksum := func() pkg.Checksum {
h := sha512.New384()
h.Write([]byte(testdata))
return (pkg.Checksum)(h.Sum(nil))
}()
var transport http.Transport
client := http.Client{Transport: &transport}
transport.RegisterProtocol("file", http.NewFileTransportFS(fstest.MapFS{
"testdata": {Data: []byte(testdata), Mode: 0400},
}))
wantIdent := func() pkg.ID {
h := sha512.New384()
// kind uint64
h.Write([]byte{byte(pkg.KindTar), 0, 0, 0, 0, 0, 0, 0})
// deps_sz uint64
h.Write([]byte{1, 0, 0, 0, 0, 0, 0, 0})
// kind uint64
h.Write([]byte{byte(pkg.KindHTTPGet), 0, 0, 0, 0, 0, 0, 0})
// ident ID
h0 := sha512.New384()
// kind uint64
h0.Write([]byte{byte(pkg.KindHTTPGet), 0, 0, 0, 0, 0, 0, 0})
// deps_sz uint64
h0.Write([]byte{0, 0, 0, 0, 0, 0, 0, 0})
// url string
h0.Write([]byte{byte(pkg.IRKindString), 0, 0, 0})
h0.Write([]byte{0x10, 0, 0, 0})
h0.Write([]byte("file:///testdata"))
// end(KnownChecksum)
h0.Write([]byte{byte(pkg.IRKindEnd), 0, 0, 0})
h0.Write([]byte{byte(pkg.IREndKnownChecksum), 0, 0, 0})
// checksum Checksum
h0.Write(testdataChecksum[:])
h.Write(h0.Sum(nil))
// compression uint32
h.Write([]byte{byte(pkg.IRKindUint32), 0, 0, 0})
h.Write([]byte{pkg.TarGzip, 0, 0, 0})
// end
h.Write([]byte{byte(pkg.IRKindEnd), 0, 0, 0})
h.Write([]byte{0, 0, 0, 0})
return pkg.ID(h.Sum(nil))
}()
a := pkg.NewHTTPGetTar(
&client,
"file:///testdata",
testdataChecksum,
pkg.TarGzip,
)
tarDir := stubArtifact{
kind: pkg.KindExec,
params: []byte("directory containing a single regular file"),
cure: func(t *pkg.TContext) error {
work := t.GetWorkDir()
if err := os.MkdirAll(work.String(), 0700); err != nil {
return err
}
return os.WriteFile(
work.Append("sample.tar.gz").String(),
[]byte(testdata),
0400,
)
},
}
tarDirMulti := stubArtifact{
kind: pkg.KindExec,
params: []byte("directory containing a multiple entries"),
cure: func(t *pkg.TContext) error {
work := t.GetWorkDir()
if err := os.MkdirAll(work.Append(
"garbage",
).String(), 0700); err != nil {
return err
}
return os.WriteFile(
work.Append("sample.tar.gz").String(),
[]byte(testdata),
0400,
)
},
}
tarDirType := stubArtifact{
kind: pkg.KindExec,
params: []byte("directory containing a symbolic link"),
cure: func(t *pkg.TContext) error {
work := t.GetWorkDir()
if err := os.MkdirAll(work.String(), 0700); err != nil {
return err
}
return os.Symlink(
work.String(),
work.Append("sample.tar.gz").String(),
)
},
}
// destroy these to avoid including it in flatten test case
defer newDestroyArtifactFunc(&tarDir)(t, base, c)
defer newDestroyArtifactFunc(&tarDirMulti)(t, base, c)
defer newDestroyArtifactFunc(&tarDirType)(t, base, c)
cureMany(t, c, []cureStep{
{"file", a, base.Append(
"identifier",
pkg.Encode(wantIdent),
), wantChecksum, nil},
{"directory", pkg.NewTar(
&tarDir,
pkg.TarGzip,
), ignorePathname, wantChecksum, nil},
{"multiple entries", pkg.NewTar(
&tarDirMulti,
pkg.TarGzip,
), nil, pkg.Checksum{}, errors.New(
"input directory does not contain a single regular file",
)},
{"bad type", pkg.NewTar(
&tarDirType,
pkg.TarGzip,
), nil, pkg.Checksum{}, errors.New(
"input directory does not contain a single regular file",
)},
{"error passthrough", pkg.NewTar(&stubArtifact{
kind: pkg.KindExec,
params: []byte("doomed artifact"),
cure: func(t *pkg.TContext) error {
return stub.UniqueError(0xcafe)
},
}, pkg.TarGzip), nil, pkg.Checksum{}, stub.UniqueError(0xcafe)},
})
}

268
internal/pkg/testdata/main.go vendored Normal file
View File

@@ -0,0 +1,268 @@
//go:build testtool
package main
import (
"encoding/gob"
"log"
"net"
"os"
"path"
"reflect"
"slices"
"strings"
"hakurei.app/container/check"
"hakurei.app/container/fhs"
"hakurei.app/container/vfs"
)
func main() {
log.SetFlags(0)
log.SetPrefix("testtool: ")
var hostNet, layers, promote bool
if len(os.Args) == 2 && os.Args[0] == "testtool" {
switch os.Args[1] {
case "net":
hostNet = true
log.SetPrefix("testtool(net): ")
break
case "layers":
layers = true
log.SetPrefix("testtool(layers): ")
break
case "promote":
promote = true
log.SetPrefix("testtool(promote): ")
default:
log.Fatalf("Args: %q", os.Args)
return
}
} else if wantArgs := []string{"testtool"}; !slices.Equal(os.Args, wantArgs) {
log.Fatalf("Args: %q, want %q", os.Args, wantArgs)
}
var overlayRoot bool
wantEnv := []string{"HAKUREI_TEST=1"}
if len(os.Environ()) == 2 {
overlayRoot = true
if !layers && !promote {
log.SetPrefix("testtool(overlay root): ")
}
wantEnv = []string{"HAKUREI_TEST=1", "HAKUREI_ROOT=1"}
}
if !slices.Equal(wantEnv, os.Environ()) {
log.Fatalf("Environ: %q, want %q", os.Environ(), wantEnv)
}
var overlayWork bool
const (
wantExec = "/opt/bin/testtool"
wantExecWork = "/work/bin/testtool"
)
var iftPath string
if got, err := os.Executable(); err != nil {
log.Fatalf("Executable: error = %v", err)
} else {
iftPath = path.Join(path.Dir(path.Dir(got)), "ift")
if got != wantExec {
switch got {
case wantExecWork:
overlayWork = true
log.SetPrefix("testtool(overlay work): ")
default:
log.Fatalf("Executable: %q, want %q", got, wantExec)
}
}
}
wantHostname := "cure"
if hostNet {
wantHostname += "-net"
}
if hostname, err := os.Hostname(); err != nil {
log.Fatalf("Hostname: error = %v", err)
} else if hostname != wantHostname {
log.Fatalf("Hostname: %q, want %q", hostname, wantHostname)
}
var m *vfs.MountInfo
if f, err := os.Open(fhs.Proc + "self/mountinfo"); err != nil {
log.Fatalf("Open: error = %v", err)
} else {
err = vfs.NewMountInfoDecoder(f).Decode(&m)
closeErr := f.Close()
if err != nil {
log.Fatalf("Decode: error = %v", err)
}
if closeErr != nil {
log.Fatalf("Close: error = %v", err)
}
}
if ift, err := net.Interfaces(); err != nil {
log.Fatal(err)
} else if !hostNet {
if len(ift) != 1 || ift[0].Name != "lo" {
log.Fatalln("got interfaces", strings.Join(slices.Collect(func(yield func(ifn string) bool) {
for _, ifi := range ift {
if !yield(ifi.Name) {
break
}
}
}), ", "))
}
} else {
var iftParent []net.Interface
var r *os.File
if r, err = os.Open(iftPath); err != nil {
log.Fatal(err)
} else {
err = gob.NewDecoder(r).Decode(&iftParent)
closeErr := r.Close()
if err != nil {
log.Fatal(err)
}
if closeErr != nil {
log.Fatal(closeErr)
}
}
if !reflect.DeepEqual(ift, iftParent) {
log.Fatalf("Interfaces: %#v, want %#v", ift, iftParent)
}
}
const checksumEmptyDir = "MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU"
ident := "dztPS6jRjiZtCF4_p8AzfnxGp6obkhrgFVsxdodbKWUoAEVtDz3MykepJB4kI_ks"
log.Println(m)
next := func() { m = m.Next; log.Println(m) }
if overlayRoot {
ident = "RdMA-mubnrHuu3Ky1wWyxauSYCO0ZH_zCPUj3uDHqkfwv5sGcByoF_g5PjlGiClb"
if m.Root != "/" || m.Target != "/" ||
m.Source != "overlay" || m.FsType != "overlay" {
log.Fatal("unexpected root mount entry")
}
var lowerdir string
for _, o := range strings.Split(m.FsOptstr, ",") {
const lowerdirKey = "lowerdir="
if strings.HasPrefix(o, lowerdirKey) {
lowerdir = o[len(lowerdirKey):]
}
}
if !layers {
if path.Base(lowerdir) != checksumEmptyDir {
log.Fatal("unexpected artifact checksum")
}
} else {
ident = "p1t_drXr34i-jZNuxDMLaMOdL6tZvQqhavNafGynGqxOZoXAUTSn7kqNh3Ovv3DT"
lowerdirsEscaped := strings.Split(lowerdir, ":")
lowerdirs := lowerdirsEscaped[:0]
// ignore the option separator since it does not appear in ident
for i, e := range lowerdirsEscaped {
if len(e) > 0 &&
e[len(e)-1] == check.SpecialOverlayEscape[0] &&
(len(e) == 1 || e[len(e)-2] != check.SpecialOverlayEscape[0]) {
// ignore escaped pathname separator since it does not
// appear in ident
e = e[:len(e)-1]
if len(lowerdirsEscaped) != i {
lowerdirsEscaped[i+1] = e + lowerdirsEscaped[i+1]
continue
}
}
lowerdirs = append(lowerdirs, e)
}
if len(lowerdirs) != 2 ||
path.Base(lowerdirs[0]) != "MGWmEfjut2QE2xPJwTsmUzpff4BN_FEnQ7T0j7gvUCCiugJQNwqt9m151fm9D1yU" ||
path.Base(lowerdirs[1]) != "nY_CUdiaUM1OL4cPr5TS92FCJ3rCRV7Hm5oVTzAvMXwC03_QnTRfQ5PPs7mOU9fK" {
log.Fatalf("unexpected lowerdirs %s", strings.Join(lowerdirs, ", "))
}
}
} else {
if hostNet {
ident = "G8qPxD9puvvoOVV7lrT80eyDeIl3G_CCFoKw12c8mCjMdG1zF7NEPkwYpNubClK3"
}
if m.Root != "/sysroot" || m.Target != "/" {
log.Fatal("unexpected root mount entry")
}
next()
if path.Base(m.Root) != "OLBgp1GsljhM2TJ-sbHjaiH9txEUvgdDTAzHv2P24donTt6_529l-9Ua0vFImLlb" {
log.Fatal("unexpected file artifact checksum")
}
next()
if path.Base(m.Root) != checksumEmptyDir {
log.Fatal("unexpected artifact checksum")
}
}
if promote {
ident = "xXTIYcXmgJWNLC91c417RRrNM9cjELwEZHpGvf8Fk_GNP5agRJp_SicD0w9aMeLJ"
}
next() // testtool artifact
next()
if overlayWork {
ident = "5hlaukCirnXE4W_RSLJFOZN47Z5RiHnacXzdFp_70cLgiJUGR6cSb_HaFftkzi0-"
if m.Root != "/" || m.Target != "/work" ||
m.Source != "overlay" || m.FsType != "overlay" {
log.Fatal("unexpected work mount entry")
}
} else {
if path.Base(m.Root) != ident || m.Target != "/work" {
log.Fatal("unexpected work mount entry")
}
}
next()
if path.Base(m.Root) != ident || m.Target != "/tmp" {
log.Fatal("unexpected temp mount entry")
}
next()
if m.Root != "/" || m.Target != "/proc" || m.Source != "proc" || m.FsType != "proc" {
log.Fatal("unexpected proc mount entry")
}
next()
if m.Root != "/" || m.Target != "/dev" || m.Source != "devtmpfs" || m.FsType != "tmpfs" {
log.Fatal("unexpected dev mount entry")
}
for i := 0; i < 9; i++ { // private /dev entries
next()
}
if m.Next != nil {
log.Println("unexpected extra mount entries")
for m.Next != nil {
next()
}
os.Exit(1)
}
checkData := []byte{0}
if hostNet {
checkData = []byte("net")
}
if err := os.WriteFile("check", checkData, 0400); err != nil {
log.Fatal(err)
}
}

90
internal/rosa/acl.go Normal file
View File

@@ -0,0 +1,90 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newAttr() pkg.Artifact {
const (
version = "2.5.2"
checksum = "YWEphrz6vg1sUMmHHVr1CRo53pFXRhq_pjN-AlG8UgwZK1y6m7zuDhxqJhD0SV0l"
)
return t.NewViaMake("attr", version, t.NewPatchedSource(
"attr", version, pkg.NewHTTPGetTar(
nil, "https://download.savannah.nongnu.org/releases/attr/"+
"attr-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), true, [2]string{"libgen-basename", `From 8a80d895dfd779373363c3a4b62ecce5a549efb2 Mon Sep 17 00:00:00 2001
From: "Haelwenn (lanodan) Monnier" <contact@hacktivis.me>
Date: Sat, 30 Mar 2024 10:17:10 +0100
Subject: tools/attr.c: Add missing libgen.h include for basename(3)
Fixes compilation issue with musl and modern C99 compilers.
See: https://bugs.gentoo.org/926294
---
tools/attr.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/attr.c b/tools/attr.c
index f12e4af..6a3c1e9 100644
--- a/tools/attr.c
+++ b/tools/attr.c
@@ -28,6 +28,7 @@
#include <errno.h>
#include <string.h>
#include <locale.h>
+#include <libgen.h>
#include <attr/attributes.h>
--
cgit v1.1`}, [2]string{"musl-errno", `diff --git a/test/attr.test b/test/attr.test
index 6ce2f9b..e9bde92 100644
--- a/test/attr.test
+++ b/test/attr.test
@@ -11,7 +11,7 @@ Try various valid and invalid names
$ touch f
$ setfattr -n user -v value f
- > setfattr: f: Operation not supported
+ > setfattr: f: Not supported
$ setfattr -n user. -v value f
> setfattr: f: Invalid argument
`},
), &MakeAttr{
ScriptEarly: `
ln -s ../../system/bin/perl /usr/bin
`,
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(Perl),
)
}
func init() { artifactsF[Attr] = Toolchain.newAttr }
func (t Toolchain) newACL() pkg.Artifact {
const (
version = "2.3.2"
checksum = "-fY5nwH4K8ZHBCRXrzLdguPkqjKI6WIiGu4dBtrZ1o0t6AIU73w8wwJz_UyjIS0P"
)
return t.NewViaMake("acl", version, pkg.NewHTTPGetTar(
nil,
"https://download.savannah.nongnu.org/releases/acl/"+
"acl-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Configure: [][2]string{
{"enable-static"},
},
// makes assumptions about uid_map/gid_map
SkipCheck: true,
},
t.Load(Attr),
)
}
func init() { artifactsF[ACL] = Toolchain.newACL }

175
internal/rosa/all.go Normal file
View File

@@ -0,0 +1,175 @@
package rosa
import (
"sync"
"hakurei.app/internal/pkg"
)
// PArtifact is a lazily-initialised [pkg.Artifact] preset.
type PArtifact int
const (
ACL PArtifact = iota
Attr
Autoconf
Automake
Bash
Binutils
CMake
Coreutils
Curl
Diffutils
Findutils
Fuse
Gawk
GMP
Gettext
Git
Go
Gperf
Grep
Gzip
Hakurei
HakureiDist
IniConfig
KernelHeaders
LibXau
Libexpat
Libpsl
Libffi
Libgd
Libtool
Libseccomp
Libucontext
Libxml2
M4
MPC
MPFR
Make
Meson
Mksh
NSS
NSSCACert
Ninja
OpenSSL
Packaging
Patch
Perl
PkgConfig
Pluggy
PyTest
Pygments
Python
Rsync
Sed
Setuptools
Toybox
toyboxEarly
Unzip
utilMacros
Wayland
WaylandProtocols
XCB
XCBProto
Xproto
XZ
Zlib
buildcatrust
// gcc is a hacked-to-pieces GCC toolchain meant for use in intermediate
// stages only. This preset and its direct output must never be exposed.
gcc
// _presetEnd is the total number of presets and does not denote a preset.
_presetEnd
)
var (
// artifactsF is an array of functions for the result of [PArtifact].
artifactsF [_presetEnd]func(t Toolchain) pkg.Artifact
// artifacts stores the result of artifactsF.
artifacts [_toolchainEnd][len(artifactsF)]pkg.Artifact
// artifactsOnce is for lazy initialisation of artifacts.
artifactsOnce [_toolchainEnd][len(artifactsF)]sync.Once
)
// Load returns the resulting [pkg.Artifact] of [PArtifact].
func (t Toolchain) Load(p PArtifact) pkg.Artifact {
artifactsOnce[t][p].Do(func() {
artifacts[t][p] = artifactsF[p](t)
})
return artifacts[t][p]
}
// ResolveName returns a [PArtifact] by name.
func ResolveName(name string) (p PArtifact, ok bool) {
p, ok = map[string]PArtifact{
"acl": ACL,
"attr": Attr,
"autoconf": Autoconf,
"automake": Automake,
"bash": Bash,
"binutils": Binutils,
"cmake": CMake,
"coreutils": Coreutils,
"curl": Curl,
"diffutils": Diffutils,
"findutils": Findutils,
"fuse": Fuse,
"gawk": Gawk,
"gmp": GMP,
"gettext": Gettext,
"git": Git,
"go": Go,
"gperf": Gperf,
"grep": Grep,
"gzip": Gzip,
"hakurei": Hakurei,
"hakurei-dist": HakureiDist,
"iniconfig": IniConfig,
"kernel-headers": KernelHeaders,
"libXau": LibXau,
"libexpat": Libexpat,
"libpsl": Libpsl,
"libseccomp": Libseccomp,
"libucontext": Libucontext,
"libxml2": Libxml2,
"libffi": Libffi,
"libgd": Libgd,
"libtool": Libtool,
"m4": M4,
"mpc": MPC,
"mpfr": MPFR,
"make": Make,
"meson": Meson,
"mksh": Mksh,
"nss": NSS,
"nss-cacert": NSSCACert,
"ninja": Ninja,
"openssl": OpenSSL,
"packaging": Packaging,
"patch": Patch,
"perl": Perl,
"pkg-config": PkgConfig,
"pluggy": Pluggy,
"pytest": PyTest,
"pygments": Pygments,
"python": Python,
"rsync": Rsync,
"sed": Sed,
"setuptools": Setuptools,
"toybox": Toybox,
"unzip": Unzip,
"wayland": Wayland,
"wayland-protocols": WaylandProtocols,
"xcb": XCB,
"xcb-proto": XCBProto,
"xproto": Xproto,
"xz": XZ,
"zlib": Zlib,
}[name]
return
}

125
internal/rosa/busybox.go Normal file
View File

@@ -0,0 +1,125 @@
package rosa
import (
"fmt"
"io"
"net/http"
"os"
"runtime"
"time"
"hakurei.app/container/fhs"
"hakurei.app/internal/pkg"
)
// busyboxBin is a busybox binary distribution installed under bin/busybox.
type busyboxBin struct {
// Underlying busybox binary.
bin pkg.FileArtifact
}
// Kind returns the hardcoded [pkg.Kind] value.
func (a busyboxBin) Kind() pkg.Kind { return kindBusyboxBin }
// Params is a noop.
func (a busyboxBin) Params(*pkg.IContext) {}
// IsExclusive returns false: Cure performs a trivial filesystem write.
func (busyboxBin) IsExclusive() bool { return false }
// Dependencies returns the underlying busybox [pkg.File].
func (a busyboxBin) Dependencies() []pkg.Artifact {
return []pkg.Artifact{a.bin}
}
func init() {
pkg.Register(kindBusyboxBin, func(r *pkg.IRReader) pkg.Artifact {
a := busyboxBin{r.Next().(pkg.FileArtifact)}
if _, ok := r.Finalise(); ok {
panic(pkg.ErrUnexpectedChecksum)
}
return a
})
}
// String returns the reporting name of the underlying file prefixed with expand.
func (a busyboxBin) String() string {
return "expand-" + a.bin.(fmt.Stringer).String()
}
// Cure installs the underlying busybox [pkg.File] to bin/busybox.
func (a busyboxBin) Cure(t *pkg.TContext) (err error) {
var r io.ReadCloser
if r, err = t.Open(a.bin); err != nil {
return
}
defer func() {
closeErr := r.Close()
if err == nil {
err = closeErr
}
}()
binDir := t.GetWorkDir().Append("bin")
if err = os.MkdirAll(binDir.String(), 0700); err != nil {
return
}
var w *os.File
if w, err = os.OpenFile(
binDir.Append("busybox").String(),
os.O_WRONLY|os.O_CREATE|os.O_EXCL,
0500,
); err != nil {
return
}
defer func() {
closeErr := w.Close()
if err == nil {
err = closeErr
}
}()
_, err = io.Copy(w, r)
return
}
// newBusyboxBin returns a [pkg.Artifact] containing a busybox installation from
// the https://busybox.net/downloads/binaries/ binary release.
func newBusyboxBin() pkg.Artifact {
var version, url, checksum string
switch runtime.GOARCH {
case "amd64":
version = "1.35.0"
url = "https://busybox.net/downloads/binaries/" +
version + "-" + linuxArch() + "-linux-musl/busybox"
checksum = "L7OBIsPu9enNHn7FqpBT1kOg_mCLNmetSeNMA3i4Y60Z5jTgnlX3qX3zcQtLx5AB"
case "arm64":
version = "1.31.0"
url = "https://busybox.net/downloads/binaries/" +
version + "-defconfig-multiarch-musl/busybox-armv8l"
checksum = "npJjBO7iwhjW6Kx2aXeSxf8kXhVgTCDChOZTTsI8ZfFfa3tbsklxRiidZQdrVERg"
default:
panic("unsupported target " + runtime.GOARCH)
}
return pkg.NewExec(
"busybox-bin-"+version, nil, pkg.ExecTimeoutMax, false,
fhs.AbsRoot, []string{
"PATH=/system/bin",
},
AbsSystem.Append("bin", "busybox"),
[]string{"hush", "-c", "" +
"busybox mkdir -p /work/system/bin/ && " +
"busybox cp /system/bin/busybox /work/system/bin/ && " +
"busybox --install -s /work/system/bin/"},
pkg.Path(AbsSystem, true, busyboxBin{pkg.NewHTTPGet(
&http.Client{Transport: &http.Transport{
// busybox website is really slow to respond
TLSHandshakeTimeout: 2 * time.Minute,
}}, url,
mustDecode(checksum),
)}),
)
}

125
internal/rosa/cmake.go Normal file
View File

@@ -0,0 +1,125 @@
package rosa
import (
"slices"
"strings"
"hakurei.app/container/check"
"hakurei.app/internal/pkg"
)
func (t Toolchain) newCMake() pkg.Artifact {
const (
version = "4.2.1"
checksum = "Y3OdbMsob6Xk2y1DCME6z4Fryb5_TkFD7knRT8dTNIRtSqbiCJyyDN9AxggN_I75"
)
return t.New("cmake-"+version, 0, []pkg.Artifact{
t.Load(Make),
t.Load(KernelHeaders),
}, nil, nil, `
cd "$(mktemp -d)"
/usr/src/cmake/bootstrap \
--prefix=/system \
--parallel="$(nproc)" \
-- \
-DCMAKE_USE_OPENSSL=OFF
make "-j$(nproc)"
make DESTDIR=/work install
`, pkg.Path(AbsUsrSrc.Append("cmake"), true, t.NewPatchedSource(
// expected to be writable in the copy made during bootstrap
"cmake", version, pkg.NewHTTPGetTar(
nil, "https://github.com/Kitware/CMake/releases/download/"+
"v"+version+"/cmake-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false,
)))
}
func init() { artifactsF[CMake] = Toolchain.newCMake }
// CMakeAttr holds the project-specific attributes that will be applied to a new
// [pkg.Artifact] compiled via [CMake].
type CMakeAttr struct {
// Path elements joined with source.
Append []string
// Use source tree as scratch space.
Writable bool
// CMake CACHE entries.
Cache [][2]string
// Additional environment variables.
Env []string
// Runs before cmake.
ScriptEarly string
// Runs after cmake, replaces default.
ScriptConfigured string
// Runs after install.
Script string
// Override the default installation prefix [AbsSystem].
Prefix *check.Absolute
// Passed through to [Toolchain.New].
Paths []pkg.ExecPath
// Passed through to [Toolchain.New].
Flag int
}
// NewViaCMake returns a [pkg.Artifact] for compiling and installing via [CMake].
func (t Toolchain) NewViaCMake(
name, version, variant string,
source pkg.Artifact,
attr *CMakeAttr,
extra ...pkg.Artifact,
) pkg.Artifact {
if name == "" || version == "" || variant == "" {
panic("names must be non-empty")
}
if attr == nil {
attr = &CMakeAttr{
Cache: [][2]string{
{"CMAKE_BUILD_TYPE", "Release"},
},
}
}
if len(attr.Cache) == 0 {
panic("CACHE must be non-empty")
}
scriptConfigured := "cmake --build .\ncmake --install .\n"
if attr.ScriptConfigured != "" {
scriptConfigured = attr.ScriptConfigured
}
prefix := attr.Prefix
if prefix == nil {
prefix = AbsSystem
}
sourcePath := AbsUsrSrc.Append(name)
return t.New(name+"-"+variant+"-"+version, attr.Flag, stage3Concat(t, extra,
t.Load(CMake),
t.Load(Ninja),
), nil, slices.Concat([]string{
"ROSA_SOURCE=" + sourcePath.String(),
"ROSA_CMAKE_SOURCE=" + sourcePath.Append(attr.Append...).String(),
"ROSA_INSTALL_PREFIX=/work" + prefix.String(),
}, attr.Env), attr.ScriptEarly+`
mkdir /cure && cd /cure
cmake -G Ninja \
-DCMAKE_C_COMPILER_TARGET="${ROSA_TRIPLE}" \
-DCMAKE_CXX_COMPILER_TARGET="${ROSA_TRIPLE}" \
-DCMAKE_ASM_COMPILER_TARGET="${ROSA_TRIPLE}" \
`+strings.Join(slices.Collect(func(yield func(string) bool) {
for _, v := range attr.Cache {
if !yield("-D" + v[0] + "=" + v[1]) {
return
}
}
}), " \\\n\t")+` \
-DCMAKE_INSTALL_PREFIX="${ROSA_INSTALL_PREFIX}" \
"${ROSA_CMAKE_SOURCE}"
`+scriptConfigured+attr.Script, slices.Concat([]pkg.ExecPath{
pkg.Path(sourcePath, attr.Writable, source),
}, attr.Paths)...)
}

32
internal/rosa/curl.go Normal file
View File

@@ -0,0 +1,32 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newCurl() pkg.Artifact {
const (
version = "8.18.0"
checksum = "YpOolP_sx1DIrCEJ3elgVAu0wTLDS-EZMZFvOP0eha7FaLueZUlEpuMwDzJNyi7i"
)
return t.NewViaMake("curl", version, pkg.NewHTTPGetTar(
nil, "https://curl.se/download/curl-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), &MakeAttr{
Env: []string{
"TFLAGS=-j256",
},
Configure: [][2]string{
{"with-openssl"},
{"with-ca-bundle", "/system/etc/ssl/certs/ca-bundle.crt"},
},
ScriptConfigured: `
make "-j$(nproc)"
`,
},
t.Load(Perl),
t.Load(Libpsl),
t.Load(OpenSSL),
)
}
func init() { artifactsF[Curl] = Toolchain.newCurl }

163
internal/rosa/etc.go Normal file
View File

@@ -0,0 +1,163 @@
package rosa
import (
"errors"
"io"
"os"
"sync"
"syscall"
"hakurei.app/container/fhs"
"hakurei.app/internal/pkg"
)
// cureEtc contains deterministic elements of /etc, made available as part of
// [Toolchain]. This silences test suites expecting certain standard files to be
// available in /etc.
type cureEtc struct {
// Optional via newIANAEtc.
iana pkg.Artifact
}
// Cure writes hardcoded configuration to files under etc.
func (a cureEtc) Cure(t *pkg.FContext) (err error) {
etc := t.GetWorkDir().Append("etc")
if err = os.MkdirAll(etc.String(), 0700); err != nil {
return
}
for _, f := range [][2]string{
{"hosts", "127.0.0.1 localhost cure cure-net\n"},
{"passwd", `root:x:0:0:System administrator:/proc/nonexistent:/bin/sh
cure:x:1023:1023:Cure:/usr/src:/bin/sh
nobody:x:65534:65534:Overflow user:/proc/nonexistent:/system/bin/false
`},
{"group", `root:x:0:
cure:x:1023:
nobody:x:65534:
`},
} {
if err = os.WriteFile(
etc.Append(f[0]).String(),
[]byte(f[1]),
0400,
); err != nil {
return
}
}
if a.iana != nil {
iana, _ := t.GetArtifact(a.iana)
buf := make([]byte, syscall.Getpagesize()<<3)
for _, name := range []string{
"protocols",
"services",
} {
var dst, src *os.File
if dst, err = os.OpenFile(
etc.Append(name).String(),
syscall.O_CREAT|syscall.O_EXCL|syscall.O_WRONLY,
0400,
); err != nil {
return
}
if src, err = os.Open(
iana.Append(name).String(),
); err != nil {
_ = dst.Close()
return
}
_, err = io.CopyBuffer(dst, src, buf)
closeErrs := [...]error{
dst.Close(),
src.Close(),
}
if err != nil {
return
} else if err = errors.Join(closeErrs[:]...); err != nil {
return
}
}
}
return os.Chmod(etc.String(), 0500)
}
// Kind returns the hardcoded [pkg.Kind] value.
func (cureEtc) Kind() pkg.Kind { return kindEtc }
// Params writes whether iana-etc is populated.
func (a cureEtc) Params(ctx *pkg.IContext) {
if a.iana != nil {
ctx.WriteUint32(1)
} else {
ctx.WriteUint32(0)
}
}
func init() {
pkg.Register(kindEtc, func(r *pkg.IRReader) pkg.Artifact {
a := cureEtc{}
if r.ReadUint32() != 0 {
a.iana = r.Next()
}
if _, ok := r.Finalise(); ok {
panic(pkg.ErrUnexpectedChecksum)
}
return a
})
}
// IsExclusive returns false: Cure performs a few trivial filesystem writes.
func (cureEtc) IsExclusive() bool { return false }
// Dependencies returns a slice containing the backing iana-etc release.
func (a cureEtc) Dependencies() []pkg.Artifact {
if a.iana != nil {
return []pkg.Artifact{a.iana}
}
return nil
}
// String returns a hardcoded reporting name.
func (a cureEtc) String() string {
if a.iana == nil {
return "cure-etc-minimal"
}
return "cure-etc"
}
// newIANAEtc returns an unpacked iana-etc release.
func newIANAEtc() pkg.Artifact {
const (
version = "20251215"
checksum = "kvKz0gW_rGG5QaNK9ZWmWu1IEgYAdmhj_wR7DYrh3axDfIql_clGRHmelP7525NJ"
)
return pkg.NewHTTPGetTar(
nil, "https://github.com/Mic92/iana-etc/releases/download/"+
version+"/iana-etc-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)
}
var (
resolvconfPath pkg.ExecPath
resolvconfOnce sync.Once
)
// resolvconf returns a hardcoded /etc/resolv.conf file.
func resolvconf() pkg.ExecPath {
resolvconfOnce.Do(func() {
resolvconfPath = pkg.Path(
fhs.AbsEtc.Append("resolv.conf"), false,
pkg.NewFile("resolv.conf", []byte(`
nameserver 1.1.1.1
nameserver 1.0.0.1
`)),
)
})
return resolvconfPath
}

45
internal/rosa/fuse.go Normal file
View File

@@ -0,0 +1,45 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newFuse() pkg.Artifact {
const (
version = "3.18.1"
checksum = "COb-BgJRWXLbt9XUkNeuiroQizpMifXqxgieE1SlkMXhs_WGSyJStrmyewAw2hd6"
)
return t.New("fuse-"+version, 0, []pkg.Artifact{
t.Load(Python),
t.Load(Meson),
t.Load(Ninja),
t.Load(IniConfig),
t.Load(Packaging),
t.Load(Pluggy),
t.Load(Pygments),
t.Load(PyTest),
t.Load(KernelHeaders),
}, nil, nil, `
cd "$(mktemp -d)"
meson setup \
--reconfigure \
--buildtype=release \
--prefix=/system \
--prefer-static \
-Dtests=true \
-Duseroot=false \
-Dinitscriptdir=/system/init.d \
-Ddefault_library=both \
. /usr/src/fuse
meson compile
python3 -m pytest test/
meson install \
--destdir=/work
`, pkg.Path(AbsUsrSrc.Append("fuse"), false, pkg.NewHTTPGetTar(
nil, "https://github.com/libfuse/libfuse/releases/download/"+
"fuse-"+version+"/fuse-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func init() { artifactsF[Fuse] = Toolchain.newFuse }

101
internal/rosa/git.go Normal file
View File

@@ -0,0 +1,101 @@
package rosa
import (
"hakurei.app/internal/pkg"
)
func (t Toolchain) newGit() pkg.Artifact {
const (
version = "2.52.0"
checksum = "uH3J1HAN_c6PfGNJd2OBwW4zo36n71wmkdvityYnrh8Ak0D1IifiAvEWz9Vi9DmS"
)
return t.NewViaMake("git", version, t.NewPatchedSource(
"git", version, pkg.NewHTTPGetTar(
nil, "https://www.kernel.org/pub/software/scm/git/"+
"git-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false,
), &MakeAttr{
// uses source tree as scratch space
Writable: true,
InPlace: true,
// test suite in subdirectory
SkipCheck: true,
Make: []string{"all"},
ScriptEarly: `
cd /usr/src/git
make configure
`,
Script: `
ln -s ../../system/bin/perl /usr/bin/ || true
function disable_test {
local test=$1 pattern=$2
if [ $# -eq 1 ]; then
rm "t/${test}.sh"
else
sed -i "t/${test}.sh" \
-e "/^\s*test_expect_.*$pattern/,/^\s*' *\$/{s/^/: #/}"
fi
}
disable_test t5319-multi-pack-index
disable_test t1305-config-include
disable_test t3900-i18n-commit
disable_test t3507-cherry-pick-conflict
disable_test t4201-shortlog
disable_test t5303-pack-corruption-resilience
disable_test t4301-merge-tree-write-tree
disable_test t8005-blame-i18n
disable_test t9350-fast-export
disable_test t9300-fast-import
disable_test t0211-trace2-perf
disable_test t1517-outside-repo
disable_test t2200-add-update
make \
-C t \
GIT_PROVE_OPTS="--jobs 32 --failures" \
prove
`,
},
t.Load(Perl),
t.Load(Diffutils),
t.Load(M4),
t.Load(Autoconf),
t.Load(Gettext),
t.Load(Zlib),
t.Load(Curl),
t.Load(OpenSSL),
t.Load(Libexpat),
)
}
func init() { artifactsF[Git] = Toolchain.newGit }
// NewViaGit returns a [pkg.Artifact] for cloning a git repository.
func (t Toolchain) NewViaGit(
name, url, rev string,
checksum pkg.Checksum,
) pkg.Artifact {
return t.New(name+"-"+rev, 0, []pkg.Artifact{
t.Load(NSSCACert),
t.Load(OpenSSL),
t.Load(Libpsl),
t.Load(Curl),
t.Load(Libexpat),
t.Load(Git),
}, &checksum, nil, `
git \
-c advice.detachedHead=false \
clone \
--revision=`+rev+` \
`+url+` \
/work
rm -rf /work/.git
`, resolvconf())
}

623
internal/rosa/gnu.go Normal file
View File

@@ -0,0 +1,623 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newM4() pkg.Artifact {
const (
version = "1.4.20"
checksum = "RT0_L3m4Co86bVBY3lCFAEs040yI1WdeNmRylFpah8IZovTm6O4wI7qiHJN3qsW9"
)
return t.NewViaMake("m4", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/m4/m4-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/m4
chmod +w tests/test-c32ispunct.sh && echo '#!/bin/sh' > tests/test-c32ispunct.sh
`,
},
t.Load(Diffutils),
)
}
func init() { artifactsF[M4] = Toolchain.newM4 }
func (t Toolchain) newSed() pkg.Artifact {
const (
version = "4.9"
checksum = "pe7HWH4PHNYrazOTlUoE1fXmhn2GOPFN_xE62i0llOr3kYGrH1g2_orDz0UtZ9Nt"
)
return t.NewViaMake("sed", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/sed/sed-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), nil,
t.Load(Diffutils),
)
}
func init() { artifactsF[Sed] = Toolchain.newSed }
func (t Toolchain) newAutoconf() pkg.Artifact {
const (
version = "2.72"
checksum = "-c5blYkC-xLDer3TWEqJTyh1RLbOd1c5dnRLKsDnIrg_wWNOLBpaqMY8FvmUFJ33"
)
return t.NewViaMake("autoconf", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/autoconf/autoconf-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Make: []string{
`TESTSUITEFLAGS="-j$(nproc)"`,
},
Flag: TExclusive,
},
t.Load(M4),
t.Load(Perl),
t.Load(Bash),
t.Load(Diffutils),
)
}
func init() { artifactsF[Autoconf] = Toolchain.newAutoconf }
func (t Toolchain) newAutomake() pkg.Artifact {
const (
version = "1.18.1"
checksum = "FjvLG_GdQP7cThTZJLDMxYpRcKdpAVG-YDs1Fj1yaHlSdh_Kx6nRGN14E0r_BjcG"
)
return t.NewViaMake("automake", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/automake/automake-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/automake
test_disable() { chmod +w "$2" && echo "$1" > "$2"; }
test_disable '#!/bin/sh' t/objcxx-minidemo.sh
test_disable '#!/bin/sh' t/objcxx-deps.sh
test_disable '#!/bin/sh' t/dist-no-built-sources.sh
test_disable '#!/bin/sh' t/distname.sh
test_disable '#!/bin/sh' t/pr9.sh
`,
},
t.Load(M4),
t.Load(Perl),
t.Load(Grep),
t.Load(Gzip),
t.Load(Autoconf),
t.Load(Diffutils),
)
}
func init() { artifactsF[Automake] = Toolchain.newAutomake }
func (t Toolchain) newLibtool() pkg.Artifact {
const (
version = "2.5.4"
checksum = "pa6LSrQggh8mSJHQfwGjysAApmZlGJt8wif2cCLzqAAa2jpsTY0jZ-6stS3BWZ2Q"
)
return t.NewViaMake("libtool", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/libtool/libtool-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Make: []string{
`TESTSUITEFLAGS=32`,
},
},
t.Load(M4),
t.Load(Diffutils),
)
}
func init() { artifactsF[Libtool] = Toolchain.newLibtool }
func (t Toolchain) newGzip() pkg.Artifact {
const (
version = "1.14"
checksum = "NWhjUavnNfTDFkZJyAUonL9aCOak8GVajWX2OMlzpFnuI0ErpBFyj88mz2xSjz0q"
)
return t.NewViaMake("gzip", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/gzip/gzip-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
// dependency loop
SkipCheck: true,
})
}
func init() { artifactsF[Gzip] = Toolchain.newGzip }
func (t Toolchain) newGettext() pkg.Artifact {
const (
version = "1.0"
checksum = "3MasKeEdPeFEgWgzsBKk7JqWqql1wEMbgPmzAfs-mluyokoW0N8oQVxPQoOnSdgC"
)
return t.NewViaMake("gettext", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/gettext/gettext-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/gettext
test_disable() { chmod +w "$2" && echo "$1" > "$2"; }
test_disable '#!/bin/sh' gettext-tools/tests/msgcat-22
test_disable '#!/bin/sh' gettext-tools/tests/msgconv-2
test_disable '#!/bin/sh' gettext-tools/tests/msgconv-8
test_disable '#!/bin/sh' gettext-tools/tests/xgettext-python-3
test_disable '#!/bin/sh' gettext-tools/tests/msgmerge-compendium-6
test_disable '#!/bin/sh' gettext-tools/tests/gettextpo-1
test_disable '#!/bin/sh' gettext-tools/tests/format-c-5
test_disable '#!/bin/sh' gettext-tools/gnulib-tests/test-c32ispunct.sh
test_disable 'int main(){return 0;}' gettext-tools/gnulib-tests/test-stdcountof-h.c
touch gettext-tools/autotools/archive.dir.tar
`,
},
t.Load(Diffutils),
t.Load(Gzip),
t.Load(Sed),
t.Load(KernelHeaders),
)
}
func init() { artifactsF[Gettext] = Toolchain.newGettext }
func (t Toolchain) newDiffutils() pkg.Artifact {
const (
version = "3.12"
checksum = "9J5VAq5oA7eqwzS1Yvw-l3G5o-TccUrNQR3PvyB_lgdryOFAfxtvQfKfhdpquE44"
)
return t.NewViaMake("diffutils", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/diffutils/diffutils-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/diffutils
test_disable() { chmod +w "$2" && echo "$1" > "$2"; }
test_disable '#!/bin/sh' gnulib-tests/test-c32ispunct.sh
test_disable 'int main(){return 0;}' gnulib-tests/test-c32ispunct.c
test_disable '#!/bin/sh' tests/cmp
`,
Flag: TEarly,
})
}
func init() { artifactsF[Diffutils] = Toolchain.newDiffutils }
func (t Toolchain) newPatch() pkg.Artifact {
const (
version = "2.8"
checksum = "MA0BQc662i8QYBD-DdGgyyfTwaeALZ1K0yusV9rAmNiIsQdX-69YC4t9JEGXZkeR"
)
return t.NewViaMake("patch", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/patch/patch-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/patch
test_disable() { chmod +w "$2" && echo "$1" > "$2"; }
test_disable '#!/bin/sh' tests/ed-style
test_disable '#!/bin/sh' tests/need-filename
`,
Flag: TEarly,
})
}
func init() { artifactsF[Patch] = Toolchain.newPatch }
func (t Toolchain) newBash() pkg.Artifact {
const (
version = "5.3"
checksum = "4LQ_GRoB_ko-Ih8QPf_xRKA02xAm_TOxQgcJLmFDT6udUPxTAWrsj-ZNeuTusyDq"
)
return t.NewViaMake("bash", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/bash/bash-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Script: "ln -s bash /work/system/bin/sh\n",
Configure: [][2]string{
{"without-bash-malloc"},
},
Flag: TEarly,
})
}
func init() { artifactsF[Bash] = Toolchain.newBash }
func (t Toolchain) newCoreutils() pkg.Artifact {
const (
version = "9.9"
checksum = "B1_TaXj1j5aiVIcazLWu8Ix03wDV54uo2_iBry4qHG6Y-9bjDpUPlkNLmU_3Nvw6"
)
return t.NewViaMake("coreutils", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/coreutils/coreutils-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/coreutils
test_disable() { chmod +w "$2" && echo "$1" > "$2"; }
test_disable '#!/bin/sh' gnulib-tests/test-c32ispunct.sh
test_disable '#!/bin/sh' tests/split/line-bytes.sh
test_disable '#!/bin/sh' tests/dd/no-allocate.sh
test_disable 'int main(){return 0;}' gnulib-tests/test-chown.c
test_disable 'int main(){return 0;}' gnulib-tests/test-fchownat.c
test_disable 'int main(){return 0;}' gnulib-tests/test-lchown.c
`,
Flag: TEarly,
},
t.Load(Perl),
t.Load(Bash),
t.Load(KernelHeaders),
)
}
func init() { artifactsF[Coreutils] = Toolchain.newCoreutils }
func (t Toolchain) newGperf() pkg.Artifact {
const (
version = "3.3"
checksum = "RtIy9pPb_Bb8-31J2Nw-rRGso2JlS-lDlVhuNYhqR7Nt4xM_nObznxAlBMnarJv7"
)
return t.NewViaMake("gperf", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gperf/gperf-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), nil,
t.Load(Diffutils),
)
}
func init() { artifactsF[Gperf] = Toolchain.newGperf }
func (t Toolchain) newGawk() pkg.Artifact {
const (
version = "5.3.2"
checksum = "uIs0d14h_d2DgMGYwrPtegGNyt_bxzG3D6Fe-MmExx_pVoVkQaHzrtmiXVr6NHKk"
)
return t.NewViaMake("gawk", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/gawk/gawk-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Flag: TEarly,
// dependency loop
SkipCheck: true,
})
}
func init() { artifactsF[Gawk] = Toolchain.newGawk }
func (t Toolchain) newGrep() pkg.Artifact {
const (
version = "3.12"
checksum = "qMB4RjaPNRRYsxix6YOrjE8gyAT1zVSTy4nW4wKW9fqa0CHYAuWgPwDTirENzm_1"
)
return t.NewViaMake("grep", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/grep/grep-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/grep
test_disable() { chmod +w "$2" && echo "$1" > "$2"; }
test_disable '#!/bin/sh' gnulib-tests/test-c32ispunct.sh
test_disable 'int main(){return 0;}' gnulib-tests/test-c32ispunct.c
`,
},
t.Load(Diffutils),
)
}
func init() { artifactsF[Grep] = Toolchain.newGrep }
func (t Toolchain) newFindutils() pkg.Artifact {
const (
version = "4.10.0"
checksum = "ZXABdNBQXL7QjTygynRRTdXYWxQKZ0Wn5eMd3NUnxR0xaS0u0VfcKoTlbo50zxv6"
)
return t.NewViaMake("findutils", version, pkg.NewHTTPGet(
nil, "https://ftpmirror.gnu.org/gnu/findutils/findutils-"+version+".tar.xz",
mustDecode(checksum),
), &MakeAttr{
SourceSuffix: ".tar.xz",
ScriptEarly: `
cd /usr/src/
tar xf findutils.tar.xz
mv findutils-` + version + ` findutils
cd findutils
echo '#!/bin/sh' > gnulib-tests/test-c32ispunct.sh
echo 'int main(){return 0;}' > tests/xargs/test-sigusr.c
`,
},
t.Load(Diffutils),
t.Load(XZ),
t.Load(Sed),
)
}
func init() { artifactsF[Findutils] = Toolchain.newFindutils }
func (t Toolchain) newBinutils() pkg.Artifact {
const (
version = "2.45"
checksum = "hlLtqqHDmzAT2OQVHaKEd_io2DGFvJkaeS-igBuK8bRRir7LUKGHgHYNkDVKaHTT"
)
return t.NewViaMake("binutils", version, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/binutils/binutils-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), &MakeAttr{
ScriptConfigured: `
make "-j$(nproc)"
`,
},
t.Load(Bash),
)
}
func init() { artifactsF[Binutils] = Toolchain.newBinutils }
func (t Toolchain) newGMP() pkg.Artifact {
const (
version = "6.3.0"
checksum = "yrgbgEDWKDdMWVHh7gPbVl56-sRtVVhfvv0M_LX7xMUUk_mvZ1QOJEAnt7g4i3k5"
)
return t.NewViaMake("gmp", version, pkg.NewHTTPGetTar(
nil, "https://gcc.gnu.org/pub/gcc/infrastructure/"+
"gmp-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), &MakeAttr{
ScriptConfigured: `
make "-j$(nproc)"
`,
},
t.Load(M4),
)
}
func init() { artifactsF[GMP] = Toolchain.newGMP }
func (t Toolchain) newMPFR() pkg.Artifact {
const (
version = "4.2.2"
checksum = "wN3gx0zfIuCn9r3VAn_9bmfvAYILwrRfgBjYSD1IjLqyLrLojNN5vKyQuTE9kA-B"
)
return t.NewViaMake("mpfr", version, pkg.NewHTTPGetTar(
nil, "https://gcc.gnu.org/pub/gcc/infrastructure/"+
"mpfr-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), nil,
t.Load(GMP),
)
}
func init() { artifactsF[MPFR] = Toolchain.newMPFR }
func (t Toolchain) newMPC() pkg.Artifact {
const (
version = "1.3.1"
checksum = "o8r8K9R4x7PuRx0-JE3-bC5jZQrtxGV2nkB773aqJ3uaxOiBDCID1gKjPaaDxX4V"
)
return t.NewViaMake("mpc", version, pkg.NewHTTPGetTar(
nil, "https://gcc.gnu.org/pub/gcc/infrastructure/"+
"mpc-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), nil,
t.Load(GMP),
t.Load(MPFR),
)
}
func init() { artifactsF[MPC] = Toolchain.newMPC }
func (t Toolchain) newGCC() pkg.Artifact {
const (
version = "15.2.0"
checksum = "TXJ5WrbXlGLzy1swghQTr4qxgDCyIZFgJry51XEPTBZ8QYbVmFeB4lZbSMtPJ-a1"
)
return t.NewViaMake("gcc", version, t.NewPatchedSource(
"gcc", version,
pkg.NewHTTPGetTar(
nil, "https://ftp.tsukuba.wide.ad.jp/software/gcc/releases/"+
"gcc-"+version+"/gcc-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), true, [2]string{"musl-off64_t-loff_t", `diff --git a/libgo/sysinfo.c b/libgo/sysinfo.c
index 180f5c31d74..44d7ea73f7d 100644
--- a/libgo/sysinfo.c
+++ b/libgo/sysinfo.c
@@ -365,11 +365,7 @@ enum {
typedef loff_t libgo_loff_t_type;
#endif
-#if defined(HAVE_OFF64_T)
-typedef off64_t libgo_off_t_type;
-#else
typedef off_t libgo_off_t_type;
-#endif
// The following section introduces explicit references to types and
// constants of interest to support bootstrapping libgo using a
`}, [2]string{"musl-legacy-lfs", `diff --git a/libgo/go/internal/syscall/unix/at_largefile.go b/libgo/go/internal/syscall/unix/at_largefile.go
index 82e0dcfd074..16151ecad1b 100644
--- a/libgo/go/internal/syscall/unix/at_largefile.go
+++ b/libgo/go/internal/syscall/unix/at_largefile.go
@@ -10,5 +10,5 @@ import (
"syscall"
)
-//extern fstatat64
+//extern fstatat
func fstatat(int32, *byte, *syscall.Stat_t, int32) int32
diff --git a/libgo/go/os/dir_largefile.go b/libgo/go/os/dir_largefile.go
index 1fc5ee0771f..0c6dffe1a75 100644
--- a/libgo/go/os/dir_largefile.go
+++ b/libgo/go/os/dir_largefile.go
@@ -11,5 +11,5 @@ package os
import "syscall"
-//extern readdir64
+//extern readdir
func libc_readdir(*syscall.DIR) *syscall.Dirent
diff --git a/libgo/go/syscall/libcall_glibc.go b/libgo/go/syscall/libcall_glibc.go
index 5c1ec483c75..5a1245ed44b 100644
--- a/libgo/go/syscall/libcall_glibc.go
+++ b/libgo/go/syscall/libcall_glibc.go
@@ -114,7 +114,7 @@ func Pipe2(p []int, flags int) (err error) {
}
//sys sendfile(outfd int, infd int, offset *Offset_t, count int) (written int, err error)
-//sendfile64(outfd _C_int, infd _C_int, offset *Offset_t, count Size_t) Ssize_t
+//sendfile(outfd _C_int, infd _C_int, offset *Offset_t, count Size_t) Ssize_t
func Sendfile(outfd int, infd int, offset *int64, count int) (written int, err error) {
if race.Enabled {
diff --git a/libgo/go/syscall/libcall_linux.go b/libgo/go/syscall/libcall_linux.go
index 03ca7261b59..ad21fd0b3ac 100644
--- a/libgo/go/syscall/libcall_linux.go
+++ b/libgo/go/syscall/libcall_linux.go
@@ -158,7 +158,7 @@ func Reboot(cmd int) (err error) {
//adjtimex(buf *Timex) _C_int
//sys Fstatfs(fd int, buf *Statfs_t) (err error)
-//fstatfs64(fd _C_int, buf *Statfs_t) _C_int
+//fstatfs(fd _C_int, buf *Statfs_t) _C_int
func Gettid() (tid int) {
r1, _, _ := Syscall(SYS_GETTID, 0, 0, 0)
@@ -245,7 +245,7 @@ func Splice(rfd int, roff *int64, wfd int, woff *int64, len int, flags int) (n i
}
//sys Statfs(path string, buf *Statfs_t) (err error)
-//statfs64(path *byte, buf *Statfs_t) _C_int
+//statfs(path *byte, buf *Statfs_t) _C_int
//sysnb Sysinfo(info *Sysinfo_t) (err error)
//sysinfo(info *Sysinfo_t) _C_int
diff --git a/libgo/go/syscall/libcall_posix_largefile.go b/libgo/go/syscall/libcall_posix_largefile.go
index f90055bb29a..334212f0af1 100644
--- a/libgo/go/syscall/libcall_posix_largefile.go
+++ b/libgo/go/syscall/libcall_posix_largefile.go
@@ -10,40 +10,40 @@
package syscall
//sys Creat(path string, mode uint32) (fd int, err error)
-//creat64(path *byte, mode Mode_t) _C_int
+//creat(path *byte, mode Mode_t) _C_int
//sys Fstat(fd int, stat *Stat_t) (err error)
-//fstat64(fd _C_int, stat *Stat_t) _C_int
+//fstat(fd _C_int, stat *Stat_t) _C_int
//sys Ftruncate(fd int, length int64) (err error)
-//ftruncate64(fd _C_int, length Offset_t) _C_int
+//ftruncate(fd _C_int, length Offset_t) _C_int
//sysnb Getrlimit(resource int, rlim *Rlimit) (err error)
-//getrlimit64(resource _C_int, rlim *Rlimit) _C_int
+//getrlimit(resource _C_int, rlim *Rlimit) _C_int
//sys Lstat(path string, stat *Stat_t) (err error)
-//lstat64(path *byte, stat *Stat_t) _C_int
+//lstat(path *byte, stat *Stat_t) _C_int
//sys mmap(addr uintptr, length uintptr, prot int, flags int, fd int, offset int64) (xaddr uintptr, err error)
-//mmap64(addr *byte, length Size_t, prot _C_int, flags _C_int, fd _C_int, offset Offset_t) *byte
+//mmap(addr *byte, length Size_t, prot _C_int, flags _C_int, fd _C_int, offset Offset_t) *byte
//sys Open(path string, mode int, perm uint32) (fd int, err error)
-//__go_open64(path *byte, mode _C_int, perm Mode_t) _C_int
+//__go_open(path *byte, mode _C_int, perm Mode_t) _C_int
//sys Pread(fd int, p []byte, offset int64) (n int, err error)
-//pread64(fd _C_int, buf *byte, count Size_t, offset Offset_t) Ssize_t
+//pread(fd _C_int, buf *byte, count Size_t, offset Offset_t) Ssize_t
//sys Pwrite(fd int, p []byte, offset int64) (n int, err error)
-//pwrite64(fd _C_int, buf *byte, count Size_t, offset Offset_t) Ssize_t
+//pwrite(fd _C_int, buf *byte, count Size_t, offset Offset_t) Ssize_t
//sys Seek(fd int, offset int64, whence int) (off int64, err error)
-//lseek64(fd _C_int, offset Offset_t, whence _C_int) Offset_t
+//lseek(fd _C_int, offset Offset_t, whence _C_int) Offset_t
//sysnb Setrlimit(resource int, rlim *Rlimit) (err error)
-//setrlimit64(resource int, rlim *Rlimit) _C_int
+//setrlimit(resource int, rlim *Rlimit) _C_int
//sys Stat(path string, stat *Stat_t) (err error)
-//stat64(path *byte, stat *Stat_t) _C_int
+//stat(path *byte, stat *Stat_t) _C_int
//sys Truncate(path string, length int64) (err error)
-//truncate64(path *byte, length Offset_t) _C_int
+//truncate(path *byte, length Offset_t) _C_int
diff --git a/libgo/runtime/go-varargs.c b/libgo/runtime/go-varargs.c
index f84860891e6..7efc9615985 100644
--- a/libgo/runtime/go-varargs.c
+++ b/libgo/runtime/go-varargs.c
@@ -84,7 +84,7 @@ __go_ioctl_ptr (int d, int request, void *arg)
int
__go_open64 (char *path, int mode, mode_t perm)
{
- return open64 (path, mode, perm);
+ return open (path, mode, perm);
}
#endif
`}), &MakeAttr{
ScriptEarly: `
ln -s system/lib /
ln -s system/lib /work/
`,
Configure: [][2]string{
{"disable-multilib"},
{"with-multilib-list", `""`},
{"enable-default-pie"},
{"disable-nls"},
{"with-gnu-as"},
{"with-gnu-ld"},
{"with-system-zlib"},
{"enable-languages", "c,c++,go"},
{"with-native-system-header-dir", "/system/include"},
},
Make: []string{
"BOOT_CFLAGS='-O2 -g'",
"bootstrap",
},
// This toolchain is hacked to pieces, it is not expected to ever work
// well in its current state. That does not matter as long as the
// toolchain it produces passes its own test suite.
SkipCheck: true,
// GCC spends most of its time in its many configure scripts, however
// it also saturates the CPU for a consequential amount of time.
Flag: TExclusive,
},
t.Load(Binutils),
t.Load(GMP),
t.Load(MPFR),
t.Load(MPC),
t.Load(Zlib),
t.Load(Libucontext),
t.Load(KernelHeaders),
)
}
func init() { artifactsF[gcc] = Toolchain.newGCC }

153
internal/rosa/go.go Normal file
View File

@@ -0,0 +1,153 @@
package rosa
import (
"runtime"
"slices"
"hakurei.app/internal/pkg"
)
// newGoBootstrap returns the Go bootstrap toolchain.
func (t Toolchain) newGoBootstrap() pkg.Artifact {
const checksum = "8o9JL_ToiQKadCTb04nvBDkp8O1xiWOolAxVEqaTGodieNe4lOFEjlOxN3bwwe23"
return t.New("go1.4-bootstrap", 0, []pkg.Artifact{
t.Load(Bash),
}, nil, []string{
"CGO_ENABLED=0",
}, `
mkdir -p /var/tmp/ /work/system/
cp -r /usr/src/go /work/system/
cd /work/system/go/src
chmod -R +w ..
./make.bash
`, pkg.Path(AbsUsrSrc.Append("go"), false, pkg.NewHTTPGetTar(
nil, "https://dl.google.com/go/go1.4-bootstrap-20171003.tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
// newGo returns a specific version of the Go toolchain.
func (t Toolchain) newGo(
version, checksum string,
env []string,
script string,
extra ...pkg.Artifact,
) pkg.Artifact {
return t.New("go"+version, 0, slices.Concat([]pkg.Artifact{
t.Load(Bash),
}, extra), nil, slices.Concat([]string{
"CC=cc",
"GOCACHE=/tmp/gocache",
"GOROOT_BOOTSTRAP=/system/go",
"TMPDIR=/dev/shm/go",
}, env), `
mkdir /work/system "${TMPDIR}"
cp -r /usr/src/go /work/system
cd /work/system/go/src
chmod -R +w ..
`+script+`
./all.bash
mkdir /work/system/bin
ln -s \
../go/bin/go \
../go/bin/gofmt \
/work/system/bin
`, pkg.Path(AbsUsrSrc.Append("go"), false, pkg.NewHTTPGetTar(
nil, "https://go.dev/dl/go"+version+".src.tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func (t Toolchain) newGoLatest() pkg.Artifact {
var (
bootstrapEnv []string
bootstrapExtra []pkg.Artifact
)
switch runtime.GOARCH {
case "amd64":
bootstrapExtra = append(bootstrapExtra, t.newGoBootstrap())
case "arm64":
bootstrapEnv = append(bootstrapEnv,
"GOROOT_BOOTSTRAP=/system",
)
bootstrapExtra = append(bootstrapExtra,
t.Load(Binutils),
t.Load(GMP),
t.Load(MPFR),
t.Load(MPC),
t.Load(Zlib),
t.Load(Libucontext),
t.Load(gcc),
)
default:
panic("unsupported target " + runtime.GOARCH)
}
go119 := t.newGo(
"1.19",
"9_e0aFHsIkVxWVGsp9T2RvvjOc3p4n9o9S8tkNe9Cvgzk_zI2FhRQB7ioQkeAAro",
append(bootstrapEnv, "CGO_ENABLED=0"), `
rm \
crypto/tls/handshake_client_test.go \
os/os_unix_test.go
sed -i \
's/os\.Getenv("GCCGO")$/"nonexistent"/' \
go/internal/gccgoimporter/importer_test.go
echo \
'type syscallDescriptor = int' >> \
os/rawconn_test.go
`, bootstrapExtra...)
go121 := t.newGo(
"1.21.13",
"YtrDka402BOAEwywx03Vz4QlVwoBiguJHzG7PuythMCPHXS8CVMLvzmvgEbu4Tzu",
[]string{"CGO_ENABLED=0"}, `
sed -i \
's,/lib/ld-musl-`+linuxArch()+`.so.1,/system/bin/linker,' \
cmd/link/internal/`+runtime.GOARCH+`/obj.go
rm \
crypto/tls/handshake_client_test.go \
crypto/tls/handshake_server_test.go \
os/os_unix_test.go
echo \
'type syscallDescriptor = int' >> \
os/rawconn_test.go
`, go119,
)
go123 := t.newGo(
"1.23.12",
"wcI32bl1tkqbgcelGtGWPI4RtlEddd-PTd76Eb-k7nXA5LbE9yTNdIL9QSOOxMOs",
nil, `
sed -i \
's,/lib/ld-musl-`+linuxArch()+`.so.1,/system/bin/linker,' \
cmd/link/internal/`+runtime.GOARCH+`/obj.go
`, go121,
)
go125 := t.newGo(
"1.25.6",
"x0z430qoDvQbbw_fftjW0rh_GSoh0VJhPzttWk_0hj9yz9AKOjuwRMupF_Q0dbt7",
nil, `
sed -i \
's,/lib/ld-musl-`+linuxArch()+`.so.1,/system/bin/linker,' \
cmd/link/internal/`+runtime.GOARCH+`/obj.go
rm \
os/root_unix_test.go
`, go123,
)
return go125
}
func init() { artifactsF[Go] = Toolchain.newGoLatest }

290
internal/rosa/hakurei.go Normal file
View File

@@ -0,0 +1,290 @@
package rosa
import (
"hakurei.app/internal/pkg"
)
func (t Toolchain) newHakurei(suffix, script string) pkg.Artifact {
const (
version = "0.3.4"
checksum = "wVwSLo75a2OnH5tgxNWXR_YhiOJUFnYM_9-sJtxAEOKhcPE0BJafs6PU8o5JzyCT"
)
return t.New("hakurei"+suffix+"-"+version, 0, []pkg.Artifact{
t.Load(Go),
t.Load(Gzip),
t.Load(PkgConfig),
t.Load(KernelHeaders),
t.Load(Libseccomp),
t.Load(ACL),
t.Load(Attr),
t.Load(Fuse),
t.Load(Xproto),
t.Load(LibXau),
t.Load(XCBProto),
t.Load(XCB),
t.Load(Libffi),
t.Load(Libexpat),
t.Load(Libxml2),
t.Load(Wayland),
t.Load(WaylandProtocols),
}, nil, []string{
"CGO_ENABLED=1",
"GOCACHE=/tmp/gocache",
"CC=clang -O3 -Werror",
}, `
echo '# Building test helper (hostname).'
go build -v -o /bin/hostname /usr/src/hostname/main.go
echo
chmod -R +w /usr/src/hakurei
cd /usr/src/hakurei
HAKUREI_VERSION='v`+version+`'
`+script, pkg.Path(AbsUsrSrc.Append("hakurei"), true, t.NewPatchedSource("hakurei", version, pkg.NewHTTPGetTar(
nil, "https://git.gensokyo.uk/security/hakurei/archive/"+
"v"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), true, [2]string{"dist", `From 67e453f5c4de915de23ecbe5980e595758f0f2fb Mon Sep 17 00:00:00 2001
From: Ophestra <cat@gensokyo.uk>
Date: Tue, 27 Jan 2026 06:49:48 +0900
Subject: [PATCH] dist: run tests
This used to be impossible due to nix jank which has been addressed.
Signed-off-by: Ophestra <cat@gensokyo.uk>
---
dist/release.sh | 21 ++++++++++++++++-----
flake.nix | 32 ++++++++++++++++++++------------
internal/acl/acl_test.go | 2 +-
package.nix | 2 +-
4 files changed, 38 insertions(+), 19 deletions(-)
diff --git a/dist/release.sh b/dist/release.sh
index 4dcb278..0ba9104 100755
--- a/dist/release.sh
+++ b/dist/release.sh
@@ -2,19 +2,30 @@
cd "$(dirname -- "$0")/.."
VERSION="${HAKUREI_VERSION:-untagged}"
pname="hakurei-${VERSION}"
-out="dist/${pname}"
+out="${DESTDIR:-dist}/${pname}"
+echo '# Preparing distribution files.'
mkdir -p "${out}"
cp -v "README.md" "dist/hsurc.default" "dist/install.sh" "${out}"
cp -rv "dist/comp" "${out}"
+echo
+echo '# Building hakurei.'
go generate ./...
-go build -trimpath -v -o "${out}/bin/" -ldflags "-s -w -buildid= -extldflags '-static'
+go build -trimpath -v -o "${out}/bin/" -ldflags "-s -w
+ -buildid= -extldflags '-static'
-X hakurei.app/internal/info.buildVersion=${VERSION}
-X hakurei.app/internal/info.hakureiPath=/usr/bin/hakurei
-X hakurei.app/internal/info.hsuPath=/usr/bin/hsu
-X main.hakureiPath=/usr/bin/hakurei" ./...
+echo
-rm -f "./${out}.tar.gz" && tar -C dist -czf "${out}.tar.gz" "${pname}"
-rm -rf "./${out}"
-(cd dist && sha512sum "${pname}.tar.gz" > "${pname}.tar.gz.sha512")
+echo '# Testing hakurei.'
+go test -ldflags='-buildid= -extldflags=-static' ./...
+echo
+
+echo '# Creating distribution.'
+rm -f "${out}.tar.gz" && tar -C "${out}/.." -vczf "${out}.tar.gz" "${pname}"
+rm -rf "${out}"
+(cd "${out}/.." && sha512sum "${pname}.tar.gz" > "${pname}.tar.gz.sha512")
+echo
diff --git a/flake.nix b/flake.nix
index 9e09c61..2340b92 100644
--- a/flake.nix
+++ b/flake.nix
@@ -143,19 +143,27 @@
"bin/mount.fuse.sharefs" = "${hakurei}/libexec/sharefs";
};
- dist = pkgs.runCommand "${hakurei.name}-dist" { buildInputs = hakurei.targetPkgs ++ [ pkgs.pkgsStatic.musl ]; } ''
- # go requires XDG_CACHE_HOME for the build cache
- export XDG_CACHE_HOME="$(mktemp -d)"
+ dist =
+ pkgs.runCommand "${hakurei.name}-dist"
+ {
+ buildInputs = hakurei.targetPkgs ++ [
+ pkgs.pkgsStatic.musl
+ ];
+ }
+ ''
+ cd $(mktemp -d) \
+ && cp -r ${hakurei.src}/. . \
+ && chmod +w cmd && cp -r ${hsu.src}/. cmd/hsu/ \
+ && chmod -R +w .
- # get a different workdir as go does not like /build
- cd $(mktemp -d) \
- && cp -r ${hakurei.src}/. . \
- && chmod +w cmd && cp -r ${hsu.src}/. cmd/hsu/ \
- && chmod -R +w .
-
- export HAKUREI_VERSION="v${hakurei.version}"
- CC="clang -O3 -Werror" ./dist/release.sh && mkdir $out && cp -v "dist/hakurei-$HAKUREI_VERSION.tar.gz"* $out
- '';
+ CC="musl-clang -O3 -Werror -Qunused-arguments" \
+ GOCACHE="$(mktemp -d)" \
+ HAKUREI_TEST_SKIP_ACL=1 \
+ PATH="${pkgs.pkgsStatic.musl.bin}/bin:$PATH" \
+ DESTDIR="$out" \
+ HAKUREI_VERSION="v${hakurei.version}" \
+ ./dist/release.sh
+ '';
}
);
diff --git a/internal/acl/acl_test.go b/internal/acl/acl_test.go
index af6da55..19ce45a 100644
--- a/internal/acl/acl_test.go
+++ b/internal/acl/acl_test.go
@@ -24,7 +24,7 @@ var (
)
func TestUpdate(t *testing.T) {
- if os.Getenv("GO_TEST_SKIP_ACL") == "1" {
+ if os.Getenv("HAKUREI_TEST_SKIP_ACL") == "1" {
t.Skip("acl test skipped")
}
diff --git a/package.nix b/package.nix
index 00c4401..2eaa2ec 100644
--- a/package.nix
+++ b/package.nix
@@ -89,7 +89,7 @@ buildGoModule rec {
CC = "clang -O3 -Werror";
# nix build environment does not allow acls
- GO_TEST_SKIP_ACL = 1;
+ HAKUREI_TEST_SKIP_ACL = 1;
};
buildInputs = [`}, [2]string{"container-tests", `From bf14a412e47344fff2681f4b24d1ecc7415bfcb0 Mon Sep 17 00:00:00 2001
From: Ophestra <cat@gensokyo.uk>
Date: Sat, 31 Jan 2026 10:59:56 +0900
Subject: [PATCH] container: fix host-dependent test cases
These are not fully controlled by hakurei and may change depending on host configuration.
Signed-off-by: Ophestra <cat@gensokyo.uk>
---
container/container_test.go | 27 +++++++++++++++------------
1 file changed, 15 insertions(+), 12 deletions(-)
diff --git a/container/container_test.go b/container/container_test.go
index d737a18..98713cb 100644
--- a/container/container_test.go
+++ b/container/container_test.go
@@ -275,12 +275,12 @@ var containerTestCases = []struct {
),
earlyMnt(
ent("/", "/dev", "ro,nosuid,nodev,relatime", "tmpfs", "devtmpfs", ignore),
- ent("/null", "/dev/null", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/zero", "/dev/zero", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/full", "/dev/full", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/random", "/dev/random", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/urandom", "/dev/urandom", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/tty", "/dev/tty", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
+ ent("/null", "/dev/null", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/zero", "/dev/zero", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/full", "/dev/full", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/random", "/dev/random", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/urandom", "/dev/urandom", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/tty", "/dev/tty", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/", "/dev/pts", "rw,nosuid,noexec,relatime", "devpts", "devpts", "rw,mode=620,ptmxmode=666"),
ent("/", "/dev/mqueue", "rw,nosuid,nodev,noexec,relatime", "mqueue", "mqueue", "rw"),
ent("/", "/dev/shm", "rw,nosuid,nodev,relatime", "tmpfs", "tmpfs", ignore),
@@ -293,12 +293,12 @@ var containerTestCases = []struct {
),
earlyMnt(
ent("/", "/dev", "ro,nosuid,nodev,relatime", "tmpfs", "devtmpfs", ignore),
- ent("/null", "/dev/null", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/zero", "/dev/zero", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/full", "/dev/full", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/random", "/dev/random", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/urandom", "/dev/urandom", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
- ent("/tty", "/dev/tty", "rw,nosuid", "devtmpfs", "devtmpfs", ignore),
+ ent("/null", "/dev/null", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/zero", "/dev/zero", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/full", "/dev/full", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/random", "/dev/random", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/urandom", "/dev/urandom", ignore, "devtmpfs", "devtmpfs", ignore),
+ ent("/tty", "/dev/tty", ignore, "devtmpfs", "devtmpfs", ignore),
ent("/", "/dev/pts", "rw,nosuid,noexec,relatime", "devpts", "devpts", "rw,mode=620,ptmxmode=666"),
ent("/", "/dev/shm", "rw,nosuid,nodev,relatime", "tmpfs", "tmpfs", ignore),
),
@@ -696,6 +696,9 @@ func init() {
mnt[i].VfsOptstr = strings.TrimSuffix(mnt[i].VfsOptstr, ",relatime")
mnt[i].VfsOptstr = strings.TrimSuffix(mnt[i].VfsOptstr, ",noatime")
+ cur.FsOptstr = strings.Replace(cur.FsOptstr, ",seclabel", "", 1)
+ mnt[i].FsOptstr = strings.Replace(mnt[i].FsOptstr, ",seclabel", "", 1)
+
if !cur.EqualWithIgnore(mnt[i], "\x00") {
fail = true
log.Printf("[FAIL] %s", cur)`}),
), pkg.Path(AbsUsrSrc.Append("hostname", "main.go"), false, pkg.NewFile(
"hostname.go",
[]byte(`
package main
import "os"
func main() {
if name, err := os.Hostname(); err != nil {
panic(err)
} else {
os.Stdout.WriteString(name)
}
}
`),
)))
}
func init() {
artifactsF[Hakurei] = func(t Toolchain) pkg.Artifact {
return t.newHakurei("", `
mkdir -p /work/system/libexec/hakurei/
echo '# Building hakurei.'
go generate -v ./...
go build -trimpath -v -o /work/system/libexec/hakurei -ldflags="-s -w
-buildid=
-extldflags=-static
-X hakurei.app/internal/info.buildVersion="$HAKUREI_VERSION"
-X hakurei.app/internal/info.hakureiPath=/system/bin/hakurei
-X hakurei.app/internal/info.hsuPath=/system/bin/hsu
-X main.hakureiPath=/system/bin/hakurei" ./...
echo
echo '# Testing hakurei.'
go test -ldflags='-buildid= -extldflags=-static' ./...
echo
mkdir -p /work/system/bin/
(cd /work/system/libexec/hakurei && mv \
hakurei \
sharefs \
../../bin/)
`)
}
artifactsF[HakureiDist] = func(t Toolchain) pkg.Artifact {
return t.newHakurei("-dist", `
export HAKUREI_VERSION
DESTDIR=/work /usr/src/hakurei/dist/release.sh
`)
}
}

44
internal/rosa/kernel.go Normal file
View File

@@ -0,0 +1,44 @@
package rosa
import (
"slices"
"hakurei.app/internal/pkg"
)
// newKernel is a helper for interacting with Kbuild.
func (t Toolchain) newKernel(
flag int,
patches [][2]string,
script string,
extra ...pkg.Artifact,
) pkg.Artifact {
const (
version = "6.18.5"
checksum = "-V1e1WWl7HuePkmm84sSKF7nLuHfUs494uNMzMqXEyxcNE_PUE0FICL0oGWn44mM"
)
return t.New("kernel-"+version, flag, slices.Concat([]pkg.Artifact{
t.Load(Make),
}, extra), nil, nil, `
export LLVM=1
export HOSTLDFLAGS="${LDFLAGS}"
cd /usr/src/linux
`+script, pkg.Path(AbsUsrSrc.Append("linux"), true, t.NewPatchedSource(
"kernel", version, pkg.NewHTTPGetTar(
nil,
"https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/"+
"snapshot/linux-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false, patches...,
)))
}
func (t Toolchain) newKernelHeaders() pkg.Artifact {
return t.newKernel(TEarly, nil, `
make "-j$(nproc)" \
INSTALL_HDR_PATH=/work/system \
headers_install
`, t.Load(Rsync))
}
func init() { artifactsF[KernelHeaders] = Toolchain.newKernelHeaders }

28
internal/rosa/libexpat.go Normal file
View File

@@ -0,0 +1,28 @@
package rosa
import (
"strings"
"hakurei.app/internal/pkg"
)
func (t Toolchain) newLibexpat() pkg.Artifact {
const (
version = "2.7.3"
checksum = "GmkoD23nRi9cMT0cgG1XRMrZWD82UcOMzkkvP1gkwSFWCBgeSXMuoLpa8-v8kxW-"
)
return t.NewViaMake("libexpat", version, pkg.NewHTTPGetTar(
nil, "https://github.com/libexpat/libexpat/releases/download/"+
"R_"+strings.ReplaceAll(version, ".", "_")+"/"+
"expat-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), &MakeAttr{
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(Bash),
)
}
func init() { artifactsF[Libexpat] = Toolchain.newLibexpat }

23
internal/rosa/libffi.go Normal file
View File

@@ -0,0 +1,23 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newLibffi() pkg.Artifact {
const (
version = "3.4.5"
checksum = "apIJzypF4rDudeRoI_n3K7N-zCeBLTbQlHRn9NSAZqdLAWA80mR0gXPTpHsL7oMl"
)
return t.NewViaMake("libffi", version, pkg.NewHTTPGetTar(
nil, "https://github.com/libffi/libffi/releases/download/"+
"v"+version+"/libffi-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(KernelHeaders),
)
}
func init() { artifactsF[Libffi] = Toolchain.newLibffi }

30
internal/rosa/libgd.go Normal file
View File

@@ -0,0 +1,30 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newLibgd() pkg.Artifact {
const (
version = "2.3.3"
checksum = "8T-sh1_FJT9K9aajgxzh8ot6vWIF-xxjcKAHvTak9MgGUcsFfzP8cAvvv44u2r36"
)
return t.NewViaMake("libgd", version, pkg.NewHTTPGetTar(
nil, "https://github.com/libgd/libgd/releases/download/"+
"gd-"+version+"/libgd-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
OmitDefaults: true,
Env: []string{
"TMPDIR=/dev/shm/gd",
},
ScriptEarly: `
mkdir /dev/shm/gd
`,
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(Zlib),
)
}
func init() { artifactsF[Libgd] = Toolchain.newLibgd }

28
internal/rosa/libpsl.go Normal file
View File

@@ -0,0 +1,28 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newLibpsl() pkg.Artifact {
const (
version = "0.21.5"
checksum = "XjfxSzh7peG2Vg4vJlL8z4JZJLcXqbuP6pLWkrGCmRxlnYUFTKNBqWGHCxEOlCad"
)
return t.NewViaMake("libpsl", version, pkg.NewHTTPGetTar(
nil, "https://github.com/rockdaboot/libpsl/releases/download/"+
version+"/libpsl-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Writable: true,
ScriptEarly: `
cd /usr/src/libpsl
test_disable() { chmod +w "$2" && echo "$1" > "$2"; }
test_disable 'int main(){return 0;}' tests/test-is-public-builtin.c
`,
},
t.Load(Python),
)
}
func init() { artifactsF[Libpsl] = Toolchain.newLibpsl }

View File

@@ -0,0 +1,33 @@
package rosa
import (
"hakurei.app/internal/pkg"
)
func (t Toolchain) newLibseccomp() pkg.Artifact {
const (
version = "2.6.0"
checksum = "mMu-iR71guPjFbb31u-YexBaanKE_nYPjPux-vuBiPfS_0kbwJdfCGlkofaUm-EY"
)
return t.NewViaMake("libseccomp", version, pkg.NewHTTPGetTar(
nil,
"https://github.com/seccomp/libseccomp/releases/download/"+
"v"+version+"/libseccomp-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
ScriptEarly: `
ln -s ../system/bin/bash /bin/
`,
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(Bash),
t.Load(Diffutils),
t.Load(Gperf),
t.Load(KernelHeaders),
)
}
func init() { artifactsF[Libseccomp] = Toolchain.newLibseccomp }

View File

@@ -0,0 +1,40 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newLibucontext() pkg.Artifact {
const (
version = "1.5"
checksum = "Ggk7FMmDNBdCx1Z9PcNWWW6LSpjGYssn2vU0GK5BLXJYw7ZxZbA2m_eSgT9TFnIG"
)
return t.New("libucontext", 0, []pkg.Artifact{
t.Load(Make),
}, nil, []string{
"ARCH=" + linuxArch(),
}, `
cd /usr/src/libucontext
make check
make DESTDIR=/work install
`, pkg.Path(AbsUsrSrc.Append("libucontext"), true,
t.NewPatchedSource("libucontext", version, pkg.NewHTTPGetTar(
nil, "https://github.com/kaniini/libucontext/archive/refs/tags/"+
"libucontext-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), true, [2]string{"rosa-prefix", `diff --git a/Makefile b/Makefile
index c80e574..4a8c1d3 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@ ifeq ($(ARCH),$(filter $(ARCH),arm64))
override ARCH = aarch64
endif
-prefix = /usr
+prefix = /system
libdir = ${prefix}/lib
shared_libdir = ${libdir}
static_libdir = ${libdir}
`}),
))
}
func init() { artifactsF[Libucontext] = Toolchain.newLibucontext }

34
internal/rosa/libxml2.go Normal file
View File

@@ -0,0 +1,34 @@
package rosa
import (
"strings"
"hakurei.app/internal/pkg"
)
func (t Toolchain) newLibxml2() pkg.Artifact {
const (
version = "2.15.1"
checksum = "pYzAR3cNrEHezhEMirgiq7jbboLzwMj5GD7SQp0jhSIMdgoU4G9oU9Gxun3zzUIU"
)
return t.NewViaMake("libxml2", version, pkg.NewHTTPGet(
nil, "https://download.gnome.org/sources/libxml2/"+
strings.Join(strings.Split(version, ".")[:2], ".")+
"/libxml2-"+version+".tar.xz",
mustDecode(checksum),
), &MakeAttr{
ScriptEarly: `
cd /usr/src/
tar xf libxml2.tar.xz
mv libxml2-` + version + ` libxml2
`,
Configure: [][2]string{
{"enable-static"},
},
SourceSuffix: ".tar.xz",
},
t.Load(Diffutils),
t.Load(XZ),
)
}
func init() { artifactsF[Libxml2] = Toolchain.newLibxml2 }

524
internal/rosa/llvm.go Normal file
View File

@@ -0,0 +1,524 @@
package rosa
import (
"runtime"
"slices"
"strconv"
"strings"
"sync"
"hakurei.app/container/check"
"hakurei.app/internal/pkg"
)
// llvmAttr holds the attributes that will be applied to a new [pkg.Artifact]
// containing a LLVM variant.
type llvmAttr struct {
flags int
// Concatenated with default environment for CMakeAttr.Env.
env []string
// Concatenated with generated entries for CMakeAttr.Cache.
cmake [][2]string
// Override CMakeAttr.Append.
append []string
// Concatenated with default dependencies for Toolchain.NewViaCMake.
extra []pkg.Artifact
// Passed through to CMakeAttr.Paths.
paths []pkg.ExecPath
// Passed through to CMakeAttr.ScriptConfigured.
scriptConfigured string
// Concatenated with default fixup for CMakeAttr.Script.
script string
// Passed through to CMakeAttr.Prefix.
prefix *check.Absolute
// Passed through to CMakeAttr.Writable.
writable bool
// Patch name and body pairs.
patches [][2]string
}
const (
llvmProjectClang = 1 << iota
llvmProjectLld
llvmProjectAll = 1<<iota - 1
llvmRuntimeCompilerRT = 1 << iota
llvmRuntimeLibunwind
llvmRuntimeLibc
llvmRuntimeLibcxx
llvmRuntimeLibcxxABI
llvmAll = 1<<iota - 1
llvmRuntimeAll = llvmAll - (2 * llvmProjectAll) - 1
)
// llvmFlagName resolves a llvmAttr.flags project or runtime flag to its name.
func llvmFlagName(flag int) string {
switch flag {
case llvmProjectClang:
return "clang"
case llvmProjectLld:
return "lld"
case llvmRuntimeCompilerRT:
return "compiler-rt"
case llvmRuntimeLibunwind:
return "libunwind"
case llvmRuntimeLibc:
return "libc"
case llvmRuntimeLibcxx:
return "libcxx"
case llvmRuntimeLibcxxABI:
return "libcxxabi"
default:
panic("invalid flag " + strconv.Itoa(flag))
}
}
// newLLVMVariant returns a [pkg.Artifact] containing a LLVM variant.
func (t Toolchain) newLLVMVariant(variant string, attr *llvmAttr) pkg.Artifact {
const (
version = "21.1.8"
checksum = "8SUpqDkcgwOPsqHVtmf9kXfFeVmjVxl4LMn-qSE1AI_Xoeju-9HaoPNGtidyxyka"
)
if attr == nil {
panic("LLVM attr must be non-nil")
}
var projects, runtimes []string
for i := 1; i < llvmProjectAll; i <<= 1 {
if attr.flags&i != 0 {
projects = append(projects, llvmFlagName(i))
}
}
for i := (llvmProjectAll + 1) << 1; i < llvmRuntimeAll; i <<= 1 {
if attr.flags&i != 0 {
runtimes = append(runtimes, llvmFlagName(i))
}
}
var script, scriptEarly string
cache := [][2]string{
{"CMAKE_BUILD_TYPE", "Release"},
{"LLVM_HOST_TRIPLE", `"${ROSA_TRIPLE}"`},
{"LLVM_DEFAULT_TARGET_TRIPLE", `"${ROSA_TRIPLE}"`},
}
if len(projects) > 0 {
cache = append(cache,
[2]string{"LLVM_ENABLE_PROJECTS", `"${ROSA_LLVM_PROJECTS}"`})
}
if len(runtimes) > 0 {
cache = append(cache,
[2]string{"LLVM_ENABLE_RUNTIMES", `"${ROSA_LLVM_RUNTIMES}"`})
}
cmakeAppend := []string{"llvm"}
if attr.append != nil {
cmakeAppend = attr.append
} else {
cache = append(cache,
[2]string{"LLVM_ENABLE_LIBCXX", "ON"},
[2]string{"LLVM_USE_LINKER", "lld"},
[2]string{"LLVM_INSTALL_BINUTILS_SYMLINKS", "ON"},
[2]string{"LLVM_INSTALL_CCTOOLS_SYMLINKS", "ON"},
)
}
if attr.flags&llvmProjectClang != 0 {
cache = append(cache,
[2]string{"CLANG_DEFAULT_LINKER", "lld"},
[2]string{"CLANG_DEFAULT_CXX_STDLIB", "libc++"},
[2]string{"CLANG_DEFAULT_RTLIB", "compiler-rt"},
[2]string{"CLANG_DEFAULT_UNWINDLIB", "libunwind"},
)
}
if attr.flags&llvmProjectLld != 0 {
script += `
ln -s ld.lld /work/system/bin/ld
`
}
if attr.flags&llvmRuntimeCompilerRT != 0 {
if attr.append == nil {
cache = append(cache,
[2]string{"COMPILER_RT_USE_LLVM_UNWINDER", "ON"})
}
}
if attr.flags&llvmRuntimeLibunwind != 0 {
cache = append(cache,
[2]string{"LIBUNWIND_USE_COMPILER_RT", "ON"})
}
if attr.flags&llvmRuntimeLibcxx != 0 {
cache = append(cache,
[2]string{"LIBCXX_HAS_MUSL_LIBC", "ON"},
[2]string{"LIBCXX_USE_COMPILER_RT", "ON"},
)
if t > toolchainStage3 {
// libcxxabi fails to compile if c++ headers not prefixed in /usr
// is found by the compiler, and doing this is easier than
// overriding CXXFLAGS; not using mv here to avoid chown failures
scriptEarly += `
cp -r /system/include /usr/include && rm -rf /system/include
`
}
}
if attr.flags&llvmRuntimeLibcxxABI != 0 {
cache = append(cache,
[2]string{"LIBCXXABI_USE_COMPILER_RT", "ON"},
[2]string{"LIBCXXABI_USE_LLVM_UNWINDER", "ON"},
)
}
return t.NewViaCMake("llvm", version, variant, t.NewPatchedSource(
"llvmorg", version, pkg.NewHTTPGetTar(
nil, "https://github.com/llvm/llvm-project/archive/refs/tags/"+
"llvmorg-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), true, attr.patches...,
), &CMakeAttr{
Cache: slices.Concat(cache, attr.cmake),
Append: cmakeAppend,
Prefix: attr.prefix,
Env: slices.Concat([]string{
"ROSA_LLVM_PROJECTS=" + strings.Join(projects, ";"),
"ROSA_LLVM_RUNTIMES=" + strings.Join(runtimes, ";"),
}, attr.env),
ScriptEarly: scriptEarly,
ScriptConfigured: attr.scriptConfigured,
Script: script + attr.script,
Writable: attr.writable,
Paths: attr.paths,
Flag: TExclusive,
}, stage3Concat(t, attr.extra,
t.Load(Libffi),
t.Load(Python),
t.Load(Perl),
t.Load(Diffutils),
t.Load(Bash),
t.Load(Gawk),
t.Load(Coreutils),
t.Load(Findutils),
t.Load(KernelHeaders),
)...)
}
// newLLVM returns LLVM toolchain across multiple [pkg.Artifact].
func (t Toolchain) newLLVM() (musl, compilerRT, runtimes, clang pkg.Artifact) {
var target string
switch runtime.GOARCH {
case "386", "amd64":
target = "X86"
case "arm64":
target = "AArch64"
default:
panic("unsupported target " + runtime.GOARCH)
}
minimalDeps := [][2]string{
{"LLVM_ENABLE_ZLIB", "OFF"},
{"LLVM_ENABLE_ZSTD", "OFF"},
{"LLVM_ENABLE_LIBXML2", "OFF"},
}
compilerRT = t.newLLVMVariant("compiler-rt", &llvmAttr{
env: stage3ExclConcat(t, []string{},
"LDFLAGS="+earlyLDFLAGS(false),
),
cmake: [][2]string{
// libc++ not yet available
{"CMAKE_CXX_COMPILER_TARGET", ""},
{"COMPILER_RT_BUILD_BUILTINS", "ON"},
{"COMPILER_RT_DEFAULT_TARGET_ONLY", "ON"},
{"COMPILER_RT_SANITIZERS_TO_BUILD", "asan"},
{"LLVM_ENABLE_PER_TARGET_RUNTIME_DIR", "ON"},
// does not work without libunwind
{"COMPILER_RT_BUILD_CTX_PROFILE", "OFF"},
{"COMPILER_RT_BUILD_LIBFUZZER", "OFF"},
{"COMPILER_RT_BUILD_MEMPROF", "OFF"},
{"COMPILER_RT_BUILD_PROFILE", "OFF"},
{"COMPILER_RT_BUILD_XRAY", "OFF"},
},
append: []string{"compiler-rt"},
extra: []pkg.Artifact{t.NewMusl(&MuslAttr{
Headers: true,
Env: []string{
"CC=clang",
},
})},
script: `
mkdir -p "${ROSA_INSTALL_PREFIX}/lib/clang/21/lib/"
ln -s \
"../../../${ROSA_TRIPLE}" \
"${ROSA_INSTALL_PREFIX}/lib/clang/21/lib/"
ln -s \
"clang_rt.crtbegin-` + linuxArch() + `.o" \
"${ROSA_INSTALL_PREFIX}/lib/${ROSA_TRIPLE}/crtbeginS.o"
ln -s \
"clang_rt.crtend-` + linuxArch() + `.o" \
"${ROSA_INSTALL_PREFIX}/lib/${ROSA_TRIPLE}/crtendS.o"
`,
})
musl = t.NewMusl(&MuslAttr{
Extra: []pkg.Artifact{compilerRT},
Env: stage3ExclConcat(t, []string{
"CC=clang",
"LIBCC=/system/lib/clang/21/lib/" +
triplet() + "/libclang_rt.builtins.a",
"AR=ar",
"RANLIB=ranlib",
},
"LDFLAGS="+earlyLDFLAGS(false),
),
})
runtimes = t.newLLVMVariant("runtimes", &llvmAttr{
env: stage3ExclConcat(t, []string{},
"LDFLAGS="+earlyLDFLAGS(false),
),
flags: llvmRuntimeLibunwind | llvmRuntimeLibcxx | llvmRuntimeLibcxxABI,
cmake: slices.Concat([][2]string{
// libc++ not yet available
{"CMAKE_CXX_COMPILER_WORKS", "ON"},
{"LIBCXX_HAS_ATOMIC_LIB", "OFF"},
{"LIBCXXABI_HAS_CXA_THREAD_ATEXIT_IMPL", "OFF"},
}, minimalDeps),
append: []string{"runtimes"},
extra: []pkg.Artifact{
compilerRT,
musl,
},
})
clang = t.newLLVMVariant("clang", &llvmAttr{
flags: llvmProjectClang | llvmProjectLld,
env: stage3ExclConcat(t, []string{},
"CFLAGS="+earlyCFLAGS,
"CXXFLAGS="+earlyCXXFLAGS(),
"LDFLAGS="+earlyLDFLAGS(false),
),
cmake: slices.Concat([][2]string{
{"LLVM_TARGETS_TO_BUILD", target},
{"CMAKE_CROSSCOMPILING", "OFF"},
{"CXX_SUPPORTS_CUSTOM_LINKER", "ON"},
}, minimalDeps),
extra: []pkg.Artifact{
musl,
compilerRT,
runtimes,
},
script: `
ln -s clang /work/system/bin/cc
ln -s clang++ /work/system/bin/c++
ninja check-all
`,
patches: [][2]string{
{"add-rosa-vendor", `diff --git a/llvm/include/llvm/TargetParser/Triple.h b/llvm/include/llvm/TargetParser/Triple.h
index 657f4230379e..12c305756184 100644
--- a/llvm/include/llvm/TargetParser/Triple.h
+++ b/llvm/include/llvm/TargetParser/Triple.h
@@ -185,6 +185,7 @@ public:
Apple,
PC,
+ Rosa,
SCEI,
Freescale,
IBM,
diff --git a/llvm/lib/TargetParser/Triple.cpp b/llvm/lib/TargetParser/Triple.cpp
index 0584c941d2e6..e4d6ef963cc7 100644
--- a/llvm/lib/TargetParser/Triple.cpp
+++ b/llvm/lib/TargetParser/Triple.cpp
@@ -269,6 +269,7 @@ StringRef Triple::getVendorTypeName(VendorType Kind) {
case NVIDIA: return "nvidia";
case OpenEmbedded: return "oe";
case PC: return "pc";
+ case Rosa: return "rosa";
case SCEI: return "scei";
case SUSE: return "suse";
}
@@ -669,6 +670,7 @@ static Triple::VendorType parseVendor(StringRef VendorName) {
.Case("suse", Triple::SUSE)
.Case("oe", Triple::OpenEmbedded)
.Case("intel", Triple::Intel)
+ .Case("rosa", Triple::Rosa)
.Default(Triple::UnknownVendor);
}
`},
{"xfail-broken-tests", `diff --git a/clang/test/Modules/timestamps.c b/clang/test/Modules/timestamps.c
index 50fdce630255..4b4465a75617 100644
--- a/clang/test/Modules/timestamps.c
+++ b/clang/test/Modules/timestamps.c
@@ -1,3 +1,5 @@
+// XFAIL: target={{.*-rosa-linux-musl}}
+
/// Verify timestamps that gets embedded in the module
#include <c-header.h>
`},
{"path-system-include", `diff --git a/clang/lib/Driver/ToolChains/Linux.cpp b/clang/lib/Driver/ToolChains/Linux.cpp
index cdbf21fb9026..dd052858700d 100644
--- a/clang/lib/Driver/ToolChains/Linux.cpp
+++ b/clang/lib/Driver/ToolChains/Linux.cpp
@@ -773,6 +773,12 @@ void Linux::AddClangSystemIncludeArgs(const ArgList &DriverArgs,
addExternCSystemInclude(
DriverArgs, CC1Args,
concat(SysRoot, "/usr/include", MultiarchIncludeDir));
+ if (!MultiarchIncludeDir.empty() &&
+ D.getVFS().exists(concat(SysRoot, "/system/include", MultiarchIncludeDir)))
+ addExternCSystemInclude(
+ DriverArgs, CC1Args,
+ concat(SysRoot, "/system/include", MultiarchIncludeDir));
+
if (getTriple().getOS() == llvm::Triple::RTEMS)
return;
@@ -783,6 +789,7 @@ void Linux::AddClangSystemIncludeArgs(const ArgList &DriverArgs,
addExternCSystemInclude(DriverArgs, CC1Args, concat(SysRoot, "/include"));
addExternCSystemInclude(DriverArgs, CC1Args, concat(SysRoot, "/usr/include"));
+ addExternCSystemInclude(DriverArgs, CC1Args, concat(SysRoot, "/system/include"));
if (!DriverArgs.hasArg(options::OPT_nobuiltininc) && getTriple().isMusl())
addSystemInclude(DriverArgs, CC1Args, ResourceDirInclude);
`},
{"path-system-libraries", `diff --git a/clang/lib/Driver/ToolChains/Linux.cpp b/clang/lib/Driver/ToolChains/Linux.cpp
index 8ac8d4eb9181..f4d1347ab64d 100644
--- a/clang/lib/Driver/ToolChains/Linux.cpp
+++ b/clang/lib/Driver/ToolChains/Linux.cpp
@@ -282,6 +282,7 @@ Linux::Linux(const Driver &D, const llvm::Triple &Triple, const ArgList &Args)
const bool IsHexagon = Arch == llvm::Triple::hexagon;
const bool IsRISCV = Triple.isRISCV();
const bool IsCSKY = Triple.isCSKY();
+ const bool IsRosa = Triple.getVendor() == llvm::Triple::Rosa;
if (IsCSKY && !SelectedMultilibs.empty())
SysRoot = SysRoot + SelectedMultilibs.back().osSuffix();
@@ -318,12 +319,23 @@ Linux::Linux(const Driver &D, const llvm::Triple &Triple, const ArgList &Args)
const std::string OSLibDir = std::string(getOSLibDir(Triple, Args));
const std::string MultiarchTriple = getMultiarchTriple(D, Triple, SysRoot);
+ if (IsRosa) {
+ ExtraOpts.push_back("-rpath");
+ ExtraOpts.push_back("/system/lib");
+ ExtraOpts.push_back("-rpath");
+ ExtraOpts.push_back(concat("/system/lib", MultiarchTriple));
+ }
+
// mips32: Debian multilib, we use /libo32, while in other case, /lib is
// used. We need add both libo32 and /lib.
if (Arch == llvm::Triple::mips || Arch == llvm::Triple::mipsel) {
Generic_GCC::AddMultilibPaths(D, SysRoot, "libo32", MultiarchTriple, Paths);
- addPathIfExists(D, concat(SysRoot, "/libo32"), Paths);
- addPathIfExists(D, concat(SysRoot, "/usr/libo32"), Paths);
+ if (!IsRosa) {
+ addPathIfExists(D, concat(SysRoot, "/libo32"), Paths);
+ addPathIfExists(D, concat(SysRoot, "/usr/libo32"), Paths);
+ } else {
+ addPathIfExists(D, concat(SysRoot, "/system/libo32"), Paths);
+ }
}
Generic_GCC::AddMultilibPaths(D, SysRoot, OSLibDir, MultiarchTriple, Paths);
@@ -341,18 +353,30 @@ Linux::Linux(const Driver &D, const llvm::Triple &Triple, const ArgList &Args)
Paths);
}
- addPathIfExists(D, concat(SysRoot, "/usr/lib", MultiarchTriple), Paths);
- addPathIfExists(D, concat(SysRoot, "/usr", OSLibDir), Paths);
+ if (!IsRosa) {
+ addPathIfExists(D, concat(SysRoot, "/usr/lib", MultiarchTriple), Paths);
+ addPathIfExists(D, concat(SysRoot, "/usr", OSLibDir), Paths);
+ } else {
+ addPathIfExists(D, concat(SysRoot, "/system/lib", MultiarchTriple), Paths);
+ addPathIfExists(D, concat(SysRoot, "/system", OSLibDir), Paths);
+ }
if (IsRISCV) {
StringRef ABIName = tools::riscv::getRISCVABI(Args, Triple);
addPathIfExists(D, concat(SysRoot, "/", OSLibDir, ABIName), Paths);
- addPathIfExists(D, concat(SysRoot, "/usr", OSLibDir, ABIName), Paths);
+ if (!IsRosa)
+ addPathIfExists(D, concat(SysRoot, "/usr", OSLibDir, ABIName), Paths);
+ else
+ addPathIfExists(D, concat(SysRoot, "/system", OSLibDir, ABIName), Paths);
}
Generic_GCC::AddMultiarchPaths(D, SysRoot, OSLibDir, Paths);
- addPathIfExists(D, concat(SysRoot, "/lib"), Paths);
- addPathIfExists(D, concat(SysRoot, "/usr/lib"), Paths);
+ if (!IsRosa) {
+ addPathIfExists(D, concat(SysRoot, "/lib"), Paths);
+ addPathIfExists(D, concat(SysRoot, "/usr/lib"), Paths);
+ } else {
+ addPathIfExists(D, concat(SysRoot, "/system/lib"), Paths);
+ }
}
ToolChain::RuntimeLibType Linux::GetDefaultRuntimeLibType() const {
@@ -457,6 +481,9 @@ std::string Linux::getDynamicLinker(const ArgList &Args) const {
return Triple.isArch64Bit() ? "/system/bin/linker64" : "/system/bin/linker";
}
if (Triple.isMusl()) {
+ if (Triple.getVendor() == llvm::Triple::Rosa)
+ return "/system/bin/linker";
+
std::string ArchName;
bool IsArm = false;
diff --git a/clang/tools/clang-installapi/Options.cpp b/clang/tools/clang-installapi/Options.cpp
index 64324a3f8b01..15ce70b68217 100644
--- a/clang/tools/clang-installapi/Options.cpp
+++ b/clang/tools/clang-installapi/Options.cpp
@@ -515,7 +515,7 @@ bool Options::processFrontendOptions(InputArgList &Args) {
FEOpts.FwkPaths = std::move(FrameworkPaths);
// Add default framework/library paths.
- PathSeq DefaultLibraryPaths = {"/usr/lib", "/usr/local/lib"};
+ PathSeq DefaultLibraryPaths = {"/usr/lib", "/system/lib", "/usr/local/lib"};
PathSeq DefaultFrameworkPaths = {"/Library/Frameworks",
"/System/Library/Frameworks"};
`},
},
})
return
}
var (
// llvm stores the result of Toolchain.newLLVM.
llvm [_toolchainEnd][4]pkg.Artifact
// llvmOnce is for lazy initialisation of llvm.
llvmOnce [_toolchainEnd]sync.Once
)
// NewLLVM returns LLVM toolchain across multiple [pkg.Artifact].
func (t Toolchain) NewLLVM() (musl, compilerRT, runtimes, clang pkg.Artifact) {
llvmOnce[t].Do(func() {
llvm[t][0], llvm[t][1], llvm[t][2], llvm[t][3] = t.newLLVM()
})
return llvm[t][0], llvm[t][1], llvm[t][2], llvm[t][3]
}

162
internal/rosa/make.go Normal file
View File

@@ -0,0 +1,162 @@
package rosa
import (
"slices"
"strings"
"hakurei.app/internal/pkg"
)
func (t Toolchain) newMake() pkg.Artifact {
const (
version = "4.4.1"
checksum = "YS_B07ZcAy9PbaK5_vKGj64SrxO2VMpnMKfc9I0Q9IC1rn0RwOH7802pJoj2Mq4a"
)
return t.New("make-"+version, TEarly, nil, nil, nil, `
cd "$(mktemp -d)"
/usr/src/make/configure \
--prefix=/system \
--build="${ROSA_TRIPLE}" \
--disable-dependency-tracking
./build.sh
./make DESTDIR=/work install check
`, pkg.Path(AbsUsrSrc.Append("make"), false, pkg.NewHTTPGetTar(
nil, "https://ftpmirror.gnu.org/gnu/make/make-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func init() { artifactsF[Make] = Toolchain.newMake }
// MakeAttr holds the project-specific attributes that will be applied to a new
// [pkg.Artifact] compiled via [Make].
type MakeAttr struct {
// Mount the source tree writable.
Writable bool
// Do not include default extras.
OmitDefaults bool
// Dependencies not provided by stage3.
NonStage3 []pkg.Artifact
// Additional environment variables.
Env []string
// Runs before configure.
ScriptEarly string
// Runs after configure.
ScriptConfigured string
// Runs after install.
Script string
// Remain in working directory set up during ScriptEarly.
InPlace bool
// Flags passed to the configure script.
Configure [][2]string
// Extra make targets.
Make []string
// Target triple, zero value is equivalent to the Rosa OS triple.
Build string
// Whether to skip the check target.
SkipCheck bool
// Name of the check target, zero value is equivalent to "check".
CheckName string
// Suffix appended to the source pathname.
SourceSuffix string
// Passed through to [Toolchain.New].
Flag int
}
// NewViaMake returns a [pkg.Artifact] for compiling and installing via [Make].
func (t Toolchain) NewViaMake(
name, version string,
source pkg.Artifact,
attr *MakeAttr,
extra ...pkg.Artifact,
) pkg.Artifact {
if name == "" || version == "" {
panic("names must be non-empty")
}
if attr == nil {
attr = new(MakeAttr)
}
build := `"${ROSA_TRIPLE}"`
if attr.Build != "" {
build = attr.Build
}
var configureFlags string
if len(attr.Configure) > 0 {
const sep = " \\\n\t"
configureFlags += sep + strings.Join(
slices.Collect(func(yield func(string) bool) {
for _, v := range attr.Configure {
s := v[0]
if v[1] == "" || (v[0] != "" &&
v[0][0] >= 'a' &&
v[0][0] <= 'z') {
s = "--" + s
}
if v[1] != "" {
s += "=" + v[1]
}
if !yield(s) {
return
}
}
}),
sep,
)
}
var buildFlag string
if attr.Build != `""` {
buildFlag = ` \
--build=` + build
}
makeTargets := make([]string, 1, 2+len(attr.Make))
if !attr.SkipCheck {
if attr.CheckName == "" {
makeTargets = append(makeTargets, "check")
} else {
makeTargets = append(makeTargets, attr.CheckName)
}
}
makeTargets = append(makeTargets, attr.Make...)
if len(makeTargets) == 1 {
makeTargets = nil
}
finalExtra := []pkg.Artifact{
t.Load(Make),
}
if attr.OmitDefaults || attr.Flag&TEarly == 0 {
finalExtra = append(finalExtra,
t.Load(Gawk),
t.Load(Coreutils),
)
}
finalExtra = append(finalExtra, extra...)
scriptEarly := attr.ScriptEarly
if !attr.InPlace {
scriptEarly += "\ncd \"$(mktemp -d)\""
} else if scriptEarly == "" {
panic("cannot remain in root")
}
return t.New(name+"-"+version, attr.Flag, stage3Concat(t,
attr.NonStage3,
finalExtra...,
), nil, attr.Env, scriptEarly+`
/usr/src/`+name+`/configure \
--prefix=/system`+buildFlag+configureFlags+attr.ScriptConfigured+`
make "-j$(nproc)"`+strings.Join(makeTargets, " ")+`
make DESTDIR=/work install
`+attr.Script, pkg.Path(AbsUsrSrc.Append(
name+attr.SourceSuffix,
), attr.Writable, source))
}

27
internal/rosa/meson.go Normal file
View File

@@ -0,0 +1,27 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newMeson() pkg.Artifact {
const (
version = "1.10.1"
checksum = "w895BXF_icncnXatT_OLCFe2PYEtg4KrKooMgUYdN-nQVvbFX3PvYWHGEpogsHtd"
)
return t.New("meson-"+version, 0, []pkg.Artifact{
t.Load(Python),
t.Load(Setuptools),
}, nil, nil, `
cd /usr/src/meson
chmod -R +w meson.egg-info
python3 setup.py \
install \
--prefix=/system \
--root=/work
`, pkg.Path(AbsUsrSrc.Append("meson"), true, pkg.NewHTTPGetTar(
nil, "https://github.com/mesonbuild/meson/releases/download/"+
version+"/meson-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func init() { artifactsF[Meson] = Toolchain.newMeson }

36
internal/rosa/mksh.go Normal file
View File

@@ -0,0 +1,36 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newMksh() pkg.Artifact {
const (
version = "59c"
checksum = "0Zj-k4nXEu3IuJY4lvwD2OrC2t27GdZj8SPy4DoaeuBRH1padWb7oREpYgwY8JNq"
)
return t.New("mksh-"+version, 0, stage3Concat(t, []pkg.Artifact{},
t.Load(Perl),
t.Load(Coreutils),
), nil, []string{
"LDSTATIC=-static",
"CPPFLAGS=-DMKSH_DEFAULT_PROFILEDIR=\\\"/system/etc\\\"",
}, `
cd "$(mktemp -d)"
sh /usr/src/mksh/Build.sh -r
CPPFLAGS="${CPPFLAGS} -DMKSH_BINSHPOSIX -DMKSH_BINSHREDUCED" \
sh /usr/src/mksh/Build.sh -r -L
./test.sh -C regress:no-ctty
mkdir -p /work/system/bin/
cp -v mksh /work/system/bin/
cp -v lksh /work/system/bin/sh
mkdir -p /work/bin/
ln -vs ../system/bin/sh /work/bin/
`, pkg.Path(AbsUsrSrc.Append("mksh"), false, pkg.NewHTTPGetTar(
nil,
"https://mbsd.evolvis.org/MirOS/dist/mir/mksh/mksh-R"+version+".tgz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func init() { artifactsF[Mksh] = Toolchain.newMksh }

64
internal/rosa/musl.go Normal file
View File

@@ -0,0 +1,64 @@
package rosa
import (
"slices"
"hakurei.app/internal/pkg"
)
// MuslAttr holds the attributes that will be applied to musl.
type MuslAttr struct {
// Install headers only.
Headers bool
// Environment variables concatenated with defaults.
Env []string
// Dependencies concatenated with defaults.
Extra []pkg.Artifact
}
// NewMusl returns a [pkg.Artifact] containing an installation of musl libc.
func (t Toolchain) NewMusl(attr *MuslAttr) pkg.Artifact {
const (
version = "1.2.5"
checksum = "y6USdIeSdHER_Fw2eT2CNjqShEye85oEg2jnOur96D073ukmIpIqDOLmECQroyDb"
)
if attr == nil {
attr = new(MuslAttr)
}
target := "install"
script := `
mkdir -p /work/system/bin
COMPAT_LINKER_NAME="ld-musl-` + linuxArch() + `.so.1"
ln -vs ../lib/libc.so /work/system/bin/linker
ln -vs ../lib/libc.so /work/system/bin/ldd
ln -vs libc.so "/work/system/lib/${COMPAT_LINKER_NAME}"
rm -v "/work/lib/${COMPAT_LINKER_NAME}"
rmdir -v /work/lib
`
if attr.Headers {
target = "install-headers"
script = ""
}
return t.New("musl-"+version, 0, stage3Concat(t, attr.Extra,
t.Load(Make),
t.Load(Coreutils),
), nil, slices.Concat([]string{
"ROSA_MUSL_TARGET=" + target,
}, attr.Env), `
cd "$(mktemp -d)"
/usr/src/musl/configure \
--prefix=/system \
--target="${ROSA_TRIPLE}"
make "-j$(nproc)" DESTDIR=/work "${ROSA_MUSL_TARGET}"
`+script, pkg.Path(AbsUsrSrc.Append("musl"), false, t.NewPatchedSource(
// expected to be writable in copies
"musl", version, pkg.NewHTTPGetTar(
nil, "https://musl.libc.org/releases/musl-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false,
)))
}

39
internal/rosa/ninja.go Normal file
View File

@@ -0,0 +1,39 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newNinja() pkg.Artifact {
const (
version = "1.13.2"
checksum = "ygKWMa0YV2lWKiFro5hnL-vcKbc_-RACZuPu0Io8qDvgQlZ0dxv7hPNSFkt4214v"
)
return t.New("ninja-"+version, 0, []pkg.Artifact{
t.Load(CMake),
t.Load(Python),
t.Load(Bash),
}, nil, nil, `
cd "$(mktemp -d)"
python3 /usr/src/ninja/configure.py \
--bootstrap \
--gtest-source-dir=/usr/src/googletest
./ninja all
./ninja_test
mkdir -p /work/system/bin/
cp ninja /work/system/bin/
`, pkg.Path(AbsUsrSrc.Append("googletest"), false,
pkg.NewHTTPGetTar(
nil, "https://github.com/google/googletest/releases/download/"+
"v1.16.0/googletest-1.16.0.tar.gz",
mustDecode("NjLGvSbgPy_B-y-o1hdanlzEzaYeStFcvFGxpYV3KYlhrWWFRcugYhM3ZMzOA9B_"),
pkg.TarGzip,
)), pkg.Path(AbsUsrSrc.Append("ninja"), true, t.NewPatchedSource(
"ninja", version, pkg.NewHTTPGetTar(
nil, "https://github.com/ninja-build/ninja/archive/refs/tags/"+
"v"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false,
)))
}
func init() { artifactsF[Ninja] = Toolchain.newNinja }

36
internal/rosa/openssl.go Normal file
View File

@@ -0,0 +1,36 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newOpenSSL() pkg.Artifact {
const (
version = "3.5.5"
checksum = "I2Hp1LxcTR8j4G6LFEQMVy6EJH-Na1byI9Ti-ThBot6EMLNRnjGXGq-WXrim3Fkz"
)
return t.New("openssl-"+version, 0, []pkg.Artifact{
t.Load(Perl),
t.Load(Make),
t.Load(Zlib),
t.Load(KernelHeaders),
}, nil, []string{
"CC=cc",
}, `
cd "$(mktemp -d)"
/usr/src/openssl/Configure \
--prefix=/system \
--libdir=lib \
--openssldir=etc/ssl
make \
"-j$(nproc)" \
HARNESS_JOBS=256 \
test
make DESTDIR=/work install
`, pkg.Path(AbsUsrSrc.Append("openssl"), false, pkg.NewHTTPGetTar(
nil, "https://github.com/openssl/openssl/releases/download/"+
"openssl-"+version+"/openssl-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func init() { artifactsF[OpenSSL] = Toolchain.newOpenSSL }

39
internal/rosa/perl.go Normal file
View File

@@ -0,0 +1,39 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newPerl() pkg.Artifact {
const (
version = "5.42.0"
checksum = "2KR7Jbpk-ZVn1a30LQRwbgUvg2AXlPQZfzrqCr31qD5-yEsTwVQ_W76eZH-EdxM9"
)
return t.New("perl-"+version, TEarly, []pkg.Artifact{
t.Load(Make),
}, nil, nil, `
cd /usr/src/perl
echo 'print STDOUT "1..0 # Skip broken test\n";' > ext/Pod-Html/t/htmldir3.t
rm -f /system/bin/ps # perl does not like toybox ps
./Configure \
-des \
-Dprefix=/system \
-Dcc="clang" \
-Dcflags='--std=gnu99' \
-Dldflags="${LDFLAGS}" \
-Doptimize='-O2 -fno-strict-aliasing' \
-Duseithreads
make \
"-j$(nproc)" \
TEST_JOBS=256 \
test_harness
make DESTDIR=/work install
`, pkg.Path(AbsUsrSrc.Append("perl"), true, t.NewPatchedSource(
"perl", version, pkg.NewHTTPGetTar(
nil, "https://www.cpan.org/src/5.0/perl-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false,
)))
}
func init() { artifactsF[Perl] = Toolchain.newPerl }

View File

@@ -0,0 +1,23 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newPkgConfig() pkg.Artifact {
const (
version = "0.29.2"
checksum = "gi7yAvkwo20Inys1tHbeYZ3Wjdm5VPkrnO0Q6_QZPCAwa1zrA8F4a63cdZDd-717"
)
return t.NewViaMake("pkg-config", version, pkg.NewHTTPGetTar(
nil,
"https://pkgconfig.freedesktop.org/releases/"+
"pkg-config-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Configure: [][2]string{
{"CFLAGS", "'-Wno-int-conversion'"},
{"with-internal-glib"},
},
})
}
func init() { artifactsF[PkgConfig] = Toolchain.newPkgConfig }

151
internal/rosa/python.go Normal file
View File

@@ -0,0 +1,151 @@
package rosa
import (
"slices"
"strings"
"hakurei.app/internal/pkg"
)
func (t Toolchain) newPython() pkg.Artifact {
const (
version = "3.14.2"
checksum = "7nZunVMGj0viB-CnxpcRego2C90X5wFsMTgsoewd5z-KSZY2zLuqaBwG-14zmKys"
)
return t.NewViaMake("python", version, t.NewPatchedSource("python", version, pkg.NewHTTPGetTar(
nil, "https://www.python.org/ftp/python/"+version+
"/Python-"+version+".tgz",
mustDecode(checksum),
pkg.TarGzip,
), false), &MakeAttr{
// test_synopsis_sourceless assumes this is writable and checks __pycache__
Writable: true,
Env: []string{
"EXTRATESTOPTS=-j0 -x " + strings.Join([]string{
// requires internet access (http://www.pythontest.net/)
"test_asyncio",
"test_socket",
"test_urllib2",
"test_urllibnet",
"test_urllib2net",
// makes assumptions about uid_map/gid_map
"test_os",
"test_subprocess",
// somehow picks up mtime of source code
"test_zipfile",
// requires gcc
"test_ctypes",
// breaks on llvm
"test_dbm_gnu",
}, " -x "),
// _ctypes appears to infer something from the linker name
"LDFLAGS=-Wl,--dynamic-linker=/system/lib/" +
"ld-musl-" + linuxArch() + ".so.1",
},
ScriptEarly: `
export HOME="$(mktemp -d)"
`,
CheckName: "test",
},
t.Load(Zlib),
t.Load(Libffi),
)
}
func init() { artifactsF[Python] = Toolchain.newPython }
// newViaPip is a helper for installing python dependencies via pip.
func (t Toolchain) newViaPip(
name, version, abi, platform, checksum, prefix string,
extra ...pkg.Artifact,
) pkg.Artifact {
wname := name + "-" + version + "-py3-" + abi + "-" + platform + ".whl"
return t.New(name+"-"+version, 0, slices.Concat([]pkg.Artifact{
t.Load(Python),
}, extra), nil, nil, `
pip3 install \
--no-index \
--prefix=/system \
--root=/work \
/usr/src/`+wname+`
`, pkg.Path(AbsUsrSrc.Append(wname), false, pkg.NewHTTPGet(
nil, prefix+wname,
mustDecode(checksum),
)))
}
func (t Toolchain) newSetuptools() pkg.Artifact {
const (
version = "80.10.1"
checksum = "p3rlwEmy1krcUH1KabprQz1TCYjJ8ZUjOQknQsWh3q-XEqLGEd3P4VrCc7ouHGXU"
)
return t.New("setuptools-"+version, 0, []pkg.Artifact{
t.Load(Python),
}, nil, nil, `
pip3 install \
--no-index \
--prefix=/system \
--root=/work \
/usr/src/setuptools
`, pkg.Path(AbsUsrSrc.Append("setuptools"), true, pkg.NewHTTPGetTar(
nil, "https://github.com/pypa/setuptools/archive/refs/tags/"+
"v"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func init() { artifactsF[Setuptools] = Toolchain.newSetuptools }
func (t Toolchain) newPygments() pkg.Artifact {
return t.newViaPip("pygments", "2.19.2", "none", "any",
"ak_lwTalmSr7W4Mjy2XBZPG9I6a0gwSy2pS87N8x4QEuZYif0ie9z0OcfRfi9msd",
"https://files.pythonhosted.org/packages/"+
"c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/")
}
func init() { artifactsF[Pygments] = Toolchain.newPygments }
func (t Toolchain) newPluggy() pkg.Artifact {
return t.newViaPip("pluggy", "1.6.0", "none", "any",
"2HWYBaEwM66-y1hSUcWI1MyE7dVVuNNRW24XD6iJBey4YaUdAK8WeXdtFMQGC-4J",
"https://files.pythonhosted.org/packages/"+
"54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/")
}
func init() { artifactsF[Pluggy] = Toolchain.newPluggy }
func (t Toolchain) newPackaging() pkg.Artifact {
return t.newViaPip("packaging", "26.0", "none", "any",
"iVVXcqdwHDskPKoCFUlh2x8J0Gyq-bhO4ns9DvUJ7oJjeOegRYtSIvLV33Bki-pP",
"https://files.pythonhosted.org/packages/"+
"b7/b9/c538f279a4e237a006a2c98387d081e9eb060d203d8ed34467cc0f0b9b53/")
}
func init() { artifactsF[Packaging] = Toolchain.newPackaging }
func (t Toolchain) newIniConfig() pkg.Artifact {
const version = "2.3.0"
return t.newViaPip("iniconfig", version, "none", "any",
"SDgs4S5bXi77aVOeKTPv2TUrS3M9rduiK4DpU0hCmDsSBWqnZcWInq9lsx6INxut",
"https://github.com/pytest-dev/iniconfig/releases/download/"+
"v"+version+"/")
}
func init() { artifactsF[IniConfig] = Toolchain.newIniConfig }
func (t Toolchain) newPyTest() pkg.Artifact {
const version = "9.0.2"
return t.newViaPip("pytest", version, "none", "any",
"IM2wDbLke1EtZhF92zvAjUl_Hms1uKDtM7U8Dt4acOaChMnDg1pW7ib8U0wYGDLH",
"https://github.com/pytest-dev/pytest/releases/download/"+
version+"/",
t.Load(IniConfig),
t.Load(Packaging),
t.Load(Pluggy),
t.Load(Pygments),
)
}
func init() { artifactsF[PyTest] = Toolchain.newPyTest }

334
internal/rosa/rosa.go Normal file
View File

@@ -0,0 +1,334 @@
// Package rosa provides Rosa OS toolchain artifacts and miscellaneous software.
package rosa
import (
"log"
"runtime"
"slices"
"strconv"
"strings"
"hakurei.app/container/fhs"
"hakurei.app/internal/pkg"
)
const (
// kindEtc is the kind of [pkg.Artifact] of cureEtc.
kindEtc = iota + pkg.KindCustomOffset
// kindBusyboxBin is the kind of [pkg.Artifact] of busyboxBin.
kindBusyboxBin
)
// mustDecode is like [pkg.MustDecode], but replaces the zero value and prints
// a warning.
func mustDecode(s string) pkg.Checksum {
var fallback = pkg.Checksum{}
if s == "" {
log.Println(
"falling back to",
pkg.Encode(fallback),
"for unpopulated checksum",
)
return fallback
}
return pkg.MustDecode(s)
}
var (
// AbsUsrSrc is the conventional directory to place source code under.
AbsUsrSrc = fhs.AbsUsr.Append("src")
// AbsSystem is the Rosa OS installation prefix.
AbsSystem = fhs.AbsRoot.Append("system")
)
// linuxArch returns the architecture name used by linux corresponding to
// [runtime.GOARCH].
func linuxArch() string {
switch runtime.GOARCH {
case "amd64":
return "x86_64"
case "arm64":
return "aarch64"
default:
panic("unsupported target " + runtime.GOARCH)
}
}
// triplet returns the Rosa OS host triple corresponding to [runtime.GOARCH].
func triplet() string {
return linuxArch() + "-rosa-linux-musl"
}
const (
// EnvTriplet holds the return value of triplet.
EnvTriplet = "ROSA_TRIPLE"
)
// earlyLDFLAGS returns LDFLAGS corresponding to triplet.
func earlyLDFLAGS(static bool) string {
s := "-fuse-ld=lld " +
"-L/system/lib -Wl,-rpath=/system/lib " +
"-L/system/lib/" + triplet() + " " +
"-Wl,-rpath=/system/lib/" + triplet() + " " +
"-rtlib=compiler-rt " +
"-unwindlib=libunwind " +
"-Wl,--as-needed"
if !static {
s += " -Wl,--dynamic-linker=/system/bin/linker"
}
return s
}
// earlyCFLAGS is reference CFLAGS for the stage3 toolchain.
const earlyCFLAGS = "-Qunused-arguments " +
"-isystem/system/include"
// earlyCXXFLAGS returns reference CXXFLAGS for the stage3 toolchain
// corresponding to [runtime.GOARCH].
func earlyCXXFLAGS() string {
return "--start-no-unused-arguments " +
"-stdlib=libc++ " +
"--end-no-unused-arguments " +
"-isystem/system/include/c++/v1 " +
"-isystem/system/include/" + triplet() + "/c++/v1 " +
"-isystem/system/include "
}
// Toolchain denotes the infrastructure to compile a [pkg.Artifact] on.
type Toolchain uintptr
const (
// toolchainBusybox denotes a busybox installation from the busyboxBin
// binary distribution. This is for decompressing unsupported formats.
toolchainBusybox Toolchain = iota
// toolchainStage3 denotes the Gentoo stage3 toolchain. Special care must be
// taken to compile correctly against this toolchain.
toolchainStage3
// toolchainIntermediate denotes the intermediate toolchain compiled against
// toolchainStage3. This toolchain should be functionally identical to [Std]
// and is used to bootstrap [Std].
toolchainIntermediate
// Std denotes the standard Rosa OS toolchain.
Std
// _toolchainEnd is the total number of toolchains available and does not
// denote a valid toolchain.
_toolchainEnd
)
// stage3Concat concatenates s and values. If the current toolchain is
// toolchainStage3, stage3Concat returns s as is.
func stage3Concat[S ~[]E, E any](t Toolchain, s S, values ...E) S {
if t == toolchainStage3 {
return s
}
return slices.Concat(s, values)
}
// stage3ExclConcat concatenates s and values. If the current toolchain is not
// toolchainStage3, stage3ExclConcat returns s as is.
func stage3ExclConcat[S ~[]E, E any](t Toolchain, s S, values ...E) S {
if t == toolchainStage3 {
return slices.Concat(s, values)
}
return s
}
// lastIndexFunc is like [strings.LastIndexFunc] but for [slices].
func lastIndexFunc[S ~[]E, E any](s S, f func(E) bool) (i int) {
if i = slices.IndexFunc(s, f); i < 0 {
return
}
if i0 := lastIndexFunc[S](s[i+1:], f); i0 >= 0 {
i = i0
}
return
}
// fixupEnviron fixes up PATH, prepends extras and returns the resulting slice.
func fixupEnviron(env, extras []string, paths ...string) []string {
const pathPrefix = "PATH="
pathVal := strings.Join(paths, ":")
if i := lastIndexFunc(env, func(s string) bool {
return strings.HasPrefix(s, pathPrefix)
}); i < 0 {
env = append(env, pathPrefix+pathVal)
} else {
if len(env[i]) == len(pathPrefix) {
env[i] = pathPrefix + pathVal
} else {
env[i] += ":" + pathVal
}
}
return append(extras, env...)
}
// absCureScript is the absolute pathname [Toolchain.New] places the fixed-up
// build script under.
var absCureScript = fhs.AbsUsrBin.Append(".cure-script")
const (
// TExclusive denotes an exclusive [pkg.Artifact].
TExclusive = 1 << iota
// TEarly hints for an early variant of [Toybox] to be used when available.
TEarly
)
// New returns a [pkg.Artifact] compiled on this toolchain.
func (t Toolchain) New(
name string,
flag int,
extra []pkg.Artifact,
checksum *pkg.Checksum,
env []string,
script string,
paths ...pkg.ExecPath,
) pkg.Artifact {
const lcMessages = "LC_MESSAGES=C.UTF-8"
var (
path = AbsSystem.Append("bin", "sh")
args = []string{"sh", absCureScript.String()}
support []pkg.Artifact
)
switch t {
case toolchainBusybox:
name += "-early"
support = slices.Concat([]pkg.Artifact{newBusyboxBin()}, extra)
path = AbsSystem.Append("bin", "busybox")
args[0] = "hush"
env = fixupEnviron(env, nil, "/system/bin")
case toolchainStage3:
name += "-boot"
var seed string
switch runtime.GOARCH {
case "amd64":
seed = "c5_FwMnRN8RZpTdBLGYkL4RR8ampdaZN2JbkgrFLe8-QHQAVQy08APVvIL6eT7KW"
case "arm64":
seed = "79uRbRI44PyknQQ9RlFUQrwqplup7vImiIk6klefL8TN-fT42TXMS_v4XszwexCb"
default:
panic("unsupported target " + runtime.GOARCH)
}
path = fhs.AbsRoot.Append("bin", "bash")
args[0] = "bash"
support = slices.Concat([]pkg.Artifact{
cureEtc{},
toolchainBusybox.New("stage3", 0, nil, nil, nil, `
tar -C /work -xf /usr/src/stage3.tar.xz
rm -rf /work/dev/ /work/proc/
ln -vs ../usr/bin /work/bin
`, pkg.Path(AbsUsrSrc.Append("stage3.tar.xz"), false,
pkg.NewHTTPGet(
nil, "https://basement.gensokyo.uk/seed/"+seed,
mustDecode(seed),
),
)),
}, extra)
env = fixupEnviron(env, []string{
EnvTriplet + "=" + triplet(),
lcMessages,
"LDFLAGS=" + earlyLDFLAGS(true),
}, "/system/bin",
"/usr/bin",
"/usr/lib/llvm/21/bin",
)
case toolchainIntermediate, Std:
if t < Std {
name += "-std"
}
boot := t - 1
musl, compilerRT, runtimes, clang := boot.NewLLVM()
toybox := Toybox
if flag&TEarly != 0 {
toybox = toyboxEarly
}
support = slices.Concat(extra, []pkg.Artifact{
cureEtc{newIANAEtc()},
musl,
compilerRT,
runtimes,
clang,
boot.Load(Mksh),
boot.Load(toybox),
})
env = fixupEnviron(env, []string{
EnvTriplet + "=" + triplet(),
lcMessages,
"AR=ar",
"RANLIB=ranlib",
"LIBCC=/system/lib/clang/21/lib/" + triplet() +
"/libclang_rt.builtins.a",
}, "/system/bin", "/bin")
default:
panic("unsupported toolchain " + strconv.Itoa(int(t)))
}
return pkg.NewExec(
name, checksum, pkg.ExecTimeoutMax, flag&TExclusive != 0,
fhs.AbsRoot, env,
path, args,
slices.Concat([]pkg.ExecPath{pkg.Path(
fhs.AbsRoot, true,
support...,
), pkg.Path(
absCureScript, false,
pkg.NewFile(".cure-script", []byte("set -e\n"+script)),
)}, paths)...,
)
}
// NewPatchedSource returns [pkg.Artifact] of source with patches applied. If
// passthrough is true, source is returned as is for zero length patches.
func (t Toolchain) NewPatchedSource(
name, version string,
source pkg.Artifact,
passthrough bool,
patches ...[2]string,
) pkg.Artifact {
if passthrough && len(patches) == 0 {
return source
}
paths := make([]pkg.ExecPath, len(patches)+1)
for i, p := range patches {
paths[i+1] = pkg.Path(
AbsUsrSrc.Append(name+"-patches", p[0]+".patch"), false,
pkg.NewFile(p[0]+".patch", []byte(p[1])),
)
}
paths[0] = pkg.Path(AbsUsrSrc.Append(name), false, source)
aname := name + "-" + version + "-src"
script := `
cp -r /usr/src/` + name + `/. /work/.
chmod -R +w /work && cd /work
`
if len(paths) > 1 {
script += `
cat /usr/src/` + name + `-patches/* | \
patch \
-p 1 \
--ignore-whitespace
`
aname += "-patched"
}
return t.New(aname, 0, stage3Concat(t, []pkg.Artifact{},
t.Load(Patch),
), nil, nil, script, paths...)
}

31
internal/rosa/rsync.go Normal file
View File

@@ -0,0 +1,31 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newRsync() pkg.Artifact {
const (
version = "3.4.1"
checksum = "VBlTsBWd9z3r2-ex7GkWeWxkUc5OrlgDzikAC0pK7ufTjAJ0MbmC_N04oSVTGPiv"
)
return t.NewViaMake("rsync", version, pkg.NewHTTPGetTar(
nil, "https://download.samba.org/pub/rsync/src/"+
"rsync-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Configure: [][2]string{
{"disable-openssl"},
{"disable-xxhash"},
{"disable-zstd"},
{"disable-lz4"},
},
// circular dependency
SkipCheck: true,
Flag: TEarly,
},
t.Load(Gawk),
)
}
func init() { artifactsF[Rsync] = Toolchain.newRsync }

84
internal/rosa/ssl.go Normal file
View File

@@ -0,0 +1,84 @@
package rosa
import (
"hakurei.app/internal/pkg"
)
func (t Toolchain) newNSS() pkg.Artifact {
const (
version = "3_120"
checksum = "9M0SNMrj9BJp6RH2rQnMm6bZWtP0Kgj64D5JNPHF7Cxr2_8kfy3msubIcvEPwC35"
version0 = "4_38_2"
checksum0 = "25x2uJeQnOHIiq_zj17b4sYqKgeoU8-IsySUptoPcdHZ52PohFZfGuIisBreWzx0"
)
return t.New("nss-"+version, 0, []pkg.Artifact{
t.Load(Perl),
t.Load(Python),
t.Load(Unzip),
t.Load(Make),
t.Load(Gawk),
t.Load(Coreutils),
t.Load(Zlib),
t.Load(KernelHeaders),
}, nil, nil, `
unzip /usr/src/nspr.zip -d /usr/src
mv '/usr/src/nspr-NSPR_`+version0+`_RTM' /usr/src/nspr
cd /usr/src/nss
make \
"-j$(nproc)" \
CCC="clang++" \
NSDISTMODE=copy \
BUILD_OPT=1 \
USE_64=1 \
nss_build_all
mkdir -p /work/system/nss
cp -r \
/usr/src/dist/. \
lib/ckfw/builtins/certdata.txt \
/work/system/nss
`, pkg.Path(AbsUsrSrc.Append("nss"), true, t.NewPatchedSource(
"nss", version, pkg.NewHTTPGetTar(
nil, "https://github.com/nss-dev/nss/archive/refs/tags/"+
"NSS_"+version+"_RTM.tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false,
)), pkg.Path(AbsUsrSrc.Append("nspr.zip"), false, pkg.NewHTTPGet(
nil, "https://hg-edge.mozilla.org/projects/nspr/archive/"+
"NSPR_"+version0+"_RTM.zip",
mustDecode(checksum0),
)))
}
func init() { artifactsF[NSS] = Toolchain.newNSS }
func (t Toolchain) newBuildCATrust() pkg.Artifact {
const version = "0.4.0"
return t.newViaPip("buildcatrust", version, "none", "any",
"k_FGzkRCLjbTWBkuBLzQJ1S8FPAz19neJZlMHm0t10F2Y0hElmvVwdSBRc03Rjo1",
"https://github.com/nix-community/buildcatrust/"+
"releases/download/v"+version+"/")
}
func init() { artifactsF[buildcatrust] = Toolchain.newBuildCATrust }
func (t Toolchain) newNSSCACert() pkg.Artifact {
return t.New("nss-cacert", 0, []pkg.Artifact{
t.Load(Bash),
t.Load(Python),
t.Load(NSS),
t.Load(buildcatrust),
}, nil, nil, `
mkdir -p /work/system/etc/ssl/{certs/unbundled,certs/hashed,trust-source}
buildcatrust \
--certdata_input /system/nss/certdata.txt \
--ca_bundle_output /work/system/etc/ssl/certs/ca-bundle.crt \
--ca_standard_bundle_output /work/system/etc/ssl/certs/ca-no-trust-rules-bundle.crt \
--ca_unpacked_output /work/system/etc/ssl/certs/unbundled \
--ca_hashed_unpacked_output /work/system/etc/ssl/certs/hashed \
--p11kit_output /work/system/etc/ssl/trust-source/ca-bundle.trust.p11-kit
`)
}
func init() { artifactsF[NSSCACert] = Toolchain.newNSSCACert }

64
internal/rosa/toybox.go Normal file
View File

@@ -0,0 +1,64 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newToybox(suffix, script string) pkg.Artifact {
const (
version = "0.8.13"
checksum = "rZ1V1ATDte2WeQZanxLVoiRGdfPXhMlEo5-exX-e-ml8cGn9qOv0ABEUVZpX3wTI"
)
return t.New("toybox-"+version+suffix, TEarly, stage3Concat(t, []pkg.Artifact{},
t.Load(Make),
t.Load(Bash),
t.Load(Gzip),
t.Load(KernelHeaders),
), nil, stage3Concat(t, []string{},
"ROSA_CHECK=make USER=cure tests",
), `
ln -s ../system/bin/bash /bin/ || true
cd /usr/src/toybox
chmod +w kconfig tests
rm \
tests/du.test \
tests/sed.test \
tests/tar.test \
tests/ls.test \
tests/taskset.test
make defconfig
sed -i \
's/^CONFIG_TOYBOX_ZHELP=y$/CONFIG_TOYBOX_ZHELP=0/' \
.config
`+script+`
make \
"-j$(nproc)" \
LDFLAGS="${LDFLAGS} -static"
${ROSA_CHECK}
PREFIX=/work/system/bin make install_flat
mkdir -p /work/usr/bin
ln -s ../../system/bin/env /work/usr/bin
`, pkg.Path(AbsUsrSrc.Append("toybox"), true, pkg.NewHTTPGetTar(
nil,
"https://landley.net/toybox/downloads/toybox-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
)))
}
func init() {
artifactsF[Toybox] = func(t Toolchain) pkg.Artifact {
return t.newToybox("", "")
}
artifactsF[toyboxEarly] = func(t Toolchain) pkg.Artifact {
return t.newToybox("-early", `
echo '
CONFIG_EXPR=y
CONFIG_TR=y
CONFIG_AWK=y
CONFIG_DIFF=y
' >> .config
`)
}
}

34
internal/rosa/unzip.go Normal file
View File

@@ -0,0 +1,34 @@
package rosa
import (
"strings"
"hakurei.app/internal/pkg"
)
func (t Toolchain) newUnzip() pkg.Artifact {
const (
version = "6.0"
checksum = "fcqjB1IOVRNJ16K5gTGEDt3zCJDVBc7EDSra9w3H93stqkNwH1vaPQs_QGOpQZu1"
)
return t.New("unzip-"+version, 0, []pkg.Artifact{
t.Load(Make),
t.Load(Coreutils),
}, nil, nil, `
cd /usr/src/unzip/
unix/configure
make -f unix/Makefile generic1
mkdir -p /work/system/bin/
mv unzip /work/system/bin/
`, pkg.Path(AbsUsrSrc.Append("unzip"), true, t.NewPatchedSource(
"unzip", version, pkg.NewHTTPGetTar(
nil, "https://downloads.sourceforge.net/project/infozip/"+
"UnZip%206.x%20%28latest%29/UnZip%20"+version+"/"+
"unzip"+strings.ReplaceAll(version, ".", "")+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), false,
)))
}
func init() { artifactsF[Unzip] = Toolchain.newUnzip }

84
internal/rosa/wayland.go Normal file
View File

@@ -0,0 +1,84 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newWayland() pkg.Artifact {
const (
version = "1.24.0"
checksum = "JxgLiFRRGw2D3uhVw8ZeDbs3V7K_d4z_ypDog2LBqiA_5y2vVbUAk5NT6D5ozm0m"
)
return t.New("wayland-"+version, 0, []pkg.Artifact{
t.Load(Python),
t.Load(Meson),
t.Load(PkgConfig),
t.Load(CMake),
t.Load(Ninja),
t.Load(Gawk),
t.Load(Diffutils),
t.Load(Libffi),
t.Load(Libexpat),
t.Load(Libxml2),
}, nil, nil, `
cd /usr/src/wayland
chmod +w tests tests/sanity-test.c
echo 'int main(){}' > tests/sanity-test.c
cd "$(mktemp -d)"
meson setup \
--reconfigure \
--buildtype=release \
--prefix=/system \
--prefer-static \
-Ddocumentation=false \
-Dtests=true \
-Ddefault_library=both \
. /usr/src/wayland
meson compile
meson test
meson install \
--destdir=/work
`, pkg.Path(AbsUsrSrc.Append("wayland"), true, pkg.NewHTTPGetTar(
nil, "https://gitlab.freedesktop.org/wayland/wayland/"+
"-/archive/"+version+"/wayland-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
)))
}
func init() { artifactsF[Wayland] = Toolchain.newWayland }
func (t Toolchain) newWaylandProtocols() pkg.Artifact {
const (
version = "1.47"
checksum = "B_NodZ7AQfCstcx7kgbaVjpkYOzbAQq0a4NOk-SA8bQixAE20FY3p1-6gsbPgHn9"
)
return t.New("wayland-protocols-"+version, 0, []pkg.Artifact{
t.Load(Python),
t.Load(Meson),
t.Load(PkgConfig),
t.Load(CMake),
t.Load(Ninja),
t.Load(Wayland),
t.Load(Libffi),
t.Load(Libexpat),
t.Load(Libxml2),
}, nil, nil, `
cd "$(mktemp -d)"
meson setup \
--reconfigure \
--buildtype=release \
--prefix=/system \
--prefer-static \
. /usr/src/wayland-protocols
meson compile
meson install \
--destdir=/work
`, pkg.Path(AbsUsrSrc.Append("wayland-protocols"), false, pkg.NewHTTPGetTar(
nil, "https://gitlab.freedesktop.org/wayland/wayland-protocols/"+
"-/archive/"+version+"/wayland-protocols-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
)))
}
func init() { artifactsF[WaylandProtocols] = Toolchain.newWaylandProtocols }

83
internal/rosa/x.go Normal file
View File

@@ -0,0 +1,83 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newUtilMacros() pkg.Artifact {
const (
version = "1.17"
checksum = "vYPO4Qq3B_WGcsBjG0-lfwZ6DZ7ayyrOLqfDrVOgTDcyLChuMGOAAVAa_UXLu5tD"
)
return t.NewViaMake("util-macros", version, pkg.NewHTTPGetTar(
nil, "https://www.x.org/releases/X11R7.7/src/util/"+
"util-macros-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), nil)
}
func init() { artifactsF[utilMacros] = Toolchain.newUtilMacros }
func (t Toolchain) newXproto() pkg.Artifact {
const (
version = "7.0.23"
checksum = "goxwWxV0jZ_3pNczXFltZWHAhq92x-aEreUGyp5Ns8dBOoOmgbpeNIu1nv0Zx07z"
)
return t.NewViaMake("xproto", version, pkg.NewHTTPGetTar(
nil, "https://www.x.org/releases/X11R7.7/src/proto/"+
"xproto-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), &MakeAttr{
Writable: true,
// ancient configure script
ScriptEarly: `
cd /usr/src/xproto
autoreconf -if
`,
},
t.Load(M4),
t.Load(Perl),
t.Load(Autoconf),
t.Load(Automake),
t.Load(PkgConfig),
t.Load(utilMacros),
)
}
func init() { artifactsF[Xproto] = Toolchain.newXproto }
func (t Toolchain) newLibXau() pkg.Artifact {
const (
version = "1.0.7"
checksum = "bm768RoZZnHRe9VjNU1Dw3BhfE60DyS9D_bgSR-JLkEEyUWT_Hb_lQripxrXto8j"
)
return t.NewViaMake("libXau", version, pkg.NewHTTPGetTar(
nil, "https://www.x.org/releases/X11R7.7/src/lib/"+
"libXau-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), &MakeAttr{
Writable: true,
// ancient configure script
ScriptEarly: `
cd /usr/src/libXau
autoreconf -if
`,
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(M4),
t.Load(Perl),
t.Load(Autoconf),
t.Load(Automake),
t.Load(Libtool),
t.Load(PkgConfig),
t.Load(utilMacros),
t.Load(Xproto),
)
}
func init() { artifactsF[LibXau] = Toolchain.newLibXau }

46
internal/rosa/xcb.go Normal file
View File

@@ -0,0 +1,46 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newXCBProto() pkg.Artifact {
const (
version = "1.17.0"
checksum = "_NtbKaJ_iyT7XiJz25mXQ7y-niTzE8sHPvLXZPcqtNoV_-vTzqkezJ8Hp2U1enCv"
)
return t.NewViaMake("xcb-proto", version, pkg.NewHTTPGetTar(
nil, "https://xcb.freedesktop.org/dist/xcb-proto-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(Python),
)
}
func init() { artifactsF[XCBProto] = Toolchain.newXCBProto }
func (t Toolchain) newXCB() pkg.Artifact {
const (
version = "1.17.0"
checksum = "hjjsc79LpWM_hZjNWbDDS6qRQUXREjjekS6UbUsDq-RR1_AjgNDxhRvZf-1_kzDd"
)
return t.NewViaMake("xcb", version, pkg.NewHTTPGetTar(
nil, "https://xcb.freedesktop.org/dist/libxcb-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
Configure: [][2]string{
{"enable-static"},
},
},
t.Load(Python),
t.Load(PkgConfig),
t.Load(XCBProto),
t.Load(Xproto),
t.Load(LibXau),
)
}
func init() { artifactsF[XCB] = Toolchain.newXCB }

19
internal/rosa/xz.go Normal file
View File

@@ -0,0 +1,19 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newXZ() pkg.Artifact {
const (
version = "5.8.2"
checksum = "rXT-XCp9R2q6cXqJ5qenp0cmGPfiENQiU3BWtUVeVgArfRmSsISeUJgvCR3zI0a0"
)
return t.NewViaMake("xz", version, pkg.NewHTTPGetTar(
nil, "https://github.com/tukaani-project/xz/releases/download/"+
"v"+version+"/xz-"+version+".tar.bz2",
mustDecode(checksum),
pkg.TarBzip2,
), nil,
t.Load(Diffutils),
)
}
func init() { artifactsF[XZ] = Toolchain.newXZ }

22
internal/rosa/zlib.go Normal file
View File

@@ -0,0 +1,22 @@
package rosa
import "hakurei.app/internal/pkg"
func (t Toolchain) newZlib() pkg.Artifact {
const (
version = "1.3.1"
checksum = "E-eIpNzE8oJ5DsqH4UuA_0GDKuQF5csqI8ooDx2w7Vx-woJ2mb-YtSbEyIMN44mH"
)
return t.NewViaMake("zlib", version, pkg.NewHTTPGetTar(
nil, "https://zlib.net/zlib-"+version+".tar.gz",
mustDecode(checksum),
pkg.TarGzip,
), &MakeAttr{
OmitDefaults: true,
Env: []string{
"CC=clang -fPIC",
},
Build: `""`,
})
}
func init() { artifactsF[Zlib] = Toolchain.newZlib }

View File

@@ -6,6 +6,7 @@ import (
"path"
"syscall"
"testing"
"time"
"hakurei.app/container/stub"
"hakurei.app/internal/acl"
@@ -497,6 +498,12 @@ type stubPipeWireConn struct {
curSendmsg int
}
func (conn *stubPipeWireConn) MightBlock(timeout time.Duration) {
if timeout != 5*time.Second {
panic("unexpected timeout " + timeout.String())
}
}
// Recvmsg marshals and copies a stubMessage prepared ahead of time.
func (conn *stubPipeWireConn) Recvmsg(p, _ []byte, _ int) (n, _, recvflags int, err error) {
defer func() { conn.curRecvmsg++ }()

View File

@@ -36,7 +36,7 @@ libzstd.so.1 = /usr/lib/libzstd.so.1 (0x7ff71bfd2000)
{"path not absolute", `
libzstd.so.1 => usr/lib/libzstd.so.1 (0x7ff71bfd2000)
`, &check.AbsoluteError{Pathname: "usr/lib/libzstd.so.1"}},
`, check.AbsoluteError("usr/lib/libzstd.so.1")},
{"unexpected segments", `
meow libzstd.so.1 => /usr/lib/libzstd.so.1 (0x7ff71bfd2000)

View File

@@ -24,11 +24,38 @@ let
getsubuid = userid: appid: userid * 100000 + 10000 + appid;
getsubname = userid: appid: "u${toString userid}_a${toString appid}";
getsubhome = userid: appid: "${cfg.stateDir}/u${toString userid}/a${toString appid}";
mountpoints = {
${cfg.sharefs.name} = mkIf (cfg.sharefs.source != null) {
depends = [ cfg.sharefs.source ];
device = "sharefs";
fsType = "fuse.sharefs";
noCheck = true;
options = [
"rw"
"noexec"
"nosuid"
"nodev"
"noatime"
"allow_other"
"mkdir"
"source=${cfg.sharefs.source}"
"setuid=${toString config.users.users.${cfg.sharefs.user}.uid}"
"setgid=${toString config.users.groups.${cfg.sharefs.group}.gid}"
];
};
};
in
{
imports = [ (import ./options.nix packages) ];
options = {
# Forward declare a dummy option for VM filesystems since the real one won't exist
# unless the VM module is actually imported.
virtualisation.fileSystems = lib.mkOption { };
};
config = mkIf cfg.enable {
assertions = [
(
@@ -66,6 +93,10 @@ in
) "" cfg.users;
};
environment.systemPackages = optional (cfg.sharefs.source != null) cfg.sharefs.package;
fileSystems = mountpoints;
virtualisation.fileSystems = mountpoints;
home-manager =
let
privPackages = mapAttrs (_: userid: {
@@ -322,25 +353,57 @@ in
in
{
users = mkMerge (
foldlAttrs (
acc: _: fid:
acc
++ foldlAttrs (
acc': _: app:
acc' ++ [ { ${getsubname fid app.identity} = getuser fid app.identity; } ]
) [ { ${getsubname fid 0} = getuser fid 0; } ] cfg.apps
) [ ] cfg.users
foldlAttrs
(
acc: _: fid:
acc
++ foldlAttrs (
acc': _: app:
acc' ++ [ { ${getsubname fid app.identity} = getuser fid app.identity; } ]
) [ { ${getsubname fid 0} = getuser fid 0; } ] cfg.apps
)
(
if (cfg.sharefs.source != null) then
[
{
${cfg.sharefs.user} = {
uid = lib.mkDefault 1023;
inherit (cfg.sharefs) group;
isSystemUser = true;
home = cfg.sharefs.source;
};
}
]
else
[ ]
)
cfg.users
);
groups = mkMerge (
foldlAttrs (
acc: _: fid:
acc
++ foldlAttrs (
acc': _: app:
acc' ++ [ { ${getsubname fid app.identity} = getgroup fid app.identity; } ]
) [ { ${getsubname fid 0} = getgroup fid 0; } ] cfg.apps
) [ ] cfg.users
foldlAttrs
(
acc: _: fid:
acc
++ foldlAttrs (
acc': _: app:
acc' ++ [ { ${getsubname fid app.identity} = getgroup fid app.identity; } ]
) [ { ${getsubname fid 0} = getgroup fid 0; } ] cfg.apps
)
(
if (cfg.sharefs.source != null) then
[
{
${cfg.sharefs.group} = {
gid = lib.mkDefault 1023;
};
}
]
else
[ ]
)
cfg.users
);
};
};

View File

@@ -35,7 +35,7 @@ package
*Default:*
` <derivation hakurei-static-x86_64-unknown-linux-musl-0.3.3> `
` <derivation hakurei-static-x86_64-unknown-linux-musl-0.3.4> `
@@ -805,7 +805,97 @@ package
*Default:*
` <derivation hakurei-hsu-0.3.3> `
` <derivation hakurei-hsu-0.3.4> `
## environment\.hakurei\.sharefs\.package
The sharefs package to use\.
*Type:*
package
*Default:*
` <derivation sharefs> `
## environment\.hakurei\.sharefs\.group
Name of the group to run the sharefs daemon as\.
*Type:*
string
*Default:*
` "sharefs" `
## environment\.hakurei\.sharefs\.name
Host path to mount sharefs on\.
*Type:*
string
*Default:*
` "/sdcard" `
## environment\.hakurei\.sharefs\.source
Writable backing directory\. Setting this to null disables sharefs\.
*Type:*
null or string
*Default:*
` null `
## environment\.hakurei\.sharefs\.user
Name of the user to run the sharefs daemon as\.
*Type:*
string
*Default:*
` "sharefs" `

View File

@@ -1,8 +1,15 @@
packages:
{ lib, pkgs, ... }:
{
lib,
pkgs,
config,
...
}:
let
inherit (lib) types mkOption mkEnableOption;
cfg = config.environment.hakurei;
in
{
@@ -40,6 +47,49 @@ in
'';
};
sharefs = {
package = mkOption {
type = types.package;
default = pkgs.linkFarm "sharefs" {
"bin/sharefs" = "${cfg.package}/libexec/sharefs";
"bin/mount.fuse.sharefs" = "${cfg.package}/libexec/sharefs";
};
description = "The sharefs package to use.";
};
user = mkOption {
type = types.str;
default = "sharefs";
description = ''
Name of the user to run the sharefs daemon as.
'';
};
group = mkOption {
type = types.str;
default = "sharefs";
description = ''
Name of the group to run the sharefs daemon as.
'';
};
name = mkOption {
type = types.str;
default = "/sdcard";
description = ''
Host path to mount sharefs on.
'';
};
source = mkOption {
type = types.nullOr types.str;
default = null;
description = ''
Writable backing directory. Setting this to null disables sharefs.
'';
};
};
apps = mkOption {
type =
let

View File

@@ -13,6 +13,9 @@
wayland-scanner,
xorg,
# for sharefs
fuse3,
# for hpkg
zstd,
gnutar,
@@ -32,7 +35,7 @@
buildGoModule rec {
pname = "hakurei";
version = "0.3.3";
version = "0.3.4";
srcFiltered = builtins.path {
name = "${pname}-src";
@@ -86,12 +89,13 @@ buildGoModule rec {
CC = "clang -O3 -Werror";
# nix build environment does not allow acls
GO_TEST_SKIP_ACL = 1;
HAKUREI_TEST_SKIP_ACL = 1;
};
buildInputs = [
libffi
libseccomp
fuse3
acl
wayland
]

View File

@@ -1,8 +1,9 @@
{ pkgs, ... }:
{
environment.hakurei = {
environment.hakurei = rec {
enable = true;
stateDir = "/var/lib/hakurei";
sharefs.source = "${stateDir}/sdcard";
users.alice = 0;
apps = {
"cat.gensokyo.extern.foot.noEnablements" = {