getting ziggy with it

[nick black]

previous efforts have been made to package the zig programming language [0].
the most recent release of the zig programming language was 0.13.0 [1]
on 2024-06-07. bug #1012286 declares an ITP for zig 0.9.1 [2], but the work
there seems to have trailed off in the 0.10 era. jonathan carter started
working on it independently a few weeks back, but the work i've
done this weekend goes beyond his afaik [5]. i am interested in getting
zig in debian both on its own account, and as part of packaging ghostty [4].

i am not an expert regarding zig, but i believe the following to be true:

later that year, zig changed the way it bootstraps [3]. there was already a
"zig-bootstrap" repository and "zig" repository, both of which are released
together. the relevant elements: "zig-bootstrap" now ships with a binary WASM
file (zig1.wasm), which is converted to a c file (zig1.c) using a tool built
using the local c compiler (wasm2c). this c file is compiled using the system
compiler to produce zig.

this wasm file is, again, distributed with the zig-bootstrap and zig sources.
it is a binary and knocks us out of main. this file can be recreated, using
sources from within the zig repository, with a working zig. but even the
"zig-bootstrap" source cannot move forward without either this file or
a working zig with which to create it ("zig-bootstrap" is more about
vendoring LLVM).

the last release without this wasm component is 0.10.1, which i've been
targeting as "zig-10-bootstrap". it requires neither the wasm binary nor
a working zig. unfortunately, it does require llvm 15--not 14, not 17,
but 15. supporting a different llvm would require changing quite a bit
of tightly-coupled C++ that is no longer being maintained upstream.
llvm-15 is not in sid, btw, though llvm-14 is. so we would need bring
llvm-15 back into sid. attempting to build llvm-15 1:15.0.7-15 from source
on sid blew up; i didn't look deeply into it yet (i can confirm that
"zig-10-bootstrap" using its vendored LLVM 15 builds on sid, fwiw).

note the upstream's statement:

"Now, there is this WebAssembly binary, which is not source code, but is in
fact a build artifact. Some people, rightly, take these things very seriously -
see for example the Debian Free Software Guidelines. I openly acknowledge that
this cost is being paid, however, I strongly believe that it is worth it in the
end. I suspect that combined with an official language specification and a
growing popularity of Zig, we will see a third-party project start an alternate
Zig implementation in C, similar to how mrustc exists for Rust (despite not yet
having a spec). This would fill the necessary role to solve O(1) source
bootstrapping again."

perhaps.

so the choices as i see them:

- wait on this zig ex machina foretold by legend, perhaps until the sun
expands, consuming earth
- put zig in non-free. anything building with it is stuck in contrib.
pretty unappealing.
- maintain the ~20K lines of zigcpp myself. this will need to be
kept up with the historically rapidly evolving LLVM API.
similarly unappealing. maybe more.
- reintroduce llvm-15 to sid, get it working, introduce zig-10-bootstrap
to sid, make it a build-dep of zig, devendor zig, build with system llvm.
the devendoring has already been done. not terrible, but i'd like to
hear llvm team's thoughts and anyone else's thoughts. so far as i can
tell, this would require llvm-15 and zig-10-bootstrap to be carried
indefinitely into the future.

but perhaps someone else can think of something better?

[0] https://ziglang.org/
[1] https://ziglang.org/download/
[2] https://lists.debian.org/debian-mentors/2022/06/msg00023.html
[3] https://ziglang.org/news/goodbye-cpp/
[4] https://ghostty.org/
[5] https://salsa.debian.org/nickblack/zig-bootstrap,
https://salsa.debian.org/nickblack/zig-10-bootstrap,
https://salsa.debian.org/nickblack/zig

[nick black]

Post by nick black
this wasm file is, again, distributed with the zig-bootstrap and zig sources.
it is a binary and knocks us out of main. this file can be recreated, using
sources from within the zig repository, with a working zig. but even the
"zig-bootstrap" source cannot move forward without either this file or
a working zig with which to create it ("zig-bootstrap" is more about
vendoring LLVM).

i looked at a few other distros:

- arch just builds it and is done with it
- guix, from what i can tell, is picking up a previous zig build of
their own and using that to bootstrap. i am 0% certain of this
interpretation [0].
- fedora looks promising; they certainly remove zig1.wasm before
building; i will examine this further.

[0] https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/zig.scm
[1] https://src.fedoraproject.org/rpms/zig.git

[Russ Allbery]

Post by nick black
later that year, zig changed the way it bootstraps [3]. there was
already a "zig-bootstrap" repository and "zig" repository, both of which
are released together. the relevant elements: "zig-bootstrap" now ships
with a binary WASM file (zig1.wasm), which is converted to a c file
(zig1.c) using a tool built using the local c compiler (wasm2c). this c
file is compiled using the system compiler to produce zig.
this wasm file is, again, distributed with the zig-bootstrap and zig
sources. it is a binary and knocks us out of main. this file can be
recreated, using sources from within the zig repository, with a working
zig. but even the "zig-bootstrap" source cannot move forward without
either this file or a working zig with which to create it
("zig-bootstrap" is more about vendoring LLVM).

I think I'm missing something. Why does the use of this file knock the
package out of main? We are distributing the source code for this file,
and it is presumably under a free software license.

The bootstrap is circular and has to be kicked off with a binary blob that
can't be recreated until one has finished the bootstrap, which is
certainly not ideal, but is also not that atypical for compiler
bootstrapping problems. I don't see how this is a violation of the DFSG or
of the requirement that main be self-contained. It's obnoxious from a
process standpoint (like a lot of compiler bootstrapping), and we may not
want things to work this way, but I don't think that makes it non-free?

[Julien Plissonneau Duquène]

Post by Russ Allbery
I think I'm missing something. Why does the use of this file knock the
package out of main? We are distributing the source code for this file,
and it is presumably under a free software license.
The bootstrap is circular and has to be kicked off with a binary blob that
can't be recreated until one has finished the bootstrap, which is
certainly not ideal, but is also not that atypical for compiler
bootstrapping problems. I don't see how this is a violation of the DFSG or
of the requirement that main be self-contained. It's obnoxious from a
process standpoint (like a lot of compiler bootstrapping), and we may not
want things to work this way, but I don't think that makes it non-free?

I second this.

FWIW I'm going to need Zig soon to build one of the native dependencies
of newer Gradle releases so I'm happy that someone is already working on
it, and might contribute to sort out issues later.

Cheers,

[nick black]

Post by Russ Allbery
I think I'm missing something. Why does the use of this file knock the
package out of main? We are distributing the source code for this file,
and it is presumably under a free software license.

but we are not distributing it in a way such that it can be
built, as the compiler is available.

Post by Russ Allbery
The bootstrap is circular and has to be kicked off with a binary blob that
can't be recreated until one has finished the bootstrap, which is
certainly not ideal, but is also not that atypical for compiler
bootstrapping problems. I don't see how this is a violation of the DFSG or
of the requirement that main be self-contained. It's obnoxious from a
process standpoint (like a lot of compiler bootstrapping), and we may not
want things to work this way, but I don't think that makes it non-free?

what you say here makes perfect sense; i'll look at what gcc and
friends are doing, thanks!

[nick black]

Post by nick black

Post by Russ Allbery
The bootstrap is circular and has to be kicked off with a binary blob that
can't be recreated until one has finished the bootstrap, which is
certainly not ideal, but is also not that atypical for compiler
bootstrapping problems. I don't see how this is a violation of the DFSG or
of the requirement that main be self-contained. It's obnoxious from a
process standpoint (like a lot of compiler bootstrapping), and we may not
want things to work this way, but I don't think that makes it non-free?

what you say here makes perfect sense; i'll look at what gcc and
friends are doing, thanks!

i had thought that there was a Policy directive prohibiting
generated files in source packages, but rereading 4.7.0.2, i
don't see such a thing. we do see in debmake-doc [0]:

"Extraneous auto-generated contents in the upstream source.
Debian package should rebuild them under the latest system."

so yeah, this seems pretty reasonable, and i'm surprised i
didn't think of it. i do think that it would be best to rebuild
the binary wasm file after having bootstrapped up zig, though
that wouldn't be strictly necessary to bootstrap. that way we're
shipping something we built, and verify the toolchain can build
it.

thanks as always, russ!

[0] https://www.debian.org/doc/manuals/debmake-doc/ch07.en.html

[Stéphane Glondu]

Post by Russ Allbery
I think I'm missing something. Why does the use of this file knock the
package out of main? We are distributing the source code for this file,
and it is presumably under a free software license.

+1

OCaml uses a similar scheme.


Cheers,