Troubleshooting experimental build - how to replicate the sbuild?

[Steven Robbins]

Hi,

I recently uploaded a package to experimental and was suprised at the build
failures [1]. I have built it locally and on a porter box without issue --
each time using a clean unstable chroot.

[1] https://buildd.debian.org/status/fetch.php?
pkg=insighttoolkit5&arch=amd64&ver=5.4.3-2&stamp=1745014062&raw=0

I am trying to run sbuild in an attempt to reproduce the precise environment.
I am using this invocation:

sbuild --purge=successful --dist=experimental --build-dep-resolver=aspcud
insighttoolkit5_5.4.3-2.dsc

However, this also succeeds locally. So clearly I'm not reproducing exactly
the buildd environment. I have asked both google and chatgpt but haven't yet
been able to uncover exactly what options are used on the buildd. Anyone know
how to find this?

Comparing the logs, I can see a couple differences:

Buildd uses sbuild 0.88.4~bpo12+2 whereas I have 0.89.0.
Buildd sets Build Type: any, whereas mine is binary.

Beyond that, there's a bunch of packages installed that I haven't waded
through yet, but it's tough to believe package versions matter. Except
perhaps for fftw, which contains the header where the error occurs.
All fftw packages are the same version in both buildd and local.

Any help appreciated,
-Steve

[NoisyCoil]

Hi Steven,

Looks like the distribution is being picked up correctly, but you may be
missing the aspcud criteria used by the buildd's, see the
`--aspcud-criteria` option in [1].

Cheers!

[1] https://wiki.debian.org/sbuild#Enabling_experimental

[Otto Kekäläinen]

Hi,

I am not able to reproduce the test failure visible at
https://buildd.debian.org/status/fetch.php?pkg=insighttoolkit5&arch=amd64&ver=5.4.3-2&stamp=1745014062&raw=0
in a local build myself, so I can't help with that. I did however
notice that the size of the resulting packages were quite small 2-6 MB
compared to the massive 425 MB source package:

1.7M insighttoolkit5-examples_5.4.3-2_all.deb
24K insighttoolkit5_5.4.3-2.debian.tar.xz
4.0K insighttoolkit5_5.4.3-2.dsc
40M insighttoolkit5_5.4.3-2_amd64.build
28K insighttoolkit5_5.4.3-2_amd64.buildinfo
4.0K insighttoolkit5_5.4.3-2_amd64.changes
425M insighttoolkit5_5.4.3.orig.tar.gz
3.2M libinsighttoolkit5-dev_5.4.3-2_amd64.deb
6.2M libinsighttoolkit5.4_5.4.3-2_amd64.deb

In unpacked sources 531 MB is binary blob directory .ExternalData/.
This massive source package size is currently making the Salsa CI at
https://salsa.debian.org/med-team/insighttoolkit fail, and preventing
you from potentially seeing the build failure in Salsa CI before the
upload to experimental. It would ideal if you can test the builds with
Salsa CI before upload instead of having to rely on the actual Debian
archive for testing purposes like you have now with the last couple of
uploads in recent days.

Have you considered putting .ExternalData/ in debian/copyright
Exclude-Files so that the source package in Debian would be without
those binary blobs? That might allow you to run Salsa CI without
hitting artifact size limits.

[Steven Robbins]

On Sunday, April 20, 2025 12:18:28 p.m. Central Daylight Saving Time Otto

Post by Otto Kekäläinen
Hi,
I am not able to reproduce the test failure visible at
https://buildd.debian.org/status/fetch.php?pkg=insighttoolkit5&arch=amd64&ve
r=5.4.3-2&stamp=1745014062&raw=0 in a local build myself, so I can't help
with that.

I am able to reproduce it thanks to NoisyCoil's hint about --aspcud-criteria.
I'm working on a fix now.

Post by Otto Kekäläinen
I did however [...] notice [...] 531 MB is binary blob
directory .ExternalData/. [...]
Have you considered putting .ExternalData/ in debian/copyright
Exclude-Files so that the source package in Debian would be without
those binary blobs? That might allow you to run Salsa CI without
hitting artifact size limits.

This is testing data. Some time ago we deliberately decided to *include* this
data in order to run the test suite after build. It's too bad that it exceeds
the CI limits, but testing is important.

-Steve

[Steven Robbins]

On Sunday, April 20, 2025 7:50:07 p.m. Central Daylight Saving Time Steven

Post by Steven Robbins
I am able to reproduce it thanks to NoisyCoil's hint about
--aspcud-criteria. I'm working on a fix now.

Okay, I don't know how to solve this. The package is uploaded now for
unstable so the --aspcud thing no longer matters.

Using sbuild --dist=sid --chroot-mode=schroot insighttoolkit5_5.4.3-3.dsc, the
package will build seemingly fine.

Using sbuild --dist=sid --chroot-mode=unshare insighttoolkit5_5.4.3-3.dsc, the
errors seen in the buildd logs appear ("error: ‘FFTW_ESTIMATE’ was not
declared in this scope")

Best I can figure is that the sbuild "unshare" mode does something tricky that
foils something somewhere. Maybe an include doesn't show up at the right
path?

I also can't figure out how to examine the chroot after a failure: using "--
purge=successful" doesn't work with unshare.

Thoughts?
-Steve

[Andrey Rakhmatullin]

Post by Steven Robbins
On Sunday, April 20, 2025 7:50:07 p.m. Central Daylight Saving Time Steven

Post by Steven Robbins
I am able to reproduce it thanks to NoisyCoil's hint about
--aspcud-criteria. I'm working on a fix now.

Okay, I don't know how to solve this. The package is uploaded now for
unstable so the --aspcud thing no longer matters.
Using sbuild --dist=sid --chroot-mode=schroot insighttoolkit5_5.4.3-3.dsc, the
package will build seemingly fine.
Using sbuild --dist=sid --chroot-mode=unshare insighttoolkit5_5.4.3-3.dsc, the
errors seen in the buildd logs appear ("error: ‘FFTW_ESTIMATE’ was not
declared in this scope")
Best I can figure is that the sbuild "unshare" mode does something tricky that
foils something somewhere. Maybe an include doesn't show up at the right
path?

unshare doesn't have network access, though I can't easily imagine a CMake
build script downloading C headers or something like that.
It should also be relatively easy to compare build logs.

[Johannes Schauer Marin Rodrigues]

Hi Steven,

Post by Steven Robbins

Post by Steven Robbins
I am able to reproduce it thanks to NoisyCoil's hint about
--aspcud-criteria. I'm working on a fix now.

Okay, I don't know how to solve this. The package is uploaded now for
unstable so the --aspcud thing no longer matters.
Using sbuild --dist=sid --chroot-mode=schroot
insighttoolkit5_5.4.3-3.dsc, the
package will build seemingly fine.
Using sbuild --dist=sid --chroot-mode=unshare
insighttoolkit5_5.4.3-3.dsc, the
errors seen in the buildd logs appear ("error: ‘FFTW_ESTIMATE’ was not
declared in this scope")
Best I can figure is that the sbuild "unshare" mode does something tricky that
foils something somewhere. Maybe an include doesn't show up at the right
path?

Santiago Vila did an archive rebuild with the new sbuild version (thank
you!!) before we uploaded it to unstable. It's unlikely we introduced
new FTBFS bugs with the 0.89.0 upload of sbuild.

I'm interested in getting to the bottom of this, especially if you say
that the only difference is in the chroot backend of sbuild. If you
would like to talk about this more, please open a bug against sbuild
that includes all the information that I need to reproduce your issue.

It would be great if you would be able to come up with a more minimal
reproducer than insighttoolkit5. Not only does the package take ages to
build, my laptop is an arm64 machine and insighttoolkit5 is limited to
amd64 and I cannot run sbuild on the porterboxes. I can run sbuild on
amd64 on salsaci but as Otto has pointed out in an earlier message, your
package is too large for salsaci. If you could create a more minimal
(and maybe even architecture independent?) reproducer that would be very
helpful and it might even already give you an idea where the problem is.

Post by Steven Robbins
I also can't figure out how to examine the chroot after a failure: using "--
purge=successful" doesn't work with unshare.

I usually use --anything-failed-commands=%s which gives you a shell that
you can use to investigate the build independent of the chroot backend.

Thanks!

cheers, josch

[Steven Robbins]

On Monday, April 21, 2025 2:46:27 a.m. Central Daylight Saving Time Johannes

Post by Johannes Schauer Marin Rodrigues

Post by Steven Robbins
I also can't figure out how to examine the chroot after a failure: using "--
purge=successful" doesn't work with unshare.

I usually use --anything-failed-commands=%s which gives you a shell that
you can use to investigate the build independent of the chroot backend.

That was the hint I needed, many thanks.

I was able to confirm that the cmake configuration was identical when using the
unshare backend. So I started looking at the sources. I have managed to
solve this by adding an #include in one of the files. To me, this looks like a
true bug in ITK so I am completely baffled by why it works in a regular build --
or with the schroot backend of sbuild.

I also confirmed that the previously uploaded version of ITK does build with
both schroot and unshare backends of sbuild. I looked at the diffs and none of
the changes looked suspicious -- indeed the code that was failing wasn't even
changed.

So I'll take the win of a successful build, but I have no idea what the root
cause was.

As ever - very grateful for the suggestions received here!

Regards,
-Steve

[Peter Pentchev]

Post by Steven Robbins
On Sunday, April 20, 2025 7:50:07 p.m. Central Daylight Saving Time Steven

Post by Steven Robbins
I am able to reproduce it thanks to NoisyCoil's hint about
--aspcud-criteria. I'm working on a fix now.

Okay, I don't know how to solve this. The package is uploaded now for
unstable so the --aspcud thing no longer matters.
Using sbuild --dist=sid --chroot-mode=schroot insighttoolkit5_5.4.3-3.dsc, the
package will build seemingly fine.
Using sbuild --dist=sid --chroot-mode=unshare insighttoolkit5_5.4.3-3.dsc, the
errors seen in the buildd logs appear ("error: ‘FFTW_ESTIMATE’ was not
declared in this scope")
Best I can figure is that the sbuild "unshare" mode does something tricky that
foils something somewhere. Maybe an include doesn't show up at the right
path?

That may very well be a red herring, but there was some trouble with
some of the differences between an schroot environment and an unshared
process/network/whatever namespace previously (tracked down and quickly
fixed by Johannes and others, thanks!); the part that I remember was
related to supplementary groups and their handling in different pid
namespaces. However, schroot has a fix for that now, so that (specifically
supplementary groups) should not be the problem in your case.

Have you compared the build logs, especially the configure-like step
where CMake examines the environment and looks for various syscalls,
functions, etc, to see if there were differences between the schroot
and unshare backend builds?

G'luck,
Peter

[Santiago Vila]

Hi.

Given that the buildds are able to build the Arch:all packages
but not the arch:amd64 packages, the way to reproduce this
would be to do this in a regular directory chroot:

dpkg-buildpackage -B

In particular, I see that debian/rules does different things
depending on the type of build:

override_dh_auto_configure-indep:

override_dh_auto_configure-arch: pre-build
dh_auto_configure -- $(CMAKE_FLAGS)

That sort of thing makes possible that a full build works
while an arch-specific build (only "all", or only "amd64") does not.

Hope this helps.

Thanks.

[Santiago Vila]

Post by Santiago Vila
dpkg-buildpackage -B

Note: I tried that (once) and unfortunately it built ok for me,
so maybe it's a makefile bug. You might want to try
GNUMAKEFLAGS=--shuffle, which sometimes amplifies the
probability that an existing makefile bug manifests itself.

If it's a makefile bug, dh --no-parallel might help.

Thanks.