BUG: 8.3.0+ builds break compatibility with Debian Buster and Ubuntu <= 18.04

[rathboma]

From 8.3.0 onward the build image is now Ubuntu 20.04 as GitHub Actions has removed Ubuntu 18.04 support.

Unfortunately this brings back an old set of issues -- any app using the prebuilds of better-sqlite3 will experience crashes when distributed through Snap, or when running on a system that contains a lower version of glibc (which is a LOT of systems).

This will specifically be a problem for those running prebuilds on servers with Ubuntu 16.04, 18.04, 14.04, or Debian Buster, all of which are still supported and used in many businesses. It is also a problem for Electron apps (mine included)

Context: Glibc 2.29 is a breaking change, but is only present on newer OSes.

SQLite3 is perfectly happy being built with older versions of glibc, so maybe there's a way to get those older versions on the build machines, or to build inside a docker container using a Ubuntu-18.04 or Debian buster image?

Related, & previously resolved by downgrading to Ubuntu 18.04 build images:

• glibc_2.29 not found beekeeper-studio/beekeeper-studio#1566
• Use of GLIBC 2.29 prevents pre-builds from being usable #586
• Installation error: version `GLIBC_2.29' not found #403

[mceachen]

Thanks for taking the time to do this research--apologies for the regression.

Can you submit a PR that reverts #982 ? @JoshuaWise has granted me a small subset of permissions for this repo--I can't self-approve any changes, for example.

[rathboma]

@mceachen I don't think it can be reverted. GitHub no longer supports the 18.04 build image

[mceachen]

SADNESS actions/runner-images#6002

[rathboma]

Yeah, that's why I wanted to start the discussion again, because I'm not sure this is going to be super easy to solve without some bigger build changes :-/

[mceachen]

I was hoping that glibc had some sort of autoconf/environment variable magicks for GLIBC_V2.21_MODE, but it seems that newer glibc removed older kernel support, so that's not going to happen.

All my googling seems to point to using a VM/docker instance to restore ubuntu 18 builds.

[rathboma]

Yeah same. Seems like it is either that or a CHROOT with a specific version of glibc installed.

I think 2.29 introduced a bunch of new stuff that isn't backwards compatible, because usually they're very good about backwards compatibility

[rathboma]

Or I guess installing a local version of an older glibc somewhere and using that one?

[mceachen]

I looked at this a bit more.

I would suggest we drop Ubuntu 18 support. EOL is at the end of this month. This will be @JoshuaWise 's call.

If we do want to try to kick the can down the road a couple months, we can push this out until september by adding this GHA job:

  prebuild-glibc:
    name: Prebuild on ancient glibc
    runs-on: ubuntu-latest
    container: node:16-stretch
    needs: publish
    steps:
      - uses: actions/checkout@v3
      - run: apt-get update && apt-get install build-essential git python3
      - run: npm install --ignore-scripts
      - run: ${{ NODE_BUILD_CMD }} -u ${{ secrets.GITHUB_TOKEN }}

The stretch build has 2.24:

$ docker run -it node:16-stretch sh
# head -1 /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
# ldd --version
ldd (Debian GLIBC 2.24-11+deb9u4) 2.24

[rathboma]

Great! I think pushing it out is a great idea.

Standard support for 18.04 ends soon, but ESM support runs for a few more years, many individuals get this for free, and most enterprises use it also. 14.04 is still supported with ESM until 2024!

[rathboma]

Never seen that container directive before. I'm going to try that myself

[mceachen]

I didn't test the new draft job yet, but see c99fca6 for what I was thinking.

[rathboma]

Looks great. It's very frustrating to have to micro manage builds like this right? I have the same frustrations, lol

[m4heshd]

@mceachen This is why I explicitly mentioned this HUGE breaking change in the PR note. (#982)

This change is necessary since GitHub has dropped support for the ubuntu-18.04 image. (actions/runner-images#6002)

This will also mean that Linux distributions with only glibc version 2.29 and above will be supported by the prebuilt binaries.

I've shopped around for many workarounds for this before the change. Unfortunately, we have to go along with all these deprecations and there's no way around it. I've even looked at using a custom runner image (I usually waste a ton of time looking for workarounds even when they're quite unnecessary). But that's practically useless since Node 18 itself requires glibc >= 2.28 and Node 16 is soon EOL. That timeline unfortunately didn't line up with GH's support drop for the Ubuntu image. Polluting the build workflow for mere short-term backward compatibility just felt wrong. But that's just my way of doing things. I would love some input from @JoshuaWise on this.

@rathboma If this is a necessity for your application, I think a solution on the application's scope would fit much better since you have full control of the building, packaging, and distribution of your app. This is just my own opinion. I never use prebuilt binaries on my distributions. I always rebuild them while packaging. You can use a self-hosted runner while doing that to please your requirements. Please correct me if I'm missing anything here because I have no experience in distributing in packmans like Snap.

[lyswhut]

@mceachen Hi, i tried this configuration:

https://github.com/mceachen/better-sqlite3/blob/c99fca6b901201505537da1628c8b8e6f688faf7/.github/workflows/build.yml#L86-L96

But it failed to build because the highest Python version of Debian 9's official package manager source is v3.5.x, and node-gyp v8 needs Python 3.6 or higher to work. Here are some reference links

• https://unix.stackexchange.com/a/743865
• [BUG] Can not build binary packages on Debian Stretch npm/run-script#40

[rathboma]

@m4heshd Yeah I'm going to have to build the package myself in my runner. I've had a hard time in the past telling my build tools to NOT use prepacked binaries, I don't think the tooling does this very well.

For now I'm just going to lock to a specific version. I don't work on my app full time, so this type of stuff is a net negative on productivity :-(.

Is there a way to mark prebuilt binaries as not working for certain operating systems?

All the better-sqlite3 contributors - thanks for doing such an amazing job. I realize this stuff isn't your fault. :-)

[JounQin]

Can we republish v8.2.0 as v8.3.1 and then release current v8.3.0 as v9.0.0 for its breaking change?

[pixdrift]

I appreciate that this thread is Ubuntu/Debian focused, but I wanted to include a note that this change also breaks install for EL8 variants as they ship with GLIBC 2.28.

This includes Red Hat Enterprise Linux, CentOS Stream, Oracle Linux, Alma, Rocky etc.

Red Hat Enterprise Linux 8 has full support from Red Hat until May 2024 and at that date it goes into extended support (but is still supported). Not that vendor support governs decisions of support of these builds, but wanted to mention it.
https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Product_life_cycle

I have been able to replicate the install issue in RHEL 8.x and that is what brought me to this ticket.

Something that may also be worth mentioning is the assumptions made in the fallback behaviour when an incompatible GLIBC is detected. This led to a cascade of issues as it attempted to build as it expects python (mentioned above), make, gcc, and open Internet access to pull sources.

[flaviut]

I also want to note that the AWS Lambda node 18.x runtime runs on Amazon Linux 2, which only has glibc 2.26.

This is outside of node18's support (glibc 2.28 is required), and I definitely don't expect this project to take any action based on that, but I wanted to bring it up as an example of another distro with old glibc versions.

[anonghuser]

Getting this error on Oracle Linux 8 which has glibc 2.28

Aside of waiting for you to fix the prebuilt binaries, can you describe how people should configure it to be built locally with their own version of glibc? The preinstall script from docs/compilation.md seems incorrect for people that prefer pnpm or other package managers.

[EDIT: Apologies, I only saw the first error in the error log and did not notice a further error about missing g++. I see now that if the prebuild binaries are not usable, you attempt a local build and it was just failing. Installing gcc-c++ worked to fix it.]

[anonghuser]

It seems to me even though github may have dropped support for old ubuntu versions as the base of its github actions runners, it still supports basically any docker image as an additional service:
https://docs.github.com/en/actions/using-containerized-services/about-service-containers
https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container
So you should still be able to build on older ubuntu docker images.

And just for completeness, though I'm sure your google-fu is no worse than mine and you've probably already seen this, there are several other alternatives described here:
https://stackoverflow.com/questions/2856438/how-can-i-link-to-a-specific-glibc-version
The more interesting-sounding ones are actually not the top answers but further down, i.e. using zig c++ or glibc_version_header. The most practical solution still seems building in docker though.