fix(NODE-5143): linking with mongocrypt in node bindings

[YakoYakoYokuYoku]

This PR fixes an issue when installing mongodb-client-encryption with npm install --build-from-source.

make: Entering directory '/home/yakoyoku/Codes/JavaScript/Clones/mongosh/node_modules/mongodb-client-encryption/build'
  CXX(target) Release/obj.target/mongocrypt/src/mongocrypt.o
  SOLINK_MODULE(target) Release/obj.target/mongocrypt.node
/usr/bin/ld: cannot find /home/yakoyoku/Codes/JavaScript/Clones/mongosh/node_modules/mongodb-client-encryption/deps/lib/libmongocrypt-static.a: No such file or directory
/usr/bin/ld: cannot find /home/yakoyoku/Codes/JavaScript/Clones/mongosh/node_modules/mongodb-client-encryption/deps/lib/libkms_message-static.a: No such file or directory
/usr/bin/ld: cannot find /home/yakoyoku/Codes/JavaScript/Clones/mongosh/node_modules/mongodb-client-encryption/deps/lib/libbson-static-for-libmongocrypt.a: No such file or directory
collect2: error: ld returned 1 exit status
make: *** [mongocrypt.target.mk:142: Release/obj.target/mongocrypt.node] Error 1
make: Leaving directory '/home/yakoyoku/Codes/JavaScript/Clones/mongosh/node_modules/mongodb-client-encryption/build'

Now with this PR when the shared_libmongocrypt npm config is set to true it will link against a system supplied libmongocrypt, otherwise it uses the static libraries from the release build.

[nbbeeken]

Thank you @YakoYakoYokuYoku for helping us here! The team is going to investigate and reply back with more info after the investigation is completed.

We are tracking the issue: https://jira.mongodb.org/browse/NODE-5143

[durran]

Hi @YakoYakoYokuYoku - is there a reason that you are unable to use the static libraries in the release build? One reason we currently don't allow dynamic linking to the system installed libmongocrypt is that due to our rapid development we have a lot of API changes happening which most common libraries do not have. This creates a high probability risk that the system installed libmongocrypt is not compatible with our Node bindings and will error. By not allowing it we can guarantee that it will always work.

[YakoYakoYokuYoku]

Hi @YakoYakoYokuYoku - is there a reason that you are unable to use the static libraries in the release build?

This happened while I trying to install the dependencies of mongosh with npm install --build-from-source, with each time seeing that mongodb-client-encryption failed to do so due to the above compilation errors. I was expecting that it'd link with my system install of libmongocrypt but I saw that the bindings.gyp wasn't doing it.

libmongocrypt/bindings/node/binding.gyp

Lines 40 to 47 in 677b5d8

	['build_type=="dynamic"', {
	'link_settings': {
	'libraries': [
	'-lmongocrypt'
	]
	}
	}],
	['build_type!="dynamic"', {

Then I've noticed that it uses the build_type variable and variables in node-gyp can be defined with npm configs, so I've tried to run npm install --build-type="release" or npm_config_build_type="release" npm install, but sadly with no avail (though in both cases build_type was set to "release" inside the config.gyp). I think that build_type means a Debug/Release build or something else. But because of it I had no other choice but to use another variable, in this case shared_libmongocrypt and with that the build worked out.

So, when you want to build from source mongodb-client-encryption with this PR, and if you want to link with the system install of libmongocrypt then you should run npm install --build-from-source --shared-libmongocrypt (which is the default), otherwise, if you want to use the static version of the libraries then npm install --build-from-source --no-shared-libmongocrypt is your choice.

One reason we currently don't allow dynamic linking to the system installed libmongocrypt is that due to our rapid development we have a lot of API changes happening which most common libraries do not have. This creates a high probability risk that the system installed libmongocrypt is not compatible with our Node bindings and will error. By not allowing it we can guarantee that it will always work.

This PR makes sure that the package prebuilt in the Evergreen CI uses shared_libmongocrypt set to false using the --no-shared-libmongocrypt flag. But I can set the behaviour to use the static libs by default if desired.

[durran]

Thanks @YakoYakoYokuYoku for the information. We've discussed this and think what we are ideally looking for is not to dynamically link to the system libmongocrypt, but we would like --build-from-source without any other options to work, so that the correct libmongocrypt is built. Is that something you could modify this PR to do instead?

[YakoYakoYokuYoku]

If I say so myself, I have an idea. If cmake-js, which is supported by prebuild, is used instead of node-gyp then bindings/node can be converted into a CMake project and build libmongocrypt as a subproject or the package.json can be moved at the root of the project and then have bindings/node as a subdirectory, while using build.js with npm configs to set libraries paths for example. How does it sounds?

[durran]

If I say so myself, I have an idea. If cmake-js, which is supported by prebuild, is used instead of node-gyp then bindings/node can be converted into a CMake project and build libmongocrypt as a subproject or the package.json can be moved at the root of the project and then have bindings/node as a subdirectory, while using build.js with npm configs to set libraries paths for example. How does it sounds?

@YakoYakoYokuYoku we were actually talking about this as well, as using cmake-js would probably simplify this instead of using node-gyp. So that sounds like a great idea. :)

[YakoYakoYokuYoku]

• Changed the build system of the bindings to CMake so many options were added to customize the build to make it suitable for an npm package.
• Fixed some issues with the compilation and linkage of the bindings so that it only produces a nodejs native module when building the bindings.
• The library sources are vendored for npm pack.
• Also I've made this PR a series of commits so that it can be bisectable, and because of that DO NOT SQUASH THIS, please.

[durran]

@YakoYakoYokuYoku Starting to have a look at this. Are there specific CMake features that are being used here that require this version? Our Evergreen build images currently have Cmake 2.8.12.2 installed so they currently have errors on the version check.

[YakoYakoYokuYoku]

Root CMakeLists.txt uses this version, so that's the reason why I've used this one.

libmongocrypt/CMakeLists.txt

Line 1 in 2f48f35

cmake_minimum_required (VERSION 3.12)

Also isn't 2.8.12.2 a bit too old?

[durran]

Yeah it's old, but that's what our some of our CI images have. I'll need to open an internal ticket to get those updated - thanks so far for the work.

[YakoYakoYokuYoku]

I've now removed the option to use system libmongocrypt due to the problems described in JIRA, you can still use system libbson by running npm i --build-from-source --mongocrypt-deps=shared or npm i --build-from-source --mongocrypt-deps=static.