Make build_lgpl.sh work with emscripten 1.31.0

[duanyao]

Fix zlib and ffmpeg build issue #29.

[bgrins]

As you mentioned in #29 (comment), it seems x264 isn't working yet. So we shouldn't merge these changes to master yet, until we get that sorted out, correct?

[bgrins]

I wonder if we update to the latest sources for x264 if it will be worked out

[duanyao]

It seems x264's configure doesn't detect compilers very well; I modified the file and x264 was compiled OK. Will try latest x264 later.

[bgrins]

I modified the file and x264 was compiled OK

Which file are you talking about? If it's the build script, can you send up the change to this PR and then I will try and see if I can get a successful build running these changes?

[duanyao]

I modified x264/configure like this:

@@ -761,11 +761,11 @@ define SYS_$SYS

 define STACK_ALIGNMENT $stack_alignment
 ASFLAGS="$ASFLAGS -DSTACK_ALIGNMENT=$stack_alignment"

 # skip endianness check for Intel Compiler, as all supported platforms are little. the -ipo flag will also cause the check to fail
-if [ $compiler = GNU ]; then
+if [ $compiler = GNU -a `basename "$CC"` != emcc ]; then
     echo "int i[2] = {0x42494745,0}; double f[2] = {0x1.0656e6469616ep+102,0};" > conftest.c
     $CC $CFLAGS conftest.c -c -o conftest.o 2>/dev/null || die "endian test failed"
     if (${cross_prefix}strings -a conftest.o | grep -q BIGE) && (${cross_prefix}strings -a conftest.o | grep -q FPendian) ; then
         define WORDS_BIGENDIAN
     elif !(${cross_prefix}strings -a conftest.o | grep -q EGIB && ${cross_prefix}strings -a conftest.o | grep -q naidnePF) ; then
@@ -1012,11 +1012,11 @@ if [ "$debug" = "yes" ]; then
 elif [ $ARCH = ARM ]; then
     # arm-gcc-4.2 produces incorrect output with -ffast-math
     # and it doesn't save any speed anyway on 4.4, so disable it
     CFLAGS="-O3 -fno-fast-math $CFLAGS"
 else
-    CFLAGS="-O3 -ffast-math $CFLAGS"
+    CFLAGS="-O2 -ffast-math $CFLAGS"
 fi

 if cc_check '' -fno-tree-vectorize ; then
     CFLAGS="$CFLAGS -fno-tree-vectorize"
 fi

Compiling seems OK, but x264 is actually unusable. The demo reported "missing function: x264_cqm_init" when encoding with x264.

[duanyao]

The error "missing function: x264_cqm_init" has been fixed by building x264 as shared library. Now every thing should be OK.

[duanyao]

@bgrins

Some updates:

1. Because of a bug of emscripten 1.31.0, ffmpeg actually needs 1.32.01.32.1 to compile properly (well, has another issue, see bellow).
2. Added a new build solution in build_edge directory, you can build latest ffmpeg and friends with latest emscripten here. In build_edge, firstly run pull_source.sh to clone latest ffmpeg, x264, openh264, vpx, lame, opus, and zlib, etc., and apply patches to make them compile properly, and then run build.sh to build. You can edit build.sh to customize ffmpeg.
3. Note that if you build full version of ffmpeg by selecting config_ffmpeg_full in build.sh, emcc 1.32.0 would generate invalid code. Currently config_ffmpeg_custom is OK. Update: This issue has been fixed, config_ffmpeg_full works fine now.
4. Lame uses SIMD intrinsics, which would be compiled into SIMD.js. This is a good thing, but it would invalidate asm.js in some browsers that don't support SIMD.js (FireFox 37), and make the code run much slower :(. You'll need FireFox nightly (support SIMD.js) to bring asm.js back.

[bgrins]

@duanyao this sounds great!! I'm taking a look now

[bgrins]

Because of a bug of emscripten 1.31.0, ffmpeg actually needs 1.32.1 to compile properly

I ran emsdk update emsdk install latest and emsdk activate latest and it's running 1.30.0 only. What are you doing to run 1.32.1?

[duanyao]

Sorry for the delay. I build emscripten 1.32.1 from its repo's incoming branch. See this instruction: http://kripken.github.io/emscripten-site/docs/building_from_source/

[Kagami]

You could also check ffmpeg.js. I added MP4 build with x264 and LAME encoders recently.

[bgrins]

Alright, so I got this build (mostly) working with emscripten 1.34.12 and it looks great! I'm not sure the workflow when I want to run ./pull_source again - I get errors like this:

Pulling https://github.com/rbrito/lame.git to lame
Already up-to-date.
lame.patch saved.
Pulling https://github.com/webmproject/libvpx.git to libvpx
Updating 670b2c0..7dd7a7d
error: Your local changes to the following files would be overwritten by merge:
  build/make/configure.sh
Please, commit your changes or stash them before you can merge.
Aborting
libvpx.patch saved.
Pulling git://git.videolan.org/x264.git to x264
Updating 121396c..7599210
error: Your local changes to the following files would be overwritten by merge:
  configure
Please, commit your changes or stash them before you can merge.
Aborting
x264.patch saved.
Pulling https://github.com/cisco/openh264.git to openh264
Updating 2d2c5d0..fb61733
error: Your local changes to the following files would be overwritten by merge:
  codec/common/src/WelsThreadLib.cpp

@duanyao, am I supposed to git checkout or stash the changes in each directory before running pull_source?

[duanyao]

@bgrins You may delete the cloned sources, revert *.patch files and re-run ./pull_source,sh. Maybe some patches are out of date, I'll check them later.

[bgrins]

OK, I had issues with running the build when libopus, libmp3lame, and openh264 were enabled, so I've commented them out for now and the build seems to work fine (this is at https://github.com/bgrins/videoconverter.js/tree/duanyao-master). Even in this state, it's just as good as the current build, and we have a set up that will make it easy to extend so I'd be happy to merge.

One thing that would be nice though is if there were a way to either check out a particular release from the repos or at least save the current rev in a file in this project so someone could reconstruct the exact build without having the same local checkout.

[bgrins]

I'm going to look into the .mem file that's generated and see if we can disable that since it seems to make it harder to include in a worker (seems you need to put both files in the same dir as the worker).

[duanyao]

I updated the patches so that build_edge should work with latest x264, libvpx, openh264, lame, opus and emscripten 1.34.12.

Maybe we can use git submodules to pin to specific versions.

[duanyao]

I also started a related project called codecbox.js:

• Build with latest ffmpeg, x264, libvpx, openh264, lame, and opus.
• Provide fine-grained APIs for video/audio processing (currently only decoding is supported).
• Has a demo media player on top of these APIs.

[nsi88]

Hi! Don't you know what's reason of error
missing function: ff_mqc_init_context_tables ?
It occurs when i run ffmpeg.js somehow, using node or browser.
I use emscripten 1.35.7, latest ffmpeg (0.1.0-5-gd15ffff) and libx264
I build full version of ffmpeg, but without libmp3lame, libopus and libopenh264
I don't see any errors during building.
Thanks

[duanyao]

@nsi88 I'm not sure. It seems ff_mqc_init_context_tables is used by jpeg2000 codec (
http://ffmpeg.org/doxygen/trunk/mqc_8c.html#a1eb65714543ab60140e7f7f2ec9a6dff ). Do you have problem with other codecs?

[nsi88]

If to disable jpeg2000 encoder/decoder all is good
Thank you!

[DickyT]

@duanyao Hi, I am trying to build this version only with full_ffmpeg and lame, and I find

WARNING:root:LLVM version appears incorrect (seeing "7.0", expected "3.8")
WARNING:root:could not check fastcomp: [Errno 2] No such file or directory
INFO:root:(Emscripten: Running sanity checks)
CRITICAL:root:Cannot find /usr/bin/llvm-link, check the paths in ~/.emscripten

I am using Mac OS X 10.11 and I could not find solution on this, can u help me to find whats wrong with this?

Also, the ./configure commands in build.sh always return non-zero result, I need to run ./configure commands myself to let build continue.

Thanks a lot!

[duanyao]

@DickyT It seems emscripten and fastcomp are not install properly. Can you build a simple C/C++ program with emscripten on your machine?

[DickyT]

@duanyao it seems doesn't work well. Thank you for your infomation, I'll try to re-configure emscripten!!!

[DickyT]

@duanyao I try to follow the official link to compile fastcomp and emscripten again, and I can use emscritpen to build my simple program, but the build.sh still print errors.

WARNING:root:LLVM version appears incorrect (seeing "3.9", expected "3.8")
ERROR:root:Emscripten, llvm and clang repo versions do not match, this is dangerous (1.35.9, 1.36.0, 1.36.0)
ERROR:root:Make sure to use the same branch in each repo, and to be up-to-date on each. See http://kripken.github.io/emscripten-site/docs/building_from_source/LLVM-Backend.html

Does it means that I need a older version of LLVM to build this videoconverter.js? or there still some configuration error?

[duanyao]

@DickyT : When building emscripten from source, there are 3 repos should be at a same version:

fastcomp: https://github.com/kripken/emscripten-fastcomp.git

fastcomp-clang: https://github.com/kripken/emscripten-fastcomp-clang.git

emscripten: https://github.com/kripken/emscripten.git

Can you check this?

[amilajack]

What is the status of this?

[ngivoni]

Guys,

I'm running into an issue with when I try to compile using Emscripten the x264 process. I am using the latest version 1.38.11 and I keep getting this error:

endian test failed
ERROR:root:Configure step failed with non-zero return code: 1. Command line: ./configure --prefix=/home/ec2-user/emsdk/emscripten/1.38.11/dist --disable-thread --disable-asm at /home/ec2-user/emsdk/emscripten/1.38.11/x264

Is there any thoughts/suggestions as to how to resolve it. I've tried getting get the older SDK mentioned in this thread, but it's no longer available in the EMScripten list of SDKs (./emsdk list --old). Any help would be appreciated.

[initHeart]

@ngivoni @duanyao I have the same problem, I'm running into an issue with when I try to compile using Emscripten the x264 process.
this is my emscripten 1.38.11
my error:
endian test failed
ERROR:root:Configure step failed with non-zero return code: 1. Command line: ./configure --prefix=/home/mini/Downloads/videoconverter.js-master/build/x264/../dist --disable-asm at /home/mini/Downloads/videoconverter.js-master/build/x264

Is there any thoughts/suggestions as to how to resolve it. Any help would be appreciated.

[ngivoni]

@duanyao

Right, I've ran into the same issue (I see you were unsuccessful getting it to work a few years back). I've tried to compile the videoconverter.js using an older version of Emscripten 1.34.10, but I didn't have much luck. I've reached out to Kagami, bgrins and the other contributors, but without much luck. I've also tried the build Kagami made but without any luck either.

Perhaps we can work through the issue together, as I'm eager to get this issue resolved. If you want to reach out off forum maybe we can get on Skype and come up with a game plan. My email is nadavgivoni@gmail.com

[vijaykumarchettiar]

@duanyao
i pulled your branch and compile it with options of --enable-protocols --enable-network in order to input the rtmp or rtsp url and output as webm or mp4 files.

however the compiled with error as below

'/home/ubuntu/emi/emsdk-portable/clang/e1.38.28_64bit/binaryen/bin/asm2wasm ../ffmpeg.temp.asm.js --total-memory=33554432 --trap-mode=allow -O3 --mem-init=../ffmpeg.js.mem --mem-base=1024 --wasm-only -o ../ffmpeg.wasm --mvp-features' failed (-9)

did you try these options?

the ffmpeg is 2014 build as it's fetched in the fetch_sources.sh

[nsi88]

hello
i pulled your branch and compile it with options of --enable-protocols --enable-network in order to input the rtmp or rtsp url and output as webm or mp4 files.

however the compiled with error as below

'/home/ubuntu/emi/emsdk-portable/clang/e1.38.28_64bit/binaryen/bin/asm2wasm ../ffmpeg.temp.asm.js --total-memory=33554432 --trap-mode=allow -O3 --mem-init=../ffmpeg.js.mem --mem-base=1024 --wasm-only -o ../ffmpeg.wasm --mvp-features' failed (-9)

did you try these options?

the ffmpeg is 2014 build as it's fetched in the fetch_sources.sh

No, I haven't tried. Sorry.

Does the compiler show any details about the error?

[vijaykumarchettiar]

hello
i pulled your branch and compile it with options of --enable-protocols --enable-network in order to input the rtmp or rtsp url and output as webm or mp4 files.
however the compiled with error as below
'/home/ubuntu/emi/emsdk-portable/clang/e1.38.28_64bit/binaryen/bin/asm2wasm ../ffmpeg.temp.asm.js --total-memory=33554432 --trap-mode=allow -O3 --mem-init=../ffmpeg.js.mem --mem-base=1024 --wasm-only -o ../ffmpeg.wasm --mvp-features' failed (-9)
did you try these options?
the ffmpeg is 2014 build as it's fetched in the fetch_sources.sh

No, I haven't tried. Sorry.

Does the compiler show any details about the error?
below is the error
/home/ubuntu/emi/emsdk-portable/clang/e1.38.28_64bit/binaryen/bin/asm2wasm ../ffmpeg.temp.asm.js --total-memory=33554432 --trap-mode=allow -O3 --mem-init=../ffmpeg.js.mem --mem-base=1024 --wasm-only -o ../ffmpeg.wasm --mvp-features' failed (-9)

The files ffmpeg.js and ffmpeg_mem.js both updated and times stambs changed. However when input the rtmp in the player, still shows protocol not supported...ffmpeg stdout still shows --disable-network and --disable-protocols

[duanyao]

@vijaykumarchettiar I don't think you can make network and protocols of ffmpeg work out-of-box with emscipten, see Kagami/ffmpeg.js#11.