Merge pull request #1808 from peterdmv/howto/xcomp-raspberry-pi3

Add how-to for cross compilation to Raspberry Pi 3

2 files changed, 599 insertions, 0 deletions

diff --git a/HOWTO/INSTALL-RASPBERRYPI3.md b/HOWTO/INSTALL-RASPBERRYPI3.md
new file mode 100644
index 0000000000..536d095cb4
--- /dev/null
+++ b/HOWTO/INSTALL-RASPBERRYPI3.md
@@ -0,0 +1,323 @@
+# Cross Compiling Erlang/OTP - Raspberry Pi 3
+
+
+## Introduction
+
+This document describes how to build a toolchain and cross compile Erlang/OTP
+to Raspberry Pi 3 on macOS High Sierra. It is recommended to consult
+[Building and Installing Erlang/OTP](https://github.com/erlang/otp/blob/master/HOWTO/INSTALL.md) and [Cross Compiling Erlang/OTP](https://github.com/erlang/otp/blob/master/HOWTO/INSTALL-CROSS.md) before attempting to follow the instructions in this guide.
+
+The whole process takes several hours and depending on the package versions different problems may arise that require additional
+fixes not described in this document. In other words, it is not fun to build a toolchain. I assume that you have a Mac and would
+like to develop Erlang/OTP applications based on the latest OTP release (or master) that not yet released as a binary for
+Raspberry Pi 3.
+
+The first and most time consuming step is building the toolchain from scratch. Once your cross compiler is ready you cross compile
+all library dependencies and create the sysroot file system. In the last step you cross compile Erlang/OTP using the new
+toolchain and sysroot.
+
+#### Tested Configuration
+
+macOS High Sierra 10.13.2<br>
+Raspberry Pi Model B Rev 1.2<br>
+Crosstools-NG 1.23.0_1
+
+> Note: /proc/device/tree/model contains model information of your
+> Raspberry Pi.
+
+
+#### Install Crosstool NG
+
+  (1)
+
+    $ brew install crosstool-ng
+    $ brew install grep --default-names # needed by crosstools-ng scripts
+    $ brew install md5sha1sum           # needed by crosstools-ng populate script
+
+
+#### Create case-sensitive disk images
+
+  (2)
+
+Create two case-sensitive disk images using Disk Utility:
+
+`File -> New Image -> Blank Image...`
+
+Format: `Mac OS Extended (Case-sensitive, Journaled)`
+
+```
+/Volumes/xtools-build-env  15 GB
+/Volumes/xtools           500 MB
+```
+
+The first image holds all source and object files while building the toolchain. The second image houses the compiled
+toolchain.
+
+## Building the Toolchain
+
+
+#### Configure crosstool-ng
+
+  (4)
+
+    $ ct-ng armv8-rpi3-linux-gnueabihf
+    $ ct-ng menuconfig
+
+#### Modify *path* section
+
+* Local tarballs directory: `/Volumes/xtools-build-env/src`
+* Working directory: `/Volumes/xtools-build-env/.build`
+* Prefix directory: `/Volumes/xtools/${CT_TARGET}`
+
+#### Modify *Extracting* section
+
+* Check option: _Stop after extracting tarballs_.
+
+> Note: The build shall stop after the tarballs have been extracted to give us time to fix source code problems.
+
+#### Enable STOP / RESTART
+
+Edit  /Volumes/xtools-build-env/.config
+  `CT_DEBUG_CT_SAVE_STEPS=y`
+
+Should the build break at a particular build step, you can fix the problem and continue the build from where it broke.
+
+Short summary of the most common `ct-ng` commands:
+
+* Listing all build steps
+
+```
+    $ ct-ng list-steps
+
+    Available build steps, in order:
+    - companion_tools_for_build
+    - companion_libs_for_build
+    - binutils_for_build
+    - companion_tools_for_host
+    - companion_libs_for_host
+    - binutils_for_host
+    - cc_core_pass_1
+    - kernel_headers
+    - libc_start_files
+    - cc_core_pass_2
+    - libc
+    - cc_for_build
+    - cc_for_host
+    - libc_post_cc
+    - companion_libs_for_target
+    - binutils_for_target
+    - debug
+    - test_suite
+    - finish
+```
+
+* Re-run step
+```
+    $ ct-ng step
+```
+
+* Restart from step
+```
+    $ ct-ng step+
+```
+
+* Run until step
+```
+    $ ct-ng +step
+```
+
+#### Fix file permissions on crosstool-NG.sh
+
+  (5)
+
+    $ chmod 744 /usr/local/Cellar/crosstool-ng/1.23.0_1/lib/crosstool-ng-1.23.0/scripts/crosstool-NG.sh
+
+#### Run build command
+
+Build process stops just after the tarballs have been extracted.
+
+  (6)
+
+    $ ct-ng build
+
+    Retrieving needed toolchain components' tarballs
+    [EXTRA]    Retrieving 'make-4.2.1'
+    [EXTRA]    Retrieving 'm4-1.4.18'
+    [EXTRA]    Retrieving 'linux-4.10.8'
+    [EXTRA]    Retrieving 'gmp-6.1.2'
+    [EXTRA]    Retrieving 'mpfr-3.1.5'
+    [EXTRA]    Retrieving 'isl-0.16.1'
+    [EXTRA]    Retrieving 'mpc-1.0.3'
+    [EXTRA]    Retrieving 'expat-2.2.0'
+    [EXTRA]    Retrieving 'ncurses-6.0'
+    [EXTRA]    Retrieving 'libiconv-1.15'
+    [EXTRA]    Retrieving 'gettext-0.19.8.1'
+    [EXTRA]    Retrieving 'binutils-2.28'
+    [EXTRA]    Retrieving 'gcc-6.3.0'
+    [EXTRA]    Retrieving 'glibc-2.25'
+    [EXTRA]    Retrieving 'gdb-7.12.1'
+
+#### Fix source files
+
+  (7)
+
+Add macro to /Volumes/xtools-build-env/.build/src/gdb-7.12.1/gdb/doublest.c:
+```C
+#define min(a,b) \
+  ({ typeof (a) _a = (a); \
+      typeof (b) _b = (b); \
+    _a < _b ? _a : _b; })
+```
+
+  (8) Update ulimit
+
+    $ ulimit -n 1024
+
+#### Modify *extract* section
+
+  (8)
+
+    $ ct-ng menuconfig
+
+ Uncheck option: _Stop after extracting tarballs_
+
+#### Re-run build command
+
+Restarts build process from where it previously stopped.
+
+  (9)
+
+    $ ct-ng build
+
+#### Fix gettext
+
+Build will fail at step `companion_tools_for_build` but it can be fixed by running autoreconf:
+
+  (10)
+
+    $ cd .build/src/gettext-0.19.8.1/
+    $ ./autoreconf
+    $ ct-ng companion_tools_for_build+
+
+#### Test the toolchain
+
+  (11)
+
+    $ cat > test.c
+    $ int main() { printf("Hello, world!\n"); return 0; }
+    $ /Volumes/xtools/arm-unknown-linux-gnueabi-gcc -o test test.c
+
+  (12) OPTIONAL
+
+    “Render the toolchain read-only” from crosstool-NG’s “Paths and misc options” configuration page.
+
+
+## Cross compiling dependencies
+
+  (13)
+
+    $ export PATH=/Volumes/xtools/armv8-rpi3-linux-gnueabihf/bin:$PATH
+
+#### Cross compiling zlib
+
+  (14)
+
+    $ wget http://zlib.net/zlib-1.2.11.tar.gz
+    $ tar xf zlib-1.2.11.tar.gz
+    $ cd zlib-1.2.11
+    $ CHOST=armv8-rpi3-linux-gnueabihf ./configure --prefix=/Users/<username>/git/raspberrypi/arm
+    $ make
+    $ make install
+
+#### Cross compiling openssl
+
+  (15)
+
+    $ wget http://openssl.org/source/openssl-1.1.0g.tar.gz
+    $ tar xf openssl-1.1.0g.tar.gz
+    $ cd openssl-1.1.0g
+    $ ./Configure linux-generic32 --prefix=/Users/<username>/git/raspberrypi/arm --openssldir=/Users/<username>/git/raspberrypi/arm/openssl --cross-compile-prefix=armv8-rpi3-linux-gnueabihf
+    $ make
+    $ make install
+
+#### Cross compiling ncurses
+
+  (16)
+
+    $ wget http://ftp.gnu.org/pub/gnu/ncurses/ncurses-5.9.tar.gz
+
+
+  (17)
+
+Apply patch:
+
+```patch
+--- a/ncurses/base/MKlib_gen.sh
++++ b/ncurses/base/MKlib_gen.sh
+@@ -474,11 +474,22 @@ sed -n -f $ED1 \
+    -e 's/gen_$//' \
+    -e 's/  / /g' >>$TMP
+
++cat >$ED1 <<EOF
++s/  / /g
++s/^ //
++s/ $//
++s/P_NCURSES_BOOL/NCURSES_BOOL/g
++EOF
++
++# A patch discussed here:
++#  https://gcc.gnu.org/ml/gcc-patches/2014-06/msg02185.html
++# introduces spurious #line markers.  Work around that by ignoring the system's
++# attempt to define "bool" and using our own symbol here.
++sed -e 's/bool/P_NCURSES_BOOL/g' $TMP > $ED2
++cat $ED2 >$TMP
++
+ $preprocessor $TMP 2>/dev/null \
+-| sed \
+-   -e 's/  / /g' \
+-   -e 's/^ //' \
+-   -e 's/_Bool/NCURSES_BOOL/g' \
++| sed -f $ED1 \
+ | $AWK -f $AW2 \
+ | sed -f $ED3 \
+ | sed \
+```
+
+  (18)
+
+    $ ./configure --build=x86_64-apple-darwin17.3.0 --host=armv8-rpi3-linux-gnueabihf --without-ada --without-cxx --without-cxx-binding --without-manpages --without-progs --without-tests --prefix=/usr --libdir=/lib --with-build-cc="gcc -D_GNU_SOURCE" --with-shared
+    $ make
+    $ make DESTDIR=/Users/<username>/git/raspberrypi/arm install
+
+  (19)
+
+Compile ncurses test program:
+
+    $ cd test
+    $ armv8-rpi3-linux-gnueabihf-gcc -o nctest ncurses.c -I${RPI_SYSROOT}/usr/include -L${RPI_SYSROOT}/lib -lncursesw
+
+
+## Populating sysroot
+
+  (19)
+
+    Edit /Volumes/xtools/armv8-rpi3-linux-gnueabihf/bin/armv8-rpi3-linux-gnueabihf-populate:
+
+    sed="gsed"
+
+  (20)
+
+    $ armv8-rpi3-linux-gnueabihf-populate -s /Users/<username>/git/raspberrypi/arm -d /Users/<username>/git/raspberrypi/sysroot
+    $ export RPI_SYSROOT=/Users/<username>/git/raspberrypi/sysroot
+
+
+## Cross compiling Erlang/OTP
+
+  (21)
+
+    $ LC_CTYPE=C && LANG=C && ./otp_build autoconf
+    $ ./otp_build configure --disable-dynamic-ssl-lib --xcomp-conf=./xcomp/erl-xcomp-armv8-rpi3-linux-gnueabihf.conf
+    $ ./otp_build boot -a
+    $ ./otp_build release -a /Users/<username>/git/raspberrypi/erlang
+    $ tar czf erlang.tgz ./erlang
+
diff --git a/xcomp/erl-xcomp-armv8-rpi3-linux-gnueabihf.conf b/xcomp/erl-xcomp-armv8-rpi3-linux-gnueabihf.conf
new file mode 100644
index 0000000000..dffc4a4b56
--- /dev/null
+++ b/xcomp/erl-xcomp-armv8-rpi3-linux-gnueabihf.conf
@@ -0,0 +1,276 @@
+## -*-shell-script-*-
+##
+## %CopyrightBegin%
+##
+## Copyright Ericsson AB 2009-2018. All Rights Reserved.
+##
+## Licensed under the Apache License, Version 2.0 (the "License");
+## you may not use this file except in compliance with the License.
+## You may obtain a copy of the License at
+##
+##     http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+##
+## %CopyrightEnd%
+##
+## File: erl-xcomp-armv8-rpi3-linux-gnueabihf.conf
+## Author: Péter Dimitrov
+## Tested: macOS High Sierra 10.13.2 /
+##         Raspberry Pi 3 Model B Rev 1.2
+##
+## -----------------------------------------------------------------------------
+## When cross compiling Erlang/OTP using `otp_build', copy this file and set
+## the variables needed below. Then pass the path to the copy of this file as
+## an argument to `otp_build' in the configure stage:
+##   `otp_build configure --xcomp-conf=<FILE>'
+## -----------------------------------------------------------------------------
+
+## Note that you cannot define arbitrary variables in a cross compilation
+## configuration file. Only the ones listed below will be guaranteed to be
+## visible throughout the whole execution of all `configure' scripts. Other
+## variables needs to be defined as arguments to `configure' or exported in
+## the environment.
+
+## -- Variables for `otp_build' Only -------------------------------------------
+
+## Variables in this section are only used, when configuring Erlang/OTP for
+## cross compilation using `$ERL_TOP/otp_build configure'.
+
+## *NOTE*! These variables currently have *no* effect if you configure using
+## the `configure' script directly.
+
+# * `erl_xcomp_build' - The build system used. This value will be passed as
+#   `--build=$erl_xcomp_build' argument to the `configure' script. It does
+#   not have to be a full `CPU-VENDOR-OS' triplet, but can be. The full
+#   `CPU-VENDOR-OS' triplet will be created by
+#   `$ERL_TOP/erts/autoconf/config.sub $erl_xcomp_build'. If set to `guess',
+#   the build system will be guessed using
+#   `$ERL_TOP/erts/autoconf/config.guess'.
+erl_xcomp_build=guess
+
+# * `erl_xcomp_host' - Cross host/target system to build for. This value will
+#   be passed as `--host=$erl_xcomp_host' argument to the `configure' script.
+#   It does not have to be a full `CPU-VENDOR-OS' triplet, but can be. The
+#   full `CPU-VENDOR-OS' triplet will be created by
+#   `$ERL_TOP/erts/autoconf/config.sub $erl_xcomp_host'.
+erl_xcomp_host=armv8-rpi3-linux-gnueabihf
+
+# * `erl_xcomp_configure_flags' - Extra configure flags to pass to the
+#   `configure' script.
+#erl_xcomp_configure_flags=
+
+## -- Cross Compiler and Other Tools -------------------------------------------
+
+## If the cross compilation tools are prefixed by `<HOST>-' you probably do
+## not need to set these variables (where `<HOST>' is what has been passed as
+## `--host=<HOST>' argument to `configure').
+
+## All variables in this section can also be used when native compiling.
+
+# * `CC' - C compiler.
+CC=armv8-rpi3-linux-gnueabihf-gcc
+
+# * `CFLAGS' - C compiler flags.
+#CFLAGS=
+
+# * `STATIC_CFLAGS' - Static C compiler flags.
+#STATIC_CFLAGS=
+
+# * `CFLAG_RUNTIME_LIBRARY_PATH' - This flag should set runtime library
+#   search path for the shared libraries. Note that this actually is a
+#   linker flag, but it needs to be passed via the compiler.
+#CFLAG_RUNTIME_LIBRARY_PATH=
+
+# * `CPP' - C pre-processor.
+CPP=armv8-rpi3-linux-gnueabihf-cpp
+
+# * `CPPFLAGS' - C pre-processor flags.
+#CPPFLAGS=
+
+# * `CXX' - C++ compiler.
+CXX=armv8-rpi3-linux-gnueabihf-g++
+
+# * `CXXFLAGS' - C++ compiler flags.
+#CXXFLAGS=
+
+# * `LD' - Linker.
+LD=armv8-rpi3-linux-gnueabihf-ld
+
+# * `LDFLAGS' - Linker flags.
+LDFLAGS="-L$RPI_SYSROOT/lib"
+
+# * `LIBS' - Libraries.
+#LIBS=
+
+## -- *D*ynamic *E*rlang *D*river Linking --
+
+## *NOTE*! Either set all or none of the `DED_LD*' variables.
+
+# * `DED_LD' - Linker for Dynamically loaded Erlang Drivers.
+#DED_LD=
+
+# * `DED_LDFLAGS' - Linker flags to use with `DED_LD'.
+#DED_LDFLAGS=
+
+# * `DED_LD_FLAG_RUNTIME_LIBRARY_PATH' - This flag should set runtime library
+#   search path for shared libraries when linking with `DED_LD'.
+#DED_LD_FLAG_RUNTIME_LIBRARY_PATH=
+
+## -- Large File Support --
+
+## *NOTE*! Either set all or none of the `LFS_*' variables.
+
+# * `LFS_CFLAGS' - Large file support C compiler flags.
+#LFS_CFLAGS=
+
+# * `LFS_LDFLAGS' - Large file support linker flags.
+#LFS_LDFLAGS=
+
+# * `LFS_LIBS' - Large file support libraries.
+#LFS_LIBS=
+
+## -- Other Tools --
+
+# * `RANLIB' - `ranlib' archive index tool.
+RANLIB=armv8-rpi3-linux-gnueabihf-ranlib
+
+# * `AR' - `ar' archiving tool.
+AR=armv8-rpi3-linux-gnueabihf-ar
+
+# * `GETCONF' - `getconf' system configuration inspection tool. `getconf' is
+#   currently used for finding out large file support flags to use, and
+#   on Linux systems for finding out if we have an NPTL thread library or
+#   not.
+#GETCONF=
+
+## -- Cross System Root Locations ----------------------------------------------
+
+# * `erl_xcomp_sysroot' - The absolute path to the system root of the cross
+#   compilation environment. Currently, the `crypto', `odbc', `ssh' and
+#   `ssl' applications need the system root. These applications will be
+#   skipped if the system root has not been set. The system root might be
+#   needed for other things too. If this is the case and the system root
+#   has not been set, `configure' will fail and request you to set it.
+erl_xcomp_sysroot="$RPI_SYSROOT"
+
+# * `erl_xcomp_isysroot' - The absolute path to the system root for includes
+#   of the cross compilation environment. If not set, this value defaults
+#   to `$erl_xcomp_sysroot', i.e., only set this value if the include system
+#   root path is not the same as the system root path.
+#erl_xcomp_isysroot=
+
+## -- Optional Feature, and Bug Tests ------------------------------------------
+
+## These tests cannot (always) be done automatically when cross compiling. You
+## usually do not need to set these variables. Only set these if you really
+## know what you are doing.
+
+## Note that some of these values will override results of tests performed
+## by `configure', and some will not be used until `configure' is sure that
+## it cannot figure the result out.
+
+## The `configure' script will issue a warning when a default value is used.
+## When a variable has been set, no warning will be issued.
+
+# * `erl_xcomp_after_morecore_hook' - `yes|no'. Defaults to `no'. If `yes',
+#   the target system must have a working `__after_morecore_hook' that can be
+#   used for tracking used `malloc()' implementations core memory usage.
+#   This is currently only used by unsupported features.
+#erl_xcomp_after_morecore_hook=
+
+# * `erl_xcomp_bigendian' - `yes|no'. No default. If `yes', the target system
+#   must be big endian. If `no', little endian. This can often be
+#   automatically detected, but not always. If not automatically detected,
+#   `configure' will fail unless this variable is set. Since no default
+#   value is used, `configure' will try to figure this out automatically.
+#erl_xcomp_bigendian=
+
+# * `erl_xcomp_double_middle` - `yes|no`. No default. If `yes`, the
+#   target system must have doubles in "middle-endian" format. If
+#   `no`, it has "regular" endianness. This can often be automatically
+#   detected, but not always. If not automatically detected,
+#   `configure` will fail unless this variable is set. Since no
+#   default value is used, `configure` will try to figure this out
+#   automatically.
+#erl_xcomp_double_middle_endian
+
+# * `erl_xcomp_clock_gettime_cpu_time' - `yes|no'. Defaults to `no'. If `yes',
+#   the target system must have a working `clock_gettime()' implementation
+#   that can be used for retrieving process CPU time.
+#erl_xcomp_clock_gettime_cpu_time=
+
+# * `erl_xcomp_getaddrinfo' - `yes|no'. Defaults to `no'. If `yes', the target
+#   system must have a working `getaddrinfo()' implementation that can
+#   handle both IPv4 and IPv6.
+#erl_xcomp_getaddrinfo=
+
+# * `erl_xcomp_gethrvtime_procfs_ioctl' - `yes|no'. Defaults to `no'. If `yes',
+#   the target system must have a working `gethrvtime()' implementation and
+#   is used with procfs `ioctl()'.
+#erl_xcomp_gethrvtime_procfs_ioctl=
+
+# * `erl_xcomp_dlsym_brk_wrappers' - `yes|no'. Defaults to `no'. If `yes', the
+#   target system must have a working `dlsym(RTLD_NEXT, <S>)' implementation
+#   that can be used on `brk' and `sbrk' symbols used by the `malloc()'
+#   implementation in use, and by this track the `malloc()' implementations
+#   core memory usage. This is currently only used by unsupported features.
+#erl_xcomp_dlsym_brk_wrappers=
+
+# * `erl_xcomp_kqueue' - `yes|no'. Defaults to `no'. If `yes', the target
+#   system must have a working `kqueue()' implementation that returns a file
+#   descriptor which can be used by `poll()' and/or `select()'. If `no' and
+#   the target system has not got `epoll()' or `/dev/poll', the kernel-poll
+#   feature will be disabled.
+#erl_xcomp_kqueue=
+
+# * `erl_xcomp_linux_clock_gettime_correction' - `yes|no'. Defaults to `yes' on
+#   Linux; otherwise, `no'. If `yes', `clock_gettime(CLOCK_MONOTONIC, _)' on
+#   the target system must work. This variable is recommended to be set to
+#   `no' on Linux systems with kernel versions less than 2.6.
+#erl_xcomp_linux_clock_gettime_correction=
+
+# * `erl_xcomp_linux_nptl' - `yes|no'. Defaults to `yes' on Linux; otherwise,
+#   `no'. If `yes', the target system must have NPTL (Native POSIX Thread
+#   Library). Older Linux systems have LinuxThreads instead of NPTL (Linux
+#   kernel versions typically less than 2.6).
+#erl_xcomp_linux_nptl=
+
+# * `erl_xcomp_linux_usable_sigaltstack' - `yes|no'. Defaults to `yes' on Linux;
+#   otherwise, `no'. If `yes', `sigaltstack()' must be usable on the target
+#   system. `sigaltstack()' on Linux kernel versions less than 2.4 are
+#   broken.
+#erl_xcomp_linux_usable_sigaltstack=
+
+# * `erl_xcomp_linux_usable_sigusrx' - `yes|no'. Defaults to `yes'. If `yes',
+#   the `SIGUSR1' and `SIGUSR2' signals must be usable by the ERTS. Old
+#   LinuxThreads thread libraries (Linux kernel versions typically less than
+#   2.2) used these signals and made them unusable by the ERTS.
+#erl_xcomp_linux_usable_sigusrx=
+
+# * `erl_xcomp_poll' - `yes|no'. Defaults to `no' on Darwin/MacOSX; otherwise,
+#   `yes'. If `yes', the target system must have a working `poll()'
+#   implementation that also can handle devices. If `no', `select()' will be
+#   used instead of `poll()'.
+#erl_xcomp_poll=
+
+# * `erl_xcomp_putenv_copy' - `yes|no'. Defaults to `no'. If `yes', the target
+#   system must have a `putenv()' implementation that stores a copy of the
+#   key/value pair.
+#erl_xcomp_putenv_copy=
+
+# * `erl_xcomp_reliable_fpe' - `yes|no'. Defaults to `no'. If `yes', the target
+#   system must have reliable floating point exceptions.
+#erl_xcomp_reliable_fpe=
+
+# * `erl_xcomp_posix_memalign' - `yes|no'. Defaults to `yes' if `posix_memalign'
+#   system call exists; otherwise `no'. If `yes', the target system must have a
+#   `posix_memalign' implementation that accepts larger than page size
+#   alignment.
+#erl_xcomp_posix_memalign=
+
+## -----------------------------------------------------------------------------