diff options
Diffstat (limited to 'HOWTO')
| -rw-r--r-- | HOWTO/BENCHMARKS.md | 73 | ||||
| -rw-r--r-- | HOWTO/BOOTSTRAP.md | 61 | ||||
| -rw-r--r-- | HOWTO/DTRACE.md | 391 | ||||
| -rw-r--r-- | HOWTO/INSTALL-ANDROID.md | 52 | ||||
| -rw-r--r-- | HOWTO/INSTALL-CROSS.md | 552 | ||||
| -rw-r--r-- | HOWTO/INSTALL-WIN32.md | 913 | ||||
| -rw-r--r-- | HOWTO/INSTALL.md | 934 | ||||
| -rw-r--r-- | HOWTO/MARKDOWN.md | 273 | ||||
| -rw-r--r-- | HOWTO/OTP-PATCH-APPLY.md | 144 | ||||
| -rw-r--r-- | HOWTO/SYSTEMTAP.md | 75 | ||||
| -rw-r--r-- | HOWTO/TESTING.md | 150 |
11 files changed, 3618 insertions, 0 deletions
diff --git a/HOWTO/BENCHMARKS.md b/HOWTO/BENCHMARKS.md new file mode 100644 index 0000000000..28a590dfd2 --- /dev/null +++ b/HOWTO/BENCHMARKS.md @@ -0,0 +1,73 @@ +Benchmarking Erlang/OTP +======================= + +The Erlang/OTP source tree contains a number of benchmarks. The same framework +is used to run these benchmarks as is used to run tests. Therefore in order to +run benchmarks you have to [release the tests][] just as you normally would. + +Note that many of these benchmarks were developed to test a specific feature +under a specific setting. We strive to keep the benchmarks up-to-date, but alas +time is not an endless resource so some benchmarks will be outdated and +irrelevant. + +Running the benchmarks +---------------------- + +As with testing, `ts` is used to run the benchmarks. Before running any +benchmarks you have to [install the tests][]. To get a listing of all +benchmarks you have available call `ts:benchmarks()`. + +To run all benchmarks call `ts:bench()`. This will run all benchmarks using +the emulator which is in your `$PATH` (Note that this does not have to be the +same as from which the benchmarks were built from). All the results of the +benchmarks are put in a folder in `$TESTROOT/test_server/` called +`YYYY_MO_DDTHH_MI_SS`. + +Each benchmark is run multiple times and the data for all runs is collected in +the files within the benchmark folder. All benchmarks are written so that +higher values are better. + +Writing benchmarks +------------------ + +Benchmarks are just normal testcases in Common Test suites. They are marked as +benchmarks by being included in the `AppName_bench.spec` which is located in +`lib/AppName/test/` for the applications which have benchmarks. Note that you +might want to add a skip clause to `AppName.spec` for the benchmarks if you do +not want them to be run in the nightly tests. + +Results of benchmarks are sent using the ct_event mechanism and automatically +collected and formatted by ts. + + ct_event:notify( + #event{name = benchmark_data, + data = [{value,TPS}]}). + +The application, suite and testcase associated with the value is automatically +detected. If you want to supply your own you can include `suite` andor `name` +with the data. i.e. + + ct_event:notify( + #event{name = benchmark_data, + data = [{suite,"erts_bench"}, + {name,"ets_transactions_per_sec"}, + {value,TPS}]}). + +The reason for using the internal ct_event and not ct is because the benchmark +code has to be backwards compatible with at least R14. + +The value which is reported should be as raw as possible. i.e. you should not +do any averaging of the value before reporting. The tools we use to collect the +benchmark data over time will do averages, means, stddev and more with the data. +So the more data which is sent using `ct_event` the better. + +Viewing benchmarks +------------------ + +At the moment of writing this HOWTO the tool for viewing benchmark results is +not available as opensource. This will hopefully change in the near future. + + + [release the tests]: TESTING.md#releasing-tests + [install the tests]: TESTING.md#configuring-the-test-environment + diff --git a/HOWTO/BOOTSTRAP.md b/HOWTO/BOOTSTRAP.md new file mode 100644 index 0000000000..59e31165cf --- /dev/null +++ b/HOWTO/BOOTSTRAP.md @@ -0,0 +1,61 @@ +Notes about prebuilt beam files under version control +===================================================== + +This information applies mostly to developers, some parts only +to developers of the main branch i.e. Ericsson and HiPE personel. + +There are two types of derived code under version control, namely: + +primary bootstrap - Resides in the `$ERL_TOP/bootstrap/{lib,bin}` directories. +preloaded code - Resides in the `$ERL_TOP/erts/preloaded` directory. + +Primary bootstrap +----------------- + +The two types of version controlled code are fundamentally +different. The primary bootstrap is code compiled from source files in +the lib/{kernel,stdlib,compiler} (and header files from +lib/orber/include. They are checked in in the version control system +to make it possible to build directly from the code base tree without +the need for an earlier version of the compiler. When a new version of +OTP is released, these files are updated manually (or rather, by using +the `$ERL_TOP/otp_build` script) and checked in. The files can also be +updated due to changes in the compiler during the development +process. The primary bootstrap is always updated as a separate +deliberate process, never during a normal development build. + +If a prebuilt open source version of erlang is used, the directory +bootstrap initially does not contain any beam files, the directory is +instead populated by copying beam files from the +`$ERL_TOP/lib/{kernel,stdlib,compiler}/ebin` directories. This +construction is to save space in the distribution, but the result +would be the same. Open source developers need not provide patches for +the precompiled beam files in the primary bootstrap, the bootstrap +update is always performed by the main developers. + +Preloaded code +-------------- + +The directory `$ERL_TOP/preloaded` contains both src and ebin +subdirectories. The preloaded code is compiled into the virtual +machine and always present. When compiling the virtual machine, those +beam files need to be present and they are considered a part of the +virtual machine rather than a part of the kernel application. When +preloaded files are to be updated, the source code is built using a +special Makefile in the `$ERL_TOP/preloaded/src` directory, which +creates beam files in the same directory. When they seem to compile +successfully, they can be used in an emulator build by being copied +to the ebin directory. `otp_build update_preloaded` can be used to +ease the process (there are also similar targets in the +`$ERL_TOP/preloaded/src/Makefile`). + +In prebuilt open source distributions, these beam files are also +present, but to update them one might need to change permission on the +`$ERL_TOP/preloaded/ebin` directory, then build and then manually copy +the beam files from the source directory to ../ebin. If patches are +created that involve the source files used to build preloaded code, +always note this specially as the preloaded/ebin directory needs +updating, or provide the new derived files in the patch or as complete +binaries. + +/Patrik, OTP diff --git a/HOWTO/DTRACE.md b/HOWTO/DTRACE.md new file mode 100644 index 0000000000..90f4addefd --- /dev/null +++ b/HOWTO/DTRACE.md @@ -0,0 +1,391 @@ +DTrace and Erlang/OTP +===================== + +History +------- + +The first implementation of DTrace probes for the Erlang virtual +machine was presented at the [2008 Erlang User Conference] [1]. That +work, based on the Erlang/OTP R12 release, was discontinued due to +what appears to be miscommunication with the original developers. + +Several users have created Erlang port drivers, linked-in drivers, or +NIFs that allow Erlang code to try to activate a probe, +e.g. `foo_module:dtrace_probe("message goes here!")`. + +Goals +----- + +* Annotate as much of the Erlang VM as is practical. + * The initial goal is to trace file I/O operations. +* Support all platforms that implement DTrace: OS X, Solaris, + and (I hope) FreeBSD and NetBSD. +* To the extent that it's practical, support SystemTap on Linux + via DTrace provider compatibility. +* Allow Erlang code to supply annotations. + +Supported platforms +------------------- + +* OS X 10.6.x / Snow Leopard, OS X 10.7.x / Lion and probably newer versions. +* Solaris 10. I have done limited testing on Solaris 11 and + OpenIndiana release 151a, and both appear to work. +* FreeBSD 9.0 and 10.0. +* Linux via SystemTap compatibility. Please see + [$ERL_TOP/HOWTO/SYSTEMTAP.md][] for more details. + +Just add the `--with-dynamic-trace=dtrace` option to your command when you +run the `configure` script. If you are using systemtap, the configure option +is `--with-dynamic-trace=systemtap` + +Status +------ + +As of R15B01, the dynamic trace code is included in the OTP source distribution, +although it's considered experimental. The main development of the dtrace code +still happens outside of Ericsson, but there is no need to fetch a patched +version of the OTP source to get the basic funtionality. + +Implementation summary +---------------------- + +So far, most effort has been focused on the `efile_drv.c` code, +which implements most file I/O on behalf of the Erlang virtual +machine. This driver also presents a big challenge: its use of an I/O +worker pool (enabled by using the `erl +A 8` flag, for example) makes +it much more difficult to trace I/O activity because each of the +following may be executed in a different Pthread: + +* I/O initiation (Erlang code) +* I/O proxy process handling, e.g. read/write when file is not opened + in `raw` mode, operations executed by the code & file server processes. + (Erlang code) +* `efile_drv` command setup (C code) +* `efile_drv` command execution (C code) +* `efile_drv` status return (C code) + +Example output from `lib/runtime_tools/examples/efile_drv.d` while executing +`file:rename("old-name", "new-name")`: + + efile_drv enter tag={3,84} user tag some-user-tag | RENAME (12) | args: old-name new-name ,\ + 0 0 (port #Port<0.59>) + async I/O worker tag={3,83} | RENAME (12) | efile_drv-int_entry + async I/O worker tag={3,83} | RENAME (12) | efile_drv-int_return + efile_drv return tag={3,83} user tag | RENAME (12) | errno 2 + +... where the following key can help decipher the output: + +* `{3,83}` is the Erlang scheduler thread number (3) and operation + counter number (83) assigned to this I/O operation. Together, + these two numbers form a unique ID for the I/O operation. +* `12` is the command number for the rename operation. See the + definition for `FILE_RENAME` in the source code file `efile_drv.c` + or the `BEGIN` section of the D script `lib/runtime_tools/examples/efile_drv.d`. +* `old-name` and `new-name` are the two string arguments for the + source and destination of the `rename(2)` system call. + The two integer arguments are unused; the simple formatting code + prints the arguments anyway, 0 and 0. +* The worker pool code was called on behalf of Erlang port `#Port<0.59>`. +* The system call failed with a POSIX errno value of 2: `ENOENT`, + because the path `old-name` does not exist. +* The `efile_drv-int_entry` and `efile_drv_int_return` probes are + provided in case the user is + interested in measuring only the latency of code executed by + `efile_drv` asynchronous functions by I/O worker pool threads + and the OS system call that they encapsulate. + +So, where does the `some-user-tag` string come from? + +At the moment, the user tag comes from code like the following: + + dyntrace:put_tag("some-user-tag"), + file:rename("old-name", "new-name"), + +This method of tagging I/O at the Erlang level is subject to change. + +Example DTrace probe specification +---------------------------------- + + /** + * Fired when a message is sent from one local process to another. + * + * NOTE: The 'size' parameter is in machine-dependent words and + * that the actual size of any binary terms in the message + * are not included. + * + * @param sender the PID (string form) of the sender + * @param receiver the PID (string form) of the receiver + * @param size the size of the message being delivered (words) + * @param token_label for the sender's sequential trace token + * @param token_previous count for the sender's sequential trace token + * @param token_current count for the sender's sequential trace token + */ + probe message__send(char *sender, char *receiver, uint32_t size, + int token_label, int token_previous, int token_current); + + /** + * Fired when a message is sent from a local process to a remote process. + * + * NOTE: The 'size' parameter is in machine-dependent words and + * that the actual size of any binary terms in the message + * are not included. + * + * @param sender the PID (string form) of the sender + * @param node_name the Erlang node name (string form) of the receiver + * @param receiver the PID/name (string form) of the receiver + * @param size the size of the message being delivered (words) + * @param token_label for the sender's sequential trace token + * @param token_previous count for the sender's sequential trace token + * @param token_current count for the sender's sequential trace token + */ + probe message__send__remote(char *sender, char *node_name, char *receiver, + uint32_t size, + int token_label, int token_previous, int token_current); + + /** + * Fired when a message is queued to a local process. This probe + * will not fire if the sender's pid == receiver's pid. + * + * NOTE: The 'size' parameter is in machine-dependent words and + * that the actual size of any binary terms in the message + * are not included. + * + * @param receiver the PID (string form) of the receiver + * @param size the size of the message being delivered (words) + * @param queue_len length of the queue of the receiving process + * @param token_label for the sender's sequential trace token + * @param token_previous count for the sender's sequential trace token + * @param token_current count for the sender's sequential trace token + */ + probe message__queued(char *receiver, uint32_t size, uint32_t queue_len, + int token_label, int token_previous, int token_current); + + /** + * Fired when a message is 'receive'd by a local process and removed + * from its mailbox. + * + * NOTE: The 'size' parameter is in machine-dependent words and + * that the actual size of any binary terms in the message + * are not included. + * + * @param receiver the PID (string form) of the receiver + * @param size the size of the message being delivered (words) + * @param queue_len length of the queue of the receiving process + * @param token_label for the sender's sequential trace token + * @param token_previous count for the sender's sequential trace token + * @param token_current count for the sender's sequential trace token + */ + probe message__receive(char *receiver, uint32_t size, uint32_t queue_len, + int token_label, int token_previous, int token_current); + + /* ... */ + + /* Async driver pool */ + + /** + * Show the post-add length of the async driver thread pool member's queue. + * + * NOTE: The port name is not available: additional lock(s) must + * be acquired in order to get the port name safely in an SMP + * environment. The same is true for the aio__pool_get probe. + * + * @param port the Port (string form) + * @param new queue length + */ + probe aio_pool__add(char *, int); + + /** + * Show the post-get length of the async driver thread pool member's queue. + * + * @param port the Port (string form) + * @param new queue length + */ + probe aio_pool__get(char *, int); + + /* Probes for efile_drv.c */ + + /** + * Entry into the efile_drv.c file I/O driver + * + * For a list of command numbers used by this driver, see the section + * "Guide to probe arguments" in ../../../README.md. That section + * also contains explanation of the various integer and string + * arguments that may be present when any particular probe fires. + * + * TODO: Adding the port string, args[10], is a pain. Making that + * port string available to all the other efile_drv.c probes + * will be more pain. Is the pain worth it? If yes, then + * add them everywhere else and grit our teeth. If no, then + * rip it out. + * + * @param thread-id number of the scheduler Pthread arg0 + * @param tag number: {thread-id, tag} uniquely names a driver operation + * @param user-tag string arg2 + * @param command number arg3 + * @param string argument 1 arg4 + * @param string argument 2 arg5 + * @param integer argument 1 arg6 + * @param integer argument 2 arg7 + * @param integer argument 3 arg8 + * @param integer argument 4 arg9 + * @param port the port ID of the busy port args[10] + */ + probe efile_drv__entry(int, int, char *, int, char *, char *, + int64_t, int64_t, int64_t, int64_t, char *); + + /** + * Entry into the driver's internal work function. Computation here + * is performed by a async worker pool Pthread. + * + * @param thread-id number + * @param tag number + * @param command number + */ + probe efile_drv__int_entry(int, int, int); + + /** + * Return from the driver's internal work function. + * + * @param thread-id number + * @param tag number + * @param command number + */ + probe efile_drv__int_return(int, int, int); + + /** + * Return from the efile_drv.c file I/O driver + * + * @param thread-id number arg0 + * @param tag number arg1 + * @param user-tag string arg2 + * @param command number arg3 + * @param Success? 1 is success, 0 is failure arg4 + * @param If failure, the errno of the error. arg5 + */ + probe efile_drv__return(int, int, char *, int, int, int); + +Guide to efile_drv.c probe arguments +------------------------------------ + + /* Driver op code: used by efile_drv-entry arg3 */ + /* used by efile_drv-int_entry arg3 */ + /* used by efile_drv-int_return arg3 */ + /* used by efile_drv-return arg3 */ + + #define FILE_OPEN 1 (probe arg3) + probe arg6 = C driver dt_i1 = flags; + probe arg4 = C driver dt_s1 = path; + + #define FILE_READ 2 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + probe arg8 = C driver dt_i3 = size; + + #define FILE_LSEEK 3 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = offset; + probe arg8 = C driver dt_i3 = origin; + + #define FILE_WRITE 4 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + probe arg8 = C driver dt_i3 = size; + + #define FILE_FSTAT 5 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + + #define FILE_PWD 6 (probe arg3) + none + + #define FILE_READDIR 7 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_CHDIR 8 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_FSYNC 9 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + + #define FILE_MKDIR 10 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_DELETE 11 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_RENAME 12 (probe arg3) + probe arg4 = C driver dt_s1 = old_name; + probe arg5 = C driver dt_s2 = new_name; + + #define FILE_RMDIR 13 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_TRUNCATE 14 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + + #define FILE_READ_FILE 15 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_WRITE_INFO 16 (probe arg3) + probe arg6 = C driver dt_i1 = mode; + probe arg7 = C driver dt_i2 = uid; + probe arg8 = C driver dt_i3 = gid; + + #define FILE_LSTAT 19 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_READLINK 20 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_LINK 21 (probe arg3) + probe arg4 = C driver dt_s1 = existing_path; + probe arg5 = C driver dt_s2 = new_path; + + #define FILE_SYMLINK 22 (probe arg3) + probe arg4 = C driver dt_s1 = existing_path; + probe arg5 = C driver dt_s2 = new_path; + + #define FILE_CLOSE 23 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + + #define FILE_PWRITEV 24 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + probe arg8 = C driver dt_i3 = size; + + #define FILE_PREADV 25 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + probe arg8 = C driver dt_i3 = size; + + #define FILE_SETOPT 26 (probe arg3) + probe arg6 = C driver dt_i1 = opt_name; + probe arg7 = C driver dt_i2 = opt_specific_value; + + #define FILE_IPREAD 27 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + probe arg8 = C driver dt_i3 = offsets[0]; + probe arg9 = C driver dt_i4 = size; + + #define FILE_ALTNAME 28 (probe arg3) + probe arg4 = C driver dt_s1 = path; + + #define FILE_READ_LINE 29 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = flags; + probe arg8 = C driver dt_i3 = read_offset; + probe arg9 = C driver dt_i4 = read_ahead; + + #define FILE_FDATASYNC 30 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + + #define FILE_FADVISE 31 (probe arg3) + probe arg6 = C driver dt_i1 = fd; + probe arg7 = C driver dt_i2 = offset; + probe arg8 = C driver dt_i3 = length; + probe arg9 = C driver dt_i4 = advise_type; + + [1]: http://www.erlang.org/euc/08/ + [$ERL_TOP/HOWTO/SYSTEMTAP.md]: SYSTEMTAP.md diff --git a/HOWTO/INSTALL-ANDROID.md b/HOWTO/INSTALL-ANDROID.md new file mode 100644 index 0000000000..31698d4ce3 --- /dev/null +++ b/HOWTO/INSTALL-ANDROID.md @@ -0,0 +1,52 @@ +Cross Compiling Erlang/OTP - ANDROID +==================================== + +Introduction +------------ + +This document describes how to cross compile Erlang OTP to Android/Rasberry Pi platforms. + +### Download and Install Android NDK ### + +https://developer.android.com/tools/sdk/ndk/index.html + +### Define System Variables ### + +export NDK_ROOT=/usr/local/android +export NDK_PLAT=android-9 +export PATH=$NDK_ROOT/toolchains/arm-linux-androideabi-4.8/prebuilt/darwin-x86_64/bin:$PATH + +### Configure OTP ### + +./otp_build configure \ + --xcomp-conf=./xcomp/erl-xcomp-arm-android.conf \ + --without-ssl + +### Compile OTP ### + +make noboot [-j4] + +### Make Release ### + +./otp_build release -a /usr/local/otp_R16B03_arm + +### Target Deployment ### + +Make a tarball out of /usr/local/otp_R16B03_arm and copy it to target device +(e.g. Raspberry Pi). Extract it and install + +./Install /usr/local/otp_R16B03_arm + +Android SDK (adb tool) is used to deploy OTP/Erlang to target device for +evaluation purpose only. + +adb push /usr/local/otp_R16B03_arm /mnt/sdcard/otp_R16B03_arm +adb shell + +### Known Issues ### + + * native inet:gethostbyname/1 return {error, nxdomain} on Raspberry PI. Use dns resolver to by-pass the issue (see http://www.erlang.org/doc/apps/erts/inet_cfg.html) + +### References ### + + The port derives some solutions from https://code.google.com/p/erlang4android/ diff --git a/HOWTO/INSTALL-CROSS.md b/HOWTO/INSTALL-CROSS.md new file mode 100644 index 0000000000..29614966b8 --- /dev/null +++ b/HOWTO/INSTALL-CROSS.md @@ -0,0 +1,552 @@ +Cross Compiling Erlang/OTP +========================== + +Introduction +------------ + +This document describes how to cross compile Erlang/OTP-%OTP-REL%. +You are advised to read the whole document before attempting to cross +compile Erlang/OTP. However, before reading this document, you should read +the [$ERL_TOP/HOWTO/INSTALL.md][] document which describes building and installing +Erlang/OTP in general. `$ERL_TOP` is the top directory in the source tree. + +### otp\_build Versus configure/make ### + +Building Erlang/OTP can be done either by using the `$ERL_TOP/otp_build` +script, or by invoking `$ERL_TOP/configure` and `make` directly. Building using +`otp_build` is easier since it involves fewer steps, but the `otp_build` build +procedure is not as flexible as the `configure`/`make` build procedure. Note +that `otp_build configure` will produce a default configuration that differs +from what `configure` will produce by default. For example, currently +`--disable-dynamic-ssl-lib` is added to the `configure` command line arguments +unless `--enable-dynamic-ssl-lib` has been explicitly passed. The binary +releases that we deliver are built using `otp_build`. The defaults used by +`otp_build configure` may change at any time without prior notice. + +### Cross Configuration ### + +The `$ERL_TOP/xcomp/erl-xcomp.conf.template` file contains all available cross +configuration variables and can be used as a template when creating a cross +compilation configuration. All [cross configuration variables][] are also +listed at the end of this document. For examples of working cross +configurations see the `$ERL_TOP/xcomp/erl-xcomp-TileraMDE2.0-tilepro.conf` +file and the `$ERL_TOP/xcomp/erl-xcomp-x86_64-saf-linux-gnu.conf` file. If the +default behavior of a variable is satisfactory, the variable does not need to +be set. However, the `configure` script will issue a warning when a default +value is used. When a variable has been set, no warning will be issued. + +A cross configuration file can be passed to `otp_build configure` using the +`--xcomp-conf` command line argument. Note that `configure` does not accept +this command line argument. When using the `configure` script directly, pass +the configuration variables as arguments to `configure` using a +`<VARIABLE>=<VALUE>` syntax. Variables can also be passed as environment +variables to `configure`. However, if you pass the configuration in the +environment, make sure to unset all of these environment variables before +invoking `make`; otherwise, the environment variables might set make variables +in some applications, or parts of some applications, and you may end up with +an erroneously configured build. + +### What can be Cross Compiled? ### + +All Erlang/OTP applications except the `wx` application can be cross compiled. +The build of the `wx` driver will currently be automatically disabled when +cross compiling. + +### Compatibility ### + +The build system, including cross compilation configuration variables used, +may be subject to non backward compatible changes without prior notice. +Current cross build system has been tested when cross compiling some Linux/GNU +systems, but has only been partly tested for more esoteric platforms. The +VxWorks example file is highly dependent on our environment and is here more +or less only for internal use. + +### Patches ### + +Please submit any patches for cross compiling in a way consistent with this +system. All input is welcome as we have a very limited set of cross compiling +environments to test with. If a new configuration variable is needed, add it +to `$ERL_TOP/xcomp/erl-xcomp.conf.template`, and use it in `configure.in`. +Other files that might need to be updated are: + +- `$ERL_TOP/xcomp/erl-xcomp-vars.sh` +- `$ERL_TOP/erl-build-tool-vars.sh` +- `$ERL_TOP/erts/aclocal.m4` +- `$ERL_TOP/xcomp/README.md` +- `$ERL_TOP/xcomp/erl-xcomp-*.conf` + +Note that this might be an incomplete list of files that need to be updated. + +General information on how to submit patches can be found at: + <http://wiki.github.com/erlang/otp/submitting-patches> + +Build and Install Procedure +--------------------------- + +If you are building in Git, you want to read the [Building in Git][] section +of [$ERL_TOP/HOWTO/INSTALL.md][] before proceeding. + +We will first go through the `configure`/`make` build procedure which people +probably are most familiar with. + +### Building With configure/make Directly ### + + (1) + +Change directory into the top directory of the Erlang/OTP source tree. + + $ cd $ERL_TOP + +In order to compile Erlang code, a small Erlang bootstrap system has to be +built, or an Erlang/OTP system of the same release as the one being built +has to be provided in the `$PATH`. The Erlang/OTP for the target system will +be built using this Erlang system, together with the cross compilation tools +provided. + +If you want to build using a compatible Erlang/OTP system in the `$PATH`, +jump to (3). + +#### Building a Bootstrap System #### + + (2) + + $ ./configure --enable-bootstrap-only + $ make + +The `--enable-bootstrap-only` argument to `configure` isn't strictly necessary, +but will speed things up. It will only run `configure` in applications +necessary for the bootstrap, and will disable a lot of things not needed by +the bootstrap system. If you run `configure` without `--enable-boostrap-only` +you also have to run make as `make bootstrap`; otherwise, the whole system will +be built. + +#### Cross Building the System #### + + (3) + + $ ./configure --host=<HOST> --build=<BUILD> [Other Config Args] + $ make + +`<HOST>` is the host/target system that you build for. It does not have to be +a full `CPU-VENDOR-OS` triplet, but can be. The full `CPU-VENDOR-OS` triplet +will be created by executing `$ERL_TOP/erts/autoconf/config.sub <HOST>`. If +`config.sub` fails, you need to be more specific. + +`<BUILD>` should equal the `CPU-VENDOR-OS` triplet of the system that you +build on. If you execute `$ERL_TOP/erts/autoconf/config.guess`, it will in +most cases print the triplet you want to use for this. + +Pass the cross compilation variables as command line arguments to `configure` +using a `<VARIABLE>=<VALUE>` syntax. + +> *NOTE*: You can *not* pass a configuration file using the `--xcomp-conf` +> argument when you invoke `configure` directly. The `--xcomp-conf` argument +> can only be passed to `otp_build configure`. + +`make` will verify that the Erlang/OTP system used when building is of the +same release as the system being built, and will fail if this is not the case. +It is possible, however not recommended, to force the cross compilation even +though the wrong Erlang/OTP system is used. This by invoking `make` like this: +`make ERL_XCOMP_FORCE_DIFFERENT_OTP=yes`. + +> *WARNING*: Invoking `make ERL_XCOMP_FORCE_DIFFERENT_OTP=yes` might fail, +> silently produce suboptimal code, or silently produce erroneous code. + +#### Installing #### + +You can either install using the installation paths determined by `configure` +(4), or install manually using (5). + +##### Installing Using Paths Determined by configure ##### + + (4) + + $ make install DESTDIR=<TEMPORARY_PREFIX> + +`make install` will install at a location specified when doing `configure`. +`configure` arguments specifying where the installation should reside are for +example: `--prefix`, `--exec-prefix`, `--libdir`, `--bindir`, etc. By default +it will install under `/usr/local`. You typically do not want to install your +cross build under `/usr/local` on your build machine. Using [DESTDIR][] +will cause the installation paths to be prefixed by `$DESTDIR`. This makes it +possible to install and package the installation on the build machine without +having to place the installation in the same directory on the build machine as +it should be executed from on the target machine. + +When `make install` has finished, change directory into `$DESTDIR`, package +the system, move it to the target machine, and unpack it. Note that the +installation will only be working on the target machine at the location +determined by `configure`. + +##### Installing Manually ##### + + (5) + + $ make release RELEASE_ROOT=<RELEASE_DIR> + +`make release` will copy what you have built for the target machine to +`<RELEASE_DIR>`. The `Install` script will not be run. The content of +`<RELEASE_DIR>` is what by default ends up in `/usr/local/lib/erlang`. + +The `Install` script used when installing Erlang/OTP requires common Unix +tools such as `sed` to be present in your `$PATH`. If your target system +does not have such tools, you need to run the `Install` script on your +build machine before packaging Erlang/OTP. The `Install` script should +currently be invoked as follows in the directory where it resides +(the top directory): + + $ ./Install [-cross] [-minimal|-sasl] <ERL_ROOT> + +where: + +* `-minimal` Creates an installation that starts up a minimal amount + of applications, i.e., only `kernel` and `stdlib` are started. The + minimal system is normally enough, and is what `make install` uses. +* `-sasl` Creates an installation that also starts up the `sasl` + application. +* `-cross` For cross compilation. Informs the install script that it + is run on the build machine. +* `<ERL_ROOT>` - The absolute path to the Erlang installation to use + at run time. This is often the same as the current working directory, + but does not have to be. It can follow any other path through the file + system to the same directory. + +If neither `-minimal`, nor `-sasl` is passed as argument you will be +prompted. + +You can now either do: + + (6) + +* Decide where the installation should be located on the target machine, + run the `Install` script on the build machine, and package the installed + installation. The installation just need to be unpacked at the right + location on the target machine: + + $ cd <RELEASE_DIR> + $ ./Install -cross [-minimal|-sasl] <ABSOLUTE_INSTALL_DIR_ON_TARGET> + +or: + + (7) + +* Package the installation in `<RELEASE_DIR>`, place it wherever you want + on your target machine, and run the `Install` script on your target + machine: + + $ cd <ABSOLUTE_INSTALL_DIR_ON_TARGET> + $ ./Install [-minimal|-sasl] <ABSOLUTE_INSTALL_DIR_ON_TARGET> + +### Building With the otp\_build Script ### + + (8) + + $ cd $ERL_TOP + + (9) + + $ ./otp_build configure --xcomp-conf=<FILE> [Other Config Args] + +alternatively: + + $ ./otp_build configure --host=<HOST> --build=<BUILD> [Other Config Args] + +If you have your cross compilation configuration in a file, pass it using the +`--xcomp-conf=<FILE>` command line argument. If not, pass `--host=<HOST>`, +`--build=<BUILD>`, and the configuration variables using a `<VARIABLE>=<VALUE>` +syntax on the command line (same as in (3)). Note that `<HOST>` and `<BUILD>` +have to be passed one way or the other; either by using `erl_xcomp_host=<HOST>` +and `erl_xcomp_build=<BUILD>` in the configuration file, or by using the +`--host=<HOST>`, and `--build=<BUILD>` command line arguments. + +`otp_build configure` will configure both for the boostrap system on the +build machine and the cross host system. + + (10) + + $ ./otp_build boot -a + +`otp_build boot -a` will first build a bootstrap system for the build machine +and then do the cross build of the system. + + (11) + + $ ./otp_build release -a <RELEASE_DIR> + +`otp_build release -a` will do the same as (5), and you will after this have +to do a manual install either by doing (6), or (7). + +Building and Installing the Documentation +----------------------------------------- + +After the system has been cross built you can build and install the +documentation the same way as after a native build of the system. See the +[How to Build the Documentation][] section in the [$ERL_TOP/HOWTO/INSTALL.md][] +document for information on how to build the documentation. + +Testing the cross compiled system +--------------------------------- + +Some of the tests that come with erlang use native code to test. This means +that when cross compiling erlang you also have to cross compile test suites +in order to run tests on the target host. To do this you first have to release +the tests as usual. + + $ make release_tests + +or + + $ ./otp_build tests + +The tests will be released into `$ERL_TOP/release/tests`. After releasing the +tests you have to install the tests on the build machine. You supply the same +xcomp file as to `./otp_build` in (9). + + $ cd $ERL_TOP/release/tests/test_server/ + $ $ERL_TOP/bootstrap/bin/erl -eval 'ts:install([{xcomp,"<FILE>"}])' -s ts compile_testcases -s init stop + +You should get a lot of printouts as the testcases are compiled. Once done you +should copy the entire `$ERL_TOP/release/tests` folder to the cross host system. + +Then go to the cross host system and setup the erlang installed in (4) or (5) +to be in your `$PATH`. Then go to what previously was +`$ERL_TOP/release/tests/test_server` and issue the following command. + + $ erl -s ts install -s ts run all_tests -s init stop + +The configure should be skipped and all tests should hopefully pass. For more +details about how to use ts run `erl -s ts help -s init stop` + +Currently Used Configuration Variables +-------------------------------------- + +Note that you cannot define arbitrary variables in a cross compilation +configuration file. Only the ones listed below will be guaranteed to be +visible throughout the whole execution of all `configure` scripts. Other +variables needs to be defined as arguments to `configure` or exported in +the environment. + +### Variables for otp\_build Only ### + +Variables in this section are only used, when configuring Erlang/OTP for +cross compilation using `$ERL_TOP/otp_build configure`. + +> *NOTE*: These variables currently have *no* effect if you configure using +> the `configure` script directly. + +* `erl_xcomp_build` - The build system used. This value will be passed as + `--build=$erl_xcomp_build` argument to the `configure` script. It does + not have to be a full `CPU-VENDOR-OS` triplet, but can be. The full + `CPU-VENDOR-OS` triplet will be created by + `$ERL_TOP/erts/autoconf/config.sub $erl_xcomp_build`. If set to `guess`, + the build system will be guessed using + `$ERL_TOP/erts/autoconf/config.guess`. + +* `erl_xcomp_host` - Cross host/target system to build for. This value will + be passed as `--host=$erl_xcomp_host` argument to the `configure` script. + It does not have to be a full `CPU-VENDOR-OS` triplet, but can be. The + full `CPU-VENDOR-OS` triplet will be created by + `$ERL_TOP/erts/autoconf/config.sub $erl_xcomp_host`. + +* `erl_xcomp_configure_flags` - Extra configure flags to pass to the + `configure` script. + +### Cross Compiler and Other Tools ### + +If the cross compilation tools are prefixed by `<HOST>-` you probably do +not need to set these variables (where `<HOST>` is what has been passed as +`--host=<HOST>` argument to `configure`). + +All variables in this section can also be used when native compiling. + +* `CC` - C compiler. + +* `CFLAGS` - C compiler flags. + +* `STATIC_CFLAGS` - Static C compiler flags. + +* `CFLAG_RUNTIME_LIBRARY_PATH` - This flag should set runtime library + search path for the shared libraries. Note that this actually is a + linker flag, but it needs to be passed via the compiler. + +* `CPP` - C pre-processor. + +* `CPPFLAGS` - C pre-processor flags. + +* `CXX` - C++ compiler. + +* `CXXFLAGS` - C++ compiler flags. + +* `LD` - Linker. + +* `LDFLAGS` - Linker flags. + +* `LIBS` - Libraries. + +#### Dynamic Erlang Driver Linking #### + +> *NOTE*: Either set all or none of the `DED_LD*` variables. + +* `DED_LD` - Linker for Dynamically loaded Erlang Drivers. + +* `DED_LDFLAGS` - Linker flags to use with `DED_LD`. + +* `DED_LD_FLAG_RUNTIME_LIBRARY_PATH` - This flag should set runtime library + search path for shared libraries when linking with `DED_LD`. + +#### Large File Support #### + +> *NOTE*: Either set all or none of the `LFS_*` variables. + +* `LFS_CFLAGS` - Large file support C compiler flags. + +* `LFS_LDFLAGS` - Large file support linker flags. + +* `LFS_LIBS` - Large file support libraries. + +#### Other Tools #### + +* `RANLIB` - `ranlib` archive index tool. + +* `AR` - `ar` archiving tool. + +* `GETCONF` - `getconf` system configuration inspection tool. `getconf` is + currently used for finding out large file support flags to use, and + on Linux systems for finding out if we have an NPTL thread library or + not. + +### Cross System Root Locations ### + +* `erl_xcomp_sysroot` - The absolute path to the system root of the cross + compilation environment. Currently, the `crypto`, `odbc`, `ssh` and + `ssl` applications need the system root. These applications will be + skipped if the system root has not been set. The system root might be + needed for other things too. If this is the case and the system root + has not been set, `configure` will fail and request you to set it. + +* `erl_xcomp_isysroot` - The absolute path to the system root for includes + of the cross compilation environment. If not set, this value defaults + to `$erl_xcomp_sysroot`, i.e., only set this value if the include system + root path is not the same as the system root path. + +### Optional Feature, and Bug Tests ### + +These tests cannot (always) be done automatically when cross compiling. You +usually do not need to set these variables. + +> *WARNING*: Setting these variables wrong may cause hard to detect +> runtime errors. If you need to change these values, *really* make sure +> that the values are correct. + +> *NOTE*: Some of these values will override results of tests performed +> by `configure`, and some will not be used until `configure` is sure that +> it cannot figure the result out. + +The `configure` script will issue a warning when a default value is used. +When a variable has been set, no warning will be issued. + +* `erl_xcomp_after_morecore_hook` - `yes|no`. Defaults to `no`. If `yes`, + the target system must have a working `__after_morecore_hook` that can be + used for tracking used `malloc()` implementations core memory usage. + This is currently only used by unsupported features. + +* `erl_xcomp_bigendian` - `yes|no`. No default. If `yes`, the target system + must be big endian. If `no`, little endian. This can often be + automatically detected, but not always. If not automatically detected, + `configure` will fail unless this variable is set. Since no default + value is used, `configure` will try to figure this out automatically. + +* `erl_xcomp_double_middle` - `yes|no`. Defaults to `no`. + If `yes`, the target system must have doubles in "middle-endian" format. If + `no`, it has "regular" endianness. + +* `erl_xcomp_clock_gettime_cpu_time` - `yes|no`. Defaults to `no`. If `yes`, + the target system must have a working `clock_gettime()` implementation + that can be used for retrieving process CPU time. + +* `erl_xcomp_getaddrinfo` - `yes|no`. Defaults to `no`. If `yes`, the target + system must have a working `getaddrinfo()` implementation that can + handle both IPv4 and IPv6. + +* `erl_xcomp_gethrvtime_procfs_ioctl` - `yes|no`. Defaults to `no`. If `yes`, + the target system must have a working `gethrvtime()` implementation and + is used with procfs `ioctl()`. + +* `erl_xcomp_dlsym_brk_wrappers` - `yes|no`. Defaults to `no`. If `yes`, the + target system must have a working `dlsym(RTLD_NEXT, <S>)` implementation + that can be used on `brk` and `sbrk` symbols used by the `malloc()` + implementation in use, and by this track the `malloc()` implementations + core memory usage. This is currently only used by unsupported features. + +* `erl_xcomp_kqueue` - `yes|no`. Defaults to `no`. If `yes`, the target + system must have a working `kqueue()` implementation that returns a file + descriptor which can be used by `poll()` and/or `select()`. If `no` and + the target system has not got `epoll()` or `/dev/poll`, the kernel-poll + feature will be disabled. + +* `erl_xcomp_linux_clock_gettime_correction` - `yes|no`. Defaults to `yes` on + Linux; otherwise, `no`. If `yes`, `clock_gettime(CLOCK_MONOTONIC, _)` on + the target system must work. This variable is recommended to be set to + `no` on Linux systems with kernel versions less than 2.6. + +* `erl_xcomp_linux_nptl` - `yes|no`. Defaults to `yes` on Linux; otherwise, + `no`. If `yes`, the target system must have NPTL (Native POSIX Thread + Library). Older Linux systems have LinuxThreads instead of NPTL (Linux + kernel versions typically less than 2.6). + +* `erl_xcomp_linux_usable_sigaltstack` - `yes|no`. Defaults to `yes` on Linux; + otherwise, `no`. If `yes`, `sigaltstack()` must be usable on the target + system. `sigaltstack()` on Linux kernel versions less than 2.4 are + broken. + +* `erl_xcomp_linux_usable_sigusrx` - `yes|no`. Defaults to `yes`. If `yes`, + the `SIGUSR1` and `SIGUSR2` signals must be usable by the ERTS. Old + LinuxThreads thread libraries (Linux kernel versions typically less than + 2.2) used these signals and made them unusable by the ERTS. + +* `erl_xcomp_poll` - `yes|no`. Defaults to `no` on Darwin/MacOSX; otherwise, + `yes`. If `yes`, the target system must have a working `poll()` + implementation that also can handle devices. If `no`, `select()` will be + used instead of `poll()`. + +* `erl_xcomp_putenv_copy` - `yes|no`. Defaults to `no`. If `yes`, the target + system must have a `putenv()` implementation that stores a copy of the + key/value pair. + +* `erl_xcomp_reliable_fpe` - `yes|no`. Defaults to `no`. If `yes`, the target + system must have reliable floating point exceptions. + +* `erl_xcomp_posix_memalign` - `yes|no`. Defaults to `yes` if `posix_memalign` + system call exists; otherwise `no`. If `yes`, the target system must have a + `posix_memalign` implementation that accepts larger than page size + alignment. + +Copyright and License +--------------------- + +%CopyrightBegin% + +Copyright Ericsson AB 2009-2014. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +%CopyrightEnd% + + + + [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md + [Building in Git]: INSTALL.md#How-to-Build-and-Install-ErlangOTP + [How to Build the Documentation]: INSTALL.md#How-to-Build-and-Install-ErlangOTP_How-to-Build-the-Documentation + [cross configuration variables]: #Currently-Used-Configuration-Variables + [DESTDIR]: http://www.gnu.org/prep/standards/html_node/DESTDIR.html + [?TOC]: true diff --git a/HOWTO/INSTALL-WIN32.md b/HOWTO/INSTALL-WIN32.md new file mode 100644 index 0000000000..c74107d749 --- /dev/null +++ b/HOWTO/INSTALL-WIN32.md @@ -0,0 +1,913 @@ +How to Build Erlang/OTP on Windows +================================== + +Introduction +------------ + +This section describes how to build the Erlang emulator and the OTP +libraries on Windows. Note that the Windows binary releases are still +a preferred alternative if one does not have Microsoft’s development +tools and/or don’t want to install Cygwin, MSYS or MSYS2. + +The instructions apply to versions of Windows supporting the Cygwin +emulated gnuish environment or the MSYS or MSYS2 ditto. We’ve built on +the following platforms: Windows 2012, Windows 7, Windows 8 and Windows 10. +It’s probably possible to build on older platforms too, but you might +not be able to install the appropriate Microsoft SDK, Visual Studio or +OpenSSL, in which case you will need to go back to earlier compilers etc. + +The procedure described uses either Cygwin, MSYS or MSYS2 as a build +environment. You run the bash shell in Cygwin/MSYS/MSYS2 and use the gnu +make/configure/autoconf etc to do the build. The emulator C-source code +is, however, mostly compiled with Microsoft Visual C++™, producing a +native Windows binary. This is the same procedure as we use to build the +pre-built binaries. Why we use VC++ and not gcc is explained further in +the FAQ section. + +If you are not familiar with Cygwin, MSYS, MSYS2 or a Unix environment, +you’ll probably need to read up a bit on how that works. There are plenty of +documentation about this online. + +These instructions apply for both 32-bit and 64-bit Windows. Note that even +if you build a 64-bit version of Erlang, most of the directories and files +involved are still named win32. Some occurances of the name win64 are +however present. The installation file for a 64-bit Windows version of +Erlang, for example, is `otp_win64_%OTP-REL%.exe`. + +If you feel comfortable with the environment and build +system, and have all the necessary tools, you have a great opportunity +to make the Erlang/OTP distribution for Windows better. Please submit +any suggestions to our [JIRA] [2] and patches to our [git project] [3] to let +them find their way into the next version of Erlang. If making changes +to the build system (like makefiles etc) please bear in mind that the +same makefiles are used on Unix/VxWorks, so that your changes +don't break other platforms. That of course goes for C-code too; system +specific code resides in the `$ERL_TOP/erts/emulator/sys/win32` and +`$ERL_TOP/erts/etc/win32` directories mostly. The +`$ERL_TOP/erts/emulator/beam` directory is for common code. + +We've used this build procedure for a couple of +releases, and it has worked fine for us. Still, there might be all +sorts of troubles on different machines and with different +setups. We'll try to give hints wherever we've encountered difficulties, +but please share your experiences by using the [erlang-questions] [1] +mailing list. We cannot, of course, help everyone with all +their issues, so please try to solve such issues and submit +solutions/workarounds. + +Lets go then! We’ll start with a short version of the setup procedure, +followed by some FAQ, and then we’ll go into more details of the setup. + + +Short Version +------------- + +In the following sections, we've described as much as we could about the +installation of the tools needed. Once the tools are installed, building +is quite easy. We have also tried to make these instructions understandable +for people with limited Unix experience. Cygwin/MSYS/MSYS2 is a whole new +environment to some Windows users, why careful explanation of environment +variables etc seemed to be in place. + +This is the short story though, for the experienced and impatient: + + * Get and install complete Cygwin (latest), complete MinGW with MSYS or + complete MSYS2 + + * Install Visual Studio 12.0 (2013) + + * Install Microsofts Windows SDK 8.1 + + * Get and install Sun's JDK 1.6.0 or later + + * Get and install NSIS 2.01 or later (up to 2.46 tried and working) + + * Get, build and install OpenSSL 0.9.8r or later (up to 1.0.2d + tried & working) with static libs. + + * Get the Erlang source distribution (from + <http://www.erlang.org/download.html>) and unpack with + Cygwin's/MSYS's/MSYS2's `tar`. + + * Set `ERL_TOP` to where you unpacked the source distribution + + * `$ cd $ERL_TOP` + + * Modify PATH and other environment variables so that all these tools + are runnable from a bash shell. Still standing in `$ERL_TOP`, issue + the following commands (for 32-bit Windows, remove the x64 from the + first row and change `otp_win64_%OTP-REL%` to `otp_win32_%OTP-REL%` on + the last row): + + $ eval `./otp_build env_win32 x64` + $ ./otp_build autoconf + $ ./otp_build configure + $ ./otp_build boot -a + $ ./otp_build release -a + $ ./otp_build installer_win32 + $ release/win32/otp_win64_%OTP-REL% /S + + Voila! `Start->Programs->Erlang OTP %OTP-REL%->Erlang` starts the Erlang + Windows shell. + + +Frequently Asked Questions +-------------------------- + +* Q: So, now I can build Erlang using GCC on Windows? + + A: No, unfortunately not. You'll need Microsoft's Visual C++ + still. A Bourne-shell script (cc.sh) wraps the Visual C++ compiler + and runs it from within the Cygwin environment. All other tools + needed to build Erlang are free-ware/open source, but not the C + compiler. The Windows SDK is however enough to build Erlang, you + do not need to buy Visual C++, just download the SDK (SDK version + 8.1 == Visual studio 2013). + +* Q: Why haven't you got rid of VC++ then, you \*\*\*\*\*\*? + + A: Well, partly because it's a good compiler - really! Actually it's + been possible in late R11-releases to build using mingw instead of + visual C++ (you might see the remnants of that in some scripts and + directories). Unfortunately the development of the SMP version for + Windows broke the mingw build and we chose to focus on the VC++ build + as the performance has been much better in the VC++ versions. The + mingw build will possibly be back, but as long as VC++ gives better + performance, the commercial build will be a VC++ one. + +* Q: OK, you need VC++, but now you've started to demand a quite recent + (and expensive) version of Visual Studio. Why? + + A: Well, it's not expensive, it's free (as in free beer). Just + download and install the latest Windows SDK from Microsoft and all + the tools you need are there. The included debugger (WinDbg) is + also quite usable. That's what I used when porting Erlang to 64bit + Windows. Another reason to use later Microsoft compilers is + DLL compatibility. DLL's using a new version of the standard + library might not load if the VM is compiled with an old VC++ + version. So we should aim to use the latest freely available SDK + and compiler. + +* Q: Can/will I build a Cygwin binary with the procedure you describe? + + A: No, the result will be a pure Windows binary, and as far as I know, + it's not possible to make a Cygwin binary yet. That is of course + something desirable, but there are still some problems with the + dynamic linking (dynamic Erlang driver loading) as well as the TCP/IP + emulation in Cygwin, which, I'm sure of, will improve, but still has + some problems. Fixing those problems might be easy or might be hard. + I suggest you try yourself and share your experience. No one would be + happier if a simple `./configure && make` would produce a fully fledged + Cygwin binary. + +* Q: Hah, I saw you, you used GCC even though you said you didn't! + + A: OK, I admit, one of the files is compiled using Cygwin's or + MinGW's GCC and the resulting object code is then converted to MS + VC++ compatible coff using a small C hack. It's because that + particular file, `beam_emu.c` benefits immensely from being able + to use the GCC labels-as-values extension, which boosts emulator + performance by up to 50%. That does unfortunately not (yet) mean + that all of OTP could be compiled using GCC. That particular + source code does not do anything system specific and actually is + adopted to the fact that GCC is used to compile it on Windows. + +* Q: So now there's a MS VC++ project file somewhere and I can build OTP + using the nifty VC++ GUI? + + A: No, never. The hassle of keeping the project files up to date and + do all the steps that constitute an OTP build from within the VC++ GUI + is simply not worth it, maybe even impossible. A VC++ project + file for Erlang/OTP will never happen. + +* Q: So how does it all work then? + + A: Cygwin, MSYS or MSYS2 is the environment, which closely resembles the + environment found on any Unix machine. It's almost like you had a + virtual Unix machine inside Windows. Configure, given certain + parameters, then creates makefiles that are used by the + environment's gnu-make to built the system. Most of the actual + compilers etc are not, however, Cygwin/MSYS/MSYS2 tools, so we've written + a couple of wrappers (Bourne-shell scripts), which reside in + `$ERL_TOP/etc/win32/cygwin_tools` and + `$ERL_TOP/etc/win32/msys_tools`. They all do conversion of + parameters and switches common in the Unix environment to fit the + native Windows tools. Most notable is of course the paths, which + in Cygwin/MSYS/MSYS2 are Unix-like paths with "forward slashes" (/) and + no drive letters. The Cygwin specific command `cygpath` is used + for most of the path conversions in a Cygwin environment. Other + tools are used (when needed) in the corresponding MSYS and MSYS2 + environment. Luckily most compilers accept forward slashes instead + of backslashes as path separators, but one still have to get the drive + letters etc right, though. The wrapper scripts are not general in + the sense that, for example, cc.sh would understand and translate + every possible gcc option and pass correct options to + cl.exe. The principle is that the scripts are powerful enough to + allow building of Erlang/OTP, no more, no less. They might need + extensions to cope with changes during the development of Erlang, and + that's one of the reasons we made them into shell-scripts and not + Perl-scripts. We believe they are easier to understand and change + that way. + + In `$ERL_TOP`, there is a script called `otp_build`. That script handles + the hassle of giving all the right parameters to `configure`/`make` and + also helps you set up the correct environment variables to work with + the Erlang source under Cygwin/MSYS/MSYS2. + +* Q: You use and need Cygwin, but then you haven't taken the time to + port Erlang to the Cygwin environment but instead focus on your + commercial release, is that really ethical? + + A: No, not really, but see this as a step in the right direction. + +* Q: Can I build something that looks exactly as the commercial release? + + A: Yes, we use the exact same build procedure. + +* Q: Which version of Cygwin/MSYS/MSYS2 and other tools do you use then? + + A: For Cygwin, MSYS and MSYS2 alike, we try to use the latest releases + available when building. What versions you use shouldn't really + matter. We try to include workarounds for the bugs we've found in + different Cygwin/MSYS/MSYS2 releases. Please help us add workarounds + for new Cygwin/MSYS/MSYS2-related bugs as soon as you encounter + them. Also please do submit bug reports to the appropriate Cygwin, MSYS + and/or MSYS2 developers. The GCC we used for %OTP-REL% was version + 4.8.1 (MinGW 32bit) and 4.8.5 (MSYS2 64bit). We used VC++ 12.0 + (i.e. Visual studio 2013), Sun's JDK 1.6.0\_45 (32bit) and Sun's + JDK 1.7.0\_1 (64bit), NSIS 2.46, and Win32 OpenSSL 1.0.2d. Please + read the next section for details on what you need. + +* Q: Can you help me setup X in Cygwin/MSYS/MSYS2? + + A: No, unfortunately we haven't got time to help with Cygwin/MSYS/MSYS2 + related user problems, please read related websites, newsgroups and + mailing lists. + + +Tools you Need and Their Environment +------------------------------------ + +You need some tools to be able to build Erlang/OTP on Windows. Most +notably you'll need Cygwin, MSYS or MSYS2, Visual Studio and Microsofts +Windows SDK, but you might also want a Java compiler, the NSIS install +system and OpenSSL. Well, here's some information about the different +tools: + +* Cygwin, the very latest is usually best. Get all the development + tools and of course all the basic ditto. Make sure to get jar and + also make sure *not* to install a Cygwin'ish Java, since the Cygwin + jar command is used but Sun's Java compiler and virtual machine. + + If you are going to build a 64bit Windows version, you should make + sure to get MinGW's 64bit gcc installed with Cygwin. It's in one of + the development packages. + + URL: <http://www.cygwin.com> + + Get the installer from the website and use it to install + Cygwin. Be sure to have fair privileges. If you're on an NT domain you + should consider running `mkpasswd -d` and `mkgroup -d` after the + installation to get the user databases correct. See their respective + manual pages. + + When you start your first bash shell, you will get an awful prompt. You + might also have a `PATH` environment variable that contains backslashes + and such. Edit `$HOME/.profile` and `$HOME/.bashrc` to set fair prompts + and a correct PATH. Also do an `export SHELL` in `.profile`. For some + non-obvious reason the environment variable `$SHELL` is not exported in + bash. Also note that `.profile` is run at login time and `.bashrc` when + sub shells are created. You'll need to explicitly source `.bashrc` from + `.profile` if you want the commands there to be run at login time (like + setting up aliases, shell functions and the like). You can for example + do like this at the end of `.profile`: + + ENV=$HOME/.bashrc + export ENV + . $ENV + + You might also want to setup X-windows (XFree86). That might be as easy + as running startx from the command prompt and it might be much harder. + Use Google to find help. + + If you don't use X-windows, you might want to setup the Windows + console window by selecting properties in the console system menu + (upper left corner of the window, the Cygwin icon in the title + bar). Especially setting a larger screen buffer size (lines) is useful + as it gets you a scrollbar so you can see whatever error messages + that might appear. + + There are a few other shells available, but in all examples below we assume + that you use bash. + +* Alternatively you download MinGW and MSYS. You'll find the latest + installer at: + + URL: <http://sourceforge.net/projects/mingw/files/Installer/mingw-get-inst/> + + Make sure to install the basic dev tools, but avoid the MinGW autoconf and + install the msys one instead. + + To be able to build the 64bit VM, you will also need the 64bit + MinGW compiler from: + + URL: <http://sourceforge.net/projects/mingw-w64/files/latest/download?source=files> + + We've tried up to 1.0, but the latest version should do. Make sure you + download the `mingw-w64-bin_i686-mingw_<something>.zip`, not a linux + version. You unzip the package on top of your MinGW installation + (`c:\MinGW`) and that's it. + +* A third alternative is to download and install MSYS2 from: + + URL: <https://msys2.github.io/> + + When you've followed the instructions there, you also need to install + these packages: autoconf, make, perl, and tar. You do so by running + the following in the msys console: + + pacman -S msys/autoconf msys/make msys/perl msys/tar + + You also need a gcc. If you installed the 64 bit MSYS2 you run: + + mingw64/mingw-w64-x86_64-gcc + + And for 32 bit MSYS2: + + pacman -S mingw32/mingw-w64-i686-gcc + pacman -S mingw-w64-i686-editrights + +* Visual Studio 2013 (Visual Studio 12.0). Download and run the web + installer from: + + https://www.visualstudio.com/ + +* Microsofts Windows SDK version 8.1 (corresponding to VC++ 12.0 and + Visual Studio 2013). You'll find it here: + + URL: <https://msdn.microsoft.com/en-us/windows/desktop/bg162891.aspx> + +* To help setup the environment, there is a bat file, + `%PROGRAMFILES%\Mirosoft Visual Studio 12.0\VC\vcvarsall.bat`, + that set's the appropriate + environment for a Windows command prompt. This is not appropriate + for bash, so you'll need to convert it to bash-style environments + by editing your `.bash_profile`. In my case, where the SDK is + installed in the default directory and `%PROGRAMFILES%` is + `C:\Program Files`, the commands for setting up a 32bit build + environment (on a 64bit or 32bit machine) look like this (in Cygwin): + + # Some common paths + C_DRV=/cygdrive/c + PRG_FLS=$C_DRV/Program\ Files + + # nsis + NSIS_BIN=$PRG_FLS/NSIS + # java + JAVA_BIN=$PROGRAMFILES/Java/jdk1.7.0_02/bin + + ## + ## MS SDK + ## + + CYGWIN=nowinsymlinks + + VISUAL_STUDIO_ROOT=$PRG_FLS/Microsoft\ Visual\ Studio\ 12.0 + WIN_VISUAL_STUDIO_ROOT="C:\\Program Files\\Microsoft Visual Studio 12.0" + SDK=$PRG_FLS/Windows\ Kits/8.1 + WIN_SDK="C:\\Program Files\\Windows Kits\\8.1" + + PATH="$NSIS_BIN:\ + $VISUAL_STUDIO_ROOT/VC/bin:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86 + /usr/local/bin:/usr/bin:/bin:\ + /cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS:\ + /cygdrive/c/WINDOWS/system32/Wbem:\ + $JAVA_BIN" + + LIBPATH="$WIN_VISUAL_STUDIO_ROOT\\VC\\lib" + + LIB="$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\;$WIN_SDK\\lib\\winv6.3\\um\\x86" + + INCLUDE="$WIN_VISUAL_STUDIO_ROOT\\VC\\include\\;$WIN_SDK\\include\\shared\\;\ + $WIN_SDK\\include\\um;$WIN_SDK\\include\\winrt\\;$WIN_SDK\\include\\um\\gl" + + export CYGWIN PATH LIBPATH LIB INCLUDE + + If you're using MinGW's MSYS instead, you need to change the `C_DRV` setting, + which would read: + + C_DRV=/c + + and you also need to change the PATH environment variable to: + + MINGW_BIN=/c/MinGW/bin + + + PATH="$NSIS_BIN:\ + $VISUAL_STUDIO_ROOT/VC/bin:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86:/usr/local/bin:\ + $MINGW_BIN:\ + /bin:/c/Windows/system32:/c/Windows:\ + /c/Windows/System32/Wbem:\ + $JAVA_BIN" + + For MSYS2 you use the same `C_DRV` and PATH as for MSYS, only update the `MINGW_BIN`: + + MINGW_BIN=/mingw32/bin + + + If you are building a 64 bit version of Erlang, you should set up + PATHs etc a little differently. We have two templates to make things + work in both Cygwin and MSYS but needs editing to work with MSYS2 (see the + comments in the script). + The following one is for 32 bits: + + make_winpath() + { + P=$1 + if [ "$IN_CYGWIN" = "true" ]; then + cygpath -d "$P" + else + (cd "$P" && /bin/cmd //C "for %i in (".") do @echo %~fsi") + fi + } + + make_upath() + { + P=$1 + if [ "$IN_CYGWIN" = "true" ]; then + cygpath "$P" + else + echo "$P" | /bin/sed 's,^\([a-zA-Z]\):\\,/\L\1/,;s,\\,/,g' + fi + } + + # Some common paths + if [ -x /usr/bin/msys-?.0.dll ]; then + # Without this the path conversion won't work + COMSPEC='C:\Windows\System32\cmd.exe' + MSYSTEM=MINGW32 # Comment out this line if in MSYS2 + export MSYSTEM COMSPEC + # For MSYS2: Change /mingw/bin to the msys bin dir on the line below + PATH=/usr/local/bin:/mingw/bin:/bin:/c/Windows/system32:\ + /c/Windows:/c/Windows/System32/Wbem + C_DRV=/c + IN_CYGWIN=false + else + PATH=/ldisk/overrides:/usr/local/bin:/usr/bin:/bin:\ + /usr/X11R6/bin:/cygdrive/c/windows/system32:\ + /cygdrive/c/windows:/cygdrive/c/windows/system32/Wbem + C_DRV=/cygdrive/c + IN_CYGWIN=true + fi + + obe_otp_gcc_vsn_map=" + .*=>default + " + obe_otp_64_gcc_vsn_map=" + .*=>default + " + # Program Files + PRG_FLS=$C_DRV/Program\ Files + + # Visual Studio + VISUAL_STUDIO_ROOT=$PRG_FLS/Microsoft\ Visual\ Studio\ 12.0 + WIN_VISUAL_STUDIO_ROOT="C:\\Program Files\\Microsoft Visual Studio 12.0" + + # SDK + SDK=$PRG_FLS/Windows\ Kits/8.1 + WIN_SDK="C:\\Program Files\\Windows Kits\\8.1" + + # NSIS + NSIS_BIN=$PROGRAMFILES/NSIS + + # Java + JAVA_BIN=$PROGRAMFILES/Java/jdk1.7.0_02/bin + + ## The PATH variable should be Cygwin'ish + VCPATH= + $VISUAL_STUDIO_ROOT/VC/bin:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86 + + ## Microsoft SDK libs + LIBPATH=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib + + LIB=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\;$WIN_KITS\\lib\\winv6.3\\um\\x86 + + INCLUDE=$WIN_VISUAL_STUDIO_ROOT\\VC\\include\\;\ + $WIN_KITS\\include\\shared\\;$WIN_KITS\\include\\um;\ + $WIN_KITS\\include\\winrt\\;$WIN_KITS\\include\\um\\gl + + # Put nsis, c compiler and java in path + export PATH=$VCPATH:$PATH:$JAVA_BIN:$NSIS_BIN + + # Make sure LIB and INCLUDE is available for others + export LIBPATH LIB INCLUDE + + + + The first part of the 64 bit template is identical to the 32 bit one, + but there are some environment variable differences: + + # Program Files + PRG_FLS64=$C_DRV/Program\ Files + PRG_FLS32=$C_DRV/Program\ Files\ \(x86\) + + # Visual Studio + VISUAL_STUDIO_ROOT=$PRG_FLS32/Microsoft\ Visual\ Studio\ 12.0 + WIN_VISUAL_STUDIO_ROOT="C:\\Program Files (x86)\\Microsoft Visual Studio 12.0" + + # SDK + SDK=$PRG_FLS32/Windows\ Kits/8.1 + WIN_SDK="C:\\Program Files (x86)\\Windows Kits\\8.1" + + # NSIS + NSIS_BIN=$PROGRAMFILES/NSIS + # Java + JAVA_BIN=$PROGRAMFILES/Java/jdk1.7.0_02/bin + + ## The PATH variable should be Cygwin'ish + VCPATH= + $VISUAL_STUDIO_ROOT/VC/bin/amd64:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86 + + ## Microsoft SDK libs + LIBPATH=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\amd64 + + LIB=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\amd64\\;\ + $WIN_KITS\\lib\\winv6.3\\um\\x64 + + INCLUDE=$WIN_VISUAL_STUDIO_ROOT\\VC\\include\\;\ + $WIN_KITS\\include\\shared\\;$WIN_KITS\\include\\um;\ + $WIN_KITS\\include\\winrt\\;$WIN_KITS\\include\\um\\gl + + # Put nsis, c compiler and java in path + export PATH=$VCPATH:$PATH:$JAVA_BIN:$NSIS_BIN + + # Make sure LIB and INCLUDE is available for others + export LIBPATH LIB INCLUDE + + + Make sure to set the PATH so that NSIS and Microsoft SDK is found + before the MSYS/Cygwin tools and that Java is last in the PATH. + + Make a simple hello world and try to compile it with the `cl` + command from within bash. If that does not work, your environment + needs fixing. Remember, there should be + no backslashes in your path environment variable in Cygwin bash, + but LIB and INCLUDE should contain Windows style paths with + semicolon, drive letters and backslashes. + +* Sun's Java JDK 1.6.0 or later. Our Java code (jinterface, ic) is + written for JDK 1.6.0. Get it for Windows and install it, the JRE is + not enough. If you don't care about Java, you can skip this step. The + result will be that jinterface is not built. + + URL: <http://java.sun.com> + + Add javac *LAST* to your path environment in bash, in my case this means: + + `PATH="$PATH:/cygdrive/c/Program Files/Java/jdk1.7.0_02/bin"` + + No `CLASSPATH` or anything is needed. Type `javac` in the bash prompt + and you should get a list of available Java options. Make sure, e.g by + typing `type java`, that you use the Java you installed. Note however that + Cygwin's/MinGW's/MSYS2's `jar.exe` is used. That's why the JDK bin-directory should be + added last in the `PATH`. + +* Nullsoft NSIS installer system. You need this to build the self + installing package. It's a free open source installer that's much + nicer to use than the commercial Wise and Install shield + installers. This is the installer we use for commercial releases as + well. + + URL: <http://nsis.sourceforge.net/download> + + Install the lot, especially the modern user interface components, as + it's definitely needed. Put `makensis` in your path, in my case: + + PATH=/cygdrive/c/Program\ Files/NSIS:$PATH + + Type makensis at the bash prompt and you should get a list of options + if everything is OK. + +* OpenSSL. This is if you want the SSL and crypto applications to + compile (and run). There are prebuilt binaries, which you can just + download and install, available here: + + URL: <http://openssl.org/community/binaries.html> + + We would recommend using 1.0.2d. + +* Building with wxWidgets. Download wxWidgets-3.0.2 or higher. + + Install or unpack it to the pgm folder: + Cygwin: + `DRIVE:/PATH/cygwin/opt/local/pgm` + MSYS: + `DRIVE:/PATH/MinGW/msys/1.0/opt/local/pgm` + MSYS2: + `DRIVE:/PATH/msys<32/64>/opt/local/pgm` + + If the `wxUSE_POSTSCRIPT` isn't enabled in `<path\to\pgm>\wxMSW-3.0.2\include\wx\msw\setup.h`, + enable it. + + build: From a command prompt with the VC tools available (See the + instructions for OpenSSL build above for help on starting the + proper command prompt in RELEASE mode): + + C:\...\> cd <path\to\pgm>\wxMSW-3.0.2\build\msw + C:\...\> nmake BUILD=release SHARED=0 DIR_SUFFIX_CPU= -f makefile.vc + + Or - if building a 64bit version: + + C:\...\> cd <path\to\pgm>\wxMSW-3.0.2\build\msw + C:\...\> nmake TARGET_CPU=amd64 BUILD=release SHARED=0 DIR_SUFFIX_CPU= -f makefile.vc + +* Get the Erlang source distribution (from <http://www.erlang.org/download.html>). + The same as for Unix platforms. Preferably use tar from within Cygwin, MSYS or MSYS2 to + unpack the source tar.gz (`tar zxf otp_src_%OTP-REL%.tar.gz`). + + Set the environment `ERL_TOP` to point to the root directory of the + source distribution. Let's say I stood in `$HOME/src` and unpacked + `otp_src_%OTP-REL%.tar.gz`, I then add the following to `.profile`: + + ERL_TOP=$HOME/src/otp_src_%OTP-REL% + export $ERL_TOP + + +The Shell Environment +--------------------- + +So, if you have followed the instructions above, when you start a bash +shell, you should have an INCLUDE environment with a Windows style +path, a LIB environment variable also in Windows style, and finally a +PATH that let's you reach cl, makensis, javac etc from the +command prompt (use `which cl` etc to verify from bash). + +You should also have an `ERL_TOP` environment variable that is *Cygwin +style*, and points to a directory containing, among other files, the +script `otp_build`. + +A final massage of the environment is needed, and that is done by +the script `$ERL_TOP/otp_build`. Start bash and do the following, note +the "back-ticks" (\`), can be quite hard to get on some keyboards, but +pressing the back-tick key followed by the space bar might do it... + + $ cd $ERL_TOP + $ eval `./otp_build env_win32` + +If you're unable to produce back-ticks on your keyboard, you can use +the ksh variant: + + $ cd $ERL_TOP + $ eval $(./otp_build env_win32) + +If you are building a 64 bit version, you supply `otp_build` with an architecture parameter: + + $ cd $ERL_TOP + $ eval `./otp_build env_win32 x64` + + +This should do the final touch to the environment and building should +be easy after this. You could run `./otp_build env_win32` without +`eval` just to see what it does, and to see that the environment it +sets seems OK. The path is cleaned of spaces if possible (using DOS +style short names instead), the variables `OVERRIDE_TARGET`, `CC`, `CXX`, +`AR` and `RANLIB` are set to their respective wrappers and the directories +`$ERL_TOP/erts/etc/win32/<cygwin/msys>_tools/vc` and +`$ERL_TOP/erts/etc/win32/<cygwin/msys>_tool` are added first in the PATH. + +Now you can check which erlc you have by writing `type erlc` in your shell. +It should reside in `$ERL_TOP/erts/etc/win32/cygwin_tools` +or `$ERL_TOP/erts/etc/win32/msys_tools`. + + +Building and Installing +----------------------- + +Building is easiest using the `otp_build` script: + + $ ./otp_build autoconf # Ignore the warning blob about versions of autoconf + $ ./otp_build configure <optional configure options> + $ ./otp_build boot -a + $ ./otp_build release -a <installation directory> + $ ./otp_build installer_win32 <installation directory> # optional + +Now you will have a file called `otp_win32_%OTP-REL%.exe` or `otp_win64_%OTP-REL%.exe` +in the `<installation directory>`, i.e. `$ERL_TOP/release/win32`. + +Lets get into more detail: + +1. `$ ./otp_build autoconf` - This step rebuilds the configure scripts + to work correctly in your environment. In an ideal world, this + would not be needed, but alas, we have encountered several + incompatibilities between our distributed configure scripts (generated + on a Linux platform) and the Cygwin/MSYS/MSYS2 environment over the + years. Running autoconf in Cygwin/MSYS/MSYS2 ensures that the configure + scripts are generated in a compatible way and that they will work well + in the next step. + +2. `$ ./otp_build configure` - This runs the newly generated configure + scripts with options making configure behave nicely. The target machine + type is plainly `win32`, so a lot of the configure-scripts recognize + this awkward target name and behave accordingly. The CC variable also + makes the compiler be `cc.sh`, which wraps MSVC++, so all configure + tests regarding the C compiler gets to run the right compiler. A lot of + the tests are not needed on Windows, but we thought it best to run the + whole configure anyway. + +3. `$ ./otp_build boot -a` - This uses the bootstrap directory (shipped + with the source, `$ERL_TOP/bootstrap`) to build a complete OTP + system. When this is done you can run erl from within the source tree; + just type `$ERL_TOP/bin/erl` and you whould have the prompt. + +4. `$ ./otp_build release -a` - Builds a commercial release tree from the + source tree. The default is to put it in `$ERL_TOP/release/win32`. You can + give any directory as parameter (Cygwin style), but it doesn't really + matter if you're going to build a self extracting installer too. + +5. `$ ./otp_build installer_win32` - Creates the self extracting installer executable. + The executable `otp_win32_%OTP-REL%.exe` or `otp_win64_%OTP-REL%.exe` will be placed + in the top directory of the release created in the previous step. If + no release directory is specified, the release is expected to have + been built to `$ERL_TOP/release/win32`, which also will be the place + where the installer executable will be placed. If you specified some + other directory for the release (i.e. `./otp_build release -a + /tmp/erl_release`), you're expected to give the same parameter here, + (i.e. `./otp_build installer_win32 /tmp/erl_release`). You need to have + a full NSIS installation and `makensis.exe` in your path for this to + work. Once you have created the installer, you can run it to + install Erlang/OTP in the regular way, just run the executable and + follow the steps in the installation wizard. To get all default settings + in the installation without any questions asked, you run the executable + with the parameter `/S` (capital S) like in: + + $ cd $ERL_TOP + $ release/win32/otp_win32_%OTP-REL% /S + ... + + or + + $ cd $ERL_TOP + $ release/win32/otp_win64_%OTP-REL% /S + ... + + + and after a while Erlang/OTP-%OTP-REL% will have been installed in + `C:\Program Files\erl%ERTS-VSN%\`, with shortcuts in the menu etc. + + +Development +----------- + +Once the system is built, you might want to change it. Having a test +release in some nice directory might be useful, but you can also run +Erlang from within the source tree. The target `local_setup`, makes +the program `$ERL_TOP/bin/erl.exe` usable and it also uses all the OTP +libraries in the source tree. + +If you hack the emulator, you can build the emulator executable +by standing in `$ERL_TOP/erts/emulator` and do a simple + + $ make opt + +Note that you need to have run ``(cd $ERL_TOP && eval `./otp_build env_win32`)`` +in the particular shell before building anything on Windows. After +doing a make opt you can test your result by running `$ERL_TOP/bin/erl`. +If you want to copy the result to a release directory (say +`/tmp/erl_release`), you do this (still in `$ERL_TOP/erts/emulator`) + + $ make TESTROOT=/tmp/erl_release release + +That will copy the emulator executables. + +To make a debug build of the emulator, you need to recompile both +`beam.dll` (the actual runtime system) and `erlexec.dll`. Do like this + + $ cd $ERL_TOP + $ rm bin/win32/erlexec.dll + $ cd erts/emulator + $ make debug + $ cd ../etc + $ make debug + +and sometimes + + $ cd $ERL_TOP + $ make local_setup + +So now when you run `$ERL_TOP/erl.exe`, you should have a debug compiled +emulator, which you will see if you do a: + + 1> erlang:system_info(system_version). + +in the erlang shell. If the returned string contains `[debug]`, you +got a debug compiled emulator. + +To hack the erlang libraries, you simply do a `make opt` in the +specific "applications" directory, like: + + $ cd $ERL_TOP/lib/stdlib + $ make opt + +or even in the source directory... + + $ cd $ERL_TOP/lib/stdlib/src + $ make opt + +Note that you're expected to have a fresh Erlang in your path when +doing this, preferably the plain %OTP-REL% you have built in the previous +steps. You could also add `$ERL_TOP/bootstrap/bin` to your `PATH` before +rebuilding specific libraries. That would give you a good enough +Erlang system to compile any OTP erlang code. Setting up the path +correctly is a little bit tricky. You still need to have +`$ERL_TOP/erts/etc/win32/cygwin_tools/vc` and +`$ERL_TOP/erts/etc/win32/cygwin_tools` *before* the actual emulator +in the path. A typical setting of the path for using the bootstrap +compiler would be: + + $ export PATH=$ERL_TOP/erts/etc/win32/cygwin_tools/vc\ + :$ERL_TOP/erts/etc/win32/cygwin_tools:$ERL_TOP/bootstrap/bin:$PATH + +That should make it possible to rebuild any library without hassle... + +If you want to copy a library (an application) newly built, to a +release area, you do like with the emulator: + + $ cd $ERL_TOP/lib/stdlib + $ make TESTROOT=/tmp/erlang_release release + +Remember that: + +* Windows specific C-code goes in the `$ERL_TOP/erts/emulator/sys/win32`, + `$ERL_TOP/erts/emulator/drivers/win32` or `$ERL_TOP/erts/etc/win32`. + +* Windows specific erlang code should be used conditionally and the + host OS tested in *runtime*, the exactly same beam files should be + distributed for every platform! So write code like: + + case os:type() of + {win32,_} -> + do_windows_specific(); + Other -> + do_fallback_or_exit() + end, + +That's basically all you need to get going. + + +Using GIT +--------- + +You might want to check out versions of the source code from GitHUB. That is possible directly in Cygwin, but not in MSYS. There is a project MsysGIT: + +URL:<http://code.google.com/p/msysgit/> + +that makes a nice Git port. The msys prompt you get from MsysGIT is +however not compatible with the full version from MinGW, so you will +need to check out files using MsysGIT's command prompt and then switch +to a common MSYS command prompt for building. Also all test suites +cannot be built as MsysGIT/MSYS does not handle symbolic links. + + +Copyright and License +--------------------- + +%CopyrightBegin% + +Copyright Ericsson AB 2003-2015. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +%CopyrightEnd% + + + [1]: http://www.erlang.org/static/doc/mailinglist.html + [2]: http://bugs.erlang.org + [3]: https://github.com/erlang/otp + + [?TOC]: true diff --git a/HOWTO/INSTALL.md b/HOWTO/INSTALL.md new file mode 100644 index 0000000000..447a23128f --- /dev/null +++ b/HOWTO/INSTALL.md @@ -0,0 +1,934 @@ +Building and Installing Erlang/OTP +================================== + +Introduction +------------ + +This document describes how to build and install Erlang/OTP-%OTP-REL%. +Erlang/OTP should be possible to build from source on any Unix/Linux system, +including OS X. You are advised to read the whole document +before attempting to build and install Erlang/OTP. + +The source code can be downloaded from the official site of Erlang/OTP or GitHub. +* <http://www.erlang.org> +* <https://github.com/erlang/otp> + +Required Utilities +------------------ + +These are the tools you need in order to unpack and build Erlang/OTP. + +> *WARNING*: Please have a look at the [Known platform issues][] chapter +> before you start. + +### Unpacking ### + +* GNU unzip, or a modern uncompress. +* A TAR program that understands the GNU TAR format for long filenames. + +### Building ### + +* GNU `make` +* Compiler -- GNU C Compiler, `gcc` or the C compiler frontend for LLVM, `clang`. +* Perl 5 +* GNU `m4` -- If HiPE (native code) support is enabled. HiPE can be + disabled using `--disable-hipe` +* `ncurses`, `termcap`, or `termlib` -- The development headers and + libraries are needed, often known as `ncurses-devel`. Use + `--without-termcap` to build without any of these libraries. Note that + in this case only the old shell (without any line editing) can be used. +* `sed` -- Stream Editor for basic text transformation. + +#### Building in Git #### + +* GNU `autoconf` of at least version 2.59. Note that `autoconf` is not + needed when building an unmodified version of the released source. + +#### Building on OS X #### + +* Xcode -- Download and install via the Mac App Store. + Read about [Building on a Mac][] before proceeding. + +### Installing ### + +* An `install` program that can take multiple file names. + + +Optional Utilities +------------------ + +Some applications are automatically skipped if the dependencies aren't met. +Here is a list of utilities needed for those applications. You will +also find the utilities needed for building the documentation. + +### Building ### + +* OpenSSL -- The opensource toolkit for Secure Socket Layer + and Transport Layer Security. + Required for building the application `crypto`. + Further, `ssl` and `ssh` require a working crypto application and + will also be skipped if OpenSSL is missing. The `public_key` + application is available without `crypto`, but the functionality + will be very limited. + + The development package of OpenSSL including the header files are needed as well + as the binary command program `openssl`. At least version 0.9.8 of OpenSSL is required. + Read more and download from <http://www.openssl.org>. +* Oracle Java SE JDK -- The Java Development Kit (Standard Edition). + Required for building the application `jinterface` and parts of `ic` and `orber`. + At least version 1.6.0 of the JDK is required. + + Download from <http://www.oracle.com/technetwork/java/javase/downloads>. + We have also tested with IBM's JDK 1.6.0. +* X Windows -- Development headers and libraries are needed + to build the Erlang/OTP application `gs` on Unix/Linux. +* `flex` -- Headers and libraries are needed to build the flex + scanner for the `megaco` application on Unix/Linux. +* wxWidgets -- Toolkit for GUI applications. + Required for building the `wx` application. At least + version 3.0 of wxWidgets is required. + + Download from <http://sourceforge.net/projects/wxwindows/files/3.0.0/> + or get it from GitHub: <https://github.com/wxWidgets/wxWidgets> + + Further instructions on wxWidgets, read [Building with wxErlang][]. + + + +### Building Documentation ### + +* `xsltproc` -- A command line XSLT processor. + + A tool for applying XSLT stylesheets + to XML documents. Download xsltproc from + <http://xmlsoft.org/XSLT/xsltproc2.html>. + +* `fop` -- Apache FOP print formatter (requires Java). Can be downloaded + from <http://xmlgraphics.apache.org/fop>. + + + +How to Build and Install Erlang/OTP +----------------------------------- + +The following instructions are for building [the released source tar ball][]. + +The variable `$ERL_TOP` will be mentioned a lot of times. It refers to +the top directory in the source tree. More information about `$ERL_TOP` +can be found in the [make and $ERL_TOP][] section below. If you are +building in git you probably want to take a look at the [Building in Git][] +section below before proceeding. + +### Unpacking ### + +Start by unpacking the Erlang/OTP distribution file with your GNU +compatible TAR program. + + $ tar -zxf otp_src_%OTP-VSN%.tar.gz # Assuming bash/sh + +Now change directory into the base directory and set the `$ERL_TOP` variable. + + $ cd otp_src_%OTP-VSN% + $ export ERL_TOP=`pwd` # Assuming bash/sh + +### Configuring ### + +Run the following commands to configure the build: + + $ ./configure [ options ] + +> *NOTE*: If you are building Erlang/OTP from git you will need to run `./otp_build autoconf` to generate +> the configure scripts. + +By default, Erlang/OTP release will be installed in `/usr/local/{bin,lib/erlang}`. +If you for instance don't have the permission to install in the standard location, + you can install Erlang/OTP somewhere else. For example, to install in +`/opt/erlang/%OTP-VSN%/{bin,lib/erlang}`, use the `--prefix=/opt/erlang/%OTP-VSN%` option. + +On some platforms Perl may behave strangely if certain locales are +set. If you get errors when building, try setting the LANG variable: + + $ export LANG=C # Assuming bash/sh + + +### Building ### + +Build the Erlang/OTP release. + + $ make + + +### Testing ### + +Before installation you should test whether your build is working properly +by running our smoke test. The smoke test is a subset of the complete Erlang/OTP test suites. +First you will need to build and release the test suites. + + $ make release_tests + +This creates an additional folder in `$ERL_TOP/release` called `tests`. +Now, it's time to start the smoke test. + + $ cd release/tests/test_server + $ $ERL_TOP/bin/erl -s ts install -s ts smoke_test batch -s init stop + +To verify that everything is ok you should open `$ERL_TOP/release/tests/test_server/index.html` +in your web browser and make sure that there are zero failed test cases. + +> *NOTE*: On builds without `crypto`, `ssl` and `ssh` there is a failed test case +> for undefined functions. Verify that the failed test case log only shows calls +> to skipped applications. + +### Installing ### + +You are now ready to install the Erlang/OTP release! +The following command will install the release on your system. + + $ make install + + +### Running ### + +You should now have a working release of Erlang/OTP! +Jump to [System Principles][] for instructions on running Erlang/OTP. + + +### How to Build the Documentation ### + +Make sure you're in the top directory in the source tree. + + $ cd $ERL_TOP + +If you have just built Erlang/OTP in the current source tree, you have +already ran `configure` and do not need to do this again; otherwise, run +`configure`. + + $ ./configure [Configure Args] + +When building the documentation you need a full Erlang/OTP-%OTP-VSN% system in +the `$PATH`. + + $ export PATH=$ERL_TOP/bin:$PATH # Assuming bash/sh + +For the FOP print formatter, two steps must be taken: + +* Adding the location of your installation of `fop` in `$FOP_HOME`. + + $ export FOP_HOME=/path/to/fop/dir # Assuming bash/sh + +* Adding the `fop` script (in `$FOP_HOME`) to your `$PATH`, either by adding `$FOP_HOME` to `$PATH`, or by copying the `fop` script to a directory already in your `$PATH`. + +Build the documentation. + + $ make docs + +#### Build Issues #### + +We have sometimes experienced problems with Oracle's `java` running out of +memory when running `fop`. Increasing the amount of memory available +as follows has in our case solved the problem. + + $ export FOP_OPTS="-Xmx<Installed amount of RAM in MB>m" + +More information can be found at +* <http://xmlgraphics.apache.org/fop/0.95/running.html#memory>. + + +### How to Install the Documentation ### + +The documentation can be installed either using the `install-docs` target, +or using the `release_docs` target. + +* If you have installed Erlang/OTP using the `install` target, install + the documentation using the `install-docs` target. Install locations + determined by `configure` will be used. `$DESTDIR` can be used the + same way as when doing `make install`. + + $ make install-docs + +* If you have installed Erlang/OTP using the `release` target, install + the documentation using the `release_docs` target. You typically want + to use the same `RELEASE_ROOT` as when invoking `make release`. + + $ make release_docs RELEASE_ROOT=<release dir> + + +### Accessing the Documentation ### + +After installation you can access the documentation by + +* Reading man pages. Make sure that `erl` is referring to the + installed version. For example `/usr/local/bin/erl`. + Try viewing at the man page for Mnesia + + $ erl -man mnesia + +* Browsing the html pages by loading the page `/usr/local/lib/erlang/doc/erlang/index.html` + or `<BaseDir>/lib/erlang/doc/erlang/index.html` if the prefix option has been used. + + +### How to Install the Pre-formatted Documentation ### + +Pre-formatted [html documentation][] and [man pages][] can be downloaded from +* <http://www.erlang.org/download.html>. + +Extract the html archive in the installation directory. + + $ cd <ReleaseDir> + $ tar -zxf otp_html_%OTP-VSN%.tar.gz + +For `erl -man <page>` to work the Unix manual pages have to be +installed in the same way, i.e. + + $ cd <ReleaseDir> + $ tar -zxf otp_man_%OTP-VSN%.tar.gz + +Where `<ReleaseDir>` is + +* `<PrefixDir>/lib/erlang` if you have installed Erlang/OTP using + `make install`. +* `$DESTDIR<PrefixDir>/lib/erlang` if you have installed Erlang/OTP + using `make install DESTDIR=<TmpInstallDir>`. +* `RELEASE_ROOT` if you have installed using + `make release RELEASE_ROOT=<ReleaseDir>`. + + +Advanced configuration and build of Erlang/OTP +---------------------------------------------- + +If you want to tailor your Erlang/OTP build and installation, please read +on for detailed information about the individual steps. + +### make and $ERL\_TOP ### + +All the makefiles in the entire directory tree use the environment +variable `ERL_TOP` to find the absolute path of the installation. The +`configure` script will figure this out and set it in the top level +Makefile (which, when building, it will pass on). However, when +developing it is sometimes convenient to be able to run make in a +subdirectory. To do this you must set the `ERL_TOP` variable +before you run make. + +For example, assume your GNU make program is called `make` and you +want to rebuild the application `STDLIB`, then you could do: + + $ cd lib/stdlib; env ERL_TOP=<Dir> make + +where `<Dir>` would be what you find `ERL_TOP` is set to in the top level +Makefile. + +### otp\_build vs configure/make ### + +Building Erlang/OTP can be done either by using the `$ERL_TOP/otp_build` +script, or by invoking `$ERL_TOP/configure` and `make` directly. Building using +`otp_build` is easier since it involves fewer steps, but the `otp_build` build +procedure is not as flexible as the `configure`/`make` build procedure. The binary +releases for Windows that we deliver are built using `otp_build`. + +### Configuring ### + +The configure script is created by the GNU autoconf utility, which +checks for system specific features and then creates a number of makefiles. + +The configure script allows you to customize a number of parameters; +type `./configure --help` or `./configure --help=recursive` for details. +`./configure --help=recursive` will give help for all `configure` scripts in +all applications. + +One of the things you can specify is where Erlang/OTP should be installed. By +default Erlang/OTP will be installed in `/usr/local/{bin,lib/erlang}`. +To keep the same structure but install in a different place, `<Dir>` say, +use the `--prefix` argument like this: `./configure --prefix=<Dir>`. + +Some of the available `configure` options are: + +* `--prefix=PATH` - Specify installation prefix. + +* `--{enable,disable}-threads` - Thread support. This is enabled by default if possible. +* `--{enable,disable}-smp-support` - SMP support (enabled by default if + a usable POSIX thread library or native Windows threads is found) +* `--{enable,disable}-kernel-poll` - Kernel poll support (enabled by + default if possible) +* `--{enable,disable}-hipe` - HiPE support (enabled by default on supported + platforms) +* `--{enable,disable}-fp-exceptions` - Floating point exceptions (an + optimization for floating point operations). The default differs + depending on operating system and hardware platform. Note that by + enabling this you might get a seemingly working system that sometimes + fail on floating point operations. +* `--enable-m64-build` - Build 64-bit binaries using the `-m64` flag to + `(g)cc` +* `--enable-m32-build` - Build 32-bit binaries using the `-m32` flag to + `(g)cc` +* `--with-assumed-cache-line-size=SIZE` - Set assumed cache-line size in + bytes. Default is 64. Valid values are powers of two between and + including 16 and 8192. The runtime system use this value in order to + try to avoid false sharing. A too large value wastes memory. A to + small value will increase the amount of false sharing. +* `--{with,without}-termcap` - termcap (without implies that only the old + Erlang shell can be used) +* `--with-javac=JAVAC` - Specify Java compiler to use +* `--{with,without}-javac` - Java compiler (without implies that the + `jinterface` application won't be built) +* `--{enable,disable}-dynamic-ssl-lib` - Dynamic OpenSSL libraries +* `--{enable,disable}-builtin-zlib` - Use the built-in source for zlib. +* `--{with,without}-ssl` - OpenSSL (without implies that the `crypto`, + `ssh`, and `ssl` won't be built) +* `--with-ssl=PATH` - Specify location of OpenSSL include and lib +* `--with-ssl-incl=PATH` - Location of OpenSSL `include` directory, + if different than specified by `--with-ssl=PATH` +* `--with-ssl-rpath=yes|no|PATHS` - Runtime library path for OpenSSL. + Default is `yes`, which equates to a number of standard locations. If + `no`, then no runtime library paths will be used. Anything else should be + a comma separated list of paths. +* `--with-libatomic_ops=PATH` - Use the `libatomic_ops` library for atomic + memory accesses. If `configure` should inform you about no native atomic + implementation available, you typically want to try using the + `libatomic_ops` library. It can be downloaded from + <https://github.com/ivmai/libatomic_ops/>. +* `--disable-smp-require-native-atomics` - By default `configure` will + fail if an SMP runtime system is about to be built, and no implementation + for native atomic memory accesses can be found. If this happens, you are + encouraged to find a native atomic implementation that can be used, e.g., + using `libatomic_ops`, but by passing `--disable-smp-require-native-atomics` + you can build using a fallback implementation based on mutexes or spinlocks. + Performance of the SMP runtime system will however suffer immensely without + an implementation for native atomic memory accesses. +* `--enable-static-{nifs,drivers}` - To allow usage of nifs and drivers on OSs + that do not support dynamic linking of libraries it is possible to statically + link nifs and drivers with the main Erlang VM binary. This is done by passing + a comma separated list to the archives that you want to statically link. e.g. + `--enable-static-nifs=/home/$USER/my_nif.a`. The path has to be absolute and the + name of the archive has to be the same as the module, i.e. `my_nif` in the + example above. This is also true for drivers, but then it is the driver name + that has to be the same as the filename. You also have to define + `STATIC_ERLANG_{NIF,DRIVER}` when compiling the .o files for the nif/driver. + If your nif/driver depends on some other dynamic library, you now have to link + that to the Erlang VM binary. This is easily achieved by passing `LIBS=-llibname` + to configure. +* `--without-$app` - By default all applications in Erlang/OTP will be included + in a release. If this is not wanted it is possible to specify that Erlang/OTP + should be compiled without one or more applications, i.e. `--without-wx`. There is + no automatic dependency handling between applications. If you disable + an application that another application depends on, you also have to disable the + dependant application. +* `--enable-gettimeofday-as-os-system-time` - Force usage of `gettimeofday()` for + OS system time. +* `--enable-prefer-elapsed-monotonic-time-during-suspend` - Prefer an OS monotonic + time source with elapsed time during suspend. +* `--disable-prefer-elapsed-monotonic-time-during-suspend` - Do not prefer an OS + monotonic time source with elapsed time during suspend. +* `--with-clock-resolution=high|low` - Try to find clock sources for OS system + time, and OS monotonic time with higher or lower resolution than chosen by + default. Note that both alternatives may have a negative impact on the performance + and scalability compared to the default clock sources chosen. +* `--disable-saved-compile-time` - Disable saving of compile date and time + in the emulator binary. +* `--enable-dirty-schedulers` - Enable the **experimental** dirty schedulers + functionality. Note that the dirty schedulers functionality is experimental, + and **not supported**. This functionality **will** be subject to backward + incompatible changes. Note that you should **not** enable the dirty scheduler + functionality on production systems. It is only provided for testing. + This switch also imply `--enable-new-purge-strategy` (see below). +* `--enable-new-purge-strategy` - Enable the purge strategy that will be + introduced in ERTS version 9.0 (OTP 20). Note that this switch will be + removed in OTP 20. + +If you or your system has special requirements please read the `Makefile` for +additional configuration information. + +#### Atomic Memory Operations and the VM #### + +The VM with SMP support makes quite a heavy use of atomic memory operations. +An implementation providing native atomic memory operations is therefore very +important when building Erlang/OTP. By default the VM will refuse to build +if native atomic memory operations are not available. + +Erlang/OTP itself provides implementations of native atomic memory operations +that can be used when compiling with a `gcc` compatible compiler for 32/64-bit +x86, 32/64-bit SPARC V9, 32-bit PowerPC, or 32-bit Tile. When compiling with +a `gcc` compatible compiler for other architectures, the VM may be able to make +use of native atomic operations using the `__atomic_*` builtins (may be +available when using a `gcc` of at least version 4.7) and/or using the +`__sync_*` builtins (may be available when using a `gcc` of at least version +4.1). If only the `gcc`'s `__sync_*` builtins are available, the performance +will suffer. Such a configuration should only be used as a last resort. When +compiling on Windows using a MicroSoft Visual C++ compiler native atomic +memory operations are provided by Windows APIs. + +Native atomic implementation in the order preferred: +1. The implementation provided by Erlang/OTP. +2. The API provided by Windows. +3. The implementation based on the `gcc` `__atomic_*` builtins. +4. If none of the above are available for your architecture/compiler, you + are recommended to build and install [libatomic_ops][] before building + Erlang/OTP. The `libatomic_ops` library provides native atomic memory + operations for a variety of architectures and compilers. When building + Erlang/OTP you need to inform the build system of where the + `libatomic_ops` library is installed using the + `--with-libatomic_ops=PATH` `configure` switch. +5. As a last resort, the implementation solely based on the `gcc` + `__sync_*` builtins. This will however cause lots of expensive and + unnecessary memory barrier instructions to be issued. That is, + performance will suffer. The `configure` script will warn at the end + of its execution if it cannot find any other alternative than this. + +### Building ### + +Building Erlang/OTP on a relatively fast computer takes approximately +5 minutes. To speed it up, you can utilize parallel make with the `-j<num_jobs>` option. + + $ export MAKEFLAGS=-j8 # Assuming bash/sh + $ make + +If you've upgraded the source with a patch you may need to clean up from previous +builds before the new build. +Make sure to read the [Pre-built Source Release][] section below before doing a `make clean`. + +#### Within Git #### + +When building in a Git working directory you also have to have a GNU `autoconf` +of at least version 2.59 on your system, because you need to generate the +`configure` scripts before you can start building. + +The `configure` scripts are generated by invoking `./otp_build autoconf` in +the `$ERL_TOP` directory. The `configure` scripts also have to be regenerated +when a `configure.in` or `aclocal.m4` file has been modified. Note that when +checking out a branch a `configure.in` or `aclocal.m4` file may change +content, and you may therefore have to regenerate the `configure` scripts +when checking out a branch. Regenerated `configure` scripts imply that you +have to run `configure` and build again. + +> *NOTE*: Running `./otp_build autoconf` is **not** needed when building +> an unmodified version of the released source. + +Other useful information can be found at our GitHub wiki: +* <http://wiki.github.com/erlang/otp> + +#### OS X (Darwin) #### + +Make sure that the command `hostname` returns a valid fully qualified host +name (this is configured in `/etc/hostconfig`). Otherwise you might experience +problems when running distributed systems. + +If you develop linked-in drivers (shared library) you need to link using +`gcc` and the flags `-bundle -flat_namespace -undefined suppress`. You also +include `-fno-common` in `CFLAGS` when compiling. Use `.so` as the library +suffix. + +If you have Xcode 4.3, or later, you will also need to download +"Command Line Tools" via the Downloads preference pane in Xcode. + +#### Building with wxErlang #### + +If you want to build the `wx` application, you will need to get wxWidgets-3.0 +(`wxWidgets-3.0.0.tar.bz2` from <http://sourceforge.net/projects/wxwindows/files/3.0.0/>) or get it from github with bug fixes: + + $ git clone --branch WX_3_0_BRANCH [email protected]:wxWidgets/wxWidgets.git + +Be aware that the wxWidgets-3.0 is a new release of wxWidgets, it is not as +mature as the old releases and the OS X port still lags behind the other ports. + +Configure and build wxWidgets (on Mavericks - 10.9): + + $ ./configure --with-cocoa --prefix=/usr/local + or without support for old versions and with static libs + $ ./configure --with-cocoa --prefix=/usr/local --with-macosx-version-min=10.9 --disable-shared + $ make + $ sudo make install + $ export PATH=/usr/local/bin:$PATH + +Check that you got the correct wx-config + + $ which wx-config && wx-config --version-full + +Build Erlang/OTP + + $ export PATH=/usr/local/bin:$PATH + $ cd $ERL_TOP + $ ./configure + $ make + $ sudo make install + + +#### Pre-built Source Release #### + +The source release is delivered with a lot of platform independent +build results already pre-built. If you want to remove these pre-built +files, invoke `./otp_build remove_prebuilt_files` from the `$ERL_TOP` +directory. After you have done this, you can build exactly the same way +as before, but the build process will take a much longer time. + +> *WARNING*: Doing `make clean` in an arbitrary directory of the source +> tree, may remove files needed for bootstrapping the build. +> +> Doing `./otp_build save_bootstrap` from the `$ERL_TOP` directory before +> doing `make clean` will ensure that it will be possible to build after +> doing `make clean`. `./otp_build save_bootstrap` will be invoked +> automatically when `make` is invoked from `$ERL_TOP` with either the +> `clean` target, or the default target. It is also automatically invoked +> if `./otp_build remove_prebuilt_files` is invoked. +> +> If you need to verify the bootstrap beam files match the provided +> source files, use `./otp_build update_primary` to create a new commit that +> contains differences, if any exist. + +#### How to Build a Debug Enabled Erlang RunTime System #### + +After completing all the normal building steps described above a debug +enabled runtime system can be built. To do this you have to change +directory to `$ERL_TOP/erts/emulator`. + +In this directory execute: + + $ make debug FLAVOR=$FLAVOR + +where `$FLAVOR` is either `plain` or `smp`. The flavor options will +produce a beam.debug and beam.smp.debug executable respectively. The +files are installed along side with the normal (opt) versions `beam.smp` +and `beam`. + +To start the debug enabled runtime system execute: + + $ $ERL_TOP/bin/cerl -debug + +The debug enabled runtime system features lock violation checking, +assert checking and various sanity checks to help a developer ensure +correctness. Some of these features can be enabled on a normal beam +using appropriate configure options. + +There are other types of runtime systems that can be built as well +using the similar steps just described. + + $ make $TYPE FLAVOR=$FLAVOR + +where `$TYPE` is `opt`, `gcov`, `gprof`, `debug`, `valgrind`, or `lcnt`. +These different beam types are useful for debugging and profiling +purposes. + + +### Installing ### + +* Staged install using [DESTDIR][]. You can perform the install + phase in a temporary directory and later move the installation into + its correct location by use of the `DESTDIR` variable: + + $ make DESTDIR=<tmp install dir> install + + The installation will be created in a location prefixed by `$DESTDIR`. + It can, however, not be run from there. It needs to be moved into the + correct location before it can be run. If `DESTDIR` have not been set + but `INSTALL_PREFIX` has been set, `DESTDIR` will be set to + `INSTALL_PREFIX`. Note that `INSTALL_PREFIX` in pre R13B04 was buggy + and behaved as `EXTRA_PREFIX` (see below). There are lots of areas of + use for an installation procedure using `DESTDIR`, e.g. when creating + a package, cross compiling, etc. Here is an example where the + installation should be located under `/opt/local`: + + $ ./configure --prefix=/opt/local + $ make + $ make DESTDIR=/tmp/erlang-build install + $ cd /tmp/erlang-build/opt/local + $ # gnu-tar is used in this example + $ tar -zcf /home/me/my-erlang-build.tgz * + $ su - + Password: ***** + $ cd /opt/local + $ tar -zxf /home/me/my-erlang-build.tgz + +* Install using the `release` target. Instead of doing `make install` you + can create the installation in whatever directory you like using the + `release` target and run the `Install` script yourself. `RELEASE_ROOT` + is used for specifying the directory where the installation should be + created. This is what by default ends up under `/usr/local/lib/erlang` + if you do the install using `make install`. All installation paths + provided in the `configure` phase are ignored, as well as `DESTDIR`, + and `INSTALL_PREFIX`. If you want links from a specific `bin` directory + to the installation you have to set those up yourself. An example where + Erlang/OTP should be located at `/home/me/OTP`: + + $ ./configure + $ make + $ make RELEASE_ROOT=/home/me/OTP release + $ cd /home/me/OTP + $ ./Install -minimal /home/me/OTP + $ mkdir -p /home/me/bin + $ cd /home/me/bin + $ ln -s /home/me/OTP/bin/erl erl + $ ln -s /home/me/OTP/bin/erlc erlc + $ ln -s /home/me/OTP/bin/escript escript + ... + + The `Install` script should currently be invoked as follows in the + directory where it resides (the top directory): + + $ ./Install [-cross] [-minimal|-sasl] <ERL_ROOT> + + where: + + * `-minimal` Creates an installation that starts up a minimal amount + of applications, i.e., only `kernel` and `stdlib` are started. The + minimal system is normally enough, and is what `make install` uses. + * `-sasl` Creates an installation that also starts up the `sasl` + application. + * `-cross` For cross compilation. Informs the install script that it + is run on the build machine. + * `<ERL_ROOT>` - The absolute path to the Erlang installation to use + at run time. This is often the same as the current working directory, + but does not have to be. It can follow any other path through the + file system to the same directory. + + If neither `-minimal`, nor `-sasl` is passed as argument you will be + prompted. + +* Test install using `EXTRA_PREFIX`. The content of the `EXTRA_PREFIX` + variable will prefix all installation paths when doing `make install`. + Note that `EXTRA_PREFIX` is similar to `DESTDIR`, but it does *not* have + the same effect as `DESTDIR`. The installation can and have to be run + from the location specified by `EXTRA_PREFIX`. That is, it can be useful + if you want to try the system out, running test suites, etc, before doing + the real install without `EXTRA_PREFIX`. + +#### Symbolic Links in --bindir #### + +When doing `make install` and the default installation prefix is used, +relative symbolic links will be created from `/usr/local/bin` to all public +Erlang/OTP executables in `/usr/local/lib/erlang/bin`. The installation phase +will try to create relative symbolic links as long as `--bindir` and the +Erlang bin directory, located under `--libdir`, both have `--exec-prefix` as +prefix. Where `--exec-prefix` defaults to `--prefix`. `--prefix`, +`--exec-prefix`, `--bindir`, and `--libdir` are all arguments that can be +passed to `configure`. One can force relative, or absolute links by passing +`BINDIR_SYMLINKS=relative|absolute` as arguments to `make` during the install +phase. Note that such a request might cause a failure if the request cannot +be satisfied. + + +### Running ### + +#### Using HiPE #### + +HiPE supports the following system configurations: + +* x86: All 32-bit and 64-bit mode processors should work. + + * Linux: Fedora Core is supported. Both 32-bit and 64-bit modes are + supported. + + NPTL glibc is strongly preferred, or a LinuxThreads + glibc configured for "floating stacks". Old non-floating + stacks glibcs have a fundamental problem that makes HiPE + support and threads support mutually exclusive. + + * Solaris: Solaris 10 (32-bit and 64-bit) and 9 (32-bit) are supported. + The build requires a version of the GNU C compiler (gcc) + that has been configured to use the GNU assembler (gas). + Sun's x86 assembler is emphatically **not** supported. + + * FreeBSD: FreeBSD 6.1 and 6.2 in 32-bit and 64-bit modes should work. + + * OS X/Darwin: Darwin 9.8.0 in 32-bit mode should work. + +* PowerPC: All 32-bit 6xx/7xx(G3)/74xx(G4) processors should work. 32-bit + mode on 970 (G5) and POWER5 processors should work. + + * Linux (Yellow Dog) and OS X 10.4 are supported. + +* SPARC: All UltraSPARC processors running 32-bit user code should work. + + * Solaris 9 is supported. The build requires a `gcc` that has been + configured to use Sun's assembler and linker. Using the GNU assembler + but Sun's linker has been known to cause problems. + + * Linux (Aurora) is supported. + +* ARM: ARMv5TE (i.e. XScale) processors should work. Both big-endian and + little-endian modes are supported. + + * Linux is supported. + +HiPE is automatically enabled on the following systems: + +* x86 in 32-bit mode: Linux, Solaris, FreeBSD +* x86 in 64-bit mode: Linux, Solaris, FreeBSD +* PowerPC: Linux, Mac OSX +* SPARC: Linux +* ARM: Linux + +On other supported systems, see [Advanced Configure][] on how to enable HiPE. + +If you are running on a platform supporting HiPE and if you have not disabled +HiPE, you can compile a module into native code like this from the Erlang +shell: + + 1> c(Module, native). + +or + + 1> c(Module, [native|OtherOptions]). + +Using the erlc program, write like this + + $ erlc +native Module.erl + +The native code will be placed into the beam file and automatically loaded +when the beam file is loaded. + +To add hipe options, write like this from the Erlang shell: + + 1> c(Module, [native,{hipe,HipeOptions}|MoreOptions]). + +Use `hipe:help_options/0` to print out the available options. + + 1> hipe:help_options(). + +#### Running with GS #### + +The `gs` application requires the GUI toolkit Tcl/Tk to run. At least +version 8.4 is required. + +Known platform issues +--------------------- + +* Suse linux 9.1 is shipped with a patched GCC version 3.3.3, having the + rpm named `gcc-3.3.3-41`. That version has a serious optimization bug + that makes it unusable for building the Erlang emulator. Please + upgrade GCC to a newer version before building on Suse 9.1. Suse Linux + Enterprise edition 9 (SLES9) has `gcc-3.3.3-43` and is not affected. + +* `gcc-4.3.0` has a serious optimizer bug. It produces an Erlang emulator + that will crash immediately. The bug is supposed to be fixed in + `gcc-4.3.1`. + +* FreeBSD had a bug which caused `kqueue`/`poll`/`select` to fail to detect + that a `writev()` on a pipe has been made. This bug should have been fixed + in FreeBSD 6.3 and FreeBSD 7.0. NetBSD and DragonFlyBSD probably have or + have had the same bug. More information can be found at: + + * <http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sys_pipe.c> + * <http://lists.freebsd.org/pipermail/freebsd-arch/2007-September/006790.html> + +* `getcwd()` on Solaris 9 can cause an emulator crash. If you have + async-threads enabled you can increase the stack size of the + async-threads as a temporary workaround. See the `+a` command-line + argument in the documentation of `erl(1)`. Without async-threads the + emulator is not as vulnerable to this bug, but if you hit it without + async-threads the only workaround available is to enable async-threads + and increase the stack size of the async-threads. Oracle has however + released patches that fixes the issue: + + > Problem Description: 6448300 large mnttab can cause stack overrun + > during Solaris 9 getcwd + + More information can be found at: + * <https://getupdates.oracle.com/readme/112874-40> + * <https://getupdates.oracle.com/readme/114432-29> + +* `sed` on Solaris seem to have some problems. For example on + Solaris 8, the BSD `sed` and XPG4 `sed` should be avoided. + Make sure `/bin/sed` or `/usr/bin/sed` is used on the Solaris + platform. + + +Daily Build and Test +-------------------- + +At Ericsson we have a "Daily Build and Test" that runs on: + +* Solaris 8, 9 + * Sparc32 + * Sparc64 +* Solaris 10 + * Sparc32 + * Sparc64 + * x86 +* SuSE Linux/GNU 9.4, 10.1 + * x86 +* SuSE Linux/GNU 10.0, 10.1, 11.0 + * x86 + * x86\_64 +* openSuSE 11.4 (Celadon) + * x86\_64 (valgrind) +* Fedora 7 + * PowerPC +* Fedora 16 + * x86\_64 +* Gentoo Linux/GNU 1.12.11.1 + * x86 +* Ubuntu Linux/GNU 7.04, 10.04, 10.10, 11.04, 12.04 + * x86\_64 +* MontaVista Linux/GNU 4.0.1 + * PowerPC +* FreeBSD 10.0 + * x86 +* OpenBSD 5.4 + * x86\_64 +* OS X 10.5.8 (Leopard), 10.7.5 (Lion), 10.9.1 (Mavericks) + * x86 +* Windows XP SP3, 2003, Vista, 7 + * x86 +* Windows 7 + * x86\_64 + +We also have the following "Daily Cross Builds": + +* SuSE Linux/GNU 10.1 x86 -> SuSE Linux/GNU 10.1 x86\_64 +* SuSE Linux/GNU 10.1 x86\_64 -> Linux/GNU TILEPro64 + +and the following "Daily Cross Build Tests": + +* SuSE Linux/GNU 10.1 x86\_64 + + +Authors +------- + +Authors are mostly listed in the application's `AUTHORS` files, +that is `$ERL_TOP/lib/*/AUTHORS` and `$ERL_TOP/erts/AUTHORS`, +not in the individual source files. + + +Copyright and License +--------------------- + +%CopyrightBegin% + +Copyright Ericsson AB 1998-2015. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +%CopyrightEnd% + + + + + + [$ERL_TOP/HOWTO/INSTALL-CROSS.md]: INSTALL-CROSS.md + [$ERL_TOP/HOWTO/INSTALL-WIN32.md]: INSTALL-WIN32.md + [DESTDIR]: http://www.gnu.org/prep/standards/html_node/DESTDIR.html + [Building in Git]: #Advanced-configuration-and-build-of-ErlangOTP_Building_Within-Git + [Advanced Configure]: #Advanced-configuration-and-build-of-ErlangOTP_Configuring + [Pre-built Source Release]: #Advanced-configuration-and-build-of-ErlangOTP_Building_Prebuilt-Source-Release + [make and $ERL_TOP]: #Advanced-configuration-and-build-of-ErlangOTP_make-and-ERLTOP + [html documentation]: http://www.erlang.org/download/otp_doc_html_%OTP-VSN%.tar.gz + [man pages]: http://www.erlang.org/download/otp_doc_man_%OTP-VSN%.tar.gz + [the released source tar ball]: http://www.erlang.org/download/otp_src_%OTP-VSN%.tar.gz + [System Principles]: ../system_principles/system_principles + [Known platform issues]: #Known-platform-issues + [native build]: #How-to-Build-and-Install-ErlangOTP + [cross build]: INSTALL-CROSS.md + [Required Utilities]: #Required-Utilities + [Optional Utilities]: #Optional-Utilities + [Building on a Mac]: #Advanced-configuration-and-build-of-ErlangOTP_Building_OS-X-Darwin + [Building with wxErlang]: #Advanced-configuration-and-build-of-ErlangOTP_Building_Building-with-wxErlang + [libatomic_ops]: https://github.com/ivmai/libatomic_ops/ diff --git a/HOWTO/MARKDOWN.md b/HOWTO/MARKDOWN.md new file mode 100644 index 0000000000..e6773a0a8e --- /dev/null +++ b/HOWTO/MARKDOWN.md @@ -0,0 +1,273 @@ +Erlangish Markdown Text Files +============================= + +Introduction +------------ + +If you are looking for information on how to build and install Erlang/OTP you +want to read the [$ERL_TOP/HOWTO/INSTALL.md][] document instead of this document +(where `$ERL_TOP` is the top source directory in the source tree). + +All files with the `.md` suffix (as well as this file) are ordinary text files +written using a Markdown like notation, and can be read using an ordinary text +editor such as for example `emacs`. + +This document describes how `*.md` files in the Erlang/OTP source tree should +be written. + +> *NOTE*: Before modifying a `*.md` file read all of this document. + +Erlangish Markdown +------------------ + +We do not use Markdown straight out of the box. The Markdown syntax we use is +similar to the original Markdown but with a number of tweaks. The original +Markdown is documented at <http://daringfireball.net/projects/markdown/>. You +should read that documentation as well as this document before modifying any +Erlangish Markdown files in the Erlang/OTP source tree. + +The original Markdown syntax was designed for generating HTML code from an +"easy to read/easy to write" plain text. Instead of generating HTML we generate +XML that fits into our documentation, i.e. we generate Erlangish XML from +Erlangish Markdown. + +The `.md` suffix has been chosen since [github][] will generate HTML pages for +files with the `.md` suffix. Github does however not generate HTML according to +Erlangish Markdown, so some features do not work. The Erlangish Markdown +documents viewed at [our github repository][] will typically suffer from broken +links. The original Markdown script, gitub's Markdown, and our Erlangish +Markdown script will generate somewhat different results if you do not follow +indentation rules outlined in the Markdown documentation. You are encouraged to +try to write using a Markdown syntax that also looks nice on github. However, +it is *much* more important that the document is formatted correct in the +Erlang/OTP documentation. + +### Differences Between Markdown and Erlangish Markdown ### + +#### Missing Features #### + +This functionality is missing compared to Markdown version 1.0.1. Do not +depend on the fact that these features are missing. Some of them might appear +in the future. + +* No inline HTML. Currently no inline XML is allowed either. Inline XML might + be allowed in the future, but there is no real need for it since we use + Erlangish Markdown for "readme"s that have a main purpose of being readable + in plain text. + +* Backslash escapes all characters. + +* No support for "horizontal rules". + +* Links. + * No support for the "title" attribute. + * Automatic links does not support email addresses. + +* Images. + * No support for the "title" attribute. Specified "title" will however + be used as `<icaption>`. + * No support for the "alt" attribute. + +* Lists aren't supported inside block quotes. + +* Nested block quotes can be generated, but current DTD does not + support it. + +* Link and image definition names *are* case sensitive. + +#### Additional Features #### + +* Automatic anchors at each heading. + +* Optionally automatically generated table of contents. + +* Note blocks. + +* Warning blocks. + +#### Extra requirements #### + +* One and only one level 1 heading is allowed and required. + +* The level 1 heading must be the first heading in the document. + +* A level `X` heading must have a level `X-1` heading as parent heading. + +* Link and image definition names aren't allowed to begin with a + question mark (?) character. Names beginning with a question mark have + been reserved for other use. + +* The encoding of the file containing Erlangish Markdown should be + UTF-8. + +### Generated XML ### + +> *WARNING*: The `emd2exml` script will blindly generate XML code according +> to the Erlangish Markdown in a file. Successfully generated XML does **not** +> imply that the generated XML adheres to used DTDs. `emd2exml` does very +> seldom fail and can easily generate XML that will cause the documentation +> build to fail. You always have to keep in mind that the XML generated +> should fit the chapter DTD of Erlang/OTP. Also note that even though HTML +> generation succeeds the PDF generation might fail, etc. +> +> *Always build the Erlang/OTP documentation after modifying an Erlangish +> Markdown document!* + +A note about how we talk about "tags" below. When we say "generate(s) `<X>` +tags" this also imply that ending `</X>` tags will be generated at appropriate +places. Appropriate attributes to the `X` tag will also be generated. + +* Inline and reference style links will either generate `<seealso>` tags + or `<url>` tags. If a "://" character sequence is found in the URL an + `<url>` tag will be generated; otherwise, a `<seealso>` tag is generated. + +* Automatic links will only generate `<url>` tags. This since a + "://" character sequence have to be present in the URL in order + for the link to be identified. + +* Inline and reference style images will generate a `<image file="..."> + <icaption>...</icaption> </image>` sequence where the "title" will be + placed between `<icaption>` and `</icaption>`. + +* Block quotes generate `<quote>` tags. + +* If the first line of a top level block quote begins with a `> *NOTE*:` + character sequence, a `<note>` tag will be generated instead of a + `<blockquote>` tag. The note will span the entire block quote. + +* If the first line of a top level block quote begins with a `> *WARNING*:` + character sequence, a `<warning>` tag will be generated instead of a + `<blockquote>` tag. The warning will span the entire block quote. + +* Paragraphs will generate `<p>` tags. + +* Break line (double trailing white space) will generate `<br/>` tags. + +* An unordered list generates a `<list type="bulleted">` tag and `<item>` + tags for each item. + +* An ordered list generates a `<list type="ordered">` tag and `<item>` tags + for each item. + +* Code blocks will generate `<code type="none">` tags. + +* Code span (backticks) will generate `<c>` tags. + +* Emphasis (single `*` or `_`) will generate `<em>` tags. + +* Strong emphasis (double `*` or `_`) will generate `<strong>` tags. + +* The level 1 heading will cause the following to be generated: + + <?xml version="1.0" encoding="utf8" ?> + <!DOCTYPE chapter SYSTEM "chapter.dtd"> + <chapter> + <header> + <copyright> + ... + </copyright> + <legalnotice> + ... + </legalnotice> + + <title>...</title> + ... + <file>...</file> + </header> + + ... + + </chapter> + + The content of copyright section and the legalnotice section will + contain information from a \%CopyrightBegin\%, \%CopyrightEnd\% block + if such exist (see below). + +* A level `X` heading where `1 < X <= 6` will cause the the following + to be generated: + + <section> + <marker id="..."/> + <title>...</title> + ... + </section> + + The marker id is automatically generated as a combination of all parent + headings up to and including level 2 separated by a `_` character. As in + `<marker heading 2>_<marker heading 3>_ ... _<current marker heading>` + where each "marker heading" is constructed as follows. All characters a-z + and A-Z are kept as they are, space and tab characters are replaced by + `-` characters, and all other characters are dropped. + + This way it is relatively easy to make sure that all marker ids of a + document are unique, but there is of course no guarantee that they are. + + The upside of these auto generated markers is that we wont have to clutter + the document with XML or something else while being able to refer into + the document. The downside is that if you change a level 2 heading you + change a lot of marker ids which may break links into a document from + other documents. That is, *be careful* when changing headings in an + existing document. + +* A level `X` heading where `6 < X` will cause the the following + to be generated: + + <marker id="..."/> + <p><strong>...</strong></p> + ... + + Current DTD:s used don't support deeper levels of sections, and we + therefore simulate a section this way. The marker id is generated as for + a true section (see above). + +* If a section enclosed by \%CopyrightBegin\% and \%CopyrightEnd\% is + encountered, it will be interpreted as an EPL copyright and license, + and will be used in the header section of the document. The + \%CopyrightBegin\% and \%CopyrightEnd\% "tags" will be removed from + the output. + +* All occurrences of \%OTP-REL% will be replaced by current OTP release number + (e.g. 17). + +* All occurrences of \%OTP-VSN% will be replaced by current OTP version + (e.g. 17.0). + +* All occurrences of \%ERTS-VSN% will be replaced by current ERTS version + (e.g. 5.8). + +* Adding a `[?TOC]: true` line (optionally indented with three spaces) + anywhere in the document will cause a table of contents to be automatically + generated at the beginning of the generated document. + +* Unicode characters (encoded in UTF-8) are allowed and will be passed + through as is to the output file. + +Copyright and License +--------------------- + +%CopyrightBegin% + +Copyright Ericsson AB 2010-2013. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +%CopyrightEnd% + + + + [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md + [github]: http://github.com + [our github repository]: http://github.com/erlang/otp + + + [?TOC]: true diff --git a/HOWTO/OTP-PATCH-APPLY.md b/HOWTO/OTP-PATCH-APPLY.md new file mode 100644 index 0000000000..2aa31629ef --- /dev/null +++ b/HOWTO/OTP-PATCH-APPLY.md @@ -0,0 +1,144 @@ +Patching OTP Applications +========================= + +Introduction +------------ + +This document describes the process of patching an existing OTP +installation with one or more Erlang/OTP applications of newer versions +than already installed. The tool `otp_patch_apply` is available for this +specific purpose. It resides in the top directory of the Erlang/OTP +source tree. + +The `otp_patch_apply` tool utilizes the [runtime_dependencies][] tag in +the [application resource file][]. This information is used to determine +if the patch can be installed in the given Erlang/OTP installation +directory. + +Read more about the [version handling][] introduced in Erlang/OTP release +17, which also describes how to determine if an installation includes one +or more patched applications. + +If you want to apply patches of multiple OTP applications that resides +in different OTP versions, you have to apply these patches in multiple +steps. It is only possible to apply multiple OTP applications from the +same OTP version at once. + +Prerequisites +------------- + +It's assumed that the reader is familiar with +[building and installing Erlang/OTP][]. To be able to patch an +application, the following must exist: + +* An Erlang/OTP installation. + +* An Erlang/OTP source tree containing the updated applications that + you want to patch into the existing Erlang/OTP installation. + +Using otp\_patch\_apply +----------------------- + +> *WARNING*: Patching applications is a one-way process. +> Create a backup of your OTP installation directory before +> proceeding. + +First of all, build the OTP source tree at `$ERL_TOP` containing +the updated applications. + +> *NOTE*: Before applying a patch you need to do a *full* build +> of OTP in the source directory. + +If you are building in `git` you first need to generate the +`configure` scripts: + + $ ./otp_build autoconf + +Configure and build all applications in OTP: + + $ configure + $ make + +or + + $ ./otp_build configure + $ ./otp_build boot -a + +If you have installed documentation in the OTP installation, also +build the documentation: + + $ make docs + +After the successful build it's time to patch. The source tree directory, +the directory of the installation and the applications to patch are given +as arguments to `otp_patch_apply`. The dependencies of each application +are validated against the applications in the installation and the other +applications given as arguments. If a dependency error is detected, the +script will be aborted. + +The `otp_patch_apply` syntax: + + $ otp_patch_apply -s <Dir> -i <Dir> [-l <Dir>] [-c] [-f] [-h] \ + [-n] [-v] <App1> [... <AppN>] + + -s <Dir> -- OTP source directory that contains build results. + -i <Dir> -- OTP installation directory to patch. + -l <Dir> -- Alternative OTP source library directory path(s) + containing build results of OTP applications. + Multiple paths should be colon separated. + -c -- Cleanup (remove) old versions of applications + patched in the installation. + -f -- Force patch of application(s) even though + dependencies are not fulfilled (should only be + considered in a test environment). + -h -- Print help then exit. + -n -- Do not install documentation. + -v -- Print version then exit. + <AppX> -- Application to patch. + + Environment Variable: + ERL_LIBS -- Alternative OTP source library directory path(s) + containing build results of OTP applications. + Multiple paths should be colon separated. + +> *NOTE*: The complete build environment is required while running +> `otp_patch_apply`. + +> *NOTE*: All source directories identified by `-s` and `-l` should +> contain build results of OTP applications. + +For example, if the user wants to install patched versions of `mnesia` +and `ssl` built in `/home/me/git/otp` into the OTP installation +located in `/opt/erlang/my_otp` type + + $ otp_patch_apply -s /home/me/git/otp -i /opt/erlang/my_otp \ + mnesia ssl + +> *NOTE*: If the list of applications contains core applications, +> i.e `erts`, `kernel`, `stdlib` or `sasl`, the `Install` script in +> the patched Erlang/OTP installation must be rerun. + +The patched applications are appended to the list of installed +applications. Take a look at +`<InstallDir>/releases/OTP-REL/installed_application_versions`. + +Sanity check +------------ + +The application dependencies can be checked using the Erlang shell. +Application dependencies are verified among installed applications by +`otp_patch_apply`, but these are not necessarily those actually loaded. +By calling `system_information:sanity_check()` one can validate +dependencies among applications actually loaded. + + 1> system_information:sanity_check(). + ok + +Please take a look at the reference of [sanity_check()][] for more +information. + +[application resource file]: kernel:app +[runtime_dependencies]: kernel:app#runtime_dependencies +[building and installing Erlang/OTP]: INSTALL.md +[version handling]: ../system_principles/versions +[sanity_check()]: runtime_tools:system_information#sanity_check-0 diff --git a/HOWTO/SYSTEMTAP.md b/HOWTO/SYSTEMTAP.md new file mode 100644 index 0000000000..ce9c0b2f0c --- /dev/null +++ b/HOWTO/SYSTEMTAP.md @@ -0,0 +1,75 @@ +SystemTap and Erlang/OTP +======================== + +Introduction +------------ + +SystemTap is DTrace for Linux. In fact Erlang's SystemTap support +is build using SystemTap's DTrace compatibility's layer. For an +introduction to Erlang DTrace support read [$ERL_TOP/HOWTO/DTRACE.md][]. + +Requisites +---------- + +* Linux Kernel with UTRACE support + + check for UTRACE support in your current kernel: + + # grep CONFIG_UTRACE /boot/config-`uname -r` + CONFIG_UTRACE=y + + Fedora 16 is known to contain UTRACE, for most other Linux distributions + a custom build kernel will be required. + Check Fedora's SystemTap documentation for additional required packages + (e.g. Kernel Debug Symbols) + +* SystemTap > 1.6 + + A the time of writing this, the latest released version of SystemTap is + version 1.6. Erlang's DTrace support requires a MACRO that was introduced + after that release. So either get a newer release or build SystemTap from + git yourself (see: http://sourceware.org/systemtap/getinvolved.html) + +Building Erlang +--------------- + +Configure and build Erlang with SystemTap support: + + # ./configure --with-dynamic-trace=systemtap + whatever args you need + # make + +Testing +------- + +SystemTap, unlike DTrace, needs to know what binary it is tracing and has to +be able to read that binary before it starts tracing. Your probe script +therefor has to reference the correct beam emulator and stap needs to be able +to find that binary. +The examples are written for "beam", but other versions such as "beam.smp" or +"beam.debug.smp" might exist (depending on your configuration). Make sure you +either specify the full the path of the binary in the probe or your "beam" +binary is in the search path. + +All available probes can be listed like this: + + # stap -L 'process("beam").mark("*")' + +or: + + # PATH=/path/to/beam:$PATH stap -L 'process("beam").mark("*")' + + +Probes in the dtrace.so NIF library like this: + + # PATH=/path/to/dtrace/priv/lib:$PATH stap -L 'process("dtrace.so").mark("*")' + +Running SystemTap scripts +------------------------- + +Adjust the process("beam") reference to your beam version and attach the script +to a running "beam" instance: + + # stap /path/to/probe/script/port1.systemtap -x <pid of beam> + + + [$ERL_TOP/HOWTO/DTRACE.md]: DTRACE.md diff --git a/HOWTO/TESTING.md b/HOWTO/TESTING.md new file mode 100644 index 0000000000..34eaa68df8 --- /dev/null +++ b/HOWTO/TESTING.md @@ -0,0 +1,150 @@ +Testing Erlang/OTP +================== + +Before you start testing you need to have the Erlang release which you +are going to test in your path. See [$ERL_TOP/HOWTO/INSTALL.md][] for +instructions on how to build an Erlang release. + +Short version +------------- +Move to the top directory of the Erlang release you want to test, i.e. +cd /ldisk/work/otp + + export ERL_TOP=`pwd` + ./otp_build setup -a + export PATH=`pwd`/bin:$PATH + ./otp_build tests + cd release/tests/test_server + erl -s ts install -s ts run all_tests -s init stop + +Where are the tests +------------------- + +There are a lot of tests which test Erlang/OTP (as of 2012 about 12000) and +they are located in different places throughout the source tree. Below is a list +of the places where you can expect to find tests: + +* $ERL_TOP/lib/AppName/test/ +* $ERL_TOP/erts/test/ +* $ERL_TOP/erts/emulator/test/ +* $ERL_TOP/erts/epmd/test/ + +Writing tests +------------- + +All tests are [common_test][] suites and follow the same pattern as all +[common_test][] suites. However, a couple of corner cases are +handled by a test wrapper called `ts`. `ts` allows the test writer to put +`Makefile.src` and `Makefile.first` files in the [data_dir][] and a special +directory called `all_SUITE_data`. + +`Makefile.first` is run before any other Makefile and is typically used to +generate .hrl files which are needed by other test suites. At the moment only +the erl_interface tests use this feature and it should remain that way. + +`Makefile.src` is configured to a `Makefile` using the `variables` created when +[configuring the tests][]. These `Makefile`s are later run by the test suite +to compile whatever platform specific code the tests need to run. + +Releasing tests +--------------- + +If you cannot use [ct_run][] in the source tree you have to release the tests +into a common test directory. The easiest way to do this is to use `otp_build` +like this: + + export ERL_TOP=`pwd`; ./otp_build tests + +This will release all tests in Erlang/OTP to `$ERL_TOP/release/tests/`. If you +want to change the directory where the tests are released to use the `TESTROOT` +environmental variable. + +In the `$TESTROOT` you should now see *_test folders. These folders contain +everything needed to test Erlang/OTP and are platform independent; if you are +testing Erlang on multiple platforms you just have to release on one and copy +the tests to all other platforms. + +### Releasing cross tests + +For releasing tests in a cross compilation environment see [$ERL_TOP/HOWTO/INSTALL-CROSS.md][]. + +Configuring and Running tests +----------------------------- + +Running tests is done by first navigating to the `$TESTROOT/test_server` folder +created when you released the tests and then start `erl` in that directory. The +emulator flags specified will be used in the test runs. For example, if you want +to test using async threads you have to supply `+A 10` to `erl` when you start it. + +To configure and run the tests `ts` is used. `ts` is a wrapper module to +[common_test][] which takes care of configuration and build issues before +[common_test][] is started. + +`ts` has a lot of special options and functions which can be usefull when +testing Erlang/OTP. For a full listing issue `ts:help()` in the erlang shell. + +### Configuring the test environment + +Before running released tests you have to install them on the target system. +Installing the tests is done by calling `ts:install().` in the Erlang shell +which you intend to test from. `ts:install()` is basically a wrapper to a +configure script and some Erlang code which figures out what your system looks +like and what kind of emulator you are testing with. `ts:install()` can also +take some arguments if necessary, see `ts:help()` for details. + +All variables created by `ts:install()` are found in +`$TESTROOT/test_server/variables`. + +### Running the tests + +To run all test suites go to `$TESTROOT/test_server` fire up an Erlang shell and type: + + ts:run(). + +Note that running all tests will require several hours, so you may want to run +the test cases for a single application + + ts:run(Application, [batch]). + +or even part of the test suite for an application, for example + + ts:run(emulator, bs, [batch]). + +to run all test suite modules starting with `bs` (i.e. all modules that test +the bit syntax). + +To run a specific test case in a module, the full name of the module and test +case must be specified: + + ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]). + +Run `ts:help().` for more information. + +As of R14B02 it is also possibly to start all tests but the erl_interface tests +by invoking Common Test directly from the released applications test directory, +i.e. + + cd $TESTROOT/test_server + $ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse + +Running [ct_run][] from the command line still requires you to do the +`ts:install()` step above. + +Examining the results +--------------------- + +Open the file `release/tests/test_server/index.html` in a web browser. Or open +`release/tests/test_server/last_test.html` when a test suite is running to +examine the results so far for the currently executing test suite (in R14B02 and +later you want to open the `release/tests/test_server/all_runs.html` file to +get to the currently running test) + + [ct_run]: http://www.erlang.org/doc/man/ct_run.html + [ct hook]: http://www.erlang.org/doc/apps/common_test/ct_hooks_chapter.html + [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md + [$ERL_TOP/HOWTO/INSTALL-CROSS.md]: INSTALL-CROSS.md#testing-the-cross-compiled-system + [common_test]: http://www.erlang.org/doc/man/ct.html + [data_dir]: http://www.erlang.org/doc/apps/common_test/write_test_chapter.html#data_priv_dir + [configuring the tests]: #configuring-the-test-environment + + [?TOC]: true |