diff options
126 files changed, 4945 insertions, 3192 deletions
diff --git a/HOWTO/INSTALL-WIN32.md b/HOWTO/INSTALL-WIN32.md index 79d89551c0..067c939d7a 100644 --- a/HOWTO/INSTALL-WIN32.md +++ b/HOWTO/INSTALL-WIN32.md @@ -4,82 +4,110 @@ How to Build Erlang/OTP on Windows Introduction ------------ -This file describes how to build the Erlang emulator and the OTP -libraries on Windows. The instructions apply to versions of Windows -supporting the Cygwin emulated gnuish environment for Windows or the -Msys ditto. We've built on the following platforms: Windows 2003 -server, Windows XP Home/Professional, Windows Vista and Windows 7 (32 -and 64 bit). You can probably build on Windows 2000, but you will not -be able to install the latest Microsoft SDK, so you have to go back to -some earlier compiler. Any Windows95'ish platform will surely get you -into trouble, what I'm not sure of, but it certainly will... - -The procedure described uses either Cygwin or Msys as a build -environment, you run the bash shell in Cygwin/Msys and use gnu -make/configure/autoconf etc to do the build. The emulator C-source -code is, however, mostly compiled with Microsoft Visual C++™, -producing a native Windows binary. This is the same procedure as we -use to build the pre-built binaries. The fact that we use VC++ and not -gcc is explained further in the FAQ section. - -I describe the build procedure to make it possible for open source -customers to build the emulator, given that they have the needed -tools. The binary Windows releases is still a preferred alternative if -one does not have Microsoft's development tools and/or don't want to -install Cygwin or Msys. - -To use Cygwin/Msys, one needs basic experience from a Unix environment, if -one does not know how to set environment variables, run programs etc -in a Unix environment, one will be quite lost in the Cygwin os Msys -ditto. I can unfortunately not teach all the world how to use -Cygwin and bash, neither how to install Cygwin nor perform basic tasks -on a computer. Please refer to other documentation on the net for -help, or use the binary release instead if you have problems using the -tools. - -However, if you feel comfortable with the environment and build +This section describes how to build the Erlang emulator and the OTP +libraries on Windows. Note that the Windows binary releases are still +a preferred alternative if one does not have Microsoft’s development +tools and/or don’t want to install Cygwin, MSYS or MSYS2. + +The instructions apply to versions of Windows supporting the Cygwin +emulated gnuish environment or the MSYS or MSYS2 ditto. We’ve built on +the following platforms: Windows 2012, Windows 7, Windows 8 and Windows 10. +It’s probably possible to build on older platforms too, but you might +not be able to install the appropriate Microsoft SDK, Visual Studio or +OpenSSL, in which case you will need to go back to earlier compilers etc. + +The procedure described uses either Cygwin, MSYS or MSYS2 as a build +environment. You run the bash shell in Cygwin/MSYS/MSYS2 and use the gnu +make/configure/autoconf etc to do the build. The emulator C-source code +is, however, mostly compiled with Microsoft Visual C++™, producing a +native Windows binary. This is the same procedure as we use to build the +pre-built binaries. Why we use VC++ and not gcc is explained further in +the FAQ section. + +If you are not familiar with Cygwin, MSYS, MSYS2 or a Unix environment, +you’ll probably need to read up a bit on how that works. There are plenty of +documentation about this online. + +These instructions apply for both 32-bit and 64-bit Windows. Note that even +if you build a 64-bit version of Erlang, most of the directories and files +involved are still named win32. Some occurances of the name win64 are +however present. The installation file for a 64-bit Windows version of +Erlang, for example, is `otp_win64_%OTP-REL%.exe`. + +If you feel comfortable with the environment and build system, and have all the necessary tools, you have a great opportunity to make the Erlang/OTP distribution for Windows better. Please submit -any suggestions and patches to the appropriate [mailing lists] [1] to let +any suggestions to our [JIRA] [2] and patches to our [git project] [3] to let them find their way into the next version of Erlang. If making changes to the build system (like makefiles etc) please bear in mind that the same makefiles are used on Unix/VxWorks, so that your changes -don't break other platforms. That of course goes for C-code too, system +don't break other platforms. That of course goes for C-code too; system specific code resides in the `$ERL_TOP/erts/emulator/sys/win32` and `$ERL_TOP/erts/etc/win32` directories mostly. The `$ERL_TOP/erts/emulator/beam` directory is for common code. -Before the R9C release of Erlang/OTP, the Windows release was built -partly on a Unix (Solaris) box and partly on a Windows box, using Perl -hacks to communicate and sync between the two machines. R9C was the -first release ever built solely on Windows, where no Unix machine is -needed at all. Now we've used this build procedure for a couple of +We've used this build procedure for a couple of releases, and it has worked fine for us. Still, there might be all sorts of troubles on different machines and with different -setups. I'll try to give hints wherever I've encountered difficulties, +setups. We'll try to give hints wherever we've encountered difficulties, but please share your experiences by using the [erlang-questions] [1] -mailing list. I cannot of course help everyone with all -their problems, please try to solve the problems and submit -solutions/workarounds. Remember, it's all about sharing, not about -demanding... - -Starting with R15B, our build system runs both on Cygwin and Msys -(MinGW's fork of an early cygwin version). Msys is a smaller package -to install and may on some machines run slightly faster. If Cygwin -gives you trouble, try Msys instead, and v.v. Beginning with R15B -there is also a native 64bit version of Erlang for 64bit Windows 7 -(only). These instructions apply to both the 32bit VM and the 64bit -ditto. - -Note that even if you build a 64bit VM, most of the directories and -files involved are still named win32. You can view the name win32 as -meaning any windows version not beeing 16bit. A few occurences of the -name Win64 are however present in the system, for example the -installation file for a 64 bit windows version of Erlang is by default -named `otp_win64_<version>.exe`. - -Lets go then, I'll start with a little FAQ, based on in house questions -and misunderstandings. +mailing list. We cannot, of course, help everyone with all +their issues, so please try to solve such issues and submit +solutions/workarounds. + +Lets go then! We’ll start with a short version of the setup procedure, +followed by some FAQ, and then we’ll go into more details of the setup. + + +Short Version +-------------------------- +In the following sections, we've described as much as we could about the +installation of the tools needed. Once the tools are installed, building +is quite easy. We have also tried to make these instructions understandable +for people with limited Unix experience. Cygwin/MSYS/MSYS2 is a whole new +environment to some Windows users, why careful explanation of environment +variables etc seemed to be in place. + +This is the short story though, for the experienced and impatient: + + * Get and install complete Cygwin (latest), complete MinGW with MSYS or + complete MSYS2 + + * Install Visual Studio 12.0 (2013) + + * Install Microsofts Windows SDK 8.1 + + * Get and install Sun's JDK 1.6.0 or later + + * Get and install NSIS 2.01 or later (up to 2.46 tried and working) + + * Get, build and install OpenSSL 0.9.8r or later (up to 1.0.2d + tried & working) with static libs. + + * Get the Erlang source distribution (from + <http://www.erlang.org/download.html>) and unpack with + Cygwin's/MSYS's/MSYS2's `tar`. + + * Set `ERL_TOP` to where you unpacked the source distribution + + * `$ cd $ERL_TOP` + + * Modify PATH and other environment variables so that all these tools + are runnable from a bash shell. Still standing in `$ERL_TOP`, issue + the following commands (for 32-bit Windows, remove the x64 from the + first row and change `otp_win64_%OTP-REL%` to `otp_win32_%OTP-REL%` on + the last row): + + $ eval `./otp_build env_win32 x64` + $ ./otp_build autoconf + $ ./otp_build configure + $ ./otp_build boot -a + $ ./otp_build release -a + $ ./otp_build installer_win32 + $ release/win32/otp_win64_%OTP-REL% /S + + Voila! `Start->Programs->Erlang OTP %OTP-REL%->Erlang` starts the Erlang + Windows shell. Frequently Asked Questions @@ -88,12 +116,12 @@ Frequently Asked Questions * Q: So, now I can build Erlang using GCC on Windows? A: No, unfortunately not. You'll need Microsoft's Visual C++ - still, a Bourne-shell script (cc.sh) wraps the Visual C++ compiler + still. A Bourne-shell script (cc.sh) wraps the Visual C++ compiler and runs it from within the Cygwin environment. All other tools needed to build Erlang are free-ware/open source, but not the C compiler. The Windows SDK is however enough to build Erlang, you do not need to buy Visual C++, just download the SDK (SDK version - 7.1 == Visual studio 2010). + 8.1 == Visual studio 2013). * Q: Why haven't you got rid of VC++ then, you \*\*\*\*\*\*? @@ -106,18 +134,17 @@ Frequently Asked Questions mingw build will possibly be back, but as long as VC++ gives better performance, the commercial build will be a VC++ one. -* Q: OK, you need VC++, but now you've started to demand a very recent - (and expensive) version of Visual studio, not the old and stable VC++ - 6.0 that was used in earlier versions. Why? +* Q: OK, you need VC++, but now you've started to demand a quite recent + (and expensive) version of Visual Studio. Why? A: Well, it's not expensive, it's free (as in free beer). Just download and install the latest Windows SDK from Microsoft and all the tools you need are there. The included debugger (WinDbg) is - also quite usable, it's what I used when porting Erlang to 64bit - Windows. Another reason to use the latest Microsoft compilers is + also quite usable. That's what I used when porting Erlang to 64bit + Windows. Another reason to use later Microsoft compilers is DLL compatibility. DLL's using a new version of the standard library might not load if the VM is compiled with an old VC++ - version, why we should aim to use the latest freely available SDK + version. So we should aim to use the latest freely available SDK and compiler. * Q: Can/will I build a Cygwin binary with the procedure you describe? @@ -130,9 +157,7 @@ Frequently Asked Questions some problems. Fixing those problems might be easy or might be hard. I suggest you try yourself and share your experience. No one would be happier if a simple `./configure && make` would produce a fully fledged - Cygwin binary. Ericsson does however not pay me to do a Cygwin port, so - such a port would have to happen in spare time, which is a limited - resource... + Cygwin binary. * Q: Hah, I saw you, you used GCC even though you said you didn't! @@ -142,7 +167,7 @@ Frequently Asked Questions particular file, `beam_emu.c` benefits immensely from being able to use the GCC labels-as-values extension, which boosts emulator performance by up to 50%. That does unfortunately not (yet) mean - that all of OTP could be compiled using GCC, that particular + that all of OTP could be compiled using GCC. That particular source code does not do anything system specific and actually is adopted to the fact that GCC is used to compile it on Windows. @@ -152,229 +177,184 @@ Frequently Asked Questions A: No, never. The hassle of keeping the project files up to date and do all the steps that constitute an OTP build from within the VC++ GUI is simply not worth it, maybe even impossible. A VC++ project - file for Erlang/OTP will never happen, at least I will never make - one. Clicking around in super-multi-tab'd dialogs to add a file or - compiler option when it's so much easier in a makefile is simply not - my style. + file for Erlang/OTP will never happen. * Q: So how does it all work then? - A: Cygwin or Msys is the environment, which closely resembles the - environments found on any Unix machine. It's almost like you had a + A: Cygwin, MSYS or MSYS2 is the environment, which closely resembles the + environment found on any Unix machine. It's almost like you had a virtual Unix machine inside Windows. Configure, given certain parameters, then creates makefiles that are used by the - Cygwin/Msys gnu-make to built the system. Most of the actual - compilers etc are not, however, Cygwin/Msys tools, so I've written + environment's gnu-make to built the system. Most of the actual + compilers etc are not, however, Cygwin/MSYS/MSYS2 tools, so we've written a couple of wrappers (Bourne-shell scripts), which reside in `$ERL_TOP/etc/win32/cygwin_tools` and `$ERL_TOP/etc/win32/msys_tools`. They all do conversion of parameters and switches common in the Unix environment to fit the native Windows tools. Most notable is of course the paths, which - in Cygwin/Msys are Unix-like paths with "forward slashes" (/) and - no drive letters, the Cygwin specific command `cygpath` is used - for most of the path conversions in a Cygwin environment, other - tools are used (when needed) in the corresponding Msys + in Cygwin/MSYS/MSYS2 are Unix-like paths with "forward slashes" (/) and + no drive letters. The Cygwin specific command `cygpath` is used + for most of the path conversions in a Cygwin environment. Other + tools are used (when needed) in the corresponding MSYS and MSYS2 environment. Luckily most compilers accept forward slashes instead of backslashes as path separators, but one still have to get the drive letters etc right, though. The wrapper scripts are not general in - the sense that, for example, cc.sh would understand and translates - every possible gcc option and passes correct options to + the sense that, for example, cc.sh would understand and translate + every possible gcc option and pass correct options to cl.exe. The principle is that the scripts are powerful enough to allow building of Erlang/OTP, no more, no less. They might need - extensions to cope with changes during the development of Erlang, - that's one of the reasons I made them into shell-scripts and not - Perl-scripts, I believe they are easier to understand and change - that way. I might be wrong though, cause another reason I didn't - write them in Perl is because I've never liked Perl and my Perl - code is no pleasant reading... + extensions to cope with changes during the development of Erlang, and + that's one of the reasons we made them into shell-scripts and not + Perl-scripts. We believe they are easier to understand and change + that way. In `$ERL_TOP`, there is a script called `otp_build`. That script handles the hassle of giving all the right parameters to `configure`/`make` and also helps you set up the correct environment variables to work with - the Erlang source under Cygwin. + the Erlang source under Cygwin/MSYS/MSYS2. * Q: You use and need Cygwin, but then you haven't taken the time to port Erlang to the Cygwin environment but instead focus on your commercial release, is that really ethical? - A: No, not really, but see this as a step in the right direction. I'm - aiming at GCC compiled emulators and a Cygwin version, but I really - need to do other things as well... In time, but don't hold your - breath... + A: No, not really, but see this as a step in the right direction. * Q: Can I build something that looks exactly as the commercial release? - A: Yes, we use the exactly same build procedure. + A: Yes, we use the exact same build procedure. -* Q: Which version of Cygwin/Msys and other tools do you use then? +* Q: Which version of Cygwin/MSYS/MSYS2 and other tools do you use then? - A: For Cygwin and Msys alike, we try to use the latest releases + A: For Cygwin, MSYS and MSYS2 alike, we try to use the latest releases available when building. What versions you use shouldn't really - matter, I try to include workarounds for the bugs I've found in - different Cygwin/Msys releases, please help me add workarounds - for new Cygwin/Msys-related bugs as soon as you encounter - them. Also please do submit bug reports to the appropriate Cygwin - and/or Msys developers. The GCC we used for %OTP-REL% was version - 4.7.0 (MinGW 64bit) and 4.3.4 (Cygwin 32bit). We used VC++ 10.0 - (i.e. Visual studio 2010), Sun's JDK 1.5.0\_17 (32bit) and Sun's - JDK 1.7.0\_1 (64bit), NSIS 2.46, and Win32 OpenSSL 0.9.8r. Please + matter. We try to include workarounds for the bugs we've found in + different Cygwin/MSYS/MSYS2 releases. Please help us add workarounds + for new Cygwin/MSYS/MSYS2-related bugs as soon as you encounter + them. Also please do submit bug reports to the appropriate Cygwin, MSYS + and/or MSYS2 developers. The GCC we used for %OTP-REL% was version + 4.8.1 (MinGW 32bit) and 4.8.5 (MSYS2 64bit). We used VC++ 12.0 + (i.e. Visual studio 2013), Sun's JDK 1.6.0\_45 (32bit) and Sun's + JDK 1.7.0\_1 (64bit), NSIS 2.46, and Win32 OpenSSL 1.0.2d. Please read the next section for details on what you need. -* Q: Can you help me setup X in Cygwin? +* Q: Can you help me setup X in Cygwin/MSYS/MSYS2? - A: No, unfortunately I haven't got time to help with Cygwin related - user problems, please read Cygwin related web sites, newsgroups and + A: No, unfortunately we haven't got time to help with Cygwin/MSYS/MSYS2 + related user problems, please read related websites, newsgroups and mailing lists. -* Q: Why is the instruction so long? Is it really that complicated? - - A: Partly it's long because I babble too much, partly because I've - described as much as I could about the installation of the needed - tools. Once the tools are installed, building is quite easy. I also - have tried to make this instruction understandable for people with - limited Unix experience. Cygwin/Msys is a whole new environment to some - Windows users, why careful explanation of environment variables etc - seemed to be in place. The short story, for the experienced and - impatient is: - - * Get and install complete Cygwin (latest) or complete MinGW with msys - - * Install Microsofts Windows SDK 7.1 (and .Net 4) - - * Get and install Sun's JDK 1.5.0 or higher - - * Get and install NSIS 2.01 or higher (up to 2.46 tried and working) - - * Get, build and install OpenSSL 0.9.8r or higher (up to 1.0.0a - tried & working) with static libs. - - * Get the Erlang source distribution (from - <http://www.erlang.org/download.html>) and unpack with Cygwin's `tar`. - - * Set `ERL_TOP` to where you unpacked the source distribution - - * `$ cd $ERL_TOP` - - * Get (from <http://www.erlang.org/download/tcltk85_win32_bin.tar.gz>) - and unpack the prebuilt TCL/TK binaries for windows with cygwin tar, - standing in `$ERL_TOP` - - * Modify PATH and other environment variables so that all these tools - are runnable from a bash shell. Still standing in `$ERL_TOP`, issue - the following commands: - - $ eval `./otp_build env_win32` - $ ./otp_build autoconf - $ ./otp_build configure - $ ./otp_build boot -a - $ ./otp_build release -a - $ ./otp_build installer_win32 - $ release/win32/otp_win32_%OTP-REL% /S - - Voila! `Start->Programs->Erlang OTP %OTP-REL%->Erlang` starts the Erlang - Windows shell. - Tools you Need and Their Environment ------------------------------------ You need some tools to be able to build Erlang/OTP on Windows. Most -notably you'll need Cygwin or Msys and Microsofts Windows SDK, but -you also might want a Java compiler, the NSIS install system and -OpenSSL. Well' here's the list: +notably you'll need Cygwin, MSYS or MSYS2, Visual Studio and Microsofts +Windows SDK, but you might also want a Java compiler, the NSIS install +system and OpenSSL. Well, here's some information about the different +tools: * Cygwin, the very latest is usually best. Get all the development - tools and of course all the basic ditto. In fact getting the complete - package might be a good idea, as you'll start to love Cygwin after a - while if you're accustomed to Unix. Make sure to get jar and also make - sure *not* to install a Cygwin'ish Java... The Cygwin jar command is - used but Sun's Java compiler and virtual machine... + tools and of course all the basic ditto. Make sure to get jar and + also make sure *not* to install a Cygwin'ish Java, since the Cygwin + jar command is used but Sun's Java compiler and virtual machine. If you are going to build a 64bit Windows version, you should make - sure to get MinGW's 64bit gcc installed with cygwin. It's in one of + sure to get MinGW's 64bit gcc installed with Cygwin. It's in one of the development packages. URL: <http://www.cygwin.com> - Get the installer from the web site and use that to install - Cygwin. Be sure to have fair privileges. If you're on a NT domain you + Get the installer from the website and use it to install + Cygwin. Be sure to have fair privileges. If you're on an NT domain you should consider running `mkpasswd -d` and `mkgroup -d` after the installation to get the user databases correct. See their respective manual pages. - When you start you first bash shell, you will get an awful prompt. You + When you start your first bash shell, you will get an awful prompt. You might also have a `PATH` environment variable that contains backslashes and such. Edit `$HOME/.profile` and `$HOME/.bashrc` to set fair prompts - and set a correct PATH. Also do a `export SHELL` in `.profile`. For some + and a correct PATH. Also do an `export SHELL` in `.profile`. For some non-obvious reason the environment variable `$SHELL` is not exported in bash. Also note that `.profile` is run at login time and `.bashrc` when sub shells are created. You'll need to explicitly source `.bashrc` from `.profile` if you want the commands there to be run at login time (like - setting up aliases, shell functions and the like). I personally - usually do like this at the end of `.profile`: + setting up aliases, shell functions and the like). You can for example + do like this at the end of `.profile`: ENV=$HOME/.bashrc export ENV . $ENV - You might also, if you're a hard core type of person at least, want to - setup X-windows (XFree86), that might be as easy as running startx - from the command prompt and it might be much harder. Use Google to - find help... + You might also want to setup X-windows (XFree86). That might be as easy + as running startx from the command prompt and it might be much harder. + Use Google to find help. If you don't use X-windows, you might want to setup the Windows console window by selecting properties in the console system menu (upper left corner of the window, the Cygwin icon in the title bar). Especially setting a larger screen buffer size (lines) is useful as it gets you a scrollbar so you can see whatever error messages - that might appear... + that might appear. - If you want to use (t)csh instead of bash you're on your own, I - haven't tried and know of no one that has. I expect - that you use bash in all shell examples. + There are a few other shells available, but in all examples below we assume + that you use bash. -* Alternatively you download MinGW and Msys. You'll find the latest +* Alternatively you download MinGW and MSYS. You'll find the latest installer at: URL: <http://sourceforge.net/projects/mingw/files/Installer/mingw-get-inst/> - Make sure to install everything they've got. + Make sure to install the basic dev tools, but avoid the MinGW autoconf and + install the msys one instead. To be able to build the 64bit VM, you will also need the 64bit MinGW compiler from: - URL: <http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Automated%20Builds/> + URL: <http://sourceforge.net/projects/mingw-w64/files/latest/download?source=files> - The latest version should do it. Make sure you download the - `mingw-w64-bin_i686-mingw_<something>.zip`, not a linux + We've tried up to 1.0, but the latest version should do. Make sure you + download the `mingw-w64-bin_i686-mingw_<something>.zip`, not a linux version. You unzip the package on top of your MinGW installation (`c:\MinGW`) and that's it. - Setting up your environment in Msys is similar to setting it up in - Cygwin. +* A third alternative is to download and install MSYS2 from: -* Microsofts Windows SDK version 7.1 (corresponding to VC++ 10.0 and - Visual Studio 2010). You'll find it here: - - URL: <http://www.microsoft.com/download/en/details.aspx?id=8279> + URL: <https://msys2.github.io/> + + When you've followed the instructions there, you also need to install + these packages: autoconf, make, perl, and tar. You do so by running + the following in the msys console: + + pacman -S msys/autoconf msys/make msys/perl msys/tar + + You also need a gcc. If you installed the 64 bit MSYS2 you run: - but before you install that, you need to have .Net 4 installed, - you'll find that here: + mingw64/mingw-w64-x86_64-gcc - URL: <http://www.microsoft.com/download/en/details.aspx?id=17851> + And for 32 bit MSYS2: - Use the web installer for the SDK, at least when I tried - downloading the whole package as an image, I got SDK 7.0 instead, - which is not what you want... + pacman -S mingw32/mingw-w64-i686-gcc + pacman -S mingw-w64-i686-editrights - There will be a Windows command file in `%PROGRAMFILES%\Mirosoft - SDKs\Windows\v7.1\Bin\SetEnv.cmd` that set's the appropriate +* Visual Studio 2013 (Visual Studio 12.0). Download and run the web + installer from: + + https://www.visualstudio.com/ + +* Microsofts Windows SDK version 8.1 (corresponding to VC++ 12.0 and + Visual Studio 2013). You'll find it here: + + URL: <https://msdn.microsoft.com/en-us/windows/desktop/bg162891.aspx> + +* To help setup the environment, there is a bat file, + `%PROGRAMFILES%\Mirosoft Visual Studio 12.0\VC\vcvarsall.bat`, + that set's the appropriate environment for a Windows command prompt. This is not appropriate for bash, so you'll need to convert it to bash-style environments by editing your `.bash_profile`. In my case, where the SDK is installed in the default directory and `%PROGRAMFILES%` is `C:\Program Files`, the commands for setting up a 32bit build - environment (on a 64bit or 32bit machine) look like this (in cygwin): + environment (on a 64bit or 32bit machine) look like this (in Cygwin): # Some common paths C_DRV=/cygdrive/c @@ -383,309 +363,290 @@ OpenSSL. Well' here's the list: # nsis NSIS_BIN=$PRG_FLS/NSIS # java - JAVA_BIN=$PRG_FLS/Java/jdk1.6.0_16/bin + JAVA_BIN=$PROGRAMFILES/Java/jdk1.7.0_02/bin ## ## MS SDK ## - CYGWIN=nowinsymlinks - MVS10="$PRG_FILES/Microsoft Visual Studio 10.0" - WIN_MVS10="C:\\Program Files\\Microsoft Visual Studio 10.0" - SDK10="$PRG_FILES/Microsoft SDKs/Windows/v7.1" - WIN_SDK10="C:\\Program Files\\Microsoft SDKs\\Windows\\v7.1" + CYGWIN=nowinsymlinks + + VISUAL_STUDIO_ROOT=$PRG_FLS/Microsoft\ Visual\ Studio\ 12.0 + WIN_VISUAL_STUDIO_ROOT="C:\\Program Files\\Microsoft Visual Studio 12.0" + SDK=$PRG_FLS/Windows\ Kits/8.1 + WIN_SDK="C:\\Program Files\\Windows Kits\\8.1" PATH="$NSIS_BIN:\ - $MVS10/Common7/IDE:\ - $MVS10/Common7/Tools:\ - $MVS10/VC/Bin:\ - $MVS10/VC/Bin/VCPackages:\ - $SDK10/Bin/NETFX 4.0 Tools:\ - $SDK10/Bin:\ + $VISUAL_STUDIO_ROOT/VC/bin:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86 /usr/local/bin:/usr/bin:/bin:\ /cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS:\ /cygdrive/c/WINDOWS/system32/Wbem:\ $JAVA_BIN" - LIBPATH="$WIN_MVS10\\VC\\LIB" + LIBPATH="$WIN_VISUAL_STUDIO_ROOT\\VC\\lib" - LIB="$WIN_MVS10\\VC\\LIB;$WIN_SDK10\\LIB" + LIB="$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\;$WIN_SDK\\lib\\winv6.3\\um\\x86" - INCLUDE="$WIN_MVS10\\VC\\INCLUDE;$WIN_SDK10\\INCLUDE;$WIN_SDK10\\INCLUDE\\gl" + INCLUDE="$WIN_VISUAL_STUDIO_ROOT\\VC\\include\\;$WIN_SDK\\include\\shared\\; + $WIN_SDK\\include\\um;$WIN_SDK\\include\\winrt\\;$WIN_SDK\\include\\um\\gl" - export CYGWIN PATH LIBPATH LIB INCLUDE + export CYGWIN PATH LIBPATH LIB INCLUDE - If you're using Msys instead, the only thing you need to change is - the `C_DRV` setting, which would read: + If you're using MinGW's MSYS instead, you need to change the `C_DRV` setting, + which would read: C_DRV=/c - And of course you might need to change `C:\Program Files` etc if - you're using a non-english version of Windows (XP). Note that in - later versions of Windows, the national adoptions of the program - files directories etc are not on the file system but only in the - explorer, so even if explorer says that your programs reside in - e.g. `C:\Program`, they might still reside in `C:\Program Files` - in reality... - - If you are building a 64 bit version of Erlang, you should set up - PATHs etc a little differently. I use the following script to - make things work in both Cygwin and Msys: - - make_winpath() - { - P=$1 - if [ "$IN_CYGWIN" = "true" ]; then - cygpath -d "$P" - else - (cd "$P" && /bin/cmd //C "for %i in (".") do @echo %~fsi") - fi - } - - make_upath() - { - P=$1 - if [ "$IN_CYGWIN" = "true" ]; then - cygpath "$P" - else - echo "$P" | /bin/sed 's,^\([a-zA-Z]\):\\,/\L\1/,;s,\\,/,g' - fi - } - - # Some common paths - if [ -x /usr/bin/msysinfo ]; then - # Without this the path conversion won't work - COMSPEC='C:\Windows\SysWOW64\cmd.exe' - MSYSTEM=MINGW32 - export MSYSTEM COMSPEC - IN_CYGWIN=false - else - CYGWIN=nowinsymlinks - export CYGWIN - IN_CYGWIN=true - fi - - if [ "$IN_CYGWIN" = "true" ]; then - PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:\ - /cygdrive/c/windows/system32:/cygdrive/c/windows:/cygdrive/c/windows/system32/Wbem - else - PATH=/usr/local/bin:/mingw/bin:/bin:/c/Windows/system32:/c/Windows:\ - /c/Windows/System32/Wbem - fi - - if [ "$IN_CYGWIN" = "true" ]; then - C_DRV=/cygdrive/c - else - C_DRV=/c - fi - - PRG_FLS64=$C_DRV/Program\ Files - PRG_FLS32=$C_DRV/Program\ Files\ \(x86\) - VISUAL_STUDIO_ROOT32=$PRG_FLS32/Microsoft\ Visual\ Studio\ 10.0 - MS_SDK_ROOT64=$PRG_FLS64/Microsoft\ SDKs/Windows/v7.1 - - # Okay, now mangle the paths and get rid of spaces by using short names - WIN_VCROOT32=`make_winpath "$VISUAL_STUDIO_ROOT32"` - VCROOT32=`make_upath $WIN_VCROOT32` - WIN_SDKROOT64=`make_winpath "$MS_SDK_ROOT64"` - SDKROOT64=`make_upath $WIN_SDKROOT64` - WIN_PROGRAMFILES32=`make_winpath "$PRG_FLS32"` - PROGRAMFILES32=`make_upath $WIN_PROGRAMFILES32` - - WIN_PROGRAMFILES64=`make_winpath "$PRG_FLS64"` - PROGRAMFILES64=`make_upath $WIN_PROGRAMFILES64` - - # nsis - NSIS_BIN=$PROGRAMFILES32/NSIS - # java - JAVA_BIN=$PROGRAMFILES64/Java/jdk1.7.0_01/bin - - ## The PATH variable should be Unix'ish - VCPATH=$VCROOT32/Common7/IDE:$VCROOT32/VC/BIN/amd64:$VCROOT32/Common7/Tools:\ - $VCROOT32/VC/VCPackages:$SDKROOT64/bin/NETFX4~1.0TO/x64:$SDKROOT64/bin/x64:\ - $SDKROOT64/bin - - ## Microsoft SDK libs - - LIBPATH=$WIN_VCROOT32\\VC\\LIB\\amd64 - LIB=$WIN_VCROOT32\\VC\\LIB\\amd64\;$WIN_SDKROOT64\\LIB\\X64 - INCLUDE=$WIN_VCROOT32\\VC\\INCLUDE\;$WIN_SDKROOT64\\include\;\ - $WIN_SDKROOT64\\include\\gl - - # Put nsis, c compiler and java in path - PATH=$NSIS_BIN:$VCPATH:$PATH:$JAVA_BIN - - # Make sure LIB and INCLUDE is available for others - export PATH LIBPATH LIB INCLUDE - - All this is derived from the SetEnv.cmd command file mentioned - earlier. The bottom line is to set the PATH so that NSIS and - Microsoft SDK is found before the Msys/Cygwin tools and that Java - is last in the PATH. - - Make a simple hello world (maybe one that prints out - `sizeof(void *)`) and try to compile it with the `cl` command from within - bash. If that does not work, your environment needs fixing. Also - remember to fix up the PATH environment, especially old Erlang - installations might have inserted quoted paths that Cygwin/Msys - does not understand. Remove or correct such paths. There should be - no backslashes in your path environment variable in Cygwin bash, - but LIB and INCLUDE should contain Windows style paths with - semicolon, drive letters and backslashes. + and you also need to change the PATH environment variable to: -* Sun's Java JDK 1.5.0 or higher. Our Java code (jinterface, ic) is - written for JDK 1.5.0. Get it for Windows and install it, the JRE is - not enough. If you don't care about Java, you can skip this step, the - result will be that jinterface is not built. + MINGW_BIN=/c/MinGW/bin - URL: <http://java.sun.com> - Add javac *LAST* to your path environment in bash, in my case this means: + PATH="$NSIS_BIN:\ + $VISUAL_STUDIO_ROOT/VC/bin:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86:/usr/local/bin:\ + $MINGW_BIN:\ + /bin:/c/Windows/system32:/c/Windows:\ + /c/Windows/System32/Wbem:\ + $JAVA_BIN" - `PATH="$PATH:/cygdrive/c/Program Files/Java/jdk1.5.0_17/bin"` + For MSYS2 you use the same `C_DRV` and PATH as for MSYS, only update the `MINGW_BIN`: - No `CLASSPATH` or anything is needed. Type `javac` at the bash prompt - and you should get a list of available Java options. Make sure by - typing `type java` that you use the Java you installed. Note however that - Cygwin's `jar.exe` is used, that's why the JDK bin-directory should be - added last in the `PATH`. + MINGW_BIN=/mingw32/bin + -* Nullsoft NSIS installer system. You need this to build the self - installing package. It's a free open source installer that's much - nicer to use than the commercial Wise and Install shield - installers. This is the installer we use for commercial releases as - well from R9C an on. + If you are building a 64 bit version of Erlang, you should set up + PATHs etc a little differently. We have two templates to make things + work in both Cygwin and MSYS but needs editing to work with MSYS2 (see the + comments in the script). + The following one is for 32 bits: + + make_winpath() + { + P=$1 + if [ "$IN_CYGWIN" = "true" ]; then + cygpath -d "$P" + else + (cd "$P" && /bin/cmd //C "for %i in (".") do @echo %~fsi") + fi + } + + make_upath() + { + P=$1 + if [ "$IN_CYGWIN" = "true" ]; then + cygpath "$P" + else + echo "$P" | /bin/sed 's,^\([a-zA-Z]\):\\,/\L\1/,;s,\\,/,g' + fi + } - URL: <http://www.nullsoft.com/free/nsis> + # Some common paths + if [ -x /usr/bin/msys-?.0.dll ]; then + # Without this the path conversion won't work + COMSPEC='C:\Windows\System32\cmd.exe' + MSYSTEM=MINGW32 # Comment out this line if in MSYS2 + export MSYSTEM COMSPEC + # For MSYS2: Change /mingw/bin to the msys bin dir on the line below + PATH=/usr/local/bin:/mingw/bin:/bin:/c/Windows/system32:\ + /c/Windows:/c/Windows/System32/Wbem + C_DRV=/c + IN_CYGWIN=false + else + PATH=/ldisk/overrides:/usr/local/bin:/usr/bin:/bin:\ + /usr/X11R6/bin:/cygdrive/c/windows/system32:\ + /cygdrive/c/windows:/cygdrive/c/windows/system32/Wbem + C_DRV=/cygdrive/c + IN_CYGWIN=true + fi + + obe_otp_gcc_vsn_map=" + .*=>default + " + obe_otp_64_gcc_vsn_map=" + .*=>default + " + # Program Files + PRG_FLS=$C_DRV/Program\ Files - Install the lot, especially the modern user interface components, as - it's definitely needed. Put `makensis` in your path, in my case: + # Visual Studio + VISUAL_STUDIO_ROOT=$PRG_FLS/Microsoft\ Visual\ Studio\ 12.0 + WIN_VISUAL_STUDIO_ROOT="C:\\Program Files\\Microsoft Visual Studio 12.0" - PATH=/cygdrive/c/Program\ Files/NSIS:$PATH + # SDK + SDK=$PRG_FLS/Windows\ Kits/8.1 + WIN_SDK="C:\\Program Files\\Windows Kits\\8.1" - type makensis at the bash prompt and you should get a list of options - if everything is OK. + # NSIS + NSIS_BIN=$PROGRAMFILES/NSIS -* OpenSSL. This is if you want the SSL and crypto applications to - compile (and run). There are prebuilt binaries available, but I - strongly recommend building this yourself. It's quite easy. + # Java + JAVA_BIN=$PROGRAMFILES/Java/jdk1.7.0_02/bin - First get the source from + ## The PATH variable should be Cygwin'ish + VCPATH= + $VISUAL_STUDIO_ROOT/VC/bin:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86 - URL: <http://openssl.org/source/> + ## Microsoft SDK libs + LIBPATH=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib - I would recommend using 0.9.8r. + LIB=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\;$WIN_KITS\\lib\\winv6.3\\um\\x86 - Download the tar file and unpack it (using your bash prompt) into - a directory of your choise. + INCLUDE=$WIN_VISUAL_STUDIO_ROOT\\VC\\include\\;\ + $WIN_KITS\\include\\shared\\;$WIN_KITS\\include\\um;\ + $WIN_KITS\\include\\winrt\\;$WIN_KITS\\include\\um\\gl - You will need a Windowish Perl for the build. ActiveState has one: + # Put nsis, c compiler and java in path + export PATH=$VCPATH:$PATH:$JAVA_BIN:$NSIS_BIN - URL: <http://www.activestate.com/activeperl/downloads> + # Make sure LIB and INCLUDE is available for others + export LIBPATH LIB INCLUDE - Download and install that. Disable options to associate it with - the .pl suffix and/or adding things to PATH, they are not needed. - Now fire up the Microsoft Windows SDK command prompt in RELEASE - mode for the architecture you are going to build. The easiest is - to copy the shortcut from the SDKs start menu item and edit the - command line in the shortcut (Right click->Properties) to end with - `/Release`. Make sure the banner when you double click your - shortcut (the text in the resulting command window) says - `Targeting Windows XP x64 Release` if you are going to do a 64 bit - build and `Targeting Windows XP x86 Release` if you are building a - 32 bit version. - Now cd to where you unpacked the OpenSSL source using your Release - Windows command prompt (it should be on the same drive as where - you are going to install it if everything is to work smothly). + The first part of the 64 bit template is identical to the 32 bit one, + but there are some environment variable differences: - C:\> cd <some dir> + # Program Files + PRG_FLS64=$C_DRV/Program\ Files + PRG_FLS32=$C_DRV/Program\ Files\ \(x86\) - Add ActiveState (or some other windows perl, not cygwins) to your PATH: + # Visual Studio + VISUAL_STUDIO_ROOT=$PRG_FLS32/Microsoft\ Visual\ Studio\ 12.0 + WIN_VISUAL_STUDIO_ROOT="C:\\Program Files (x86)\\Microsoft Visual Studio 12.0" - C:\...\> set PATH=C:\Perl\bin;%PATH% + # SDK + SDK=$PRG_FLS32/Windows\ Kits/8.1 + WIN_SDK="C:\\Program Files (x86)\\Windows Kits\\8.1" - Or if you installed the 64bit perl: - - C:\...\> set PATH=C:\Perl64\bin;%PATH% + # NSIS + NSIS_BIN=$PROGRAMFILES/NSIS + # Java + JAVA_BIN=$PROGRAMFILES/Java/jdk1.7.0_02/bin - Configure OpenSSL for 32 bit: + ## The PATH variable should be Cygwin'ish + VCPATH= + $VISUAL_STUDIO_ROOT/VC/bin/amd64:\ + $VISUAL_STUDIO_ROOT/VC/vcpackages:\ + $VISUAL_STUDIO_ROOT/Common7/IDE:\ + $VISUAL_STUDIO_ROOT/Common7/Tools:\ + $SDK/bin/x86 - C:\...\> perl Configure VC-WIN32 --prefix=/OpenSSL + ## Microsoft SDK libs + LIBPATH=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\amd64 - Or for 64 bit: + LIB=$WIN_VISUAL_STUDIO_ROOT\\VC\\lib\\amd64\\;\ + $WIN_KITS\\lib\\winv6.3\\um\\x64 - C:\...\> perl Configure VC-WIN64A --prefix=/OpenSSL-Win64 + INCLUDE=$WIN_VISUAL_STUDIO_ROOT\\VC\\include\\;\ + $WIN_KITS\\include\\shared\\;$WIN_KITS\\include\\um;\ + $WIN_KITS\\include\\winrt\\;$WIN_KITS\\include\\um\\gl - Do some setup (for 32 bit): + # Put nsis, c compiler and java in path + export PATH=$VCPATH:$PATH:$JAVA_BIN:$NSIS_BIN - C:\...\> ms\do_ms + # Make sure LIB and INCLUDE is available for others + export LIBPATH LIB INCLUDE - The same for 64 bit: - C:\...\> ms\do_win64a + Make sure to set the PATH so that NSIS and Microsoft SDK is found + before the MSYS/Cygwin tools and that Java is last in the PATH. - Then build static libraries and install: + Make a simple hello world and try to compile it with the `cl` + command from within bash. If that does not work, your environment + needs fixing. Remember, there should be + no backslashes in your path environment variable in Cygwin bash, + but LIB and INCLUDE should contain Windows style paths with + semicolon, drive letters and backslashes. - C:\...\> nmake -f ms\nt.mak - C:\...\> nmake -f ms\nt.mak install +* Sun's Java JDK 1.6.0 or later. Our Java code (jinterface, ic) is + written for JDK 1.6.0. Get it for Windows and install it, the JRE is + not enough. If you don't care about Java, you can skip this step. The + result will be that jinterface is not built. + + URL: <http://java.sun.com> + + Add javac *LAST* to your path environment in bash, in my case this means: + + `PATH="$PATH:/cygdrive/c/Program Files/Java/jdk1.7.0_02/bin"` + + No `CLASSPATH` or anything is needed. Type `javac` in the bash prompt + and you should get a list of available Java options. Make sure, e.g by + typing `type java`, that you use the Java you installed. Note however that + Cygwin's/MinGW's/MSYS2's `jar.exe` is used. That's why the JDK bin-directory should be + added last in the `PATH`. + +* Nullsoft NSIS installer system. You need this to build the self + installing package. It's a free open source installer that's much + nicer to use than the commercial Wise and Install shield + installers. This is the installer we use for commercial releases as + well. + + URL: <http://nsis.sourceforge.net/download> + + Install the lot, especially the modern user interface components, as + it's definitely needed. Put `makensis` in your path, in my case: + + PATH=/cygdrive/c/Program\ Files/NSIS:$PATH + + Type makensis at the bash prompt and you should get a list of options + if everything is OK. + +* OpenSSL. This is if you want the SSL and crypto applications to + compile (and run). There are prebuilt binaries, which you can just + download and install, available here: + + URL: <http://openssl.org/community/binaries.html> - That's it - you now have your perfectly consistent static build of - openssl. If you want to get rid of any possibly patented - algorithms in the lib, just read up on the OpenSSL FAQ and follow - the instructions. + We would recommend using 1.0.2d. - The installation locations chosen are where configure will look - for OpenSSL, so try to keep them as is. - -* Building with wxWidgets. Download wxWidgets-3.0.2 or higher patch - release. +* Building with wxWidgets. Download wxWidgets-3.0.2 or higher. - Install or unpack it to `DRIVE:/PATH/cygwin/opt/local/pgm`. + Install or unpack it to the pgm folder: + Cygwin: + `DRIVE:/PATH/cygwin/opt/local/pgm` + MSYS: + `DRIVE:/PATH/MinGW/msys/1.0/opt/local/pgm` + MSYS2: + `DRIVE:/PATH/msys<32/64>/opt/local/pgm` - edit: `C:\cygwin\opt\local\pgm\wxMSW-3.0.2\include\wx\msw\setup.h` - enable `wxUSE_POSTSCRIPT` + If the `wxUSE_POSTSCRIPT` isn't enabled in `<path\to\pgm>\wxMSW-3.0.2\include\wx\msw\setup.h`, + enable it. build: From a command prompt with the VC tools available (See the instructions for OpenSSL build above for help on starting the proper command prompt in RELEASE mode): - C:\...\> cd C:\cygwin\opt\local\pgm\wxMSW-3.0.2\build\msw + C:\...\> cd <path\to\pgm>\wxMSW-3.0.2\build\msw C:\...\> nmake BUILD=release SHARED=0 DIR_SUFFIX_CPU= -f makefile.vc Or - if building a 64bit version: - C:\...\> cd C:\cygwin\opt\local\pgm\wxMSW-3.0.2\build\msw + C:\...\> cd <path\to\pgm>\wxMSW-3.0.2\build\msw C:\...\> nmake TARGET_CPU=amd64 BUILD=release SHARED=0 DIR_SUFFIX_CPU= -f makefile.vc -* The Erlang source distribution (from <http://www.erlang.org/download.html>). - The same as for Unix platforms. Preferably use tar from within Cygwin to +* Get the Erlang source distribution (from <http://www.erlang.org/download.html>). + The same as for Unix platforms. Preferably use tar from within Cygwin, MSYS or MSYS2 to unpack the source tar.gz (`tar zxf otp_src_%OTP-REL%.tar.gz`). - set the environment `ERL_TOP` to point to the root directory of the + Set the environment `ERL_TOP` to point to the root directory of the source distribution. Let's say I stood in `$HOME/src` and unpacked `otp_src_%OTP-REL%.tar.gz`, I then add the following to `.profile`: ERL_TOP=$HOME/src/otp_src_%OTP-REL% export $ERL_TOP -* The TCL/TK binaries. You could compile Tcl/Tk for windows yourself, - but you can get a stripped down version from our website which is - suitable to include in the final binary package. If you want to supply - tcl/tk yourself, read the instructions about how the tcl/tk tar file - used in the build is constructed under `$ERL_TOP/lib/gs/tcl`. The easy - way is to download <http://www.erlang.org/download/tcltk85_win32_bin.tar.gz> - and unpack it standing in the `$ERL_TOP` directory. This will create the - file `win32.tar.gz` in `$ERL_TOP/lib/gs/tcl/binaries`. - - One last alternative is to create a file named `SKIP` in the - `$ERL_TOP/lib/gs/` after configure is run, but that will give you an - erlang system without gs (which might be okay as you probably will use - wx anyway). - - Note that there is no special 64bit version of TCL/TK needed, you - can use the 32bit program even for a 64bit build. The Shell Environment --------------------- @@ -726,37 +687,18 @@ be easy after this. You could run `./otp_build env_win32` without sets seems OK. The path is cleaned of spaces if possible (using DOS style short names instead), the variables `OVERRIDE_TARGET`, `CC`, `CXX`, `AR` and `RANLIB` are set to their respective wrappers and the directories -`$ERL_TOP/erts/etc/win32/cygwin_tools/vc` and -`$ERL_TOP/erts/etc/win32/cygwin_tool` are added first in the PATH. - -Try now a `type erlc`. That should result in the erlc wrapper script -(which does not have the .sh extension, for reasons best kept -untold...). It should reside in `$ERL_TOP/erts/etc/win32/cygwin_tools` -or `$ERL_TOP/erts/etc/win32/msys_tools`. You could also try `which -cc.sh`, which `ar.sh` etc. +`$ERL_TOP/erts/etc/win32/<cygwin/msys>_tools/vc` and +`$ERL_TOP/erts/etc/win32/<cygwin/msys>_tool` are added first in the PATH. -Now you're ready to build... +Now you can check which erlc you have by writing `type erlc` in your shell. +It should reside in `$ERL_TOP/erts/etc/win32/cygwin_tools` +or `$ERL_TOP/erts/etc/win32/msys_tools`. Building and Installing ----------------------- -Now it's assumed that you have executed `` eval `./otp_build env_win32` `` or -`` eval `./otp_build env_win32 x64` `` for this particular shell... - -Building is easiest using the `otp_build` script. That script takes care -of running configure, bootstrapping etc on Windows in a simple -way. The `otp_build` script is the utility we use ourselves to build on -different platforms and it therefore contains code for all sorts of -platforms. The principle is, however, that for non-Unix platforms, one -uses `./otp_build env_<target>` to set up environment and then the -script knows how to build on the platform "by itself". You've already -run `./otp_build env_win32` in the step above, so now it's mostly like -we build on any platform. OK, here are then steps; Assuming you will -want to build a full installation executable with NSIS, you can omit -`<installation directory>` and the release will be copied to -`$ERL_TOP/release/win32`: and there is where the packed self installing -executable will reside too. +Building is easiest using the `otp_build` script: $ ./otp_build autoconf # Ignore the warning blob about versions of autoconf $ ./otp_build configure <optional configure options> @@ -764,18 +706,18 @@ executable will reside too. $ ./otp_build release -a <installation directory> $ ./otp_build installer_win32 <installation directory> # optional -Now you will have a file called `otp_win32_R12B.exe` in the -`<installation directory>`, i.e. `$ERL_TOP/release/win32`. +Now you will have a file called `otp_win32_%OTP-REL%.exe` or `otp_win64_%OTP-REL%.exe` +in the `<installation directory>`, i.e. `$ERL_TOP/release/win32`. Lets get into more detail: 1. `$ ./otp_build autoconf` - This step rebuilds the configure scripts - to work correctly in the cygwin environment. In an ideal world, this + to work correctly in your environment. In an ideal world, this would not be needed, but alas, we have encountered several incompatibilities between our distributed configure scripts (generated - on a Linux platform) and the cygwin environment over the - years. Running autoconf on cygwin ensures that the configure scripts - are generated in a cygwin-compatible way and that they will work well + on a Linux platform) and the Cygwin/MSYS/MSYS2 environment over the + years. Running autoconf in Cygwin/MSYS/MSYS2 ensures that the configure + scripts are generated in a compatible way and that they will work well in the next step. 2. `$ ./otp_build configure` - This runs the newly generated configure @@ -784,38 +726,21 @@ Lets get into more detail: this awkward target name and behave accordingly. The CC variable also makes the compiler be `cc.sh`, which wraps MSVC++, so all configure tests regarding the C compiler gets to run the right compiler. A lot of - the tests are not needed on Windows, but I thought it best to run the - whole configure anyway. The only configure option you might want to - supply is `--with-ssl`, which might be needed if you have built your - own OpenSSL distribution. The Shining Lights distribution should be - found automatically by `configure`, if that fails, add a - `--with-ssl=<dir>` that specifies the root directory of your OpenSSL - installation. + the tests are not needed on Windows, but we thought it best to run the + whole configure anyway. 3. `$ ./otp_build boot -a` - This uses the bootstrap directory (shipped with the source, `$ERL_TOP/bootstrap`) to build a complete OTP - system. It first builds an emulator and sets up a minimal OTP system - under `$ERL_TOP/bootstrap`, then starts to compile the different OTP - compilers to make the `$ERL_TOP/bootstrap` system potent enough to be - able to compile all Erlang code in OTP. Then, all Erlang and C code - under `$ERL_TOP/lib` is built using the bootstrap system, giving a - complete OTP system (although not installed). When this is done, one - can run Erlang from within the source tree, just type `$ERL_TOP/bin/erl` - and you should have a prompt. If you omit the -a flag, you'll get a - smaller system, that might be useful during development. Now - exit from Erlang and start making a release of the thing: + system. When this is done you can run erl from within the source tree; + just type `$ERL_TOP/bin/erl` and you whould have the prompt. 4. `$ ./otp_build release -a` - Builds a commercial release tree from the - source tree, default is to put it in `$ERL_TOP/release/win32`, you can + source tree. The default is to put it in `$ERL_TOP/release/win32`. You can give any directory as parameter (Cygwin style), but it doesn't really - matter if you're going to build a self extracting installer too. You - could of course build release to the final directory and then run - `./Install.exe` standing in the directory where the release was put, - that will create a fully functional OTP installation. But let's make - the nifty installer: - -5. `$ ./otp_build installer_win32` - Create the self extracting installer - executable. The executable `otp_win32_%OTP-REL%.exe` will be placed + matter if you're going to build a self extracting installer too. + +5. `$ ./otp_build installer_win32` - Creates the self extracting installer executable. + The executable `otp_win32_%OTP-REL%.exe` or `otp_win64_%OTP-REL%.exe` will be placed in the top directory of the release created in the previous step. If no release directory is specified, the release is expected to have been built to `$ERL_TOP/release/win32`, which also will be the place @@ -824,7 +749,7 @@ Lets get into more detail: /tmp/erl_release`), you're expected to give the same parameter here, (i.e. `./otp_build installer_win32 /tmp/erl_release`). You need to have a full NSIS installation and `makensis.exe` in your path for this to - work of course. Once you have created the installer, you can run it to + work. Once you have created the installer, you can run it to install Erlang/OTP in the regular way, just run the executable and follow the steps in the installation wizard. To get all default settings in the installation without any questions asked, you run the executable @@ -844,37 +769,17 @@ Lets get into more detail: and after a while Erlang/OTP-%OTP-REL% will have been installed in `C:\Program Files\erl%ERTS-VSN%\`, with shortcuts in the menu etc. - The necessary setup of an Erlang installation is actually done by the - program `Install.exe`, which resides in the release top. That program - creates `.ini`-files and copies the correct boot scripts. If one has - the correct directory tree (like after a `./otp_build release -a`), only - the running of `Install.exe` is necessary to get a fully functional - OTP. What the self extracting installer adds is (of course) the - possibility to distribute the binary easily, together with adding - shortcuts to the Windows start menu. There is also some adding of - entries in the registry, to associate `.erl` and `.beam` files with - Erlang and get nifty icons, but that's not something you'll really need - to run Erlang. The registry is also used to store uninstall information, - but if one has not used the self extracting installer, one cannot - (need not) do any uninstall, one just scratches the release directory - and everything is gone. Erlang/OTP does not *need* to put anything - in the Windows registry at all, and does not if you don't use the self - extracting installer. In other words the installer is pure cosmetics. - -> *NOTE*: Beginning with R9C, the Windows installer does *not* add Erlang -> to the system wide path. If one wants to have Erlang in the path, one -> has to add it by hand. Development ----------- Once the system is built, you might want to change it. Having a test -release in some nice directory might be useful, but you also can run +release in some nice directory might be useful, but you can also run Erlang from within the source tree. The target `local_setup`, makes the program `$ERL_TOP/bin/erl.exe` usable and it also uses all the OTP libraries in the source tree. -If you hack the emulator, you can then build the emulator executable +If you hack the emulator, you can build the emulator executable by standing in `$ERL_TOP/erts/emulator` and do a simple $ make opt @@ -923,12 +828,12 @@ or even in the source directory... $ cd $ERL_TOP/lib/stdlib/src $ make opt -Note that you're expected o have a fresh Erlang in your path when +Note that you're expected to have a fresh Erlang in your path when doing this, preferably the plain %OTP-REL% you have built in the previous steps. You could also add `$ERL_TOP/bootstrap/bin` to your `PATH` before -rebuilding specific libraries, that would give you a good enough +rebuilding specific libraries. That would give you a good enough Erlang system to compile any OTP erlang code. Setting up the path -correctly is a little bit tricky, you still need to have +correctly is a little bit tricky. You still need to have `$ERL_TOP/erts/etc/win32/cygwin_tools/vc` and `$ERL_TOP/erts/etc/win32/cygwin_tools` *before* the actual emulator in the path. A typical setting of the path for using the bootstrap @@ -963,57 +868,27 @@ Remember that: That's basically all you need to get going. + Using GIT --------- -You might want to check out versions of the source code from GitHUB. That is possible directly in cygwin, but not in Msys. There is a project MsysGIT: +You might want to check out versions of the source code from GitHUB. That is possible directly in Cygwin, but not in MSYS. There is a project MsysGIT: URL:<http://code.google.com/p/msysgit/> that makes a nice Git port. The msys prompt you get from MsysGIT is however not compatible with the full version from MinGW, so you will need to check out files using MsysGIT's command prompt and then switch -to a common Msys command prompt for building. Also all test suites -cannot be built as MsysGIT/Msys does not handle symbolic links. To -build test suites on Windows, you will need Cygwin for now. Hopefully -all symbolic links will disappear from our repository soon and this -issue will disappear. +to a common MSYS command prompt for building. Also all test suites +cannot be built as MsysGIT/MSYS does not handle symbolic links. -Final Words ------------ -My hope is that the possibility to build the whole system on Windows -will open up for free development on this platform too. There are many -things one might want to do better in the Windows version, like the -window-style command prompt as well as pure Cygwin porting. Although i -realize it's a much larger step to start building on Windows (with all -the software you need) than for instance on Linux, I sincerely hope -that some of you will make the effort and start submitting Windows -friendly patches. - -The first build system for Erlang using Cygwin on Windows was created -by Per Bergkvist. I haven't used his build system, but it's rumored to -be good. The idea to do this came from his work, so credit is well -deserved. - -Of course this would have been completely impossible without the -excellent Cygwin. The guys at Cygnus solutions and -Redhat deserve a huge THANKS! as well as all the other people in the -free software community who have helped in creating the magnificent -software that constitutes Cygwin. - -Also the people developing the alternative command prompt Msys and -the MinGW compiler are worth huge THANKS! The 64bit port would have -been impossible without the 64bit MinGW compiler. - -Good luck and Happy Hacking, -Patrik, OTP Copyright and License --------------------- %CopyrightBegin% -Copyright Ericsson AB 2003-2014. All Rights Reserved. +Copyright Ericsson AB 2003-2015. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -1030,6 +905,8 @@ limitations under the License. %CopyrightEnd% - [1]: http://www.erlang.org/faq.html "mailing lists" + [1]: http://www.erlang.org/static/doc/mailinglist.html + [2]: http://bugs.erlang.org + [3]: https://github.com/erlang/otp [?TOC]: true diff --git a/erts/configure.in b/erts/configure.in index 9ad1588b6c..20075b08c9 100644 --- a/erts/configure.in +++ b/erts/configure.in @@ -106,7 +106,6 @@ AC_CONFIG_HEADER($host/config.h:config.h.in include/internal/$host/ethread_heade dnl ---------------------------------------------------------------------- dnl Optional features. dnl ---------------------------------------------------------------------- -enable_child_waiter_thread=no ENABLE_ALLOC_TYPE_VARS= AC_SUBST(ENABLE_ALLOC_TYPE_VARS) @@ -1303,11 +1302,7 @@ else AC_MSG_RESULT(no) fi - disable_child_waiter_thread=no case $host_os in - solaris*) - enable_child_waiter_thread=yes - ;; linux*) AC_MSG_CHECKING([whether dlopen() needs to be called before first call to dlerror()]) if test "x$ETHR_THR_LIB_BASE_TYPE" != "xposix_nptl"; then @@ -1317,16 +1312,6 @@ else else AC_MSG_RESULT(no) fi - if test "x$ETHR_THR_LIB_BASE_TYPE" != "xposix_nptl"; then - # Child waiter thread cannot be enabled - disable_child_waiter_thread=yes - enable_child_waiter_thread=no - fi - ;; - win32) - # Child waiter thread cannot be enabled - disable_child_waiter_thread=yes - enable_child_waiter_thread=no ;; *) ;; @@ -1346,24 +1331,6 @@ else esac done EMU_THR_DEFS=$new_emu_thr_defs - - AC_MSG_CHECKING(whether the child waiter thread should be enabled) - if test $enable_child_waiter_thread = yes; then - AC_DEFINE(ENABLE_CHILD_WAITER_THREAD,[1], - [Define if you want to enable child waiter thread]) - AC_MSG_RESULT(yes) - else - case $ERTS_BUILD_SMP_EMU-$disable_child_waiter_thread in - yes-no) - AC_MSG_RESULT([yes on SMP build, but not on non-SMP build]);; - *-yes) - AC_DEFINE(DISABLE_CHILD_WAITER_THREAD,[1], - [Define if you want to disable child waiter thread]) - AC_MSG_RESULT(no);; - *) - AC_MSG_RESULT(no);; - esac - fi fi AC_SUBST(EMU_THR_LIB_NAME) @@ -1495,19 +1462,27 @@ dnl # if -lsocket doesn't work by itself. #-------------------------------------------------------------------- +tk_oldLibs=$LIBS erl_checkBoth=0 +SOCKET_LIBS="" AC_CHECK_FUNC(connect, erl_checkSocket=0, erl_checkSocket=1) if test "$erl_checkSocket" = 1; then - AC_CHECK_LIB(socket, main, LIBS="$LIBS -lsocket", erl_checkBoth=1) + AC_CHECK_LIB(socket, main, SOCKET_LIBS="-lsocket", erl_checkBoth=1) fi + if test "$erl_checkBoth" = 1; then - tk_oldLibs=$LIBS LIBS="$LIBS -lsocket -lnsl" - AC_CHECK_FUNC(accept, erl_checkNsl=0, [LIBS=$tk_oldLibs]) + AC_CHECK_FUNC(accept, SOCKET_LIBS="-lsocket -lnsl") fi -AC_CHECK_FUNC(gethostbyname, , AC_CHECK_LIB(nsl, main, [LIBS="$LIBS -lnsl"])) + +LIBS="$tk_oldLibs $SOCKET_LIBS" +AC_CHECK_FUNC(gethostbyname, , AC_CHECK_LIB(nsl, main, [SOCKET_LIBS="$SOCKET_LIBS -lnsl"])) AC_CHECK_FUNC(gethostbyname_r,have_gethostbyname_r=yes) +LIBS="$tk_oldLibs $SOCKET_LIBS" + +AC_SUBST(SOCKET_LIBS) + dnl dnl These gethostbyname thingies use old style AC_DEFINE for BC with ancient dnl autoconf... diff --git a/erts/doc/src/driver_entry.xml b/erts/doc/src/driver_entry.xml index c802693977..ae7f264d0c 100644 --- a/erts/doc/src/driver_entry.xml +++ b/erts/doc/src/driver_entry.xml @@ -437,7 +437,14 @@ typedef struct erl_drv_entry { <seealso marker="erl_driver#erl_drv_busy_msgq_limits">erl_drv_busy_msgq_limits()</seealso> function. </item> - </taglist> + <tag><c>ERL_DRV_FLAG_USE_INIT_ACK</c></tag> + <item>When this flag is given the linked-in driver has to manually + acknowledge that the port has been successfully started using + <seealso marker="erl_driver#erl_drv_init_ack">erl_drv_init_ack()</seealso>. + This allows the implementor to make the erlang:open_port exit with + badarg after some initial asynchronous initialization has been done. + </item> + </taglist> </item> <tag>void *handle2</tag> <item> diff --git a/erts/doc/src/erl_driver.xml b/erts/doc/src/erl_driver.xml index e81d38cb80..cade732c56 100644 --- a/erts/doc/src/erl_driver.xml +++ b/erts/doc/src/erl_driver.xml @@ -2131,6 +2131,53 @@ ERL_DRV_MAP int sz </func> <func> + <name><ret>void</ret><nametext>erl_drv_init_ack(ErlDrvPort port, ErlDrvData res)</nametext></name> + <fsummary>Acknowledge the start of the port</fsummary> + <desc> + <marker id="erl_drv_init_ack"></marker> + <p>Arguments:</p> + <taglist> + <tag><c>port</c></tag> + <item>The port handle of the port (driver instance) creating + doing the acknowledgment. + </item> + <tag><c>res</c></tag> + <item>The result of the port initialization. This can be the same values + as the return value of <seealso marker="driver_entry#start">start</seealso>, + i.e any of the error codes or the ErlDrvData that is to be used for this + port. + </item> + </taglist> + <p> + When this function is called the initiating erlang:open_port call is + returned as if the <seealso marker="driver_entry#start">start</seealso> + function had just been called. It can only be used when the + <seealso marker="driver_entry#driver_flags">ERL_DRV_FLAG_USE_INIT_ACK</seealso> + flag has been set on the linked-in driver. + </p> + </desc> + </func> + + <func> + <name><ret>void</ret><nametext>erl_drv_set_os_pid(ErlDrvPort port, ErlDrvSInt pid)</nametext></name> + <fsummary>Set the os_pid for the port</fsummary> + <desc> + <marker id="erl_drv_set_os_pid"></marker> + <p>Arguments:</p> + <taglist> + <tag><c>port</c></tag> + <item>The port handle of the port (driver instance) to set the pid on. + </item> + <tag><c>pid</c></tag> + <item>The pid to set.</item> + </taglist> + <p> + Set the os_pid seen when doing erlang:port_info/2 on this port. + </p> + </desc> + </func> + + <func> <name><ret>int</ret><nametext>erl_drv_thread_create(char *name, ErlDrvTid *tid, void * (*func)(void *), diff --git a/erts/doc/src/notes.xml b/erts/doc/src/notes.xml index 5a65115cb2..5cb9bdb690 100644 --- a/erts/doc/src/notes.xml +++ b/erts/doc/src/notes.xml @@ -32,6 +32,111 @@ <p>This document describes the changes made to the ERTS application.</p> +<section><title>Erts 7.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Small documentation fixes</p> + <p> + Own Id: OTP-13017</p> + </item> + <item> + <p> + Fix memory corruption bug caused by disabling + distribution and then re-enable distribution with a node + name that has previously been used by a remote node.</p> + <p> + Own Id: OTP-13076 Aux Id: seq12959 </p> + </item> + <item> + <p> + Renamed variables with name bool as Visual Studio 2015 + now treats this is a keyword.</p> + <p> + Own Id: OTP-13079</p> + </item> + <item> + <p><c>erl_prim_loader</c> has not supported custom + loaders for several releases. In the documentation for + <c>erl_prim_loader</c>, all references to custom loaders + have now been removed.</p> + <p> + Own Id: OTP-13102</p> + </item> + <item> + <p> + Fixed compilation of erts together with libc versions + that do not define __uint32_t.</p> + <p> + Own Id: OTP-13105</p> + </item> + <item> + <p> + erl -make now returns non-zero exit codes on failure</p> + <p> + Own Id: OTP-13107</p> + </item> + <item> + <p> + Fix crash on init:restart in embedded mode caused by + on_load handler process not being relaunched leading to + load failure for modules such as crypto and asn1rt_nif + that need it to be present for correct NIF loading.</p> + <p> + Own Id: OTP-13115</p> + </item> + <item> + <p> + Fix maps decode in erlang:binary_to_term/1</p> + <p>Decoding a term with a large (HAMT) map in an small + (FLAT) map could cause a critical error if the external + format was not produced by beam.</p> + <p> + Own Id: OTP-13125</p> + </item> + <item> + <p> + Fix very rare bug in GC when big maps with a lot of hash + collisions from a remote node are waiting in inner + message queue.</p> + <p> + Own Id: OTP-13146</p> + </item> + <item> + <p> + Fixed a bug that could cause a crash dump to become + almost empty.</p> + <p> + Own Id: OTP-13150</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> Updated the xmllint target to just check the xml + files with real documentation content.<br/> Corrected + some errors and added some missing target in the DTD's. + </p> + <p> + Own Id: OTP-13026</p> + </item> + <item> + <p> + Add function enif_getenv to read OS environment variables + in a portable way from NIFs.</p> + <p> + Own Id: OTP-13147</p> + </item> + </list> + </section> + +</section> + <section><title>Erts 7.1</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in index c5080d5b5d..8cf435905b 100644 --- a/erts/emulator/Makefile.in +++ b/erts/emulator/Makefile.in @@ -317,7 +317,7 @@ else CS_CFLAGS = $(CS_CFLAGS_) endif CS_LDFLAGS = $(LDFLAGS) -CS_LIBS = -L../lib/internal/$(TARGET) -lerts_internal$(TYPEMARKER) @ERTS_INTERNAL_X_LIBS@ +CS_LIBS = -L../lib/internal/$(TARGET) -lerts_internal$(TYPEMARKER) @ERTS_INTERNAL_X_LIBS@ @SOCKET_LIBS@ LIBS += @TERMCAP_LIB@ -L../lib/internal/$(TARGET) @ERTS_INTERNAL_X_LIBS@ @@ -398,7 +398,7 @@ EMULATOR_EXECUTABLE = beam$(TF_MARKER).dll else EMULATOR_EXECUTABLE = beam$(TF_MARKER) endif -CS_EXECUTABLE = child_setup$(TYPEMARKER) +CS_EXECUTABLE = erl_child_setup$(TYPEMARKER) # ---------------------------------------------------------------------- @@ -690,11 +690,11 @@ $(OBJDIR)/%.o: drivers/$(ERLANG_OSTYPE)/%.c # ---------------------------------------------------------------------- # Specials # -CS_SRC = sys/$(ERLANG_OSTYPE)/erl_child_setup.c +CS_OBJ = $(OBJDIR)/erl_child_setup.o $(OBJDIR)/sys_uds.o $(OBJDIR)/hash.o -$(BINDIR)/$(CS_EXECUTABLE): $(TTF_DIR)/GENERATED $(PRELOAD_SRC) $(CS_SRC) $(ERTS_LIB) - $(ld_verbose)$(CS_PURIFY) $(CC) $(CS_LDFLAGS) -o $(BINDIR)/$(CS_EXECUTABLE) \ - $(CS_CFLAGS) $(COMMON_INCLUDES) $(CS_SRC) $(CS_LIBS) +$(BINDIR)/$(CS_EXECUTABLE): $(TTF_DIR)/GENERATED $(PRELOAD_SRC) $(CS_OBJ) $(ERTS_LIB) + $(ld_verbose)$(CS_PURIFY) $(LD) $(CS_LDFLAGS) -o $(BINDIR)/$(CS_EXECUTABLE) \ + $(CS_CFLAGS) $(COMMON_INCLUDES) $(CS_OBJ) $(CS_LIBS) $(OBJDIR)/%.kp.o: sys/common/%.c $(V_CC) -DERTS_KERNEL_POLL_VERSION $(subst -O2, $(GEN_OPT_FLGS), $(CFLAGS)) $(INCLUDES) -c $< -o $@ @@ -798,6 +798,8 @@ OS_OBJS = \ else OS_OBJS = \ $(OBJDIR)/sys.o \ + $(OBJDIR)/sys_drivers.o \ + $(OBJDIR)/sys_uds.o \ $(OBJDIR)/driver_tab.o \ $(OBJDIR)/unix_efile.o \ $(OBJDIR)/gzio.o \ diff --git a/erts/emulator/beam/atom.c b/erts/emulator/beam/atom.c index fe91134ef4..fd2adac676 100644 --- a/erts/emulator/beam/atom.c +++ b/erts/emulator/beam/atom.c @@ -435,6 +435,9 @@ init_atom_table(void) f.cmp = (HCMP_FUN) atom_cmp; f.alloc = (HALLOC_FUN) atom_alloc; f.free = (HFREE_FUN) atom_free; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; atom_text_pos = NULL; atom_text_end = NULL; diff --git a/erts/emulator/beam/atom.names b/erts/emulator/beam/atom.names index 7424e47ec3..13c2a0f8f9 100644 --- a/erts/emulator/beam/atom.names +++ b/erts/emulator/beam/atom.names @@ -209,6 +209,7 @@ atom dsend_continue_trap atom dunlink atom duplicate_bag atom dupnames +atom einval atom elib_malloc atom emulator atom enable_trace diff --git a/erts/emulator/beam/beam_bif_load.c b/erts/emulator/beam/beam_bif_load.c index 6b6c066211..c925a8c812 100644 --- a/erts/emulator/beam/beam_bif_load.c +++ b/erts/emulator/beam/beam_bif_load.c @@ -837,10 +837,10 @@ check_process_code(Process* rp, Module* modp, int allow_gc, int *redsp) if (check_mod_funs(rp, &hfrag->off_heap, mod_start, mod_size)) return am_true; /* Should not contain any constants... */ - ASSERT(!any_heap_ref_ptrs(&hfrag->mem[0], - &hfrag->mem[hfrag->used_size], - mod_start, - mod_size)); + ASSERT(!any_heap_refs(&hfrag->mem[0], + &hfrag->mem[hfrag->used_size], + mod_start, + mod_size)); } } @@ -881,7 +881,7 @@ check_process_code(Process* rp, Module* modp, int allow_gc, int *redsp) hp = &hfrag->mem[0]; hp_end = &hfrag->mem[hfrag->used_size]; - if (any_heap_ref_ptrs(hp, hp_end, mod_start, lit_bsize)) + if (any_heap_refs(hp, hp_end, mod_start, lit_bsize)) goto try_literal_gc; } @@ -902,7 +902,7 @@ check_process_code(Process* rp, Module* modp, int allow_gc, int *redsp) hp = &hfrag->mem[0]; hp_end = &hfrag->mem[hfrag->used_size]; - ASSERT(!any_heap_ref_ptrs(hp, hp_end, mod_start, lit_bsize)); + ASSERT(!any_heap_refs(hp, hp_end, mod_start, lit_bsize)); } } diff --git a/erts/emulator/beam/bif.tab b/erts/emulator/beam/bif.tab index 07d4702b92..0aee8681c6 100644 --- a/erts/emulator/beam/bif.tab +++ b/erts/emulator/beam/bif.tab @@ -115,7 +115,7 @@ bif erlang:time_offset/0 bif erlang:time_offset/1 bif erlang:timestamp/0 -bif erlang:open_port/2 +bif erts_internal:open_port/2 bif erlang:pid_to_list/1 bif erlang:ports/0 diff --git a/erts/emulator/beam/erl_bif_port.c b/erts/emulator/beam/erl_bif_port.c index e47d7bcbbb..839abd0424 100644 --- a/erts/emulator/beam/erl_bif_port.c +++ b/erts/emulator/beam/erl_bif_port.c @@ -41,6 +41,7 @@ #include "external.h" #include "packet_parser.h" #include "erl_bits.h" +#include "erl_bif_unique.h" #include "dtrace-wrapper.h" static Port *open_port(Process* p, Eterm name, Eterm settings, int *err_typep, int *err_nump); @@ -50,10 +51,10 @@ static void free_args(char **); char *erts_default_arg0 = "default"; -BIF_RETTYPE open_port_2(BIF_ALIST_2) +BIF_RETTYPE erts_internal_open_port_2(BIF_ALIST_2) { Port *port; - Eterm port_id; + Eterm res; char *str; int err_type, err_num; @@ -61,27 +62,58 @@ BIF_RETTYPE open_port_2(BIF_ALIST_2) if (!port) { if (err_type == -3) { ASSERT(err_num == BADARG || err_num == SYSTEM_LIMIT); - BIF_ERROR(BIF_P, err_num); + if (err_num == BADARG) + res = am_badarg; + else if (err_num == SYSTEM_LIMIT) + res = am_system_limit; + else + /* this is only here to silence gcc, it should not happen */ + BIF_ERROR(BIF_P, EXC_INTERNAL_ERROR); } else if (err_type == -2) { str = erl_errno_id(err_num); + res = erts_atom_put((byte *) str, strlen(str), ERTS_ATOM_ENC_LATIN1, 1); } else { - str = "einval"; + res = am_einval; } - BIF_P->fvalue = erts_atom_put((byte *) str, strlen(str), ERTS_ATOM_ENC_LATIN1, 1); - BIF_ERROR(BIF_P, EXC_ERROR); - } + BIF_RET(res); + } + + if (port->drv_ptr->flags & ERL_DRV_FLAG_USE_INIT_ACK) { + /* Copied from erl_port_task.c */ + port->async_open_port = erts_alloc(ERTS_ALC_T_PRTSD, + sizeof(*port->async_open_port)); + erts_make_ref_in_array(port->async_open_port->ref); + port->async_open_port->to = BIF_P->common.id; + + erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCKS_MSG_RECEIVE | ERTS_PROC_LOCK_LINK); + if (ERTS_PROC_PENDING_EXIT(BIF_P)) { + /* need to exit caller instead */ + erts_smp_proc_unlock(BIF_P, ERTS_PROC_LOCKS_MSG_RECEIVE | ERTS_PROC_LOCK_LINK); + KILL_CATCHES(BIF_P); + BIF_P->freason = EXC_EXIT; + erts_port_release(port); + BIF_RET(am_badarg); + } + + ERTS_SMP_MSGQ_MV_INQ2PRIVQ(BIF_P); + BIF_P->msg.save = BIF_P->msg.last; - erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCK_LINK); + erts_smp_proc_unlock(BIF_P, ERTS_PROC_LOCKS_MSG_RECEIVE); + + res = erts_proc_store_ref(BIF_P, port->async_open_port->ref); + } else { + res = port->common.id; + erts_smp_proc_lock(BIF_P, ERTS_PROC_LOCK_LINK); + } - port_id = port->common.id; erts_add_link(&ERTS_P_LINKS(port), LINK_PID, BIF_P->common.id); - erts_add_link(&ERTS_P_LINKS(BIF_P), LINK_PID, port_id); + erts_add_link(&ERTS_P_LINKS(BIF_P), LINK_PID, port->common.id); erts_smp_proc_unlock(BIF_P, ERTS_PROC_LOCK_LINK); erts_port_release(port); - BIF_RET(port_id); + BIF_RET(res); } static ERTS_INLINE Port * diff --git a/erts/emulator/beam/erl_driver.h b/erts/emulator/beam/erl_driver.h index 1093366e08..bda4d5d1c6 100644 --- a/erts/emulator/beam/erl_driver.h +++ b/erts/emulator/beam/erl_driver.h @@ -125,7 +125,7 @@ typedef struct { #define ERL_DRV_EXTENDED_MARKER (0xfeeeeeed) #define ERL_DRV_EXTENDED_MAJOR_VERSION 3 -#define ERL_DRV_EXTENDED_MINOR_VERSION 2 +#define ERL_DRV_EXTENDED_MINOR_VERSION 3 /* * The emulator will refuse to load a driver with a major version @@ -163,6 +163,7 @@ typedef struct { #define ERL_DRV_FLAG_USE_PORT_LOCKING (1 << 0) #define ERL_DRV_FLAG_SOFT_BUSY (1 << 1) #define ERL_DRV_FLAG_NO_BUSY_MSGQ (1 << 2) +#define ERL_DRV_FLAG_USE_INIT_ACK (1 << 3) /* * Integer types @@ -690,6 +691,12 @@ EXTERN char *driver_dl_error(void); EXTERN int erl_drv_putenv(const char *key, char *value); EXTERN int erl_drv_getenv(const char *key, char *value, size_t *value_size); +/* spawn start init ack */ +EXTERN void erl_drv_init_ack(ErlDrvPort ix, ErlDrvData res); + +/* set the pid seen in port_info */ +EXTERN void erl_drv_set_os_pid(ErlDrvPort ix, ErlDrvSInt pid); + #endif /* !ERL_DRIVER_TYPES_ONLY */ #ifdef WIN32_DYNAMIC_ERL_DRIVER diff --git a/erts/emulator/beam/erl_fun.c b/erts/emulator/beam/erl_fun.c index 4268e2d40a..cff476694c 100644 --- a/erts/emulator/beam/erl_fun.c +++ b/erts/emulator/beam/erl_fun.c @@ -66,6 +66,9 @@ erts_init_fun_table(void) f.cmp = (HCMP_FUN) fun_cmp; f.alloc = (HALLOC_FUN) fun_alloc; f.free = (HFREE_FUN) fun_free; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; hash_init(ERTS_ALC_T_FUN_TABLE, &erts_fun_table, "fun_table", 16, f); } diff --git a/erts/emulator/beam/erl_init.c b/erts/emulator/beam/erl_init.c index 99d8a2a987..e3390c2769 100644 --- a/erts/emulator/beam/erl_init.c +++ b/erts/emulator/beam/erl_init.c @@ -388,6 +388,7 @@ erl_init(int ncpu, erts_mseg_late_init(); /* Must be after timer (erts_init_time()) and thread initializations */ #endif + erl_sys_late_init(); #ifdef HIPE hipe_mode_switch_init(); /* Must be after init_load/beam_catches/init */ #endif diff --git a/erts/emulator/beam/erl_lock_check.c b/erts/emulator/beam/erl_lock_check.c index 84bee976ff..f7b4bd8041 100644 --- a/erts/emulator/beam/erl_lock_check.c +++ b/erts/emulator/beam/erl_lock_check.c @@ -113,9 +113,6 @@ static erts_lc_lock_order_t erts_lock_order[] = { { "environ", NULL }, #endif { "efile_drv", "address" }, -#if defined(ENABLE_CHILD_WAITER_THREAD) || defined(ERTS_SMP) - { "child_status", NULL }, -#endif { "drv_ev_state_grow", NULL, }, { "drv_ev_state", "address" }, { "safe_hash", "address" }, diff --git a/erts/emulator/beam/erl_node_tables.c b/erts/emulator/beam/erl_node_tables.c index 106235ab30..13f14adbab 100644 --- a/erts/emulator/beam/erl_node_tables.c +++ b/erts/emulator/beam/erl_node_tables.c @@ -787,10 +787,13 @@ void erts_init_node_tables(int dd_sec) erts_smp_rwmtx_init_opt(&erts_node_table_rwmtx, &rwmtx_opt, "node_table"); erts_smp_rwmtx_init_opt(&erts_dist_table_rwmtx, &rwmtx_opt, "dist_table"); - f.hash = (H_FUN) dist_table_hash; - f.cmp = (HCMP_FUN) dist_table_cmp; - f.alloc = (HALLOC_FUN) dist_table_alloc; - f.free = (HFREE_FUN) dist_table_free; + f.hash = (H_FUN) dist_table_hash; + f.cmp = (HCMP_FUN) dist_table_cmp; + f.alloc = (HALLOC_FUN) dist_table_alloc; + f.free = (HFREE_FUN) dist_table_free; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; hash_init(ERTS_ALC_T_DIST_TABLE, &erts_dist_table, "dist_table", 11, f); f.hash = (H_FUN) node_table_hash; diff --git a/erts/emulator/beam/erl_node_tables.h b/erts/emulator/beam/erl_node_tables.h index 64278d2ea0..fb2f2a5407 100644 --- a/erts/emulator/beam/erl_node_tables.h +++ b/erts/emulator/beam/erl_node_tables.h @@ -41,6 +41,7 @@ #include "sys.h" #include "hash.h" +#include "erl_alloc.h" #include "erl_process.h" #include "erl_monitors.h" #include "erl_smp.h" diff --git a/erts/emulator/beam/erl_port.h b/erts/emulator/beam/erl_port.h index acd68ef0ad..fa97707a87 100644 --- a/erts/emulator/beam/erl_port.h +++ b/erts/emulator/beam/erl_port.h @@ -187,6 +187,11 @@ struct _erl_drv_port { ErtsPrtSD *psd; /* Port specific data */ int reds; /* Only used while executing driver callbacks */ + + struct { + Eterm to; + Uint32 ref[ERTS_MAX_REF_NUMBERS]; + } *async_open_port; /* Reference used with async open port */ }; @@ -687,7 +692,7 @@ erts_drvport2port_state(ErlDrvPort drvport, erts_aint32_t *statep) Port *prt = ERTS_ErlDrvPort2Port(drvport); erts_aint32_t state; ASSERT(prt); - ERTS_LC_ASSERT(erts_lc_is_emu_thr()); +// ERTS_LC_ASSERT(erts_lc_is_emu_thr()); if (prt == ERTS_INVALID_ERL_DRV_PORT) return ERTS_INVALID_ERL_DRV_PORT; ERTS_SMP_LC_ASSERT(erts_lc_is_port_locked(prt) @@ -944,4 +949,9 @@ ErtsPortOpResult erts_port_control(Process *, Port *, unsigned int, Eterm, Eterm ErtsPortOpResult erts_port_call(Process *, Port *, unsigned int, Eterm, Eterm *); ErtsPortOpResult erts_port_info(Process *, Port *, Eterm, Eterm *); +/* + * Signals from ports to ports. Used by sys drivers. + */ +int erl_drv_port_control(Eterm, char, char*, ErlDrvSizeT); + #endif diff --git a/erts/emulator/beam/erl_process.c b/erts/emulator/beam/erl_process.c index 6c132a7e3d..2c62685f8c 100644 --- a/erts/emulator/beam/erl_process.c +++ b/erts/emulator/beam/erl_process.c @@ -498,9 +498,6 @@ dbg_chk_aux_work_val(erts_aint32_t value) #if HAVE_ERTS_MSEG valid |= ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK; #endif -#ifdef ERTS_SMP_SCHEDULERS_NEED_TO_CHECK_CHILDREN - valid |= ERTS_SSI_AUX_WORK_CHECK_CHILDREN; -#endif #ifdef ERTS_SSI_AUX_WORK_REAP_PORTS valid |= ERTS_SSI_AUX_WORK_REAP_PORTS; #endif @@ -587,8 +584,6 @@ erts_pre_init_process(void) = "MISC_THR_PRGR"; erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_MISC_IX] = "MISC"; - erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_CHECK_CHILDREN_IX] - = "CHECK_CHILDREN"; erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_SET_TMO_IX] = "SET_TMO"; erts_aux_work_flag_descr[ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK_IX] @@ -2095,34 +2090,6 @@ erts_debug_wait_completed(Process *c_p, int flags) } -#ifdef ERTS_SMP_SCHEDULERS_NEED_TO_CHECK_CHILDREN -void -erts_smp_notify_check_children_needed(void) -{ - int i; - for (i = 0; i < erts_no_schedulers; i++) - set_aux_work_flags_wakeup_nob(ERTS_SCHED_SLEEP_INFO_IX(i), - ERTS_SSI_AUX_WORK_CHECK_CHILDREN); -#ifdef ERTS_DIRTY_SCHEDULERS - for (i = 0; i < erts_no_dirty_cpu_schedulers; i++) - set_aux_work_flags_wakeup_nob(ERTS_DIRTY_CPU_SCHED_SLEEP_INFO_IX(i), - ERTS_SSI_AUX_WORK_CHECK_CHILDREN); - for (i = 0; i < erts_no_dirty_io_schedulers; i++) - set_aux_work_flags_wakeup_nob(ERTS_DIRTY_IO_SCHED_SLEEP_INFO_IX(i), - ERTS_SSI_AUX_WORK_CHECK_CHILDREN); -#endif -} - -static ERTS_INLINE erts_aint32_t -handle_check_children(ErtsAuxWorkData *awdp, erts_aint32_t aux_work, int waiting) -{ - unset_aux_work_flags(awdp->ssi, ERTS_SSI_AUX_WORK_CHECK_CHILDREN); - erts_check_children(); - return aux_work & ~ERTS_SSI_AUX_WORK_CHECK_CHILDREN; -} - -#endif - static void notify_reap_ports_relb(void) { @@ -2276,10 +2243,6 @@ handle_aux_work(ErtsAuxWorkData *awdp, erts_aint32_t orig_aux_work, int waiting) HANDLE_AUX_WORK(ERTS_SSI_AUX_WORK_MISC, handle_misc_aux_work); -#ifdef ERTS_SMP_SCHEDULERS_NEED_TO_CHECK_CHILDREN - HANDLE_AUX_WORK(ERTS_SSI_AUX_WORK_CHECK_CHILDREN, - handle_check_children); -#endif HANDLE_AUX_WORK(ERTS_SSI_AUX_WORK_SET_TMO, handle_setup_aux_work_timer); diff --git a/erts/emulator/beam/erl_process.h b/erts/emulator/beam/erl_process.h index 71065d875a..4bc879eacb 100644 --- a/erts/emulator/beam/erl_process.h +++ b/erts/emulator/beam/erl_process.h @@ -293,7 +293,6 @@ typedef enum { ERTS_SSI_AUX_WORK_ASYNC_READY_CLEAN_IX, ERTS_SSI_AUX_WORK_MISC_THR_PRGR_IX, ERTS_SSI_AUX_WORK_MISC_IX, - ERTS_SSI_AUX_WORK_CHECK_CHILDREN_IX, ERTS_SSI_AUX_WORK_SET_TMO_IX, ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK_IX, ERTS_SSI_AUX_WORK_REAP_PORTS_IX, @@ -326,8 +325,6 @@ typedef enum { (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_MISC_THR_PRGR_IX) #define ERTS_SSI_AUX_WORK_MISC \ (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_MISC_IX) -#define ERTS_SSI_AUX_WORK_CHECK_CHILDREN \ - (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_CHECK_CHILDREN_IX) #define ERTS_SSI_AUX_WORK_SET_TMO \ (((erts_aint32_t) 1) << ERTS_SSI_AUX_WORK_SET_TMO_IX) #define ERTS_SSI_AUX_WORK_MSEG_CACHE_CHECK \ @@ -1666,11 +1663,8 @@ Eterm erts_multi_scheduling_blockers(Process *); void erts_start_schedulers(void); void erts_alloc_notify_delayed_dealloc(int); void erts_alloc_ensure_handle_delayed_dealloc_call(int); -#ifdef ERTS_SMP void erts_notify_canceled_timer(ErtsSchedulerData *, int); #endif -void erts_smp_notify_check_children_needed(void); -#endif #if ERTS_USE_ASYNC_READY_Q void erts_notify_check_async_ready_queue(void *); #endif diff --git a/erts/emulator/beam/export.c b/erts/emulator/beam/export.c index 2420df36b5..581efe6eec 100644 --- a/erts/emulator/beam/export.c +++ b/erts/emulator/beam/export.c @@ -184,6 +184,9 @@ init_export_table(void) f.cmp = (HCMP_FUN) export_cmp; f.alloc = (HALLOC_FUN) export_alloc; f.free = (HFREE_FUN) export_free; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; for (i=0; i<ERTS_NUM_CODE_IX; i++) { erts_index_init(ERTS_ALC_T_EXPORT_TABLE, &export_tables[i], "export_list", diff --git a/erts/emulator/beam/global.h b/erts/emulator/beam/global.h index 98c275a20c..0bf5988244 100644 --- a/erts/emulator/beam/global.h +++ b/erts/emulator/beam/global.h @@ -1492,6 +1492,7 @@ extern void erts_match_prog_foreach_offheap(Binary *b, extern erts_driver_t vanilla_driver; extern erts_driver_t spawn_driver; +extern erts_driver_t forker_driver; extern erts_driver_t fd_driver; int erts_beam_jump_table(void); diff --git a/erts/emulator/beam/hash.c b/erts/emulator/beam/hash.c index 75d091d11c..5a0b93f693 100644 --- a/erts/emulator/beam/hash.c +++ b/erts/emulator/beam/hash.c @@ -101,11 +101,11 @@ void hash_info(int to, void *arg, Hash* h) hash_get_info(&hi, h); - erts_print(to, arg, "=hash_table:%s\n", hi.name); - erts_print(to, arg, "size: %d\n", hi.size); - erts_print(to, arg, "used: %d\n", hi.used); - erts_print(to, arg, "objs: %d\n", hi.objs); - erts_print(to, arg, "depth: %d\n", hi.depth); + h->fun.meta_print(to, arg, "=hash_table:%s\n", hi.name); + h->fun.meta_print(to, arg, "size: %d\n", hi.size); + h->fun.meta_print(to, arg, "used: %d\n", hi.used); + h->fun.meta_print(to, arg, "objs: %d\n", hi.objs); + h->fun.meta_print(to, arg, "depth: %d\n", hi.depth); } @@ -135,22 +135,22 @@ static ERTS_INLINE void set_thresholds(Hash* h) ** init a pre allocated or static hash structure ** and allocate buckets. */ -Hash* hash_init(ErtsAlcType_t type, Hash* h, char* name, int size, HashFunctions fun) +Hash* hash_init(int type, Hash* h, char* name, int size, HashFunctions fun) { int sz; int ix = 0; - h->type = type; + h->meta_alloc_type = type; while (h_size_table[ix] != -1 && h_size_table[ix] < size) ix++; if (h_size_table[ix] == -1) - erl_exit(1, "panic: too large hash table size (%d)\n", size); + return NULL; size = h_size_table[ix]; sz = size*sizeof(HashBucket*); - h->bucket = (HashBucket**) erts_alloc(h->type, sz); + h->bucket = (HashBucket**) fun.meta_alloc(h->meta_alloc_type, sz); sys_memzero(h->bucket, sz); h->is_allocated = 0; @@ -167,11 +167,11 @@ Hash* hash_init(ErtsAlcType_t type, Hash* h, char* name, int size, HashFunctions /* ** Create a new hash table */ -Hash* hash_new(ErtsAlcType_t type, char* name, int size, HashFunctions fun) +Hash* hash_new(int type, char* name, int size, HashFunctions fun) { Hash* h; - h = erts_alloc(type, sizeof(Hash)); + h = fun.meta_alloc(type, sizeof(Hash)); h = hash_init(type, h, name, size, fun); h->is_allocated = 1; @@ -195,9 +195,9 @@ void hash_delete(Hash* h) b = b_next; } } - erts_free(h->type, h->bucket); + h->fun.meta_free(h->meta_alloc_type, h->bucket); if (h->is_allocated) - erts_free(h->type, (void*) h); + h->fun.meta_free(h->meta_alloc_type, (void*) h); } /* @@ -223,7 +223,7 @@ static void rehash(Hash* h, int grow) h->size = h_size_table[h->size_ix]; sz = h->size*sizeof(HashBucket*); - new_bucket = (HashBucket **) erts_alloc(h->type, sz); + new_bucket = (HashBucket **) h->fun.meta_alloc(h->meta_alloc_type, sz); sys_memzero(new_bucket, sz); for (i = 0; i < old_size; i++) { @@ -236,7 +236,7 @@ static void rehash(Hash* h, int grow) b = b_next; } } - erts_free(h->type, (void *) h->bucket); + h->fun.meta_free(h->meta_alloc_type, (void *) h->bucket); h->bucket = new_bucket; set_thresholds(h); } diff --git a/erts/emulator/beam/hash.h b/erts/emulator/beam/hash.h index dc7e9c10c5..e94aaa0a84 100644 --- a/erts/emulator/beam/hash.h +++ b/erts/emulator/beam/hash.h @@ -29,14 +29,17 @@ #include "sys.h" #endif -#include "erl_alloc.h" - typedef unsigned long HashValue; +typedef struct hash Hash; typedef int (*HCMP_FUN)(void*, void*); typedef HashValue (*H_FUN)(void*); typedef void* (*HALLOC_FUN)(void*); typedef void (*HFREE_FUN)(void*); +/* Meta functions */ +typedef void* (*HMALLOC_FUN)(int,size_t); +typedef void (*HMFREE_FUN)(int,void*); +typedef int (*HMPRINT_FUN)(int,void*,char*, ...); /* ** This bucket must be placed in top of @@ -55,6 +58,9 @@ typedef struct hash_functions HCMP_FUN cmp; HALLOC_FUN alloc; HFREE_FUN free; + HMALLOC_FUN meta_alloc; + HMFREE_FUN meta_free; + HMPRINT_FUN meta_print; } HashFunctions; typedef struct { @@ -65,11 +71,11 @@ typedef struct { int depth; } HashInfo; -typedef struct hash +struct hash { HashFunctions fun; /* Function block */ int is_allocated; /* 0 iff hash structure is on stack or is static */ - ErtsAlcType_t type; + int meta_alloc_type; /* argument to pass to meta_alloc and meta_free */ char* name; /* Table name (static string, for debugging) */ int size; /* Number of slots */ int shrink_threshold; @@ -78,10 +84,10 @@ typedef struct hash int min_size_ix; /* Never shrink table smaller than this */ int nobjs; /* Number of objects in table */ HashBucket** bucket; /* Vector of bucket pointers (objects) */ -} Hash; +}; -Hash* hash_new(ErtsAlcType_t, char*, int, HashFunctions); -Hash* hash_init(ErtsAlcType_t, Hash*, char*, int, HashFunctions); +Hash* hash_new(int, char*, int, HashFunctions); +Hash* hash_init(int, Hash*, char*, int, HashFunctions); void hash_delete(Hash*); void hash_get_info(HashInfo*, Hash*); diff --git a/erts/emulator/beam/index.h b/erts/emulator/beam/index.h index 14fab41026..99b2bdfab0 100644 --- a/erts/emulator/beam/index.h +++ b/erts/emulator/beam/index.h @@ -30,6 +30,10 @@ #include "hash.h" #endif +#ifndef ERL_ALLOC_H__ +#include "erl_alloc.h" +#endif + typedef struct index_slot { HashBucket bucket; diff --git a/erts/emulator/beam/io.c b/erts/emulator/beam/io.c index 56a04f9b7f..93c591b124 100644 --- a/erts/emulator/beam/io.c +++ b/erts/emulator/beam/io.c @@ -54,6 +54,9 @@ extern ErlDrvEntry fd_driver_entry; extern ErlDrvEntry vanilla_driver_entry; extern ErlDrvEntry spawn_driver_entry; +#ifndef __WIN32__ +extern ErlDrvEntry forker_driver_entry; +#endif extern ErlDrvEntry *driver_tab[]; /* table of static drivers, only used during initialization */ erts_driver_t *driver_list; /* List of all drivers, static and dynamic. */ @@ -71,6 +74,9 @@ const Port erts_invalid_port = {{ERTS_INVALID_PORT}}; erts_driver_t vanilla_driver; erts_driver_t spawn_driver; +#ifndef __WIN32__ +erts_driver_t forker_driver; +#endif erts_driver_t fd_driver; int erts_port_synchronous_ops = 0; @@ -84,6 +90,7 @@ static void deliver_result(Eterm sender, Eterm pid, Eterm res); static int init_driver(erts_driver_t *, ErlDrvEntry *, DE_Handle *); static void terminate_port(Port *p); static void pdl_init(void); +static int driver_failure_term(ErlDrvPort ix, Eterm term, int eof); #ifdef ERTS_SMP static void driver_monitor_lock_pdl(Port *p); static void driver_monitor_unlock_pdl(Port *p); @@ -305,12 +312,9 @@ static Port *create_port(char *name, size_t port_size, busy_port_queue_size, size; erts_aint32_t state = ERTS_PORT_SFLG_CONNECTED; erts_aint32_t x_pts_flgs = 0; -#ifdef DEBUG - /* Make sure the debug flags survives until port is freed */ - state |= ERTS_PORT_SFLG_PORT_DEBUG; -#endif #ifdef ERTS_SMP + ErtsRunQueue *runq; if (!driver_lock) { /* Align size for mutex following port struct */ port_size = size = ERTS_ALC_DATA_ALIGN_SIZE(sizeof(Port)); @@ -320,6 +324,12 @@ static Port *create_port(char *name, #endif port_size = size = ERTS_ALC_DATA_ALIGN_SIZE(sizeof(Port)); +#ifdef DEBUG + /* Make sure the debug flags survives until port is freed */ + state |= ERTS_PORT_SFLG_PORT_DEBUG; +#endif + + busy_port_queue_size = ((driver->flags & ERL_DRV_FLAG_NO_BUSY_MSGQ) ? 0 @@ -355,8 +365,12 @@ static Port *create_port(char *name, p += sizeof(erts_mtx_t); state |= ERTS_PORT_SFLG_PORT_SPECIFIC_LOCK; } - erts_smp_atomic_set_nob(&prt->run_queue, - (erts_aint_t) erts_get_runq_current(NULL)); + if (erts_get_scheduler_data()) + runq = erts_get_runq_current(NULL); + else + runq = ERTS_RUNQ_IX(0); + erts_smp_atomic_set_nob(&prt->run_queue, (erts_aint_t) runq); + prt->xports = NULL; #else erts_atomic32_init_nob(&prt->refc, 1); @@ -383,6 +397,7 @@ static Port *create_port(char *name, ERTS_PTMR_INIT(prt); erts_port_task_handle_init(&prt->timeout_task); prt->psd = NULL; + prt->async_open_port = NULL; prt->drv_data = (SWord) 0; prt->os_pid = -1; @@ -464,6 +479,11 @@ erts_port_free(Port *prt) erts_port_task_fini_sched(&prt->sched); + if (prt->async_open_port) { + erts_free(ERTS_ALC_T_PRTSD, prt->async_open_port); + prt->async_open_port = NULL; + } + #ifdef ERTS_SMP ASSERT(prt->lock); if (state & ERTS_PORT_SFLG_PORT_SPECIFIC_LOCK) @@ -1525,6 +1545,26 @@ erts_schedule_proc2port_signal(Process *c_p, return ERTS_PORT_OP_SCHEDULED; } +static int +erts_schedule_port2port_signal(Eterm port_num, ErtsProc2PortSigData *sigdp, + int task_flags, + ErtsProc2PortSigCallback callback) +{ + Port *prt = erts_port_lookup_raw(port_num); + + if (!prt) + return -1; + + sigdp->caller = ERTS_INVALID_PID; + + return erts_port_task_schedule(prt->common.id, + NULL, + ERTS_PORT_TASK_PROC_SIG, + sigdp, + callback, + task_flags); +} + static ERTS_INLINE void send_badsig(Port *prt) { ErtsProcLocks rp_locks = ERTS_PROC_LOCKS_XSIG_SEND; @@ -2360,6 +2400,11 @@ erts_port_exit(Process *c_p, | ERTS_PORT_SIG_FLG_BROKEN_LINK | ERTS_PORT_SIG_FLG_FORCE_SCHED)) == 0); +#ifndef __WIN32__ + if (prt->drv_ptr == &forker_driver) + return ERTS_PORT_OP_DROPPED; +#endif + if (!(flags & ERTS_PORT_SIG_FLG_FORCE_SCHED)) { ErtsTryImmDrvCallState try_call_state = ERTS_INIT_TRY_IMM_DRV_CALL_STATE(c_p, @@ -2724,6 +2769,72 @@ erts_port_link(Process *c_p, Port *prt, Eterm to, Eterm *refp) port_sig_link); } +static void +init_ack_send_reply(Port *port, Eterm resp) +{ + + if (!is_internal_port(resp)) { + Process *rp = erts_proc_lookup_raw(port->async_open_port->to); + erts_smp_proc_lock(rp, ERTS_PROC_LOCK_LINK); + erts_remove_link(&ERTS_P_LINKS(port), port->async_open_port->to); + erts_remove_link(&ERTS_P_LINKS(rp), port->common.id); + erts_smp_proc_unlock(rp, ERTS_PROC_LOCK_LINK); + } + port_sched_op_reply(port->async_open_port->to, + port->async_open_port->ref, + resp); + + erts_free(ERTS_ALC_T_PRTSD, port->async_open_port); + port->async_open_port = NULL; +} + +void +erl_drv_init_ack(ErlDrvPort ix, ErlDrvData res) { + Port *port = erts_drvport2port(ix); + SWord err_type = (SWord)res; + Eterm resp; + + if (port == ERTS_INVALID_ERL_DRV_PORT && port->async_open_port) + return; + + if (port->async_open_port) { + switch(err_type) { + case -3: + resp = am_badarg; + break; + case -2: { + char *str = erl_errno_id(errno); + resp = erts_atom_put((byte *) str, strlen(str), + ERTS_ATOM_ENC_LATIN1, 1); + break; + } + case -1: + resp = am_einval; + break; + default: + resp = port->common.id; + break; + } + + init_ack_send_reply(port, resp); + + if (err_type == -1 || err_type == -2 || err_type == -3) + driver_failure_term(ix, am_normal, 0); + port->drv_data = err_type; + } +} + +void +erl_drv_set_os_pid(ErlDrvPort ix, ErlDrvSInt pid) { + Port *port = erts_drvport2port(ix); + + if (port == ERTS_INVALID_ERL_DRV_PORT) + return; + + port->os_pid = (SWord)pid; + +} + void erts_init_io(int port_tab_size, int port_tab_size_ignore_files, int legacy_port_tab) @@ -2786,6 +2897,9 @@ void erts_init_io(int port_tab_size, init_driver(&fd_driver, &fd_driver_entry, NULL); init_driver(&vanilla_driver, &vanilla_driver_entry, NULL); init_driver(&spawn_driver, &spawn_driver_entry, NULL); +#ifndef __WIN32__ + init_driver(&forker_driver, &forker_driver_entry, NULL); +#endif erts_init_static_drivers(); for (dp = driver_tab; *dp != NULL; dp++) erts_add_driver_entry(*dp, NULL, 1); @@ -2847,6 +2961,9 @@ void erts_lcnt_enable_io_lock_count(int enable) { lcnt_enable_drv_lock_count(&vanilla_driver, enable); lcnt_enable_drv_lock_count(&spawn_driver, enable); +#ifndef __WIN32__ + lcnt_enable_drv_lock_count(&forker_driver, enable); +#endif lcnt_enable_drv_lock_count(&fd_driver, enable); /* enable lock counting in all drivers */ for (dp = driver_list; dp; dp = dp->next) { @@ -3885,7 +4002,7 @@ port_sig_control(Port *prt, Uint hsz, rsz; int control_flags; - rp = erts_proc_lookup_raw(sigdp->caller); + rp = sigdp->caller == ERTS_INVALID_PID ? NULL : erts_proc_lookup_raw(sigdp->caller); if (!rp) goto done; @@ -3921,7 +4038,8 @@ port_sig_control(Port *prt, /* failure */ - port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg); + if (sigdp->caller != ERTS_INVALID_PID) + port_sched_op_reply(sigdp->caller, sigdp->ref, am_badarg); done: @@ -3931,6 +4049,23 @@ done: return ERTS_PORT_REDS_CONTROL; } +/* + * This is an asynchronous control call. I.e. it will not return anything + * to the caller. + */ +int +erl_drv_port_control(Eterm port_num, char cmd, char* buff, ErlDrvSizeT size) +{ + ErtsProc2PortSigData *sigdp = erts_port_task_alloc_p2p_sig_data(); + + sigdp->flags = ERTS_P2P_SIG_TYPE_CONTROL | ERTS_P2P_SIG_DATA_FLG_REPLY; + sigdp->u.control.binp = NULL; + sigdp->u.control.command = cmd; + sigdp->u.control.bufp = buff; + sigdp->u.control.size = size; + + return erts_schedule_port2port_signal(port_num, sigdp, 0, port_sig_control); +} ErtsPortOpResult erts_port_control(Process* c_p, @@ -4702,6 +4837,10 @@ print_port_info(Port *p, int to, void *arg) erts_print(to, arg, "Port is a file: %s\n",p->name); } else if (p->drv_ptr == &spawn_driver) { erts_print(to, arg, "Port controls external process: %s\n",p->name); +#ifndef __WIN32__ + } else if (p->drv_ptr == &forker_driver) { + erts_print(to, arg, "Port controls forker process: %s\n",p->name); +#endif } else { erts_print(to, arg, "Port controls linked-in driver: %s\n",p->name); } @@ -6933,6 +7072,9 @@ driver_failure_term(ErlDrvPort ix, Eterm term, int eof) if (prt == ERTS_INVALID_ERL_DRV_PORT) return -1; ERTS_SMP_LC_ASSERT(erts_lc_is_port_locked(prt)); + + if (prt->async_open_port) + init_ack_send_reply(prt, prt->common.id); if (eof) flush_linebuf_messages(prt, state); if (state & ERTS_PORT_SFLG_CLOSING) { diff --git a/erts/emulator/beam/module.c b/erts/emulator/beam/module.c index f6794c012f..f5c7b177d3 100644 --- a/erts/emulator/beam/module.c +++ b/erts/emulator/beam/module.c @@ -103,6 +103,9 @@ void init_module_table(void) f.cmp = (HCMP_FUN) module_cmp; f.alloc = (HALLOC_FUN) module_alloc; f.free = (HFREE_FUN) module_free; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; for (i = 0; i < ERTS_NUM_CODE_IX; i++) { erts_index_init(ERTS_ALC_T_MODULE_TABLE, &module_tables[i], "module_code", diff --git a/erts/emulator/beam/register.c b/erts/emulator/beam/register.c index 7ade8bca0f..fdb6cbc813 100644 --- a/erts/emulator/beam/register.c +++ b/erts/emulator/beam/register.c @@ -151,6 +151,9 @@ void init_register_table(void) f.cmp = (HCMP_FUN) reg_cmp; f.alloc = (HALLOC_FUN) reg_alloc; f.free = (HFREE_FUN) reg_free; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; hash_init(ERTS_ALC_T_REG_TABLE, &process_reg, "process_reg", PREG_HASH_SIZE, f); diff --git a/erts/emulator/beam/sys.h b/erts/emulator/beam/sys.h index 2170d416c8..53f8313daa 100644 --- a/erts/emulator/beam/sys.h +++ b/erts/emulator/beam/sys.h @@ -87,15 +87,6 @@ # define NO_FPE_SIGNALS #endif -#ifdef DISABLE_CHILD_WAITER_THREAD -#undef ENABLE_CHILD_WAITER_THREAD -#endif - -#if defined(ERTS_SMP) && !defined(DISABLE_CHILD_WAITER_THREAD) -#undef ENABLE_CHILD_WAITER_THREAD -#define ENABLE_CHILD_WAITER_THREAD 1 -#endif - #define ERTS_I64_LITERAL(X) X##LL #define ErtsInArea(ptr,start,nbytes) \ @@ -746,6 +737,7 @@ void erts_sys_main_thread(void); extern int erts_sys_prepare_crash_dump(int secs); extern void erts_sys_pre_init(void); extern void erl_sys_init(void); +extern void erl_sys_late_init(void); extern void erl_sys_args(int *argc, char **argv); extern void erl_sys_schedule(int); void sys_tty_reset(int); diff --git a/erts/emulator/hipe/hipe_bif0.c b/erts/emulator/hipe/hipe_bif0.c index 00936b6b8a..6f495b8825 100644 --- a/erts/emulator/hipe/hipe_bif0.c +++ b/erts/emulator/hipe/hipe_bif0.c @@ -509,6 +509,9 @@ static void init_const_term_table(void) f.cmp = (HCMP_FUN) const_term_cmp; f.alloc = (HALLOC_FUN) const_term_alloc; f.free = (HFREE_FUN) NULL; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; hash_init(ERTS_ALC_T_HIPE, &const_term_table, "const_term_table", 97, f); } @@ -717,6 +720,9 @@ static void init_nbif_table(void) f.cmp = (HCMP_FUN) nbif_cmp; f.alloc = (HALLOC_FUN) nbif_alloc; f.free = NULL; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; hash_init(ERTS_ALC_T_NBIF_TABLE, &nbif_table, "nbif_table", 500, f); @@ -810,6 +816,9 @@ static void init_primop_table(void) f.cmp = (HCMP_FUN) primop_cmp; f.alloc = (HALLOC_FUN) primop_alloc; f.free = NULL; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; hash_init(ERTS_ALC_T_HIPE, &primop_table, "primop_table", 50, f); @@ -1828,6 +1837,9 @@ static void init_modinfo_table(void) f.cmp = (HCMP_FUN) modinfo_cmp; f.alloc = (HALLOC_FUN) modinfo_alloc; f.free = (HFREE_FUN) NULL; + f.meta_alloc = (HMALLOC_FUN) erts_alloc; + f.meta_free = (HMFREE_FUN) erts_free; + f.meta_print = (HMPRINT_FUN) erts_print; hash_init(ERTS_ALC_T_HIPE, &modinfo_table, "modinfo_table", 11, f); } diff --git a/erts/emulator/sys/unix/erl_child_setup.c b/erts/emulator/sys/unix/erl_child_setup.c index a3c5c20641..4e61530cf1 100644 --- a/erts/emulator/sys/unix/erl_child_setup.c +++ b/erts/emulator/sys/unix/erl_child_setup.c @@ -1,7 +1,7 @@ /* * %CopyrightBegin% * - * Copyright Ericsson AB 2002-2009. All Rights Reserved. + * Copyright Ericsson AB 2002-2015. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -19,109 +19,233 @@ */ /* - * After a vfork() (or fork()) the child exec()s to this program which - * sets up the child and exec()s to the user program (see spawn_start() - * in sys.c and ticket OTP-4389). + * This program is started at erts startup and all fork's that + * have to be done are done in here. This is done for a couple + * of reasons: + * - Allow usage of fork without a memory explosion. + * -- we do not want to use vfork, as it blocks the VM + * until the execv is done, and if the program that + * is to be executed is on an NFS that is unavailable, + * the execv can block for a very long time. + * -- we cannot do fork inside the VM as that would temporarily + * duplicate the memory usage of the VM per parallel exec. + * + * Some implementation notes: + * - A single Unix Domain Socket is setup in between the VM and + * this program. Over that UDS the file descriptors that should + * be used to talk to the child program are sent. + * The actual command to execute, together with options and the + * environment, is sent over the pipe represented by the + * file descriptors mentioned above. We don't send the + * command over the UDS as that would increase the likely hood + * that it's buffer would be full. + * + * - Since it is this program that execv's, it has to take care of + * all the SIGCHLD signals that the child programs generate. The + * signals are received and the pid+exit reason is sent as data + * on the UDS to the VM. The VM is then able to map the pid to the + * port of the child program that just exited and deliver the status + * code if requested. */ #ifdef HAVE_CONFIG_H # include "config.h" #endif -#define NEED_CHILD_SETUP_DEFINES -#include "sys.h" -#include "erl_misc_utils.h" +#include <stdlib.h> +#include <stdio.h> +#include <sys/wait.h> -#ifdef SIG_SIGSET /* Old SysV */ -void sys_sigrelease(int sig) +#define WANT_NONBLOCKING + +#include "erl_driver.h" +#include "sys_uds.h" +#include "hash.h" +#include "erl_child_setup.h" + +#define SET_CLOEXEC(fd) fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) | FD_CLOEXEC) + +#if defined(__ANDROID__) +#define SHELL "/system/bin/sh" +#else +#define SHELL "/bin/sh" +#endif /* __ANDROID__ */ + +//#define HARD_DEBUG +#ifdef HARD_DEBUG +#define DEBUG_PRINT(fmt, ...) fprintf(stderr, fmt "\r\n", ##__VA_ARGS__) +#else +#define DEBUG_PRINT(fmt, ...) +#endif + +#define ABORT(fmt, ...) do { \ + fprintf(stderr, "erl_child_setup: " fmt "\r\n", ##__VA_ARGS__); \ + abort(); \ + } while(0) + +#ifdef DEBUG +void +erl_assert_error(const char* expr, const char* func, const char* file, int line) { - sigrelse(sig); + fflush(stdout); + fprintf(stderr, "%s:%d:%s() Assertion failed: %s\n", + file, line, func, expr); + fflush(stderr); + abort(); } -#else /* !SIG_SIGSET */ -#ifdef SIG_SIGNAL /* Old BSD */ -sys_sigrelease(int sig) +#endif + +void sys_sigblock(int sig) { - sigsetmask(sigblock(0) & ~sigmask(sig)); + sigset_t mask; + + sigemptyset(&mask); + sigaddset(&mask, sig); + sigprocmask(SIG_BLOCK, &mask, (sigset_t *)NULL); } -#else /* !SIG_SIGNAL */ /* The True Way - POSIX!:-) */ + void sys_sigrelease(int sig) { sigset_t mask; - sigemptyset(&mask); sigaddset(&mask, sig); sigprocmask(SIG_UNBLOCK, &mask, (sigset_t *)NULL); } -#endif /* !SIG_SIGNAL */ -#endif /* !SIG_SIGSET */ - -#if defined(__ANDROID__) -static int system_properties_fd(void); -#endif /* __ANDROID__ */ -#if defined(__ANDROID__) -#define SHELL "/system/bin/sh" -#else -#define SHELL "/bin/sh" -#endif /* __ANDROID__ */ +static void add_os_pid_to_port_id_mapping(Eterm, pid_t); +static Eterm get_port_id(pid_t); +static int forker_hash_init(void); +static int max_files = -1; +static int sigchld_pipe[2]; -int -main(int argc, char *argv[]) +static int +start_new_child(int pipes[]) { - int i, from, to; - int erts_spawn_executable = 0; + int size, res, i, pos = 0; + char *buff, *o_buff; + + char *cmd, *wd, **new_environ, **args = NULL; + + Sint cnt, flags; - /* OBSERVE! - * Keep child setup after fork() (implemented in sys.c) up to date - * if changes are made here. - */ + /* only child executes here */ - if (argc != CS_ARGV_NO_OF_ARGS) { - if (argc < CS_ARGV_NO_OF_ARGS) { - return 1; - } else { - erts_spawn_executable = 1; - } + do { + res = read(pipes[0], (char*)&size, sizeof(size)); + } while(res < 0 && (errno == EINTR || errno == ERRNO_BLOCK)); + + if (res <= 0) { + goto child_error; } - if (strcmp("false", argv[CS_ARGV_UNBIND_IX]) != 0) - if (erts_unbind_from_cpu_str(argv[CS_ARGV_UNBIND_IX]) != 0) - return 1; + buff = malloc(size); + + DEBUG_PRINT("size = %d", size); + + do { + if ((res = read(pipes[0], buff + pos, size - pos)) < 0) { + if (errno == ERRNO_BLOCK || errno == EINTR) + continue; + goto child_error; + } + if (res == 0) { + errno = EPIPE; + goto child_error; + } + pos += res; + } while(size - pos != 0); + + o_buff = buff; + + flags = get_int32(buff); + buff += sizeof(Sint32); - for (i = 0; i < CS_ARGV_NO_OF_DUP2_OPS; i++) { - if (argv[CS_ARGV_DUP2_OP_IX(i)][0] == '-' - && argv[CS_ARGV_DUP2_OP_IX(i)][1] == '\0') - break; - if (sscanf(argv[CS_ARGV_DUP2_OP_IX(i)], "%d:%d", &from, &to) != 2) - return 1; - if (dup2(from, to) < 0) - return 1; + DEBUG_PRINT("flags = %d", flags); + + cmd = buff; + buff += strlen(buff) + 1; + if (*buff == '\0') { + wd = NULL; + } else { + wd = buff; + buff += strlen(buff) + 1; } + buff++; - if (sscanf(argv[CS_ARGV_FD_CR_IX], "%d:%d", &from, &to) != 2) - return 1; + DEBUG_PRINT("wd = %s", wd); -#if defined(HAVE_CLOSEFROM) - closefrom(from); -#elif defined(__ANDROID__) - if (from <= to) { - int spfd = system_properties_fd(); - for (i = from; i <= to; i++) { - if (i != spfd) { - (void) close(i); - } - } + cnt = get_int32(buff); + buff += sizeof(Sint32); + new_environ = malloc(sizeof(char*)*(cnt + 1)); + + DEBUG_PRINT("env_len = %ld", cnt); + for (i = 0; i < cnt; i++, buff++) { + new_environ[i] = buff; + while(*buff != '\0') buff++; } -#else /* !__ANDROID__ */ - for (i = from; i <= to; i++) { - (void) close(i); + new_environ[cnt] = NULL; + + if (o_buff + size != buff) { + /* This is a spawn executable call */ + cnt = get_int32(buff); + buff += sizeof(Sint32); + args = malloc(sizeof(char*)*(cnt + 1)); + for (i = 0; i < cnt; i++, buff++) { + args[i] = buff; + while(*buff != '\0') buff++; + } + args[cnt] = NULL; } -#endif /* HAVE_CLOSEFROM */ - if (!(argv[CS_ARGV_WD_IX][0] == '.' && argv[CS_ARGV_WD_IX][1] == '\0') - && chdir(argv[CS_ARGV_WD_IX]) < 0) - return 1; + if (o_buff + size != buff) { + errno = EINVAL; + goto child_error; + } + + DEBUG_PRINT("read ack"); + do { + ErtsSysForkerProto proto; + res = read(pipes[0], &proto, sizeof(proto)); + if (res > 0) { + ASSERT(proto.action == ErtsSysForkerProtoAction_Ack); + ASSERT(res == sizeof(proto)); + } + } while(res < 0 && (errno == EINTR || errno == ERRNO_BLOCK)); + if (res < 1) { + errno = EPIPE; + goto child_error; + } + + DEBUG_PRINT("Do that forking business: '%s'\n",cmd); + + /* When the dup2'ing below is done, only + fd's 0, 1, 2 and maybe 3, 4 should survive the + exec. All other fds (i.e. the unix domain sockets + and stray pipe ends) should have CLOEXEC set on them + so they will be closed when the exec happens */ + if (flags & FORKER_FLAG_USE_STDIO) { + /* stdin for process */ + if (flags & FORKER_FLAG_DO_WRITE && + dup2(pipes[0], 0) < 0) + goto child_error; + /* stdout for process */ + if (flags & FORKER_FLAG_DO_READ && + dup2(pipes[1], 1) < 0) + goto child_error; + } + else { /* XXX will fail if pipes[0] == 4 (unlikely..) */ + if (flags & FORKER_FLAG_DO_READ && dup2(pipes[1], 4) < 0) + goto child_error; + if (flags & FORKER_FLAG_DO_WRITE && dup2(pipes[0], 3) < 0) + goto child_error; + } + + if (dup2(pipes[2], 2) < 0) + goto child_error; + + if (wd && chdir(wd) < 0) + goto child_error; #if defined(USE_SETPGRP_NOARGS) /* SysV */ (void) setpgrp(); @@ -131,34 +255,301 @@ main(int argc, char *argv[]) (void) setsid(); #endif + close(pipes[0]); + close(pipes[1]); + close(pipes[2]); + sys_sigrelease(SIGCHLD); - sys_sigrelease(SIGINT); - sys_sigrelease(SIGUSR1); - - if (erts_spawn_executable) { - if (argv[CS_ARGV_NO_OF_ARGS + 1] == NULL) { - execl(argv[CS_ARGV_NO_OF_ARGS],argv[CS_ARGV_NO_OF_ARGS], - (char *) NULL); - } else { - execv(argv[CS_ARGV_NO_OF_ARGS],&(argv[CS_ARGV_NO_OF_ARGS + 1])); - } + + if (args) { + /* spawn_executable */ + execve(cmd, args, new_environ); } else { - execl(SHELL, "sh", "-c", argv[CS_ARGV_CMD_IX], (char *) NULL); + execle(SHELL, "sh", "-c", cmd, (char *) NULL, new_environ); } - return 1; +child_error: + DEBUG_PRINT("exec error: %d\r\n",errno); + _exit(128 + errno); +} + + +/* + * [OTP-3906] + * Solaris signal management gets confused when threads are used and a + * lot of child processes dies. The confusion results in that SIGCHLD + * signals aren't delivered to the emulator which in turn results in + * a lot of defunct processes in the system. + * + * The problem seems to appear when a signal is frequently + * blocked/unblocked at the same time as the signal is frequently + * propagated. The child waiter thread is a workaround for this problem. + * The SIGCHLD signal is always blocked (in all threads), and the child + * waiter thread fetches the signal by a call to sigwait(). See + * child_waiter(). + * + * This should be a non-issue since the fork:ing was moved outside of + * the emulator into erl_child_setup. I'm leaving the comment here + * for posterity. */ + +static void handle_sigchld(int sig) { + int buff[2], res; + + sys_sigblock(SIGCHLD); + + while ((buff[0] = waitpid((pid_t)(-1), buff+1, WNOHANG)) > 0) { + do { + res = write(sigchld_pipe[1], buff, sizeof(buff)); + } while (res < 0 && errno == EINTR); + if (res <= 0) + ABORT("Failed to write to sigchld_pipe (%d): %d (%d)", sigchld_pipe[1], res, errno); + DEBUG_PRINT("Reap child %d (%d)", buff[0], buff[1]); + } + + sys_sigrelease(SIGCHLD); } #if defined(__ANDROID__) static int system_properties_fd(void) { - int fd; + static int fd = -2; char *env; + if (fd != -2) return fd; env = getenv("ANDROID_PROPERTY_WORKSPACE"); if (!env) { + fd = -1; return -1; } fd = atoi(env); return fd; } #endif /* __ANDROID__ */ + +int +main(int argc, char *argv[]) +{ + /* This fd should be open from beam */ + int uds_fd = 3, max_fd = 3; +#ifndef HAVE_CLOSEFROM + int i; +#endif + struct sigaction sa; + + if (argc < 1 || sscanf(argv[1],"%d",&max_files) != 1) { + ABORT("Invalid arguments to child_setup"); + } + +/* We close all fds except the uds from beam. + All other fds from now on will have the + CLOEXEC flags set on them. This means that we + only have to close a very limited number of fds + after we fork before the exec. */ +#if defined(HAVE_CLOSEFROM) + closefrom(4); +#else + for (i = 4; i < max_files; i++) +#if defined(__ANDROID__) + if (i != system_properties_fd()) +#endif + (void) close(i); +#endif + + if (pipe(sigchld_pipe) < 0) { + ABORT("Failed to setup sigchld pipe (%d)", errno); + } + + SET_CLOEXEC(sigchld_pipe[0]); + SET_CLOEXEC(sigchld_pipe[1]); + + max_fd = max_fd < sigchld_pipe[0] ? sigchld_pipe[0] : max_fd; + + sa.sa_handler = &handle_sigchld; + sigemptyset(&sa.sa_mask); + sa.sa_flags = SA_RESTART | SA_NOCLDSTOP; + if (sigaction(SIGCHLD, &sa, 0) == -1) { + perror(0); + exit(1); + } + + forker_hash_init(); + + SET_CLOEXEC(uds_fd); + + DEBUG_PRINT("Starting forker %d", max_files); + + while (1) { + fd_set read_fds; + int res; + FD_ZERO(&read_fds); + FD_SET(uds_fd, &read_fds); + FD_SET(sigchld_pipe[0], &read_fds); + DEBUG_PRINT("child_setup selecting on %d, %d (%d)", + uds_fd, sigchld_pipe[0], max_fd); + res = select(max_fd+1, &read_fds, NULL, NULL, NULL); + + if (res < 0) { + if (errno == EINTR) continue; + ABORT("Select failed: %d (%d)",res, errno); + } + + if (FD_ISSET(uds_fd, &read_fds)) { + int pipes[3], res, os_pid; + ErtsSysForkerProto proto; + errno = 0; + if ((res = sys_uds_read(uds_fd, (char*)&proto, sizeof(proto), + pipes, 3, MSG_DONTWAIT)) < 0) { + if (errno == EINTR) + continue; + DEBUG_PRINT("erl_child_setup failed to read from uds: %d, %d", res, errno); + _exit(0); + } + + if (res == 0) { + DEBUG_PRINT("uds was closed!"); + _exit(0); + } + /* Since we use unix domain sockets and send the entire data in + one go we *should* get the entire payload at once. */ + ASSERT(res == sizeof(proto)); + ASSERT(proto.action == ErtsSysForkerProtoAction_Start); + + sys_sigblock(SIGCHLD); + + errno = 0; + + os_pid = fork(); + if (os_pid == 0) + start_new_child(pipes); + + add_os_pid_to_port_id_mapping(proto.u.start.port_id, os_pid); + + /* We write an ack here, but expect the reply on + the pipes[0] inside the fork */ + proto.action = ErtsSysForkerProtoAction_Go; + proto.u.go.os_pid = os_pid; + proto.u.go.error_number = errno; + while (write(pipes[1], &proto, sizeof(proto)) < 0 && errno == EINTR) + ; /* remove gcc warning */ + +#ifdef FORKER_PROTO_START_ACK + proto.action = ErtsSysForkerProtoAction_StartAck; + while (write(uds_fd, &proto, sizeof(proto)) < 0 && errno == EINTR) + ; /* remove gcc warning */ +#endif + + sys_sigrelease(SIGCHLD); + close(pipes[0]); + close(pipes[1]); + close(pipes[2]); + } + + if (FD_ISSET(sigchld_pipe[0], &read_fds)) { + int ibuff[2]; + ErtsSysForkerProto proto; + res = read(sigchld_pipe[0], ibuff, sizeof(ibuff)); + if (res <= 0) { + if (errno == EINTR) + continue; + ABORT("Failed to read from sigchld pipe: %d (%d)", res, errno); + } + + proto.u.sigchld.port_id = get_port_id((pid_t)(ibuff[0])); + + if (proto.u.sigchld.port_id == THE_NON_VALUE) + continue; /* exit status report not requested */ + + proto.action = ErtsSysForkerProtoAction_SigChld; + proto.u.sigchld.error_number = ibuff[1]; + DEBUG_PRINT("send %s to %d", buff, uds_fd); + if (write(uds_fd, &proto, sizeof(proto)) < 0) { + if (errno == EINTR) + continue; + /* The uds was close, which most likely means that the VM + has exited. This will be detected when we try to read + from the uds_fd. */ + DEBUG_PRINT("Failed to write to uds: %d (%d)", uds_fd, errno); + } + } + } + return 1; +} + +typedef struct exit_status { + HashBucket hb; + pid_t os_pid; + Eterm port_id; +} ErtsSysExitStatus; + +static Hash *forker_hash; + +static void add_os_pid_to_port_id_mapping(Eterm port_id, pid_t os_pid) +{ + if (port_id != THE_NON_VALUE) { + /* exit status report requested */ + ErtsSysExitStatus es; + es.os_pid = os_pid; + es.port_id = port_id; + hash_put(forker_hash, &es); + } +} + +static Eterm get_port_id(pid_t os_pid) +{ + ErtsSysExitStatus est, *es; + Eterm port_id; + est.os_pid = os_pid; + es = hash_remove(forker_hash, &est); + if (!es) return THE_NON_VALUE; + port_id = es->port_id; + free(es); + return port_id; +} + +static int fcmp(void *a, void *b) +{ + ErtsSysExitStatus *sa = a; + ErtsSysExitStatus *sb = b; + return !(sa->os_pid == sb->os_pid); +} + +static HashValue fhash(void *e) +{ + ErtsSysExitStatus *se = e; + Uint32 val = se->os_pid; + val = (val+0x7ed55d16) + (val<<12); + val = (val^0xc761c23c) ^ (val>>19); + val = (val+0x165667b1) + (val<<5); + val = (val+0xd3a2646c) ^ (val<<9); + val = (val+0xfd7046c5) + (val<<3); + val = (val^0xb55a4f09) ^ (val>>16); + return val; +} + +static void *falloc(void *e) +{ + ErtsSysExitStatus *se = e; + ErtsSysExitStatus *ne = malloc(sizeof(ErtsSysExitStatus)); + ne->os_pid = se->os_pid; + ne->port_id = se->port_id; + return ne; +} + +static void *meta_alloc(int type, size_t size) { return malloc(size); } +static void meta_free(int type, void *p) { free(p); } + +static int forker_hash_init(void) +{ + HashFunctions forker_hash_functions; + forker_hash_functions.hash = fhash; + forker_hash_functions.cmp = fcmp; + forker_hash_functions.alloc = falloc; + forker_hash_functions.free = free; + forker_hash_functions.meta_alloc = meta_alloc; + forker_hash_functions.meta_free = meta_free; + forker_hash_functions.meta_print = NULL; + + forker_hash = hash_new(0, "forker_hash", + 16, forker_hash_functions); + + return 1; +} diff --git a/erts/emulator/sys/unix/erl_child_setup.h b/erts/emulator/sys/unix/erl_child_setup.h new file mode 100644 index 0000000000..a28b136bfc --- /dev/null +++ b/erts/emulator/sys/unix/erl_child_setup.h @@ -0,0 +1,77 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2015-2015. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * %CopyrightEnd% + * + * This file defines the interface inbetween erts and child_setup. + */ + +#ifndef _ERL_UNIX_FORKER_H +#define _ERL_UNIX_FORKER_H + +#include "sys.h" + +#ifdef __FreeBSD__ +/* The freebsd sendmsg man page explicitly states that + you should not close fds before they are known + to have reached the other side, so this Ack protects + against that. */ +#define FORKER_PROTO_START_ACK 1 +#endif + +#define FORKER_ARGV_NO_OF_ARGS 3 +#define FORKER_ARGV_PROGNAME_IX 0 /* Program name */ +#define FORKER_ARGV_MAX_FILES 1 /* max_files */ + +#define FORKER_FLAG_USE_STDIO (1 << 0) /* dup the pipe to stdin/stderr */ +#define FORKER_FLAG_EXIT_STATUS (1 << 1) /* send the exit status to parent */ +#define FORKER_FLAG_DO_READ (1 << 2) /* dup write fd */ +#define FORKER_FLAG_DO_WRITE (1 << 3) /* dup read fd */ + +#if SIZEOF_VOID_P == SIZEOF_LONG +typedef unsigned long ErtsSysPortId; +#elif SIZEOF_VOID_P == SIZEOF_INT +typedef unsigned int ErtsSysPortId; +#elif SIZEOF_VOID_P == SIZEOF_LONG_LONG +typedef unsigned long long ErtsSysPortId; +#endif + +typedef struct ErtsSysForkerProto_ { + enum { + ErtsSysForkerProtoAction_Start, + ErtsSysForkerProtoAction_StartAck, + ErtsSysForkerProtoAction_Go, + ErtsSysForkerProtoAction_SigChld, + ErtsSysForkerProtoAction_Ack + } action; + union { + struct { + ErtsSysPortId port_id; + int fds[3]; + } start; + struct { + pid_t os_pid; + int error_number; + } go; + struct { + ErtsSysPortId port_id; + int error_number; + } sigchld; + } u; +} ErtsSysForkerProto; + +#endif /* #ifndef _ERL_UNIX_FORKER_H */ diff --git a/erts/emulator/sys/unix/erl_unix_sys.h b/erts/emulator/sys/unix/erl_unix_sys.h index 8d4e98bf3a..0352ee1b3c 100644 --- a/erts/emulator/sys/unix/erl_unix_sys.h +++ b/erts/emulator/sys/unix/erl_unix_sys.h @@ -30,9 +30,7 @@ #include <limits.h> #include <stdlib.h> #include <string.h> -#ifndef QNX #include <memory.h> -#endif #if defined(__sun__) && defined(__SVR4) && !defined(__EXTENSIONS__) # define __EXTENSIONS__ @@ -92,11 +90,6 @@ #include <ieeefp.h> #endif -#ifdef QNX -#include <process.h> -#include <sys/qnx_glob.h> -#endif - #include <pwd.h> #ifndef HZ @@ -136,13 +129,6 @@ # define ERTS_POLL_NEED_ASYNC_INTERRUPT_SUPPORT #endif -#ifndef ENABLE_CHILD_WAITER_THREAD -# ifdef ERTS_SMP -# define ERTS_SMP_SCHEDULERS_NEED_TO_CHECK_CHILDREN -void erts_check_children(void); -# endif -#endif - typedef void *GETENV_STATE; /* @@ -310,7 +296,6 @@ typedef void (*SIGFUNC)(int); extern SIGFUNC sys_signal(int, SIGFUNC); extern void sys_sigrelease(int); extern void sys_sigblock(int); -extern void sys_stop_cat(void); /* * Handling of floating point exceptions. @@ -425,19 +410,6 @@ void erts_sys_unblock_fpe(int); #define ERTS_FP_ERROR_THOROUGH(p, f, A) __ERTS_FP_ERROR_THOROUGH(&(p)->fp_exception, f, A) -#ifdef NEED_CHILD_SETUP_DEFINES -/* The child setup argv[] */ -#define CS_ARGV_PROGNAME_IX 0 /* Program name */ -#define CS_ARGV_UNBIND_IX 1 /* Unbind from cpu */ -#define CS_ARGV_WD_IX 2 /* Working directory */ -#define CS_ARGV_CMD_IX 3 /* Command */ -#define CS_ARGV_FD_CR_IX 4 /* Fd close range */ -#define CS_ARGV_DUP2_OP_IX(N) ((N) + 5) /* dup2 operations */ - -#define CS_ARGV_NO_OF_DUP2_OPS 3 /* Number of dup2 ops */ -#define CS_ARGV_NO_OF_ARGS 8 /* Number of arguments */ -#endif /* #ifdef NEED_CHILD_SETUP_DEFINES */ - /* Threads */ #ifdef USE_THREADS extern int init_async(int); diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c index 8d7da3e47e..2ad5f3b4d5 100644 --- a/erts/emulator/sys/unix/sys.c +++ b/erts/emulator/sys/unix/sys.c @@ -49,7 +49,6 @@ #include <sys/ioctl.h> #endif -#define NEED_CHILD_SETUP_DEFINES #define ERTS_WANT_BREAK_HANDLING #define ERTS_WANT_GOT_SIGUSR1 #define WANT_NONBLOCKING /* must define this to pull in defs from sys.h */ @@ -67,7 +66,7 @@ #include "erl_mseg.h" extern char **environ; -static erts_smp_rwmtx_t environ_rwmtx; +erts_smp_rwmtx_t environ_rwmtx; #define MAX_VSIZE 16 /* Max number of entries allowed in an I/O * vector sock_sendv(). @@ -76,89 +75,12 @@ static erts_smp_rwmtx_t environ_rwmtx; * Don't need global.h, but bif_table.h (included by bif.h), * won't compile otherwise */ -#include "global.h" +#include "global.h" #include "bif.h" -#include "erl_sys_driver.h" #include "erl_check_io.h" #include "erl_cpu_topology.h" -#ifndef DISABLE_VFORK -#define DISABLE_VFORK 0 -#endif - -#if defined IOV_MAX -#define MAXIOV IOV_MAX -#elif defined UIO_MAXIOV -#define MAXIOV UIO_MAXIOV -#else -#define MAXIOV 16 -#endif - -#ifdef USE_THREADS -# ifdef ENABLE_CHILD_WAITER_THREAD -# define CHLDWTHR ENABLE_CHILD_WAITER_THREAD -# else -# define CHLDWTHR 0 -# endif -# define FDBLOCK 1 -#else -# define CHLDWTHR 0 -# define FDBLOCK 0 -#endif -/* - * [OTP-3906] - * Solaris signal management gets confused when threads are used and a - * lot of child processes dies. The confusion results in that SIGCHLD - * signals aren't delivered to the emulator which in turn results in - * a lot of defunct processes in the system. - * - * The problem seems to appear when a signal is frequently - * blocked/unblocked at the same time as the signal is frequently - * propagated. The child waiter thread is a workaround for this problem. - * The SIGCHLD signal is always blocked (in all threads), and the child - * waiter thread fetches the signal by a call to sigwait(). See - * child_waiter(). - */ - -typedef struct ErtsSysReportExit_ ErtsSysReportExit; -struct ErtsSysReportExit_ { - ErtsSysReportExit *next; - Eterm port; - int pid; - int ifd; - int ofd; -#if CHLDWTHR && !defined(ERTS_SMP) - int status; -#endif -}; - -/* Used by the fd driver iff the fd could not be set to non-blocking */ -typedef struct ErtsSysBlocking_ { - ErlDrvPDL pdl; - int res; - int err; - unsigned int pkey; -} ErtsSysBlocking; - - -/* This data is shared by these drivers - initialized by spawn_init() */ -static struct driver_data { - ErlDrvPort port_num; - int ofd, packet_bytes; - ErtsSysReportExit *report_exit; - int pid; - int alive; - int status; - int terminating; - ErtsSysBlocking *blocking; -} *driver_data; /* indexed by fd */ - -static ErtsSysReportExit *report_exit_list; -#if CHLDWTHR && !defined(ERTS_SMP) -static ErtsSysReportExit *report_exit_transit_list; -#endif - extern int driver_interrupt(int, int); extern void do_break(void); @@ -170,33 +92,6 @@ extern void erts_sys_init_float(void); extern void erl_crash_dump(char* file, int line, char* fmt, ...); -#define DIR_SEPARATOR_CHAR '/' - -#if defined(__ANDROID__) -#define SHELL "/system/bin/sh" -#else -#define SHELL "/bin/sh" -#endif /* __ANDROID__ */ - - -#if defined(DEBUG) -#define ERL_BUILD_TYPE_MARKER ".debug" -#elif defined(PURIFY) -#define ERL_BUILD_TYPE_MARKER ".purify" -#elif defined(QUANTIFY) -#define ERL_BUILD_TYPE_MARKER ".quantify" -#elif defined(PURECOV) -#define ERL_BUILD_TYPE_MARKER ".purecov" -#elif defined(VALGRIND) -#define ERL_BUILD_TYPE_MARKER ".valgrind" -#else /* opt */ -#define ERL_BUILD_TYPE_MARKER -#endif - -#define CHILD_SETUP_PROG_NAME "child_setup" ERL_BUILD_TYPE_MARKER -#if !DISABLE_VFORK -static char *child_setup_prog; -#endif #ifdef DEBUG static int debug_log = 0; @@ -220,7 +115,7 @@ static volatile int have_prepared_crash_dump; (have_prepared_crash_dump++) #endif -static erts_smp_atomic_t sys_misc_mem_sz; +erts_smp_atomic_t sys_misc_mem_sz; #if defined(ERTS_SMP) static void smp_sig_notify(char c); @@ -233,46 +128,6 @@ static int sig_suspend_fds[2] = {-1, -1}; jmp_buf erts_sys_sigsegv_jmp; -#if CHLDWTHR || defined(ERTS_SMP) -erts_mtx_t chld_stat_mtx; -#endif -#if CHLDWTHR -static erts_tid_t child_waiter_tid; -/* chld_stat_mtx is used to protect against concurrent accesses - of the driver_data fields pid, alive, and status. */ -erts_cnd_t chld_stat_cnd; -static long children_alive; -#define CHLD_STAT_LOCK erts_mtx_lock(&chld_stat_mtx) -#define CHLD_STAT_UNLOCK erts_mtx_unlock(&chld_stat_mtx) -#define CHLD_STAT_WAIT erts_cnd_wait(&chld_stat_cnd, &chld_stat_mtx) -#define CHLD_STAT_SIGNAL erts_cnd_signal(&chld_stat_cnd) -#elif defined(ERTS_SMP) /* ------------------------------------------------- */ -#define CHLD_STAT_LOCK erts_mtx_lock(&chld_stat_mtx) -#define CHLD_STAT_UNLOCK erts_mtx_unlock(&chld_stat_mtx) - -#else /* ------------------------------------------------------------------- */ -#define CHLD_STAT_LOCK -#define CHLD_STAT_UNLOCK -static volatile int children_died; -#endif - - -static struct fd_data { - char pbuf[4]; /* hold partial packet bytes */ - int psz; /* size of pbuf */ - char *buf; - char *cpos; - int sz; - int remain; /* for input on fd */ -} *fd_data; /* indexed by fd */ - -/* static FUNCTION(int, write_fill, (int, char*, int)); unused? */ -static void note_child_death(int, int); - -#if CHLDWTHR -static void* child_waiter(void *); -#endif - static int crashdump_companion_cube_fd = -1; /********************* General functions ****************************/ @@ -453,9 +308,10 @@ MALLOC_USE_HASH(1); #ifdef USE_THREADS #ifdef ERTS_THR_HAVE_SIG_FUNCS + /* * Child thread inherits parents signal mask at creation. In order to - * guarantee that the main thread will receive all SIGINT, SIGCHLD, and + * guarantee that the main thread will receive all SIGINT, and * SIGUSR1 signals sent to the process, we block these signals in the * parent thread when creating a new thread. */ @@ -551,14 +407,11 @@ erts_sys_pre_init(void) #ifdef ERTS_THR_HAVE_SIG_FUNCS sigemptyset(&thr_create_sigmask); sigaddset(&thr_create_sigmask, SIGINT); /* block interrupt */ - sigaddset(&thr_create_sigmask, SIGCHLD); /* block child signals */ sigaddset(&thr_create_sigmask, SIGUSR1); /* block user defined signal */ #endif erts_thr_init(&eid); - report_exit_list = NULL; - #ifdef ERTS_ENABLE_LOCK_COUNT erts_lcnt_init(); #endif @@ -569,17 +422,6 @@ erts_sys_pre_init(void) #ifdef USE_THREADS -#if CHLDWTHR || defined(ERTS_SMP) - erts_mtx_init(&chld_stat_mtx, "child_status"); -#endif -#if CHLDWTHR -#ifndef ERTS_SMP - report_exit_transit_list = NULL; -#endif - erts_cnd_init(&chld_stat_cnd); - children_alive = 0; -#endif - #ifdef ERTS_SMP erts_smp_atomic32_init_nob(&erts_break_requested, 0); erts_smp_atomic32_init_nob(&erts_got_sigusr1, 0); @@ -589,9 +431,6 @@ erts_sys_pre_init(void) erts_got_sigusr1 = 0; have_prepared_crash_dump = 0; #endif -#if !CHLDWTHR && !defined(ERTS_SMP) - children_died = 0; -#endif #endif /* USE_THREADS */ @@ -628,39 +467,6 @@ erts_sys_pre_init(void) void erl_sys_init(void) { -#if !DISABLE_VFORK - { - int res; - char bindir[MAXPATHLEN]; - size_t bindirsz = sizeof(bindir); - Uint csp_path_sz; - - res = erts_sys_getenv_raw("BINDIR", bindir, &bindirsz); - if (res != 0) { - if (res < 0) - erl_exit(-1, - "Environment variable BINDIR is not set\n"); - if (res > 0) - erl_exit(-1, - "Value of environment variable BINDIR is too large\n"); - } - if (bindir[0] != DIR_SEPARATOR_CHAR) - erl_exit(-1, - "Environment variable BINDIR does not contain an" - " absolute path\n"); - csp_path_sz = (strlen(bindir) - + 1 /* DIR_SEPARATOR_CHAR */ - + sizeof(CHILD_SETUP_PROG_NAME) - + 1); - child_setup_prog = erts_alloc(ERTS_ALC_T_CS_PROG_PATH, csp_path_sz); - erts_smp_atomic_add_nob(&sys_misc_mem_sz, csp_path_sz); - erts_snprintf(child_setup_prog, csp_path_sz, - "%s%c%s", - bindir, - DIR_SEPARATOR_CHAR, - CHILD_SETUP_PROG_NAME); - } -#endif #ifdef USE_SETLINEBUF setlinebuf(stdout); @@ -978,43 +784,6 @@ int sys_max_files(void) return(max_files); } -static void block_signals(void) -{ -#if !CHLDWTHR - sys_sigblock(SIGCHLD); -#endif -#ifndef ERTS_SMP - sys_sigblock(SIGINT); -#ifndef ETHR_UNUSABLE_SIGUSRX - sys_sigblock(SIGUSR1); -#endif /* #ifndef ETHR_UNUSABLE_SIGUSRX */ -#endif /* #ifndef ERTS_SMP */ - -#if defined(ERTS_SMP) && !defined(ETHR_UNUSABLE_SIGUSRX) - sys_sigblock(ERTS_SYS_SUSPEND_SIGNAL); -#endif - -} - -static void unblock_signals(void) -{ - /* Update erl_child_setup.c if changed */ -#if !CHLDWTHR - sys_sigrelease(SIGCHLD); -#endif -#ifndef ERTS_SMP - sys_sigrelease(SIGINT); -#ifndef ETHR_UNUSABLE_SIGUSRX - sys_sigrelease(SIGUSR1); -#endif /* #ifndef ETHR_UNUSABLE_SIGUSRX */ -#endif /* #ifndef ERTS_SMP */ - -#if defined(ERTS_SMP) && !defined(ETHR_UNUSABLE_SIGUSRX) - sys_sigrelease(ERTS_SYS_SUSPEND_SIGNAL); -#endif - -} - /************************** OS info *******************************/ /* Used by erlang:info/1. */ @@ -1102,1502 +871,6 @@ void fini_getenv_state(GETENV_STATE *state) erts_smp_rwmtx_runlock(&environ_rwmtx); } - -/************************** Port I/O *******************************/ - - - -/* I. Common stuff */ - -/* - * Decreasing the size of it below 16384 is not allowed. - */ - -/* II. The spawn/fd/vanilla drivers */ - -#define ERTS_SYS_READ_BUF_SZ (64*1024) - -/* Driver interfaces */ -static ErlDrvData spawn_start(ErlDrvPort, char*, SysDriverOpts*); -static ErlDrvData fd_start(ErlDrvPort, char*, SysDriverOpts*); -#if FDBLOCK -static void fd_async(void *); -static void fd_ready_async(ErlDrvData drv_data, ErlDrvThreadData thread_data); -#endif -static ErlDrvSSizeT fd_control(ErlDrvData, unsigned int, char *, ErlDrvSizeT, - char **, ErlDrvSizeT); -static ErlDrvData vanilla_start(ErlDrvPort, char*, SysDriverOpts*); -static int spawn_init(void); -static void fd_stop(ErlDrvData); -static void fd_flush(ErlDrvData); -static void stop(ErlDrvData); -static void ready_input(ErlDrvData, ErlDrvEvent); -static void ready_output(ErlDrvData, ErlDrvEvent); -static void output(ErlDrvData, char*, ErlDrvSizeT); -static void outputv(ErlDrvData, ErlIOVec*); -static void stop_select(ErlDrvEvent, void*); - -struct erl_drv_entry spawn_driver_entry = { - spawn_init, - spawn_start, - stop, - output, - ready_input, - ready_output, - "spawn", - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - ERL_DRV_EXTENDED_MARKER, - ERL_DRV_EXTENDED_MAJOR_VERSION, - ERL_DRV_EXTENDED_MINOR_VERSION, - ERL_DRV_FLAG_USE_PORT_LOCKING, - NULL, NULL, - stop_select -}; -struct erl_drv_entry fd_driver_entry = { - NULL, - fd_start, - fd_stop, - output, - ready_input, - ready_output, - "fd", - NULL, - NULL, - fd_control, - NULL, - outputv, -#if FDBLOCK - fd_ready_async, /* ready_async */ -#else - NULL, -#endif - fd_flush, /* flush */ - NULL, /* call */ - NULL, /* event */ - ERL_DRV_EXTENDED_MARKER, - ERL_DRV_EXTENDED_MAJOR_VERSION, - ERL_DRV_EXTENDED_MINOR_VERSION, - 0, /* ERL_DRV_FLAGs */ - NULL, /* handle2 */ - NULL, /* process_exit */ - stop_select -}; -struct erl_drv_entry vanilla_driver_entry = { - NULL, - vanilla_start, - stop, - output, - ready_input, - ready_output, - "vanilla", - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, - NULL, /* flush */ - NULL, /* call */ - NULL, /* event */ - ERL_DRV_EXTENDED_MARKER, - ERL_DRV_EXTENDED_MAJOR_VERSION, - ERL_DRV_EXTENDED_MINOR_VERSION, - 0, /* ERL_DRV_FLAGs */ - NULL, /* handle2 */ - NULL, /* process_exit */ - stop_select -}; - -/* Handle SIGCHLD signals. */ -#if (defined(SIG_SIGSET) || defined(SIG_SIGNAL)) -static RETSIGTYPE onchld(void) -#else -static RETSIGTYPE onchld(int signum) -#endif -{ -#if CHLDWTHR - ASSERT(0); /* We should *never* catch a SIGCHLD signal */ -#elif defined(ERTS_SMP) - smp_sig_notify('C'); -#else - children_died = 1; - ERTS_CHK_IO_AS_INTR(); /* Make sure we don't sleep in poll */ -#endif -} - -static int set_blocking_data(struct driver_data *dd) { - - dd->blocking = erts_alloc(ERTS_ALC_T_SYS_BLOCKING, sizeof(ErtsSysBlocking)); - - erts_smp_atomic_add_nob(&sys_misc_mem_sz, sizeof(ErtsSysBlocking)); - - dd->blocking->pdl = driver_pdl_create(dd->port_num); - dd->blocking->res = 0; - dd->blocking->err = 0; - dd->blocking->pkey = driver_async_port_key(dd->port_num); - - return 1; -} - -static int set_driver_data(ErlDrvPort port_num, - int ifd, - int ofd, - int packet_bytes, - int read_write, - int exit_status, - int pid, - int is_blocking) -{ - Port *prt; - ErtsSysReportExit *report_exit; - - if (!exit_status) - report_exit = NULL; - else { - report_exit = erts_alloc(ERTS_ALC_T_PRT_REP_EXIT, - sizeof(ErtsSysReportExit)); - report_exit->next = report_exit_list; - report_exit->port = erts_drvport2id(port_num); - report_exit->pid = pid; - report_exit->ifd = read_write & DO_READ ? ifd : -1; - report_exit->ofd = read_write & DO_WRITE ? ofd : -1; -#if CHLDWTHR && !defined(ERTS_SMP) - report_exit->status = 0; -#endif - report_exit_list = report_exit; - } - - prt = erts_drvport2port(port_num); - if (prt != ERTS_INVALID_ERL_DRV_PORT) - prt->os_pid = pid; - - if (read_write & DO_READ) { - driver_data[ifd].packet_bytes = packet_bytes; - driver_data[ifd].port_num = port_num; - driver_data[ifd].report_exit = report_exit; - driver_data[ifd].pid = pid; - driver_data[ifd].alive = 1; - driver_data[ifd].status = 0; - driver_data[ifd].terminating = 0; - driver_data[ifd].blocking = NULL; - if (read_write & DO_WRITE) { - driver_data[ifd].ofd = ofd; - if (is_blocking && FDBLOCK) - if (!set_blocking_data(driver_data+ifd)) - return -1; - if (ifd != ofd) - driver_data[ofd] = driver_data[ifd]; /* structure copy */ - } else { /* DO_READ only */ - driver_data[ifd].ofd = -1; - } - (void) driver_select(port_num, ifd, (ERL_DRV_READ|ERL_DRV_USE), 1); - return(ifd); - } else { /* DO_WRITE only */ - driver_data[ofd].packet_bytes = packet_bytes; - driver_data[ofd].port_num = port_num; - driver_data[ofd].report_exit = report_exit; - driver_data[ofd].ofd = ofd; - driver_data[ofd].pid = pid; - driver_data[ofd].alive = 1; - driver_data[ofd].status = 0; - driver_data[ofd].terminating = 0; - driver_data[ofd].blocking = NULL; - if (is_blocking && FDBLOCK) - if (!set_blocking_data(driver_data+ofd)) - return -1; - return(ofd); - } -} - -static int spawn_init() -{ - int i; -#if CHLDWTHR - erts_thr_opts_t thr_opts = ERTS_THR_OPTS_DEFAULT_INITER; - - thr_opts.detached = 0; - thr_opts.suggested_stack_size = 0; /* Smallest possible */ - thr_opts.name = "child_waiter"; -#endif - - sys_signal(SIGPIPE, SIG_IGN); /* Ignore - we'll handle the write failure */ - driver_data = (struct driver_data *) - erts_alloc(ERTS_ALC_T_DRV_TAB, max_files * sizeof(struct driver_data)); - erts_smp_atomic_add_nob(&sys_misc_mem_sz, - max_files * sizeof(struct driver_data)); - - for (i = 0; i < max_files; i++) - driver_data[i].pid = -1; - -#if CHLDWTHR - sys_sigblock(SIGCHLD); -#endif - - sys_signal(SIGCHLD, onchld); /* Reap children */ - -#if CHLDWTHR - erts_thr_create(&child_waiter_tid, child_waiter, NULL, &thr_opts); -#endif - - return 1; -} - -static void close_pipes(int ifd[2], int ofd[2], int read_write) -{ - if (read_write & DO_READ) { - (void) close(ifd[0]); - (void) close(ifd[1]); - } - if (read_write & DO_WRITE) { - (void) close(ofd[0]); - (void) close(ofd[1]); - } -} - -static void init_fd_data(int fd, ErlDrvPort port_num) -{ - fd_data[fd].buf = NULL; - fd_data[fd].cpos = NULL; - fd_data[fd].remain = 0; - fd_data[fd].sz = 0; - fd_data[fd].psz = 0; -} - -static char **build_unix_environment(char *block) -{ - int i; - int j; - int len; - char *cp; - char **cpp; - char** old_env; - - ERTS_SMP_LC_ASSERT(erts_smp_lc_rwmtx_is_rlocked(&environ_rwmtx)); - - cp = block; - len = 0; - while (*cp != '\0') { - cp += strlen(cp) + 1; - len++; - } - old_env = environ; - while (*old_env++ != NULL) { - len++; - } - - cpp = (char **) erts_alloc_fnf(ERTS_ALC_T_ENVIRONMENT, - sizeof(char *) * (len+1)); - if (cpp == NULL) { - return NULL; - } - - cp = block; - len = 0; - while (*cp != '\0') { - cpp[len] = cp; - cp += strlen(cp) + 1; - len++; - } - - i = len; - for (old_env = environ; *old_env; old_env++) { - char* old = *old_env; - - for (j = 0; j < len; j++) { - char *s, *t; - - s = cpp[j]; - t = old; - while (*s == *t && *s != '=') { - s++, t++; - } - if (*s == '=' && *t == '=') { - break; - } - } - - if (j == len) { /* New version not found */ - cpp[len++] = old; - } - } - - for (j = 0; j < i; ) { - size_t last = strlen(cpp[j])-1; - if (cpp[j][last] == '=' && strchr(cpp[j], '=') == cpp[j]+last) { - cpp[j] = cpp[--len]; - if (len < i) { - i--; - } else { - j++; - } - } - else { - j++; - } - } - - cpp[len] = NULL; - return cpp; -} - -/* - [arndt] In most Unix systems, including Solaris 2.5, 'fork' allocates memory - in swap space for the child of a 'fork', whereas 'vfork' does not do this. - The natural call to use here is therefore 'vfork'. Due to a bug in - 'vfork' in Solaris 2.5 (apparently fixed in 2.6), using 'vfork' - can be dangerous in what seems to be these circumstances: - If the child code under a vfork sets the signal action to SIG_DFL - (or SIG_IGN) - for any signal which was previously set to a signal handler, the - state of the parent is clobbered, so that the later arrival of - such a signal yields a sigsegv in the parent. If the signal was - not set to a signal handler, but ignored, all seems to work. - If you change the forking code below, beware of this. - */ - -static ErlDrvData spawn_start(ErlDrvPort port_num, char* name, SysDriverOpts* opts) -{ -#define CMD_LINE_PREFIX_STR "exec " -#define CMD_LINE_PREFIX_STR_SZ (sizeof(CMD_LINE_PREFIX_STR) - 1) - - int ifd[2], ofd[2], len, pid, i; - char **volatile new_environ; /* volatile since a vfork() then cannot - cause 'new_environ' to be clobbered - in the parent process. */ - int saved_errno; - long res; - char *cmd_line; -#ifndef QNX - int unbind; -#endif -#if !DISABLE_VFORK - int no_vfork; - size_t no_vfork_sz = sizeof(no_vfork); - - no_vfork = (erts_sys_getenv_raw("ERL_NO_VFORK", - (char *) &no_vfork, - &no_vfork_sz) >= 0); -#endif - - switch (opts->read_write) { - case DO_READ: - if (pipe(ifd) < 0) - return ERL_DRV_ERROR_ERRNO; - if (ifd[0] >= max_files) { - close_pipes(ifd, ofd, opts->read_write); - errno = EMFILE; - return ERL_DRV_ERROR_ERRNO; - } - ofd[1] = -1; /* keep purify happy */ - break; - case DO_WRITE: - if (pipe(ofd) < 0) return ERL_DRV_ERROR_ERRNO; - if (ofd[1] >= max_files) { - close_pipes(ifd, ofd, opts->read_write); - errno = EMFILE; - return ERL_DRV_ERROR_ERRNO; - } - ifd[0] = -1; /* keep purify happy */ - break; - case DO_READ|DO_WRITE: - if (pipe(ifd) < 0) return ERL_DRV_ERROR_ERRNO; - errno = EMFILE; /* default for next two conditions */ - if (ifd[0] >= max_files || pipe(ofd) < 0) { - close_pipes(ifd, ofd, DO_READ); - return ERL_DRV_ERROR_ERRNO; - } - if (ofd[1] >= max_files) { - close_pipes(ifd, ofd, opts->read_write); - errno = EMFILE; - return ERL_DRV_ERROR_ERRNO; - } - break; - default: - ASSERT(0); - return ERL_DRV_ERROR_GENERAL; - } - - if (opts->spawn_type == ERTS_SPAWN_EXECUTABLE) { - /* started with spawn_executable, not with spawn */ - len = strlen(name); - cmd_line = (char *) erts_alloc_fnf(ERTS_ALC_T_TMP, len + 1); - if (!cmd_line) { - close_pipes(ifd, ofd, opts->read_write); - errno = ENOMEM; - return ERL_DRV_ERROR_ERRNO; - } - memcpy((void *) cmd_line,(void *) name, len); - cmd_line[len] = '\0'; - if (access(cmd_line,X_OK) != 0) { - int save_errno = errno; - erts_free(ERTS_ALC_T_TMP, cmd_line); - errno = save_errno; - return ERL_DRV_ERROR_ERRNO; - } - } else { - /* make the string suitable for giving to "sh" */ - len = strlen(name); - cmd_line = (char *) erts_alloc_fnf(ERTS_ALC_T_TMP, - CMD_LINE_PREFIX_STR_SZ + len + 1); - if (!cmd_line) { - close_pipes(ifd, ofd, opts->read_write); - errno = ENOMEM; - return ERL_DRV_ERROR_ERRNO; - } - memcpy((void *) cmd_line, - (void *) CMD_LINE_PREFIX_STR, - CMD_LINE_PREFIX_STR_SZ); - memcpy((void *) (cmd_line + CMD_LINE_PREFIX_STR_SZ), (void *) name, len); - cmd_line[CMD_LINE_PREFIX_STR_SZ + len] = '\0'; - } - - erts_smp_rwmtx_rlock(&environ_rwmtx); - - if (opts->envir == NULL) { - new_environ = environ; - } else if ((new_environ = build_unix_environment(opts->envir)) == NULL) { - erts_smp_rwmtx_runlock(&environ_rwmtx); - erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); - errno = ENOMEM; - return ERL_DRV_ERROR_ERRNO; - } - -#ifndef QNX - /* Block child from SIGINT and SIGUSR1. Must be before fork() - to be safe. */ - block_signals(); - - CHLD_STAT_LOCK; - - unbind = erts_sched_bind_atfork_prepare(); - -#if !DISABLE_VFORK - /* See fork/vfork discussion before this function. */ - if (no_vfork) { -#endif - - DEBUGF(("Using fork\n")); - pid = fork(); - - if (pid == 0) { - /* The child! Setup child... */ - - if (erts_sched_bind_atfork_child(unbind) != 0) - goto child_error; - - /* OBSERVE! - * Keep child setup after vfork() (implemented below and in - * erl_child_setup.c) up to date if changes are made here. - */ - - if (opts->use_stdio) { - if (opts->read_write & DO_READ) { - /* stdout for process */ - if (dup2(ifd[1], 1) < 0) - goto child_error; - if(opts->redir_stderr) - /* stderr for process */ - if (dup2(ifd[1], 2) < 0) - goto child_error; - } - if (opts->read_write & DO_WRITE) - /* stdin for process */ - if (dup2(ofd[0], 0) < 0) - goto child_error; - } - else { /* XXX will fail if ofd[0] == 4 (unlikely..) */ - if (opts->read_write & DO_READ) - if (dup2(ifd[1], 4) < 0) - goto child_error; - if (opts->read_write & DO_WRITE) - if (dup2(ofd[0], 3) < 0) - goto child_error; - } - -#if defined(HAVE_CLOSEFROM) - closefrom(opts->use_stdio ? 3 : 5); -#else - for (i = opts->use_stdio ? 3 : 5; i < max_files; i++) - (void) close(i); -#endif - - if (opts->wd && chdir(opts->wd) < 0) - goto child_error; - -#if defined(USE_SETPGRP_NOARGS) /* SysV */ - (void) setpgrp(); -#elif defined(USE_SETPGRP) /* BSD */ - (void) setpgrp(0, getpid()); -#else /* POSIX */ - (void) setsid(); -#endif - - unblock_signals(); - - if (opts->spawn_type == ERTS_SPAWN_EXECUTABLE) { - if (opts->argv == NULL) { - execle(cmd_line,cmd_line,(char *) NULL, new_environ); - } else { - if (opts->argv[0] == erts_default_arg0) { - opts->argv[0] = cmd_line; - } - execve(cmd_line, opts->argv, new_environ); - if (opts->argv[0] == cmd_line) { - opts->argv[0] = erts_default_arg0; - } - } - } else { - execle(SHELL, "sh", "-c", cmd_line, (char *) NULL, new_environ); - } - child_error: - _exit(1); - } -#if !DISABLE_VFORK - } -#define ENOUGH_BYTES (44) - else { /* Use vfork() */ - char **cs_argv= erts_alloc(ERTS_ALC_T_TMP,(CS_ARGV_NO_OF_ARGS + 1)* - sizeof(char *)); - char fd_close_range[ENOUGH_BYTES]; /* 44 bytes are enough to */ - char dup2_op[CS_ARGV_NO_OF_DUP2_OPS][ENOUGH_BYTES]; /* hold any "%d:%d" string */ - /* on a 64-bit machine. */ - - /* Setup argv[] for the child setup program (implemented in - erl_child_setup.c) */ - i = 0; - if (opts->use_stdio) { - if (opts->read_write & DO_READ){ - /* stdout for process */ - erts_snprintf(&dup2_op[i++][0], ENOUGH_BYTES, "%d:%d", ifd[1], 1); - if(opts->redir_stderr) - /* stderr for process */ - erts_snprintf(&dup2_op[i++][0], ENOUGH_BYTES, "%d:%d", ifd[1], 2); - } - if (opts->read_write & DO_WRITE) - /* stdin for process */ - erts_snprintf(&dup2_op[i++][0], ENOUGH_BYTES, "%d:%d", ofd[0], 0); - } else { /* XXX will fail if ofd[0] == 4 (unlikely..) */ - if (opts->read_write & DO_READ) - erts_snprintf(&dup2_op[i++][0], ENOUGH_BYTES, "%d:%d", ifd[1], 4); - if (opts->read_write & DO_WRITE) - erts_snprintf(&dup2_op[i++][0], ENOUGH_BYTES, "%d:%d", ofd[0], 3); - } - for (; i < CS_ARGV_NO_OF_DUP2_OPS; i++) - strcpy(&dup2_op[i][0], "-"); - erts_snprintf(fd_close_range, ENOUGH_BYTES, "%d:%d", opts->use_stdio ? 3 : 5, max_files-1); - - cs_argv[CS_ARGV_PROGNAME_IX] = child_setup_prog; - cs_argv[CS_ARGV_WD_IX] = opts->wd ? opts->wd : "."; - cs_argv[CS_ARGV_UNBIND_IX] = erts_sched_bind_atvfork_child(unbind); - cs_argv[CS_ARGV_FD_CR_IX] = fd_close_range; - for (i = 0; i < CS_ARGV_NO_OF_DUP2_OPS; i++) - cs_argv[CS_ARGV_DUP2_OP_IX(i)] = &dup2_op[i][0]; - - if (opts->spawn_type == ERTS_SPAWN_EXECUTABLE) { - int num = 0; - int j = 0; - if (opts->argv != NULL) { - for(; opts->argv[num] != NULL; ++num) - ; - } - cs_argv = erts_realloc(ERTS_ALC_T_TMP,cs_argv, (CS_ARGV_NO_OF_ARGS + 1 + num + 1) * sizeof(char *)); - cs_argv[CS_ARGV_CMD_IX] = "-"; - cs_argv[CS_ARGV_NO_OF_ARGS] = cmd_line; - if (opts->argv != NULL) { - for (;opts->argv[j] != NULL; ++j) { - if (opts->argv[j] == erts_default_arg0) { - cs_argv[CS_ARGV_NO_OF_ARGS + 1 + j] = cmd_line; - } else { - cs_argv[CS_ARGV_NO_OF_ARGS + 1 + j] = opts->argv[j]; - } - } - } - cs_argv[CS_ARGV_NO_OF_ARGS + 1 + j] = NULL; - } else { - cs_argv[CS_ARGV_CMD_IX] = cmd_line; /* Command */ - cs_argv[CS_ARGV_NO_OF_ARGS] = NULL; - } - DEBUGF(("Using vfork\n")); - pid = vfork(); - - if (pid == 0) { - /* The child! */ - - /* Observe! - * OTP-4389: The child setup program (implemented in - * erl_child_setup.c) will perform the necessary setup of the - * child before it execs to the user program. This because - * vfork() only allow an *immediate* execve() or _exit() in the - * child. - */ - execve(child_setup_prog, cs_argv, new_environ); - _exit(1); - } - erts_free(ERTS_ALC_T_TMP,cs_argv); - } -#undef ENOUGH_BYTES -#endif - - erts_sched_bind_atfork_parent(unbind); - - if (pid == -1) { - saved_errno = errno; - CHLD_STAT_UNLOCK; - erts_smp_rwmtx_runlock(&environ_rwmtx); - erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); - unblock_signals(); - close_pipes(ifd, ofd, opts->read_write); - errno = saved_errno; - return ERL_DRV_ERROR_ERRNO; - } -#else /* QNX */ - if (opts->use_stdio) { - if (opts->read_write & DO_READ) - qnx_spawn_options.iov[1] = ifd[1]; /* stdout for process */ - if (opts->read_write & DO_WRITE) - qnx_spawn_options.iov[0] = ofd[0]; /* stdin for process */ - } - else { - if (opts->read_write & DO_READ) - qnx_spawn_options.iov[4] = ifd[1]; - if (opts->read_write & DO_WRITE) - qnx_spawn_options.iov[3] = ofd[0]; - } - /* Close fds on exec */ - for (i = 3; i < max_files; i++) - fcntl(i, F_SETFD, 1); - - qnx_spawn_options.flags = _SPAWN_SETSID; - if ((pid = spawnl(P_NOWAIT, SHELL, SHELL, "-c", cmd_line, - (char *) 0)) < 0) { - erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); - reset_qnx_spawn(); - erts_smp_rwmtx_runlock(&environ_rwmtx); - close_pipes(ifd, ofd, opts->read_write); - return ERL_DRV_ERROR_GENERAL; - } - reset_qnx_spawn(); -#endif /* QNX */ - - erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); - - if (new_environ != environ) - erts_free(ERTS_ALC_T_ENVIRONMENT, (void *) new_environ); - - if (opts->read_write & DO_READ) - (void) close(ifd[1]); - if (opts->read_write & DO_WRITE) - (void) close(ofd[0]); - - if (opts->read_write & DO_READ) { - SET_NONBLOCKING(ifd[0]); - init_fd_data(ifd[0], port_num); - } - if (opts->read_write & DO_WRITE) { - SET_NONBLOCKING(ofd[1]); - init_fd_data(ofd[1], port_num); - } - - res = set_driver_data(port_num, ifd[0], ofd[1], opts->packet_bytes, - opts->read_write, opts->exit_status, pid, 0); - /* Don't unblock SIGCHLD until now, since the call above must - first complete putting away the info about our new subprocess. */ - unblock_signals(); - -#if CHLDWTHR - ASSERT(children_alive >= 0); - - if (!(children_alive++)) - CHLD_STAT_SIGNAL; /* Wake up child waiter thread if no children - was alive before we fork()ed ... */ -#endif - /* Don't unlock chld_stat_mtx until now of the same reason as above */ - CHLD_STAT_UNLOCK; - - erts_smp_rwmtx_runlock(&environ_rwmtx); - - return (ErlDrvData)res; -#undef CMD_LINE_PREFIX_STR -#undef CMD_LINE_PREFIX_STR_SZ -} - -#ifdef QNX -static reset_qnx_spawn() -{ - int i; - - /* Reset qnx_spawn_options */ - qnx_spawn_options.flags = 0; - qnx_spawn_options.iov[0] = 0xff; - qnx_spawn_options.iov[1] = 0xff; - qnx_spawn_options.iov[2] = 0xff; - qnx_spawn_options.iov[3] = 0xff; -} -#endif - -#define FD_DEF_HEIGHT 24 -#define FD_DEF_WIDTH 80 -/* Control op */ -#define FD_CTRL_OP_GET_WINSIZE 100 - -static int fd_get_window_size(int fd, Uint32 *width, Uint32 *height) -{ -#ifdef TIOCGWINSZ - struct winsize ws; - if (ioctl(fd,TIOCGWINSZ,&ws) == 0) { - *width = (Uint32) ws.ws_col; - *height = (Uint32) ws.ws_row; - return 0; - } -#endif - return -1; -} - -static ErlDrvSSizeT fd_control(ErlDrvData drv_data, - unsigned int command, - char *buf, ErlDrvSizeT len, - char **rbuf, ErlDrvSizeT rlen) -{ - int fd = (int)(long)drv_data; - char resbuff[2*sizeof(Uint32)]; - switch (command) { - case FD_CTRL_OP_GET_WINSIZE: - { - Uint32 w,h; - if (fd_get_window_size(fd,&w,&h)) - return 0; - memcpy(resbuff,&w,sizeof(Uint32)); - memcpy(resbuff+sizeof(Uint32),&h,sizeof(Uint32)); - } - break; - default: - return 0; - } - if (rlen < 2*sizeof(Uint32)) { - *rbuf = driver_alloc(2*sizeof(Uint32)); - } - memcpy(*rbuf,resbuff,2*sizeof(Uint32)); - return 2*sizeof(Uint32); -} - -static ErlDrvData fd_start(ErlDrvPort port_num, char* name, - SysDriverOpts* opts) -{ - ErlDrvData res; - int non_blocking = 0; - - if (((opts->read_write & DO_READ) && opts->ifd >= max_files) || - ((opts->read_write & DO_WRITE) && opts->ofd >= max_files)) - return ERL_DRV_ERROR_GENERAL; - - /* - * Historical: - * - * "Note about nonblocking I/O. - * - * At least on Solaris, setting the write end of a TTY to nonblocking, - * will set the input end to nonblocking as well (and vice-versa). - * If erl is run in a pipeline like this: cat | erl - * the input end of the TTY will be the standard input of cat. - * And cat is not prepared to handle nonblocking I/O." - * - * Actually, the reason for this is not that the tty itself gets set - * in non-blocking mode, but that the "input end" (cat's stdin) and - * the "output end" (erlang's stdout) are typically the "same" file - * descriptor, dup()'ed from a single fd by one of this process' - * ancestors. - * - * The workaround for this problem used to be a rather bad kludge, - * interposing an extra process ("internal cat") between erlang's - * stdout and the original stdout, allowing erlang to set its stdout - * in non-blocking mode without affecting the stdin of the preceding - * process in the pipeline - and being a kludge, it caused all kinds - * of weird problems. - * - * So, this is the current logic: - * - * The only reason to set non-blocking mode on the output fd at all is - * if it's something that can cause a write() to block, of course, - * i.e. primarily if it points to a tty, socket, pipe, or fifo. - * - * If we don't set non-blocking mode when we "should" have, and output - * becomes blocked, the entire runtime system will be suspended - this - * is normally bad of course, and can happen fairly "easily" - e.g. user - * hits ^S on tty - but doesn't necessarily happen. - * - * If we do set non-blocking mode when we "shouldn't" have, the runtime - * system will end up seeing EOF on the input fd (due to the preceding - * process dying), which typically will cause the entire runtime system - * to terminate immediately (due to whatever erlang process is seeing - * the EOF taking it as a signal to halt the system). This is *very* bad. - * - * I.e. we should take a conservative approach, and only set non- - * blocking mode when we a) need to, and b) are reasonably certain - * that it won't be a problem. And as in the example above, the problem - * occurs when input fd and output fd point to different "things". - * - * However, determining that they are not just the same "type" of - * "thing", but actually the same instance of that type of thing, is - * unreasonably complex in many/most cases. - * - * Also, with pipes, sockets, and fifos it's far from obvious that the - * user *wants* non-blocking output: If you're running erlang inside - * some complex pipeline, you're probably not running a real-time system - * that must never stop, but rather *want* it to suspend if the output - * channel is "full". - * - * So, the bottom line: We will only set the output fd non-blocking if - * it points to a tty, and either a) the input fd also points to a tty, - * or b) we can make sure that setting the output fd non-blocking - * doesn't interfere with someone else's input, via a somewhat milder - * kludge than the above. - * - * Also keep in mind that while this code is almost exclusively run as - * a result of an erlang open_port({fd,0,1}, ...), that isn't the only - * case - it can be called with any old pre-existing file descriptors, - * the relations between which (if they're even two) we can only guess - * at - still, we try our best... - * - * Added note OTP 18: Some systems seem to use stdout/stderr to log data - * using unix pipes, so we cannot allow the system to block on a write. - * Therefore we use an async thread to write the data to fd's that could - * not be set to non-blocking. When no async threads are available we - * fall back on the old behaviour. - * - * Also the guarantee about what is delivered to the OS has changed. - * Pre 18 the fd driver did no flushing of data before terminating. - * Now it does. This is because we want to be able to guarantee that things - * such as escripts and friends really have outputted all data before - * terminating. This could potentially block the termination of the system - * for a very long time, but if the user wants to terminate fast she should - * use erlang:halt with flush=false. - */ - - if (opts->read_write & DO_READ) { - init_fd_data(opts->ifd, port_num); - } - if (opts->read_write & DO_WRITE) { - init_fd_data(opts->ofd, port_num); - - /* If we don't have a read end, all bets are off - no non-blocking. */ - if (opts->read_write & DO_READ) { - - if (isatty(opts->ofd)) { /* output fd is a tty:-) */ - - if (isatty(opts->ifd)) { /* input fd is also a tty */ - - /* To really do this "right", we should also check that - input and output fd point to the *same* tty - but - this seems like overkill; ttyname() isn't for free, - and this is a very common case - and it's hard to - imagine a scenario where setting non-blocking mode - here would cause problems - go ahead and do it. */ - - non_blocking = 1; - SET_NONBLOCKING(opts->ofd); - - } else { /* output fd is a tty, input fd isn't */ - - /* This is a "problem case", but also common (see the - example above) - i.e. it makes sense to try a bit - harder before giving up on non-blocking mode: Try to - re-open the tty that the output fd points to, and if - successful replace the original one with the "new" fd - obtained this way, and set *that* one in non-blocking - mode. (Yes, this is a kludge.) - - However, re-opening the tty may fail in a couple of - (unusual) cases: - - 1) The name of the tty (or an equivalent one, i.e. - same major/minor number) can't be found, because - it actually lives somewhere other than /dev (or - wherever ttyname() looks for it), and isn't - equivalent to any of those that do live in the - "standard" place - this should be *very* unusual. - - 2) Permissions on the tty don't allow us to open it - - it's perfectly possible to have an fd open to an - object whose permissions wouldn't allow us to open - it. This is not as unusual as it sounds, one case - is if the user has su'ed to someone else (not - root) - we have a read/write fd open to the tty - (because it has been inherited all the way down - here), but we have neither read nor write - permission for the tty. - - In these cases, we finally give up, and don't set the - output fd in non-blocking mode. */ - - char *tty; - int nfd; - - if ((tty = ttyname(opts->ofd)) != NULL && - (nfd = open(tty, O_WRONLY)) != -1) { - dup2(nfd, opts->ofd); - close(nfd); - non_blocking = 1; - SET_NONBLOCKING(opts->ofd); - } - } - } - } - } - CHLD_STAT_LOCK; - res = (ErlDrvData)(long)set_driver_data(port_num, opts->ifd, opts->ofd, - opts->packet_bytes, - opts->read_write, 0, -1, - !non_blocking); - CHLD_STAT_UNLOCK; - return res; -} - -static void clear_fd_data(int fd) -{ - if (fd_data[fd].sz > 0) { - erts_free(ERTS_ALC_T_FD_ENTRY_BUF, (void *) fd_data[fd].buf); - ASSERT(erts_smp_atomic_read_nob(&sys_misc_mem_sz) >= fd_data[fd].sz); - erts_smp_atomic_add_nob(&sys_misc_mem_sz, -1*fd_data[fd].sz); - } - fd_data[fd].buf = NULL; - fd_data[fd].sz = 0; - fd_data[fd].remain = 0; - fd_data[fd].cpos = NULL; - fd_data[fd].psz = 0; -} - -static void nbio_stop_fd(ErlDrvPort prt, int fd) -{ - driver_select(prt,fd,DO_READ|DO_WRITE,0); - clear_fd_data(fd); - SET_BLOCKING(fd); -} - -static void fd_stop(ErlDrvData ev) /* Does not close the fds */ -{ - int ofd; - int fd = (int)(long)ev; - ErlDrvPort prt = driver_data[fd].port_num; - -#if FDBLOCK - if (driver_data[fd].blocking) { - erts_free(ERTS_ALC_T_SYS_BLOCKING,driver_data[fd].blocking); - driver_data[fd].blocking = NULL; - erts_smp_atomic_add_nob(&sys_misc_mem_sz, -1*sizeof(ErtsSysBlocking)); - } -#endif - - nbio_stop_fd(prt, fd); - ofd = driver_data[fd].ofd; - if (ofd != fd && ofd != -1) - nbio_stop_fd(prt, ofd); -} - -static void fd_flush(ErlDrvData fd) -{ - if (!driver_data[(int)(long)fd].terminating) - driver_data[(int)(long)fd].terminating = 1; -} - -static ErlDrvData vanilla_start(ErlDrvPort port_num, char* name, - SysDriverOpts* opts) -{ - int flags, fd; - ErlDrvData res; - - flags = (opts->read_write == DO_READ ? O_RDONLY : - opts->read_write == DO_WRITE ? O_WRONLY|O_CREAT|O_TRUNC : - O_RDWR|O_CREAT); - if ((fd = open(name, flags, 0666)) < 0) - return ERL_DRV_ERROR_GENERAL; - if (fd >= max_files) { - close(fd); - return ERL_DRV_ERROR_GENERAL; - } - SET_NONBLOCKING(fd); - init_fd_data(fd, port_num); - - CHLD_STAT_LOCK; - res = (ErlDrvData)(long)set_driver_data(port_num, fd, fd, - opts->packet_bytes, - opts->read_write, 0, -1, 0); - CHLD_STAT_UNLOCK; - return res; -} - -/* Note that driver_data[fd].ifd == fd if the port was opened for reading, */ -/* otherwise (i.e. write only) driver_data[fd].ofd = fd. */ - -static void stop(ErlDrvData fd) -{ - ErlDrvPort prt; - int ofd; - - prt = driver_data[(int)(long)fd].port_num; - nbio_stop_fd(prt, (int)(long)fd); - - ofd = driver_data[(int)(long)fd].ofd; - if (ofd != (int)(long)fd && (int)(long)ofd != -1) - nbio_stop_fd(prt, ofd); - else - ofd = -1; - - CHLD_STAT_LOCK; - - /* Mark as unused. */ - driver_data[(int)(long)fd].pid = -1; - - CHLD_STAT_UNLOCK; - - /* SMP note: Close has to be last thing done (open file descriptors work - as locks on driver_data[] entries) */ - driver_select(prt, (int)(long)fd, ERL_DRV_USE, 0); /* close(fd); */ - if (ofd >= 0) { - driver_select(prt, (int)(long)ofd, ERL_DRV_USE, 0); /* close(ofd); */ - } -} - -/* used by fd_driver */ -static void outputv(ErlDrvData e, ErlIOVec* ev) -{ - int fd = (int)(long)e; - ErlDrvPort ix = driver_data[fd].port_num; - int pb = driver_data[fd].packet_bytes; - int ofd = driver_data[fd].ofd; - ssize_t n; - ErlDrvSizeT sz; - char lb[4]; - char* lbp; - ErlDrvSizeT len = ev->size; - - /* (len > ((unsigned long)-1 >> (4-pb)*8)) */ - /* if (pb >= 0 && (len & (((ErlDrvSizeT)1 << (pb*8))) - 1) != len) {*/ - if (((pb == 2) && (len > 0xffff)) || (pb == 1 && len > 0xff)) { - driver_failure_posix(ix, EINVAL); - return; /* -1; */ - } - /* Handles 0 <= pb <= 4 only */ - put_int32((Uint32) len, lb); - lbp = lb + (4-pb); - - ev->iov[0].iov_base = lbp; - ev->iov[0].iov_len = pb; - ev->size += pb; - - if (driver_data[fd].blocking && FDBLOCK) - driver_pdl_lock(driver_data[fd].blocking->pdl); - - if ((sz = driver_sizeq(ix)) > 0) { - driver_enqv(ix, ev, 0); - - if (driver_data[fd].blocking && FDBLOCK) - driver_pdl_unlock(driver_data[fd].blocking->pdl); - - if (sz + ev->size >= (1 << 13)) - set_busy_port(ix, 1); - } - else if (!driver_data[fd].blocking || !FDBLOCK) { - /* We try to write directly if the fd in non-blocking */ - int vsize = ev->vsize > MAX_VSIZE ? MAX_VSIZE : ev->vsize; - - n = writev(ofd, (const void *) (ev->iov), vsize); - if (n == ev->size) - return; /* 0;*/ - if (n < 0) { - if ((errno != EINTR) && (errno != ERRNO_BLOCK)) { - driver_failure_posix(ix, errno); - return; /* -1;*/ - } - n = 0; - } - driver_enqv(ix, ev, n); /* n is the skip value */ - driver_select(ix, ofd, ERL_DRV_WRITE|ERL_DRV_USE, 1); - } -#if FDBLOCK - else { - if (ev->size != 0) { - driver_enqv(ix, ev, 0); - driver_pdl_unlock(driver_data[fd].blocking->pdl); - driver_async(ix, &driver_data[fd].blocking->pkey, - fd_async, driver_data+fd, NULL); - } else { - driver_pdl_unlock(driver_data[fd].blocking->pdl); - } - } -#endif - /* return 0;*/ -} - -/* Used by spawn_driver and vanilla driver */ -static void output(ErlDrvData e, char* buf, ErlDrvSizeT len) -{ - int fd = (int)(long)e; - ErlDrvPort ix = driver_data[fd].port_num; - int pb = driver_data[fd].packet_bytes; - int ofd = driver_data[fd].ofd; - ssize_t n; - ErlDrvSizeT sz; - char lb[4]; - char* lbp; - struct iovec iv[2]; - - /* (len > ((unsigned long)-1 >> (4-pb)*8)) */ - if (((pb == 2) && (len > 0xffff)) || (pb == 1 && len > 0xff)) { - driver_failure_posix(ix, EINVAL); - return; /* -1; */ - } - put_int32(len, lb); - lbp = lb + (4-pb); - - if ((sz = driver_sizeq(ix)) > 0) { - driver_enq(ix, lbp, pb); - driver_enq(ix, buf, len); - if (sz + len + pb >= (1 << 13)) - set_busy_port(ix, 1); - } - else { - iv[0].iov_base = lbp; - iv[0].iov_len = pb; /* should work for pb=0 */ - iv[1].iov_base = buf; - iv[1].iov_len = len; - n = writev(ofd, iv, 2); - if (n == pb+len) - return; /* 0; */ - if (n < 0) { - if ((errno != EINTR) && (errno != ERRNO_BLOCK)) { - driver_failure_posix(ix, errno); - return; /* -1; */ - } - n = 0; - } - if (n < pb) { - driver_enq(ix, lbp+n, pb-n); - driver_enq(ix, buf, len); - } - else { - n -= pb; - driver_enq(ix, buf+n, len-n); - } - driver_select(ix, ofd, ERL_DRV_WRITE|ERL_DRV_USE, 1); - } - return; /* 0; */ -} - -static int port_inp_failure(ErlDrvPort port_num, int ready_fd, int res) - /* Result: 0 (eof) or -1 (error) */ -{ - int err = errno; - - ASSERT(res <= 0); - (void) driver_select(port_num, ready_fd, ERL_DRV_READ|ERL_DRV_WRITE, 0); - clear_fd_data(ready_fd); - - if (driver_data[ready_fd].blocking && FDBLOCK) { - driver_pdl_lock(driver_data[ready_fd].blocking->pdl); - if (driver_sizeq(driver_data[ready_fd].port_num) > 0) { - driver_pdl_unlock(driver_data[ready_fd].blocking->pdl); - /* We have stuff in the output queue, so we just - set the state to terminating and wait for fd_async_ready - to terminate the port */ - if (res == 0) - driver_data[ready_fd].terminating = 2; - else - driver_data[ready_fd].terminating = -err; - return 0; - } - driver_pdl_unlock(driver_data[ready_fd].blocking->pdl); - } - - if (res == 0) { - if (driver_data[ready_fd].report_exit) { - CHLD_STAT_LOCK; - - if (driver_data[ready_fd].alive) { - /* - * We have eof and want to report exit status, but the process - * hasn't exited yet. When it does report_exit_status() will - * driver_select() this fd which will make sure that we get - * back here with driver_data[ready_fd].alive == 0 and - * driver_data[ready_fd].status set. - */ - CHLD_STAT_UNLOCK; - return 0; - } - else { - int status = driver_data[ready_fd].status; - CHLD_STAT_UNLOCK; - - /* We need not be prepared for stopped/continued processes. */ - if (WIFSIGNALED(status)) - status = 128 + WTERMSIG(status); - else - status = WEXITSTATUS(status); - - driver_report_exit(driver_data[ready_fd].port_num, status); - } - } - driver_failure_eof(port_num); - } else { - driver_failure_posix(port_num, err); - } - return 0; -} - -/* fd is the drv_data that is returned from the */ -/* initial start routine */ -/* ready_fd is the descriptor that is ready to read */ - -static void ready_input(ErlDrvData e, ErlDrvEvent ready_fd) -{ - int fd = (int)(long)e; - ErlDrvPort port_num; - int packet_bytes; - int res; - Uint h; - - port_num = driver_data[fd].port_num; - packet_bytes = driver_data[fd].packet_bytes; - - - if (packet_bytes == 0) { - byte *read_buf = (byte *) erts_alloc(ERTS_ALC_T_SYS_READ_BUF, - ERTS_SYS_READ_BUF_SZ); - res = read(ready_fd, read_buf, ERTS_SYS_READ_BUF_SZ); - if (res < 0) { - if ((errno != EINTR) && (errno != ERRNO_BLOCK)) - port_inp_failure(port_num, ready_fd, res); - } - else if (res == 0) - port_inp_failure(port_num, ready_fd, res); - else - driver_output(port_num, (char*) read_buf, res); - erts_free(ERTS_ALC_T_SYS_READ_BUF, (void *) read_buf); - } - else if (fd_data[ready_fd].remain > 0) { /* We try to read the remainder */ - /* space is allocated in buf */ - res = read(ready_fd, fd_data[ready_fd].cpos, - fd_data[ready_fd].remain); - if (res < 0) { - if ((errno != EINTR) && (errno != ERRNO_BLOCK)) - port_inp_failure(port_num, ready_fd, res); - } - else if (res == 0) { - port_inp_failure(port_num, ready_fd, res); - } - else if (res == fd_data[ready_fd].remain) { /* we're done */ - driver_output(port_num, fd_data[ready_fd].buf, - fd_data[ready_fd].sz); - clear_fd_data(ready_fd); - } - else { /* if (res < fd_data[ready_fd].remain) */ - fd_data[ready_fd].cpos += res; - fd_data[ready_fd].remain -= res; - } - } - else if (fd_data[ready_fd].remain == 0) { /* clean fd */ - byte *read_buf = (byte *) erts_alloc(ERTS_ALC_T_SYS_READ_BUF, - ERTS_SYS_READ_BUF_SZ); - /* We make one read attempt and see what happens */ - res = read(ready_fd, read_buf, ERTS_SYS_READ_BUF_SZ); - if (res < 0) { - if ((errno != EINTR) && (errno != ERRNO_BLOCK)) - port_inp_failure(port_num, ready_fd, res); - } - else if (res == 0) { /* eof */ - port_inp_failure(port_num, ready_fd, res); - } - else if (res < packet_bytes - fd_data[ready_fd].psz) { - memcpy(fd_data[ready_fd].pbuf+fd_data[ready_fd].psz, - read_buf, res); - fd_data[ready_fd].psz += res; - } - else { /* if (res >= packet_bytes) */ - unsigned char* cpos = read_buf; - int bytes_left = res; - - while (1) { - int psz = fd_data[ready_fd].psz; - char* pbp = fd_data[ready_fd].pbuf + psz; - - while(bytes_left && (psz < packet_bytes)) { - *pbp++ = *cpos++; - bytes_left--; - psz++; - } - - if (psz < packet_bytes) { - fd_data[ready_fd].psz = psz; - break; - } - fd_data[ready_fd].psz = 0; - - switch (packet_bytes) { - case 1: h = get_int8(fd_data[ready_fd].pbuf); break; - case 2: h = get_int16(fd_data[ready_fd].pbuf); break; - case 4: h = get_int32(fd_data[ready_fd].pbuf); break; - default: ASSERT(0); return; /* -1; */ - } - - if (h <= (bytes_left)) { - driver_output(port_num, (char*) cpos, h); - cpos += h; - bytes_left -= h; - continue; - } - else { /* The last message we got was split */ - char *buf = erts_alloc_fnf(ERTS_ALC_T_FD_ENTRY_BUF, h); - if (!buf) { - errno = ENOMEM; - port_inp_failure(port_num, ready_fd, -1); - } - else { - erts_smp_atomic_add_nob(&sys_misc_mem_sz, h); - sys_memcpy(buf, cpos, bytes_left); - fd_data[ready_fd].buf = buf; - fd_data[ready_fd].sz = h; - fd_data[ready_fd].remain = h - bytes_left; - fd_data[ready_fd].cpos = buf + bytes_left; - } - break; - } - } - } - erts_free(ERTS_ALC_T_SYS_READ_BUF, (void *) read_buf); - } -} - - -/* fd is the drv_data that is returned from the */ -/* initial start routine */ -/* ready_fd is the descriptor that is ready to read */ - -static void ready_output(ErlDrvData e, ErlDrvEvent ready_fd) -{ - int fd = (int)(long)e; - ErlDrvPort ix = driver_data[fd].port_num; - int n; - struct iovec* iv; - int vsize; - - - if ((iv = (struct iovec*) driver_peekq(ix, &vsize)) == NULL) { - driver_select(ix, ready_fd, ERL_DRV_WRITE, 0); - if (driver_data[fd].terminating) - driver_failure_atom(driver_data[fd].port_num,"normal"); - return; /* 0; */ - } - vsize = vsize > MAX_VSIZE ? MAX_VSIZE : vsize; - if ((n = writev(ready_fd, iv, vsize)) > 0) { - if (driver_deq(ix, n) == 0) - set_busy_port(ix, 0); - } - else if (n < 0) { - if (errno == ERRNO_BLOCK || errno == EINTR) - return; /* 0; */ - else { - int res = errno; - driver_select(ix, ready_fd, ERL_DRV_WRITE, 0); - driver_failure_posix(ix, res); - return; /* -1; */ - } - } - return; /* 0; */ -} - -static void stop_select(ErlDrvEvent fd, void* _) -{ - close((int)fd); -} - -#if FDBLOCK - -static void -fd_async(void *async_data) -{ - int res; - struct driver_data *dd = (struct driver_data*)async_data; - SysIOVec *iov0; - SysIOVec *iov; - int iovlen; - int err = 0; - /* much of this code is stolen from efile_drv:invoke_writev */ - driver_pdl_lock(dd->blocking->pdl); - iov0 = driver_peekq(dd->port_num, &iovlen); - iovlen = iovlen < MAXIOV ? iovlen : MAXIOV; - iov = erts_alloc_fnf(ERTS_ALC_T_SYS_WRITE_BUF, - sizeof(SysIOVec)*iovlen); - if (!iov) { - res = -1; - err = ENOMEM; - driver_pdl_unlock(dd->blocking->pdl); - } else { - memcpy(iov,iov0,iovlen*sizeof(SysIOVec)); - driver_pdl_unlock(dd->blocking->pdl); - - do { - res = writev(dd->ofd, iov, iovlen); - } while (res < 0 && errno == EINTR); - if (res < 0) - err = errno; - - erts_free(ERTS_ALC_T_SYS_WRITE_BUF, iov); - } - dd->blocking->res = res; - dd->blocking->err = err; -} - -void fd_ready_async(ErlDrvData drv_data, - ErlDrvThreadData thread_data) { - struct driver_data *dd = (struct driver_data *)thread_data; - ErlDrvPort port_num = dd->port_num; - - ASSERT(dd->blocking); - ASSERT(dd == (driver_data + (int)(long)drv_data)); - - if (dd->blocking->res > 0) { - driver_pdl_lock(dd->blocking->pdl); - if (driver_deq(port_num, dd->blocking->res) == 0) { - driver_pdl_unlock(dd->blocking->pdl); - set_busy_port(port_num, 0); - if (dd->terminating) { - /* The port is has been ordered to terminate - from either fd_flush or port_inp_failure */ - if (dd->terminating == 1) - driver_failure_atom(port_num, "normal"); - else if (dd->terminating == 2) - driver_failure_eof(port_num); - else if (dd->terminating < 0) - driver_failure_posix(port_num, -dd->terminating); - return; /* -1; */ - } - } else { - driver_pdl_unlock(dd->blocking->pdl); - /* still data left to write in queue */ - driver_async(port_num, &dd->blocking->pkey, fd_async, dd, NULL); - return /* 0; */; - } - } else if (dd->blocking->res < 0) { - if (dd->blocking->err == ERRNO_BLOCK) { - set_busy_port(port_num, 1); - /* still data left to write in queue */ - driver_async(port_num, &dd->blocking->pkey, fd_async, dd, NULL); - } else - driver_failure_posix(port_num, dd->blocking->err); - return; /* -1; */ - } - return; /* 0; */ -} - -#endif - void erts_do_break_handling(void) { struct termios temp_mode; @@ -2738,10 +1011,6 @@ erts_sys_unsetenv(char *key) void sys_init_io(void) { - fd_data = (struct fd_data *) - erts_alloc(ERTS_ALC_T_FD_TAB, max_files * sizeof(struct fd_data)); - erts_smp_atomic_add_nob(&sys_misc_mem_sz, - max_files * sizeof(struct fd_data)); } #if (0) /* unused? */ @@ -2935,179 +1204,6 @@ erl_debug(char* fmt, ...) #endif /* DEBUG */ -static ERTS_INLINE void -report_exit_status(ErtsSysReportExit *rep, int status) -{ - Port *pp; -#ifdef ERTS_SMP - CHLD_STAT_UNLOCK; - pp = erts_thr_id2port_sflgs(rep->port, - ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP); - CHLD_STAT_LOCK; -#else - pp = erts_id2port_sflgs(rep->port, - NULL, - 0, - ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP); -#endif - if (pp) { - if (rep->ifd >= 0) { - driver_data[rep->ifd].alive = 0; - driver_data[rep->ifd].status = status; - (void) driver_select(ERTS_Port2ErlDrvPort(pp), - rep->ifd, - (ERL_DRV_READ|ERL_DRV_USE), - 1); - } - if (rep->ofd >= 0) { - driver_data[rep->ofd].alive = 0; - driver_data[rep->ofd].status = status; - (void) driver_select(ERTS_Port2ErlDrvPort(pp), - rep->ofd, - (ERL_DRV_WRITE|ERL_DRV_USE), - 1); - } -#ifdef ERTS_SMP - erts_thr_port_release(pp); -#else - erts_port_release(pp); -#endif - } - erts_free(ERTS_ALC_T_PRT_REP_EXIT, rep); -} - -#if !CHLDWTHR /* ---------------------------------------------------------- */ - -#define ERTS_REPORT_EXIT_STATUS report_exit_status - -static int check_children(void) -{ - int res = 0; - int pid; - int status; - -#ifndef ERTS_SMP - if (children_died) -#endif - { - sys_sigblock(SIGCHLD); - CHLD_STAT_LOCK; - while ((pid = waitpid(-1, &status, WNOHANG)) > 0) - note_child_death(pid, status); -#ifndef ERTS_SMP - children_died = 0; -#endif - CHLD_STAT_UNLOCK; - sys_sigrelease(SIGCHLD); - res = 1; - } - return res; -} - -#ifdef ERTS_SMP - -void -erts_check_children(void) -{ - (void) check_children(); -} - -#endif - -#elif CHLDWTHR && defined(ERTS_SMP) /* ------------------------------------- */ - -#define ERTS_REPORT_EXIT_STATUS report_exit_status - -#define check_children() (0) - - -#else /* CHLDWTHR && !defined(ERTS_SMP) ------------------------------------ */ - -#define ERTS_REPORT_EXIT_STATUS initiate_report_exit_status - -static ERTS_INLINE void -initiate_report_exit_status(ErtsSysReportExit *rep, int status) -{ - rep->next = report_exit_transit_list; - rep->status = status; - report_exit_transit_list = rep; - erts_sys_schedule_interrupt(1); -} - -static int check_children(void) -{ - int res; - ErtsSysReportExit *rep; - CHLD_STAT_LOCK; - rep = report_exit_transit_list; - res = rep != NULL; - while (rep) { - ErtsSysReportExit *curr_rep = rep; - rep = rep->next; - report_exit_status(curr_rep, curr_rep->status); - } - report_exit_transit_list = NULL; - CHLD_STAT_UNLOCK; - return res; -} - -#endif /* ------------------------------------------------------------------ */ - -static void note_child_death(int pid, int status) -{ - ErtsSysReportExit **repp = &report_exit_list; - ErtsSysReportExit *rep = report_exit_list; - - while (rep) { - if (pid == rep->pid) { - *repp = rep->next; - ERTS_REPORT_EXIT_STATUS(rep, status); - break; - } - repp = &rep->next; - rep = rep->next; - } -} - -#if CHLDWTHR - -static void * -child_waiter(void *unused) -{ - int pid; - int status; - -#ifdef ERTS_ENABLE_LOCK_CHECK - erts_lc_set_thread_name("child waiter"); -#endif - - while(1) { -#ifdef DEBUG - int waitpid_errno; -#endif - pid = waitpid(-1, &status, 0); -#ifdef DEBUG - waitpid_errno = errno; -#endif - CHLD_STAT_LOCK; - if (pid < 0) { - ASSERT(waitpid_errno == ECHILD); - } - else { - children_alive--; - ASSERT(children_alive >= 0); - note_child_death(pid, status); - } - while (!children_alive) - CHLD_STAT_WAIT; /* Wait for children to wait on... :) */ - CHLD_STAT_UNLOCK; - } - - return NULL; -} - -#endif - /* * Called from schedule() when it runs out of runnable processes, * or when Erlang code has performed INPUT_REDUCTIONS reduction @@ -3116,13 +1212,8 @@ child_waiter(void *unused) void erl_sys_schedule(int runnable) { -#ifdef ERTS_SMP ERTS_CHK_IO(!runnable); -#else - ERTS_CHK_IO(runnable ? 0 : !check_children()); -#endif ERTS_SMP_LC_ASSERT(!erts_thr_progress_is_blocking()); - (void) check_children(); } @@ -3150,10 +1241,6 @@ smp_sig_notify(char c) static void * signal_dispatcher_thread_func(void *unused) { -#if !CHLDWTHR - int initialized = 0; - int notify_check_children = 0; -#endif #ifdef ERTS_ENABLE_LOCK_CHECK erts_lc_set_thread_name("signal_dispatcher"); #endif @@ -3191,19 +1278,7 @@ signal_dispatcher_thread_func(void *unused) */ switch (buf[i]) { case 0: /* Emulator initialized */ -#if !CHLDWTHR - initialized = 1; - if (!notify_check_children) -#endif - break; -#if !CHLDWTHR - case 'C': /* SIGCHLD */ - if (initialized) - erts_smp_notify_check_children_needed(); - else - notify_check_children = 1; - break; -#endif + break; case 'I': /* SIGINT */ break_requested(); break; diff --git a/erts/emulator/sys/unix/sys_drivers.c b/erts/emulator/sys/unix/sys_drivers.c new file mode 100644 index 0000000000..2a7cd91265 --- /dev/null +++ b/erts/emulator/sys/unix/sys_drivers.c @@ -0,0 +1,1862 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 1996-2014. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * %CopyrightEnd% + */ + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif + +#ifdef ISC32 +#define _POSIX_SOURCE +#define _XOPEN_SOURCE +#endif + +#include <sys/times.h> /* ! */ +#include <time.h> +#include <signal.h> +#include <sys/wait.h> +#include <sys/uio.h> +#include <termios.h> +#include <ctype.h> +#include <sys/utsname.h> +#include <sys/select.h> +#include <arpa/inet.h> + +#ifdef ISC32 +#include <sys/bsdtypes.h> +#endif + +#include <termios.h> +#ifdef HAVE_FCNTL_H +#include <fcntl.h> +#endif +#ifdef HAVE_SYS_IOCTL_H +#include <sys/ioctl.h> +#endif + +#define WANT_NONBLOCKING /* must define this to pull in defs from sys.h */ +#include "sys.h" + +#ifdef USE_THREADS +#include "erl_threads.h" +#endif + +extern char **environ; +extern erts_smp_rwmtx_t environ_rwmtx; + +extern erts_smp_atomic_t sys_misc_mem_sz; + +static Eterm forker_port; + +#define MAX_VSIZE 16 /* Max number of entries allowed in an I/O + * vector sock_sendv(). + */ +/* + * Don't need global.h, but erl_cpu_topology.h won't compile otherwise + */ +#include "global.h" +#include "erl_cpu_topology.h" + +#include "erl_sys_driver.h" +#include "sys_uds.h" + +#include "erl_child_setup.h" + +#if defined IOV_MAX +#define MAXIOV IOV_MAX +#elif defined UIO_MAXIOV +#define MAXIOV UIO_MAXIOV +#else +#define MAXIOV 16 +#endif + +#ifdef USE_THREADS +# define FDBLOCK 1 +#else +# define FDBLOCK 0 +#endif + +/* Used by the fd driver iff the fd could not be set to non-blocking */ +typedef struct ErtsSysBlocking_ { + ErlDrvPDL pdl; + int res; + int err; + unsigned int pkey; +} ErtsSysBlocking; + +typedef struct fd_data { + int fd; + char pbuf[4]; /* hold partial packet bytes */ + int psz; /* size of pbuf */ + char *buf; + char *cpos; + int sz; + int remain; /* for input on fd */ +} ErtsSysFdData; + +typedef struct driver_data { + ErlDrvPort port_num; + ErtsSysFdData *ofd; + ErtsSysFdData *ifd; + int packet_bytes; + int pid; + int alive; + int status; + int terminating; + ErtsSysBlocking *blocking; +} ErtsSysDriverData; + +#define DIR_SEPARATOR_CHAR '/' + +#if defined(__ANDROID__) +#define SHELL "/system/bin/sh" +#else +#define SHELL "/bin/sh" +#endif /* __ANDROID__ */ + +#if defined(DEBUG) +#define ERL_BUILD_TYPE_MARKER ".debug" +#elif defined(PURIFY) +#define ERL_BUILD_TYPE_MARKER ".purify" +#elif defined(QUANTIFY) +#define ERL_BUILD_TYPE_MARKER ".quantify" +#elif defined(PURECOV) +#define ERL_BUILD_TYPE_MARKER ".purecov" +#elif defined(VALGRIND) +#define ERL_BUILD_TYPE_MARKER ".valgrind" +#else /* opt */ +#define ERL_BUILD_TYPE_MARKER +#endif + +#ifdef DEBUG +#define close(fd) do { int res = close(fd); ASSERT(res > -1); } while(0) +#endif + +#define CHILD_SETUP_PROG_NAME "erl_child_setup" ERL_BUILD_TYPE_MARKER + +// #define HARD_DEBUG +#ifdef HARD_DEBUG +#define driver_select(port_num, fd, flags, onoff) \ + do { \ + if (((flags) & ERL_DRV_READ) && onoff) \ + fprintf(stderr,"%010d %p: read select %d\r\n", __LINE__, port_num, (int)fd); \ + if (((flags) & ERL_DRV_WRITE) && onoff) \ + fprintf(stderr,"%010d %p: writ select %d\r\n", __LINE__, port_num, (int)fd); \ + if (((flags) & ERL_DRV_READ) && !onoff) \ + fprintf(stderr,"%010d %p: read unsele %d\r\n", __LINE__, port_num, (int)fd); \ + if (((flags) & ERL_DRV_WRITE) && !onoff) \ + fprintf(stderr,"%010d %p: writ unsele %d\r\n", __LINE__, port_num, (int)fd); \ + driver_select_nkp(port_num, fd, flags, onoff); \ + } while(0) +#endif + +/* + * Decreasing the size of it below 16384 is not allowed. + */ + +#define ERTS_SYS_READ_BUF_SZ (64*1024) + +/* I. Initialization */ + +void +erl_sys_late_init(void) +{ + SysDriverOpts opts; +#ifdef ERTS_SMP + Port *port; +#endif + + sys_signal(SIGPIPE, SIG_IGN); /* Ignore - we'll handle the write failure */ + + opts.packet_bytes = 0; + opts.use_stdio = 1; + opts.redir_stderr = 0; + opts.read_write = 0; + opts.hide_window = 0; + opts.wd = NULL; + opts.envir = NULL; + opts.exit_status = 0; + opts.overlapped_io = 0; + opts.spawn_type = ERTS_SPAWN_ANY; + opts.argv = NULL; + opts.parallelism = erts_port_parallelism; + +#ifdef ERTS_SMP + port = +#endif + erts_open_driver(&forker_driver, make_internal_pid(0), "forker", &opts, NULL, NULL); +#ifdef ERTS_SMP + erts_mtx_unlock(port->lock); +#endif +} + +/* II. Prototypes */ + +/* II.I Spawn prototypes */ +static ErlDrvData spawn_start(ErlDrvPort, char*, SysDriverOpts*); +static ErlDrvSSizeT spawn_control(ErlDrvData, unsigned int, char *, + ErlDrvSizeT, char **, ErlDrvSizeT); + +/* II.II Vanilla prototypes */ +static ErlDrvData vanilla_start(ErlDrvPort, char*, SysDriverOpts*); + + +/* II.III FD prototypes */ +static ErlDrvData fd_start(ErlDrvPort, char*, SysDriverOpts*); +#if FDBLOCK +static void fd_async(void *); +static void fd_ready_async(ErlDrvData drv_data, ErlDrvThreadData thread_data); +#endif +static ErlDrvSSizeT fd_control(ErlDrvData, unsigned int, char *, ErlDrvSizeT, + char **, ErlDrvSizeT); +static void fd_stop(ErlDrvData); +static void fd_flush(ErlDrvData); + +/* II.IV Common prototypes */ +static void stop(ErlDrvData); +static void ready_input(ErlDrvData, ErlDrvEvent); +static void ready_output(ErlDrvData, ErlDrvEvent); +static void output(ErlDrvData, char*, ErlDrvSizeT); +static void outputv(ErlDrvData, ErlIOVec*); +static void stop_select(ErlDrvEvent, void*); + +/* II.V Forker prototypes */ +static ErlDrvData forker_start(ErlDrvPort, char*, SysDriverOpts*); +static void forker_stop(ErlDrvData); +static void forker_ready_input(ErlDrvData, ErlDrvEvent); +static void forker_ready_output(ErlDrvData, ErlDrvEvent); +static ErlDrvSSizeT forker_control(ErlDrvData, unsigned int, char *, + ErlDrvSizeT, char **, ErlDrvSizeT); + +/* III Driver entries */ + +/* III.I The spawn driver */ +struct erl_drv_entry spawn_driver_entry = { + NULL, + spawn_start, + stop, + output, + ready_input, + ready_output, + "spawn", + NULL, + NULL, + spawn_control, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + ERL_DRV_EXTENDED_MARKER, + ERL_DRV_EXTENDED_MAJOR_VERSION, + ERL_DRV_EXTENDED_MINOR_VERSION, + ERL_DRV_FLAG_USE_PORT_LOCKING | ERL_DRV_FLAG_USE_INIT_ACK, + NULL, NULL, + stop_select +}; + +/* III.II The fd driver */ +struct erl_drv_entry fd_driver_entry = { + NULL, + fd_start, + fd_stop, + output, + ready_input, + ready_output, + "fd", + NULL, + NULL, + fd_control, + NULL, + outputv, +#if FDBLOCK + fd_ready_async, /* ready_async */ +#else + NULL, +#endif + fd_flush, /* flush */ + NULL, /* call */ + NULL, /* event */ + ERL_DRV_EXTENDED_MARKER, + ERL_DRV_EXTENDED_MAJOR_VERSION, + ERL_DRV_EXTENDED_MINOR_VERSION, + 0, /* ERL_DRV_FLAGs */ + NULL, /* handle2 */ + NULL, /* process_exit */ + stop_select +}; + +/* III.III The vanilla driver */ +struct erl_drv_entry vanilla_driver_entry = { + NULL, + vanilla_start, + stop, + output, + ready_input, + ready_output, + "vanilla", + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, /* flush */ + NULL, /* call */ + NULL, /* event */ + ERL_DRV_EXTENDED_MARKER, + ERL_DRV_EXTENDED_MAJOR_VERSION, + ERL_DRV_EXTENDED_MINOR_VERSION, + 0, /* ERL_DRV_FLAGs */ + NULL, /* handle2 */ + NULL, /* process_exit */ + stop_select +}; + +/* III.III The forker driver */ +struct erl_drv_entry forker_driver_entry = { + NULL, + forker_start, + forker_stop, + NULL, + forker_ready_input, + forker_ready_output, + "spawn_forker", + NULL, + NULL, + forker_control, + NULL, + NULL, + NULL, + NULL, + NULL, + NULL, + ERL_DRV_EXTENDED_MARKER, + ERL_DRV_EXTENDED_MAJOR_VERSION, + ERL_DRV_EXTENDED_MINOR_VERSION, + 0, + NULL, NULL, + stop_select +}; + +/* Untility functions */ + +static int set_blocking_data(ErtsSysDriverData *dd) { + + dd->blocking = erts_alloc(ERTS_ALC_T_SYS_BLOCKING, sizeof(ErtsSysBlocking)); + + erts_smp_atomic_add_nob(&sys_misc_mem_sz, sizeof(ErtsSysBlocking)); + + dd->blocking->pdl = driver_pdl_create(dd->port_num); + dd->blocking->res = 0; + dd->blocking->err = 0; + dd->blocking->pkey = driver_async_port_key(dd->port_num); + + return 1; +} + +static void init_fd_data(ErtsSysFdData *fd_data, int fd) +{ + fd_data->fd = fd; + fd_data->buf = NULL; + fd_data->cpos = NULL; + fd_data->remain = 0; + fd_data->sz = 0; + fd_data->psz = 0; +} + +static ErtsSysDriverData * +create_driver_data(ErlDrvPort port_num, + int ifd, + int ofd, + int packet_bytes, + int read_write, + int exit_status, + int pid, + int is_blocking) +{ + Port *prt; + ErtsSysDriverData *driver_data; + char *data; + int size = sizeof(ErtsSysDriverData); + + if (read_write & DO_READ) + size += sizeof(ErtsSysFdData); + + if ((read_write & DO_WRITE) && + ((ifd != ofd || ofd == -1) || !(read_write & DO_READ))) + size += sizeof(ErtsSysFdData); + + data = erts_alloc(ERTS_ALC_T_DRV_TAB,size); + erts_smp_atomic_add_nob(&sys_misc_mem_sz, size); + + driver_data = (ErtsSysDriverData*)data; + data += sizeof(*driver_data); + + prt = erts_drvport2port(port_num); + if (prt != ERTS_INVALID_ERL_DRV_PORT) + prt->os_pid = pid; + + driver_data->packet_bytes = packet_bytes; + driver_data->port_num = port_num; + driver_data->pid = pid; + driver_data->alive = exit_status ? 1 : 0; + driver_data->status = 0; + driver_data->terminating = 0; + driver_data->blocking = NULL; + + if (read_write & DO_READ) { + driver_data->ifd = (ErtsSysFdData*)data; + data += sizeof(*driver_data->ifd); + init_fd_data(driver_data->ifd, ifd); + driver_select(port_num, ifd, (ERL_DRV_READ|ERL_DRV_USE), 1); + } else { + driver_data->ifd = NULL; + } + + if (read_write & DO_WRITE) { + if (ofd != -1 && ifd == ofd && read_write & DO_READ) { + /* This is for when ifd and ofd are the same fd */ + driver_data->ofd = driver_data->ifd; + } else { + driver_data->ofd = (ErtsSysFdData*)data; + data += sizeof(*driver_data->ofd); + init_fd_data(driver_data->ofd, ofd); + } + if (is_blocking && FDBLOCK) + if (!set_blocking_data(driver_data)) { + erts_free(ERTS_ALC_T_DRV_TAB, driver_data); + return NULL; + } + } else { + driver_data->ofd = NULL; + } + + return driver_data; +} + +/* Spawn driver */ + +static void close_pipes(int ifd[2], int ofd[2]) +{ + close(ifd[0]); + close(ifd[1]); + close(ofd[0]); + close(ofd[1]); +} + +static char **build_unix_environment(char *block) +{ + int i; + int j; + int len; + char *cp; + char **cpp; + char** old_env; + + ERTS_SMP_LC_ASSERT(erts_smp_lc_rwmtx_is_rlocked(&environ_rwmtx)); + + cp = block; + len = 0; + while (*cp != '\0') { + cp += strlen(cp) + 1; + len++; + } + old_env = environ; + while (*old_env++ != NULL) { + len++; + } + + cpp = (char **) erts_alloc_fnf(ERTS_ALC_T_ENVIRONMENT, + sizeof(char *) * (len+1)); + if (cpp == NULL) { + return NULL; + } + + cp = block; + len = 0; + while (*cp != '\0') { + cpp[len] = cp; + cp += strlen(cp) + 1; + len++; + } + + i = len; + for (old_env = environ; *old_env; old_env++) { + char* old = *old_env; + + for (j = 0; j < len; j++) { + char *s, *t; + + /* check if cpp[j] equals old + before the = sign, + i.e. + "TMPDIR=/tmp/" */ + s = cpp[j]; + t = old; + while (*s == *t && *s != '=') { + s++, t++; + } + if (*s == '=' && *t == '=') { + break; + } + } + + if (j == len) { /* New version not found */ + cpp[len++] = old; + } + } + + for (j = 0; j < i; ) { + size_t last = strlen(cpp[j])-1; + if (cpp[j][last] == '=' && strchr(cpp[j], '=') == cpp[j]+last) { + cpp[j] = cpp[--len]; + if (len < i) { + i--; + } else { + j++; + } + } + else { + j++; + } + } + + cpp[len] = NULL; + return cpp; +} + +static ErlDrvData spawn_start(ErlDrvPort port_num, char* name, + SysDriverOpts* opts) +{ +#define CMD_LINE_PREFIX_STR "exec " +#define CMD_LINE_PREFIX_STR_SZ (sizeof(CMD_LINE_PREFIX_STR) - 1) + + int len; + char **new_environ; + ErtsSysDriverData *dd; + char *cmd_line; + char wd_buff[MAXPATHLEN+1]; + char *wd; + int ifd[2], ofd[2], stderrfd; + + if (pipe(ifd) < 0) return ERL_DRV_ERROR_ERRNO; + errno = EMFILE; /* default for next three conditions */ + if (ifd[0] >= sys_max_files() || pipe(ofd) < 0) { + close(ifd[0]); + close(ifd[1]); + return ERL_DRV_ERROR_ERRNO; + } + if (ofd[1] >= sys_max_files()) { + close_pipes(ifd, ofd); + errno = EMFILE; + return ERL_DRV_ERROR_ERRNO; + } + + SET_NONBLOCKING(ifd[0]); + SET_NONBLOCKING(ofd[1]); + + stderrfd = opts->redir_stderr ? ifd[1] : dup(2); + + if (stderrfd >= sys_max_files() || stderrfd < 0) { + close_pipes(ifd, ofd); + if (stderrfd > -1) + close(stderrfd); + return ERL_DRV_ERROR_ERRNO; + } + + if (opts->spawn_type == ERTS_SPAWN_EXECUTABLE) { + /* started with spawn_executable, not with spawn */ + len = strlen(name); + cmd_line = (char *) erts_alloc_fnf(ERTS_ALC_T_TMP, len + 1); + if (!cmd_line) { + close_pipes(ifd, ofd); + errno = ENOMEM; + return ERL_DRV_ERROR_ERRNO; + } + memcpy((void *) cmd_line,(void *) name, len); + cmd_line[len] = '\0'; + len = len + 1; + if (access(cmd_line,X_OK) != 0) { + int save_errno = errno; + erts_free(ERTS_ALC_T_TMP, cmd_line); + close_pipes(ifd, ofd); + errno = save_errno; + return ERL_DRV_ERROR_ERRNO; + } + } else { + /* make the string suitable for giving to "sh" */ + len = strlen(name); + cmd_line = (char *) erts_alloc_fnf(ERTS_ALC_T_TMP, + CMD_LINE_PREFIX_STR_SZ + len + 1); + if (!cmd_line) { + close_pipes(ifd, ofd); + errno = ENOMEM; + return ERL_DRV_ERROR_ERRNO; + } + memcpy((void *) cmd_line, + (void *) CMD_LINE_PREFIX_STR, + CMD_LINE_PREFIX_STR_SZ); + memcpy((void *) (cmd_line + CMD_LINE_PREFIX_STR_SZ), (void *) name, len); + cmd_line[CMD_LINE_PREFIX_STR_SZ + len] = '\0'; + len = CMD_LINE_PREFIX_STR_SZ + len + 1; + } + + erts_smp_rwmtx_rlock(&environ_rwmtx); + + if (opts->envir == NULL) { + new_environ = environ; + } else if ((new_environ = build_unix_environment(opts->envir)) == NULL) { + erts_smp_rwmtx_runlock(&environ_rwmtx); + close_pipes(ifd, ofd); + erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); + errno = ENOMEM; + return ERL_DRV_ERROR_ERRNO; + } + + if (opts->wd == NULL) { + if ((wd = getcwd(wd_buff, MAXPATHLEN+1)) == NULL) { + /* on some OSs this call opens a fd in the + background which means that this can + return EMFILE */ + int err = errno; + close_pipes(ifd, ofd); + erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); + if (new_environ != environ) + erts_free(ERTS_ALC_T_ENVIRONMENT, (void *) new_environ); + erts_smp_rwmtx_runlock(&environ_rwmtx); + errno = err; + return ERL_DRV_ERROR_ERRNO; + } + } else { + wd = opts->wd; + } + + { + struct iovec *io_vector; + int iov_len = 5; + char nullbuff[] = "\0"; + int j, i = 0, res; + Sint32 buffsz = 0, env_len = 0, argv_len = 0, + flags = (opts->use_stdio ? FORKER_FLAG_USE_STDIO : 0) + | (opts->exit_status ? FORKER_FLAG_EXIT_STATUS : 0) + | (opts->read_write & DO_READ ? FORKER_FLAG_DO_READ : 0) + | (opts->read_write & DO_WRITE ? FORKER_FLAG_DO_WRITE : 0); + + /* count number of elements in environment */ + while(new_environ[env_len] != NULL) + env_len++; + iov_len += 1 + env_len; /* num envs including size int */ + + /* count number of element in argument list */ + if (opts->spawn_type == ERTS_SPAWN_EXECUTABLE) { + if (opts->argv != NULL) { + while(opts->argv[argv_len] != NULL) + argv_len++; + } else { + argv_len++; + } + iov_len += 1 + argv_len; /* num argvs including size int */ + } + + io_vector = erts_alloc_fnf(ERTS_ALC_T_TMP, sizeof(struct iovec) * iov_len); + + if (!io_vector) { + close_pipes(ifd, ofd); + erts_smp_rwmtx_runlock(&environ_rwmtx); + erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); + if (new_environ != environ) + erts_free(ERTS_ALC_T_ENVIRONMENT, (void *) new_environ); + errno = ENOMEM; + return ERL_DRV_ERROR_ERRNO; + } + + io_vector[i].iov_base = (void*)&buffsz; + io_vector[i++].iov_len = sizeof(buffsz); + + io_vector[i].iov_base = (void*)&flags; + flags = htonl(flags); + io_vector[i++].iov_len = sizeof(flags); + buffsz += sizeof(flags); + + io_vector[i].iov_base = cmd_line; + io_vector[i++].iov_len = len; + buffsz += len; + + io_vector[i].iov_base = wd; + io_vector[i].iov_len = strlen(io_vector[i].iov_base) + 1; + buffsz += io_vector[i++].iov_len; + + io_vector[i].iov_base = nullbuff; + io_vector[i++].iov_len = 1; + buffsz += io_vector[i-1].iov_len; + + io_vector[i].iov_base = (void*)&env_len; + env_len = htonl(env_len); + io_vector[i++].iov_len = sizeof(env_len); + buffsz += io_vector[i-1].iov_len; + + for (j = 0; new_environ[j] != NULL; j++) { + io_vector[i].iov_base = new_environ[j]; + io_vector[i++].iov_len = strlen(new_environ[j]) + 1; + buffsz += io_vector[i-1].iov_len; + } + + /* only append arguments if this was a spawn_executable */ + if (opts->spawn_type == ERTS_SPAWN_EXECUTABLE) { + + io_vector[i].iov_base = (void*)&argv_len; + argv_len = htonl(argv_len); + io_vector[i++].iov_len = sizeof(argv_len); + buffsz += io_vector[i-1].iov_len; + + if (opts->argv) { + /* If there are arguments we copy in the references to + them into the iov */ + for (j = 0; opts->argv[j]; j++) { + if (opts->argv[j] == erts_default_arg0) + io_vector[i].iov_base = cmd_line; + else + io_vector[i].iov_base = opts->argv[j]; + io_vector[i].iov_len = strlen(io_vector[i].iov_base) + 1; + buffsz += io_vector[i++].iov_len; + } + } else { + io_vector[i].iov_base = cmd_line; + io_vector[i].iov_len = strlen(io_vector[i].iov_base) + 1; + buffsz += io_vector[i++].iov_len; + } + } + + /* we send the request to do the fork */ + if ((res = writev(ofd[1], io_vector, iov_len > MAXIOV ? MAXIOV : iov_len)) < 0) { + if (errno == ERRNO_BLOCK) { + res = 0; + } else { + int err = errno; + close_pipes(ifd, ofd); + erts_free(ERTS_ALC_T_TMP, io_vector); + if (new_environ != environ) + erts_free(ERTS_ALC_T_ENVIRONMENT, (void *) new_environ); + erts_smp_rwmtx_runlock(&environ_rwmtx); + erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); + errno = err; + return ERL_DRV_ERROR_ERRNO; + } + } + + if (res < buffsz) { + /* we only wrote part of the command payload. Enqueue the rest. */ + for (i = 0; i < iov_len; i++) { + driver_enq(port_num, io_vector[i].iov_base, io_vector[i].iov_len); + } + driver_deq(port_num, res); + driver_select(port_num, ofd[1], ERL_DRV_WRITE|ERL_DRV_USE, 1); + } + + erts_free(ERTS_ALC_T_TMP, io_vector); + } + + erts_free(ERTS_ALC_T_TMP, (void *) cmd_line); + + if (new_environ != environ) + erts_free(ERTS_ALC_T_ENVIRONMENT, (void *) new_environ); + + erts_smp_rwmtx_runlock(&environ_rwmtx); + + dd = create_driver_data(port_num, ifd[0], ofd[1], opts->packet_bytes, + DO_WRITE | DO_READ, opts->exit_status, + 0, 0); + + { + /* send ofd[0] + ifd[1] + stderrfd to forker port */ + ErtsSysForkerProto *proto = + erts_alloc(ERTS_ALC_T_DRV_CTRL_DATA, + sizeof(ErtsSysForkerProto)); + memset(proto, 0, sizeof(ErtsSysForkerProto)); + proto->action = ErtsSysForkerProtoAction_Start; + proto->u.start.fds[0] = ofd[0]; + proto->u.start.fds[1] = ifd[1]; + proto->u.start.fds[2] = stderrfd; + proto->u.start.port_id = opts->exit_status ? erts_drvport2id(port_num) : THE_NON_VALUE; + if (erl_drv_port_control(forker_port, 'S', (char*)proto, sizeof(*proto))) { + /* The forker port has been killed, we close both fd's which will + make open_port throw an epipe error */ + close(ofd[0]); + close(ifd[1]); + } + } + + /* we set these fds to negative to mark if + they should be closed after the handshake */ + if (!(opts->read_write & DO_READ)) + dd->ifd->fd *= -1; + + if (!(opts->read_write & DO_WRITE)) + dd->ofd->fd *= -1; + + return (ErlDrvData)dd; +#undef CMD_LINE_PREFIX_STR +#undef CMD_LINE_PREFIX_STR_SZ +} + +static ErlDrvSSizeT spawn_control(ErlDrvData e, unsigned int cmd, char *buf, + ErlDrvSizeT len, char **rbuf, ErlDrvSizeT rlen) +{ + ErtsSysDriverData *dd = (ErtsSysDriverData*)e; + ErtsSysForkerProto *proto = (ErtsSysForkerProto *)buf; + + ASSERT(len == sizeof(*proto)); + ASSERT(proto->action == ErtsSysForkerProtoAction_SigChld); + + dd->status = proto->u.sigchld.error_number; + dd->alive = -1; + + if (dd->ifd) + driver_select(dd->port_num, abs(dd->ifd->fd), ERL_DRV_READ | ERL_DRV_USE, 1); + + if (dd->ofd) + driver_select(dd->port_num, abs(dd->ofd->fd), ERL_DRV_WRITE | ERL_DRV_USE, 1); + + return 0; +} + +#define FD_DEF_HEIGHT 24 +#define FD_DEF_WIDTH 80 +/* Control op */ +#define FD_CTRL_OP_GET_WINSIZE 100 + +static int fd_get_window_size(int fd, Uint32 *width, Uint32 *height) +{ +#ifdef TIOCGWINSZ + struct winsize ws; + if (ioctl(fd,TIOCGWINSZ,&ws) == 0) { + *width = (Uint32) ws.ws_col; + *height = (Uint32) ws.ws_row; + return 0; + } +#endif + return -1; +} + +static ErlDrvSSizeT fd_control(ErlDrvData drv_data, + unsigned int command, + char *buf, ErlDrvSizeT len, + char **rbuf, ErlDrvSizeT rlen) +{ + int fd = (int)(long)drv_data; + char resbuff[2*sizeof(Uint32)]; + switch (command) { + case FD_CTRL_OP_GET_WINSIZE: + { + Uint32 w,h; + if (fd_get_window_size(fd,&w,&h)) + return 0; + memcpy(resbuff,&w,sizeof(Uint32)); + memcpy(resbuff+sizeof(Uint32),&h,sizeof(Uint32)); + } + break; + default: + return 0; + } + if (rlen < 2*sizeof(Uint32)) { + *rbuf = driver_alloc(2*sizeof(Uint32)); + } + memcpy(*rbuf,resbuff,2*sizeof(Uint32)); + return 2*sizeof(Uint32); +} + +static ErlDrvData fd_start(ErlDrvPort port_num, char* name, + SysDriverOpts* opts) +{ + int non_blocking = 0; + + if (((opts->read_write & DO_READ) && opts->ifd >= sys_max_files()) || + ((opts->read_write & DO_WRITE) && opts->ofd >= sys_max_files())) + return ERL_DRV_ERROR_GENERAL; + + /* + * Historical: + * + * "Note about nonblocking I/O. + * + * At least on Solaris, setting the write end of a TTY to nonblocking, + * will set the input end to nonblocking as well (and vice-versa). + * If erl is run in a pipeline like this: cat | erl + * the input end of the TTY will be the standard input of cat. + * And cat is not prepared to handle nonblocking I/O." + * + * Actually, the reason for this is not that the tty itself gets set + * in non-blocking mode, but that the "input end" (cat's stdin) and + * the "output end" (erlang's stdout) are typically the "same" file + * descriptor, dup()'ed from a single fd by one of this process' + * ancestors. + * + * The workaround for this problem used to be a rather bad kludge, + * interposing an extra process ("internal cat") between erlang's + * stdout and the original stdout, allowing erlang to set its stdout + * in non-blocking mode without affecting the stdin of the preceding + * process in the pipeline - and being a kludge, it caused all kinds + * of weird problems. + * + * So, this is the current logic: + * + * The only reason to set non-blocking mode on the output fd at all is + * if it's something that can cause a write() to block, of course, + * i.e. primarily if it points to a tty, socket, pipe, or fifo. + * + * If we don't set non-blocking mode when we "should" have, and output + * becomes blocked, the entire runtime system will be suspended - this + * is normally bad of course, and can happen fairly "easily" - e.g. user + * hits ^S on tty - but doesn't necessarily happen. + * + * If we do set non-blocking mode when we "shouldn't" have, the runtime + * system will end up seeing EOF on the input fd (due to the preceding + * process dying), which typically will cause the entire runtime system + * to terminate immediately (due to whatever erlang process is seeing + * the EOF taking it as a signal to halt the system). This is *very* bad. + * + * I.e. we should take a conservative approach, and only set non- + * blocking mode when we a) need to, and b) are reasonably certain + * that it won't be a problem. And as in the example above, the problem + * occurs when input fd and output fd point to different "things". + * + * However, determining that they are not just the same "type" of + * "thing", but actually the same instance of that type of thing, is + * unreasonably complex in many/most cases. + * + * Also, with pipes, sockets, and fifos it's far from obvious that the + * user *wants* non-blocking output: If you're running erlang inside + * some complex pipeline, you're probably not running a real-time system + * that must never stop, but rather *want* it to suspend if the output + * channel is "full". + * + * So, the bottom line: We will only set the output fd non-blocking if + * it points to a tty, and either a) the input fd also points to a tty, + * or b) we can make sure that setting the output fd non-blocking + * doesn't interfere with someone else's input, via a somewhat milder + * kludge than the above. + * + * Also keep in mind that while this code is almost exclusively run as + * a result of an erlang open_port({fd,0,1}, ...), that isn't the only + * case - it can be called with any old pre-existing file descriptors, + * the relations between which (if they're even two) we can only guess + * at - still, we try our best... + * + * Added note OTP 18: Some systems seem to use stdout/stderr to log data + * using unix pipes, so we cannot allow the system to block on a write. + * Therefore we use an async thread to write the data to fd's that could + * not be set to non-blocking. When no async threads are available we + * fall back on the old behaviour. + * + * Also the guarantee about what is delivered to the OS has changed. + * Pre 18 the fd driver did no flushing of data before terminating. + * Now it does. This is because we want to be able to guarantee that things + * such as escripts and friends really have outputted all data before + * terminating. This could potentially block the termination of the system + * for a very long time, but if the user wants to terminate fast she should + * use erlang:halt with flush=false. + */ + + /* Try to figure out if we can use non-blocking writes */ + if (opts->read_write & DO_WRITE) { + + /* If we don't have a read end, all bets are off - no non-blocking. */ + if (opts->read_write & DO_READ) { + + if (isatty(opts->ofd)) { /* output fd is a tty:-) */ + + if (isatty(opts->ifd)) { /* input fd is also a tty */ + + /* To really do this "right", we should also check that + input and output fd point to the *same* tty - but + this seems like overkill; ttyname() isn't for free, + and this is a very common case - and it's hard to + imagine a scenario where setting non-blocking mode + here would cause problems - go ahead and do it. */ + + non_blocking = 1; + SET_NONBLOCKING(opts->ofd); + + } else { /* output fd is a tty, input fd isn't */ + + /* This is a "problem case", but also common (see the + example above) - i.e. it makes sense to try a bit + harder before giving up on non-blocking mode: Try to + re-open the tty that the output fd points to, and if + successful replace the original one with the "new" fd + obtained this way, and set *that* one in non-blocking + mode. (Yes, this is a kludge.) + + However, re-opening the tty may fail in a couple of + (unusual) cases: + + 1) The name of the tty (or an equivalent one, i.e. + same major/minor number) can't be found, because + it actually lives somewhere other than /dev (or + wherever ttyname() looks for it), and isn't + equivalent to any of those that do live in the + "standard" place - this should be *very* unusual. + + 2) Permissions on the tty don't allow us to open it - + it's perfectly possible to have an fd open to an + object whose permissions wouldn't allow us to open + it. This is not as unusual as it sounds, one case + is if the user has su'ed to someone else (not + root) - we have a read/write fd open to the tty + (because it has been inherited all the way down + here), but we have neither read nor write + permission for the tty. + + In these cases, we finally give up, and don't set the + output fd in non-blocking mode. */ + + char *tty; + int nfd; + + if ((tty = ttyname(opts->ofd)) != NULL && + (nfd = open(tty, O_WRONLY)) != -1) { + dup2(nfd, opts->ofd); + close(nfd); + non_blocking = 1; + SET_NONBLOCKING(opts->ofd); + } + } + } + } + } + return (ErlDrvData)create_driver_data(port_num, opts->ifd, opts->ofd, + opts->packet_bytes, + opts->read_write, 0, -1, + !non_blocking); +} + +static void clear_fd_data(ErtsSysFdData *fdd) +{ + if (fdd->sz > 0) { + erts_free(ERTS_ALC_T_FD_ENTRY_BUF, (void *) fdd->buf); + ASSERT(erts_smp_atomic_read_nob(&sys_misc_mem_sz) >= fdd->sz); + erts_smp_atomic_add_nob(&sys_misc_mem_sz, -1*fdd->sz); + } + fdd->buf = NULL; + fdd->sz = 0; + fdd->remain = 0; + fdd->cpos = NULL; + fdd->psz = 0; +} + +static void nbio_stop_fd(ErlDrvPort prt, ErtsSysFdData *fdd) +{ + driver_select(prt, abs(fdd->fd), DO_READ|DO_WRITE, 0); + clear_fd_data(fdd); + SET_BLOCKING(abs(fdd->fd)); + +} + +static void fd_stop(ErlDrvData ev) /* Does not close the fds */ +{ + ErtsSysDriverData* dd = (ErtsSysDriverData*)ev; + ErlDrvPort prt = dd->port_num; + int sz = sizeof(ErtsSysDriverData); + +#if FDBLOCK + if (dd->blocking) { + erts_free(ERTS_ALC_T_SYS_BLOCKING, dd->blocking); + dd->blocking = NULL; + sz += sizeof(ErtsSysBlocking); + } +#endif + + if (dd->ifd) { + sz += sizeof(ErtsSysFdData); + nbio_stop_fd(prt, dd->ifd); + } + if (dd->ofd && dd->ofd != dd->ifd) { + sz += sizeof(ErtsSysFdData); + nbio_stop_fd(prt, dd->ofd); + } + + erts_free(ERTS_ALC_T_DRV_TAB, dd); + erts_smp_atomic_add_nob(&sys_misc_mem_sz, -sz); +} + +static void fd_flush(ErlDrvData ev) +{ + ErtsSysDriverData* dd = (ErtsSysDriverData*)ev; + if (!dd->terminating) + dd->terminating = 1; +} + +static ErlDrvData vanilla_start(ErlDrvPort port_num, char* name, + SysDriverOpts* opts) +{ + int flags, fd; + ErlDrvData res; + + flags = (opts->read_write == DO_READ ? O_RDONLY : + opts->read_write == DO_WRITE ? O_WRONLY|O_CREAT|O_TRUNC : + O_RDWR|O_CREAT); + if ((fd = open(name, flags, 0666)) < 0) + return ERL_DRV_ERROR_GENERAL; + if (fd >= sys_max_files()) { + close(fd); + return ERL_DRV_ERROR_GENERAL; + } + SET_NONBLOCKING(fd); + + res = (ErlDrvData)(long)create_driver_data(port_num, fd, fd, + opts->packet_bytes, + opts->read_write, 0, -1, 0); + return res; +} + +/* Note that driver_data[fd].ifd == fd if the port was opened for reading, */ +/* otherwise (i.e. write only) driver_data[fd].ofd = fd. */ + +static void stop(ErlDrvData ev) +{ + ErtsSysDriverData* dd = (ErtsSysDriverData*)ev; + ErlDrvPort prt = dd->port_num; + + if (dd->ifd) { + nbio_stop_fd(prt, dd->ifd); + driver_select(prt, abs(dd->ifd->fd), ERL_DRV_USE, 0); /* close(ifd); */ + } + + if (dd->ofd && dd->ofd != dd->ifd) { + nbio_stop_fd(prt, dd->ofd); + driver_select(prt, abs(dd->ofd->fd), ERL_DRV_USE, 0); /* close(ofd); */ + } + + erts_free(ERTS_ALC_T_DRV_TAB, dd); +} + +/* used by fd_driver */ +static void outputv(ErlDrvData e, ErlIOVec* ev) +{ + ErtsSysDriverData *dd = (ErtsSysDriverData*)e; + ErlDrvPort ix = dd->port_num; + int pb = dd->packet_bytes; + int ofd = dd->ofd ? dd->ofd->fd : -1; + ssize_t n; + ErlDrvSizeT sz; + char lb[4]; + char* lbp; + ErlDrvSizeT len = ev->size; + + /* (len > ((unsigned long)-1 >> (4-pb)*8)) */ + /* if (pb >= 0 && (len & (((ErlDrvSizeT)1 << (pb*8))) - 1) != len) {*/ + if (((pb == 2) && (len > 0xffff)) || (pb == 1 && len > 0xff)) { + driver_failure_posix(ix, EINVAL); + return; /* -1; */ + } + /* Handles 0 <= pb <= 4 only */ + put_int32((Uint32) len, lb); + lbp = lb + (4-pb); + + ev->iov[0].iov_base = lbp; + ev->iov[0].iov_len = pb; + ev->size += pb; + + if (dd->blocking && FDBLOCK) + driver_pdl_lock(dd->blocking->pdl); + + if ((sz = driver_sizeq(ix)) > 0) { + driver_enqv(ix, ev, 0); + + if (dd->blocking && FDBLOCK) + driver_pdl_unlock(dd->blocking->pdl); + + if (sz + ev->size >= (1 << 13)) + set_busy_port(ix, 1); + } + else if (!dd->blocking || !FDBLOCK) { + /* We try to write directly if the fd in non-blocking */ + int vsize = ev->vsize > MAX_VSIZE ? MAX_VSIZE : ev->vsize; + + n = writev(ofd, (const void *) (ev->iov), vsize); + if (n == ev->size) + return; /* 0;*/ + if (n < 0) { + if ((errno != EINTR) && (errno != ERRNO_BLOCK)) { + driver_failure_posix(ix, errno); + return; /* -1;*/ + } + n = 0; + } + driver_enqv(ix, ev, n); /* n is the skip value */ + driver_select(ix, ofd, ERL_DRV_WRITE|ERL_DRV_USE, 1); + } +#if FDBLOCK + else { + if (ev->size != 0) { + driver_enqv(ix, ev, 0); + driver_pdl_unlock(dd->blocking->pdl); + driver_async(ix, &dd->blocking->pkey, + fd_async, dd, NULL); + } else { + driver_pdl_unlock(dd->blocking->pdl); + } + } +#endif + /* return 0;*/ +} + +/* Used by spawn_driver and vanilla driver */ +static void output(ErlDrvData e, char* buf, ErlDrvSizeT len) +{ + ErtsSysDriverData *dd = (ErtsSysDriverData*)e; + ErlDrvPort ix = dd->port_num; + int pb = dd->packet_bytes; + int ofd = dd->ofd ? dd->ofd->fd : -1; + ssize_t n; + ErlDrvSizeT sz; + char lb[4]; + char* lbp; + struct iovec iv[2]; + + /* (len > ((unsigned long)-1 >> (4-pb)*8)) */ + if (((pb == 2) && (len > 0xffff)) + || (pb == 1 && len > 0xff) + || dd->pid == 0 /* Attempt at output before port is ready */) { + driver_failure_posix(ix, EINVAL); + return; /* -1; */ + } + put_int32(len, lb); + lbp = lb + (4-pb); + + if ((sz = driver_sizeq(ix)) > 0) { + driver_enq(ix, lbp, pb); + driver_enq(ix, buf, len); + if (sz + len + pb >= (1 << 13)) + set_busy_port(ix, 1); + } + else { + iv[0].iov_base = lbp; + iv[0].iov_len = pb; /* should work for pb=0 */ + iv[1].iov_base = buf; + iv[1].iov_len = len; + n = writev(ofd, iv, 2); + if (n == pb+len) + return; /* 0; */ + if (n < 0) { + if ((errno != EINTR) && (errno != ERRNO_BLOCK)) { + driver_failure_posix(ix, errno); + return; /* -1; */ + } + n = 0; + } + if (n < pb) { + driver_enq(ix, lbp+n, pb-n); + driver_enq(ix, buf, len); + } + else { + n -= pb; + driver_enq(ix, buf+n, len-n); + } + driver_select(ix, ofd, ERL_DRV_WRITE|ERL_DRV_USE, 1); + } + return; /* 0; */ +} + +static int port_inp_failure(ErtsSysDriverData *dd, int res) + /* Result: 0 (eof) or -1 (error) */ +{ + int err = errno; + + ASSERT(res <= 0); + if (dd->ifd) { + driver_select(dd->port_num, dd->ifd->fd, ERL_DRV_READ|ERL_DRV_WRITE, 0); + clear_fd_data(dd->ifd); + } + + if (dd->blocking && FDBLOCK) { + driver_pdl_lock(dd->blocking->pdl); + if (driver_sizeq(dd->port_num) > 0) { + driver_pdl_unlock(dd->blocking->pdl); + /* We have stuff in the output queue, so we just + set the state to terminating and wait for fd_async_ready + to terminate the port */ + if (res == 0) + dd->terminating = 2; + else + dd->terminating = -err; + return 0; + } + driver_pdl_unlock(dd->blocking->pdl); + } + + if (res == 0) { + if (dd->alive == 1) { + /* + * We have eof and want to report exit status, but the process + * hasn't exited yet. When it does ready_input will + * driver_select() this fd which will make sure that we get + * back here with dd->alive == -1 and dd->status set. + */ + return 0; + } + else if (dd->alive == -1) { + int status = dd->status; + + /* We need not be prepared for stopped/continued processes. */ + if (WIFSIGNALED(status)) + status = 128 + WTERMSIG(status); + else + status = WEXITSTATUS(status); + driver_report_exit(dd->port_num, status); + } + driver_failure_eof(dd->port_num); + } else if (dd->ifd) { + erl_drv_init_ack(dd->port_num, ERL_DRV_ERROR_ERRNO); + } else { + driver_failure_posix(dd->port_num, err); + } + return 0; +} + +/* fd is the drv_data that is returned from the */ +/* initial start routine */ +/* ready_fd is the descriptor that is ready to read */ + +static void ready_input(ErlDrvData e, ErlDrvEvent ready_fd) +{ + ErtsSysDriverData *dd = (ErtsSysDriverData*)e; + ErlDrvPort port_num; + int packet_bytes; + int res; + Uint h; + + port_num = dd->port_num; + packet_bytes = dd->packet_bytes; + + ASSERT(abs(dd->ifd->fd) == ready_fd); + + if (dd->pid == 0) { + /* the pid is sent from erl_child_setup. spawn driver only. */ + ErtsSysForkerProto proto; + int res; + + if((res = read(ready_fd, &proto, sizeof(proto))) <= 0) { + /* hmm, child setup seems to have closed the pipe too early... + we close the port as there is not much else we can do */ + if (res < 0 && errno == ERRNO_BLOCK) + return; + driver_select(port_num, ready_fd, ERL_DRV_READ, 0); + if (res == 0) + errno = EPIPE; + port_inp_failure(dd, -1); + return; + } + + ASSERT(proto.action == ErtsSysForkerProtoAction_Go); + dd->pid = proto.u.go.os_pid; + + if (dd->pid == -1) { + /* Setup failed! The only reason why this should happen is if + the fork fails. */ + errno = proto.u.go.error_number; + port_inp_failure(dd, -1); + return; + } + + proto.action = ErtsSysForkerProtoAction_Ack; + + if (driver_sizeq(port_num) > 0) { + driver_enq(port_num, (char*)&proto, sizeof(proto)); + } else { + if (write(abs(dd->ofd->fd), &proto, sizeof(proto)) < 0) + if (errno == ERRNO_BLOCK || errno == EINTR) + driver_enq(port_num, (char*)&proto, sizeof(proto)); + /* do nothing on failure here. If the ofd is broken, then + the ifd will probably also be broken and trigger + a port_inp_failure */ + } + + if (dd->ifd->fd < 0) { + driver_select(port_num, abs(dd->ifd->fd), ERL_DRV_READ|ERL_DRV_USE, 0); + erts_smp_atomic_add_nob(&sys_misc_mem_sz, -sizeof(ErtsSysFdData)); + dd->ifd = NULL; + } + + if (dd->ofd->fd < 0 || driver_sizeq(port_num) > 0) + /* we select in order to close fd or write to queue, + child setup will close this fd if fd < 0 */ + driver_select(port_num, abs(dd->ofd->fd), ERL_DRV_WRITE|ERL_DRV_USE, 1); + + erl_drv_set_os_pid(port_num, dd->pid); + erl_drv_init_ack(port_num, e); + return; + } + + if (packet_bytes == 0) { + byte *read_buf = (byte *) erts_alloc(ERTS_ALC_T_SYS_READ_BUF, + ERTS_SYS_READ_BUF_SZ); + res = read(ready_fd, read_buf, ERTS_SYS_READ_BUF_SZ); + if (res < 0) { + if ((errno != EINTR) && (errno != ERRNO_BLOCK)) + port_inp_failure(dd, res); + } + else if (res == 0) + port_inp_failure(dd, res); + else + driver_output(port_num, (char*) read_buf, res); + erts_free(ERTS_ALC_T_SYS_READ_BUF, (void *) read_buf); + } + else if (dd->ifd->remain > 0) { /* We try to read the remainder */ + /* space is allocated in buf */ + res = read(ready_fd, dd->ifd->cpos, + dd->ifd->remain); + if (res < 0) { + if ((errno != EINTR) && (errno != ERRNO_BLOCK)) + port_inp_failure(dd, res); + } + else if (res == 0) { + port_inp_failure(dd, res); + } + else if (res == dd->ifd->remain) { /* we're done */ + driver_output(port_num, dd->ifd->buf, + dd->ifd->sz); + clear_fd_data(dd->ifd); + } + else { /* if (res < dd->ifd->remain) */ + dd->ifd->cpos += res; + dd->ifd->remain -= res; + } + } + else if (dd->ifd->remain == 0) { /* clean fd */ + byte *read_buf = (byte *) erts_alloc(ERTS_ALC_T_SYS_READ_BUF, + ERTS_SYS_READ_BUF_SZ); + /* We make one read attempt and see what happens */ + res = read(ready_fd, read_buf, ERTS_SYS_READ_BUF_SZ); + if (res < 0) { + if ((errno != EINTR) && (errno != ERRNO_BLOCK)) + port_inp_failure(dd, res); + } + else if (res == 0) { /* eof */ + port_inp_failure(dd, res); + } + else if (res < packet_bytes - dd->ifd->psz) { + memcpy(dd->ifd->pbuf+dd->ifd->psz, + read_buf, res); + dd->ifd->psz += res; + } + else { /* if (res >= packet_bytes) */ + unsigned char* cpos = read_buf; + int bytes_left = res; + + while (1) { + int psz = dd->ifd->psz; + char* pbp = dd->ifd->pbuf + psz; + + while(bytes_left && (psz < packet_bytes)) { + *pbp++ = *cpos++; + bytes_left--; + psz++; + } + + if (psz < packet_bytes) { + dd->ifd->psz = psz; + break; + } + dd->ifd->psz = 0; + + switch (packet_bytes) { + case 1: h = get_int8(dd->ifd->pbuf); break; + case 2: h = get_int16(dd->ifd->pbuf); break; + case 4: h = get_int32(dd->ifd->pbuf); break; + default: ASSERT(0); return; /* -1; */ + } + + if (h <= (bytes_left)) { + driver_output(port_num, (char*) cpos, h); + cpos += h; + bytes_left -= h; + continue; + } + else { /* The last message we got was split */ + char *buf = erts_alloc_fnf(ERTS_ALC_T_FD_ENTRY_BUF, h); + if (!buf) { + errno = ENOMEM; + port_inp_failure(dd, -1); + } + else { + erts_smp_atomic_add_nob(&sys_misc_mem_sz, h); + sys_memcpy(buf, cpos, bytes_left); + dd->ifd->buf = buf; + dd->ifd->sz = h; + dd->ifd->remain = h - bytes_left; + dd->ifd->cpos = buf + bytes_left; + } + break; + } + } + } + erts_free(ERTS_ALC_T_SYS_READ_BUF, (void *) read_buf); + } +} + + +/* fd is the drv_data that is returned from the */ +/* initial start routine */ +/* ready_fd is the descriptor that is ready to read */ + +static void ready_output(ErlDrvData e, ErlDrvEvent ready_fd) +{ + ErtsSysDriverData *dd = (ErtsSysDriverData*)e; + ErlDrvPort ix = dd->port_num; + int n; + struct iovec* iv; + int vsize; + + if ((iv = (struct iovec*) driver_peekq(ix, &vsize)) == NULL) { + driver_select(ix, ready_fd, ERL_DRV_WRITE, 0); + if (dd->pid > 0 && dd->ofd->fd < 0) { + /* The port was opened with 'in' option, which means we + should close the output fd as soon as the command has + been sent. */ + driver_select(ix, ready_fd, ERL_DRV_WRITE|ERL_DRV_USE, 0); + erts_smp_atomic_add_nob(&sys_misc_mem_sz, -sizeof(ErtsSysFdData)); + dd->ofd = NULL; + } + if (dd->terminating) + driver_failure_atom(dd->port_num,"normal"); + return; /* 0; */ + } + vsize = vsize > MAX_VSIZE ? MAX_VSIZE : vsize; + if ((n = writev(ready_fd, iv, vsize)) > 0) { + if (driver_deq(ix, n) == 0) + set_busy_port(ix, 0); + } + else if (n < 0) { + if (errno == ERRNO_BLOCK || errno == EINTR) + return; /* 0; */ + else { + int res = errno; + driver_select(ix, ready_fd, ERL_DRV_WRITE, 0); + driver_failure_posix(ix, res); + return; /* -1; */ + } + } + return; /* 0; */ +} + +static void stop_select(ErlDrvEvent fd, void* _) +{ + close((int)fd); +} + +#if FDBLOCK + +static void +fd_async(void *async_data) +{ + int res; + ErtsSysDriverData *dd = (ErtsSysDriverData *)async_data; + SysIOVec *iov0; + SysIOVec *iov; + int iovlen; + int err = 0; + /* much of this code is stolen from efile_drv:invoke_writev */ + driver_pdl_lock(dd->blocking->pdl); + iov0 = driver_peekq(dd->port_num, &iovlen); + iovlen = iovlen < MAXIOV ? iovlen : MAXIOV; + iov = erts_alloc_fnf(ERTS_ALC_T_SYS_WRITE_BUF, + sizeof(SysIOVec)*iovlen); + if (!iov) { + res = -1; + err = ENOMEM; + driver_pdl_unlock(dd->blocking->pdl); + } else { + memcpy(iov,iov0,iovlen*sizeof(SysIOVec)); + driver_pdl_unlock(dd->blocking->pdl); + + do { + res = writev(dd->ofd->fd, iov, iovlen); + } while (res < 0 && errno == EINTR); + if (res < 0) + err = errno; + err = errno; + + erts_free(ERTS_ALC_T_SYS_WRITE_BUF, iov); + } + dd->blocking->res = res; + dd->blocking->err = err; +} + +void fd_ready_async(ErlDrvData drv_data, + ErlDrvThreadData thread_data) { + ErtsSysDriverData *dd = (ErtsSysDriverData *)thread_data; + ErlDrvPort port_num = dd->port_num; + + ASSERT(dd->blocking); + + if (dd->blocking->res > 0) { + driver_pdl_lock(dd->blocking->pdl); + if (driver_deq(port_num, dd->blocking->res) == 0) { + driver_pdl_unlock(dd->blocking->pdl); + set_busy_port(port_num, 0); + if (dd->terminating) { + /* The port is has been ordered to terminate + from either fd_flush or port_inp_failure */ + if (dd->terminating == 1) + driver_failure_atom(port_num, "normal"); + else if (dd->terminating == 2) + driver_failure_eof(port_num); + else if (dd->terminating < 0) + driver_failure_posix(port_num, -dd->terminating); + return; /* -1; */ + } + } else { + driver_pdl_unlock(dd->blocking->pdl); + /* still data left to write in queue */ + driver_async(port_num, &dd->blocking->pkey, fd_async, dd, NULL); + return /* 0; */; + } + } else if (dd->blocking->res < 0) { + if (dd->blocking->err == ERRNO_BLOCK) { + set_busy_port(port_num, 1); + /* still data left to write in queue */ + driver_async(port_num, &dd->blocking->pkey, fd_async, dd, NULL); + } else + driver_failure_posix(port_num, dd->blocking->err); + return; /* -1; */ + } + return; /* 0; */ +} + +#endif + +/* Forker driver */ + +static int forker_fd; + +static ErlDrvData forker_start(ErlDrvPort port_num, char* name, + SysDriverOpts* opts) +{ + + int i; + int fds[2]; + int res, unbind; + char bindir[MAXPATHLEN]; + size_t bindirsz = sizeof(bindir); + Uint csp_path_sz; + char *child_setup_prog; + + forker_port = erts_drvport2id(port_num); + + res = erts_sys_getenv_raw("BINDIR", bindir, &bindirsz); + if (res != 0) { + if (res < 0) + erl_exit(-1, + "Environment variable BINDIR is not set\n"); + if (res > 0) + erl_exit(-1, + "Value of environment variable BINDIR is too large\n"); + } + if (bindir[0] != DIR_SEPARATOR_CHAR) + erl_exit(-1, + "Environment variable BINDIR does not contain an" + " absolute path\n"); + csp_path_sz = (strlen(bindir) + + 1 /* DIR_SEPARATOR_CHAR */ + + sizeof(CHILD_SETUP_PROG_NAME) + + 1); + child_setup_prog = erts_alloc(ERTS_ALC_T_CS_PROG_PATH, csp_path_sz); + erts_snprintf(child_setup_prog, csp_path_sz, + "%s%c%s", + bindir, + DIR_SEPARATOR_CHAR, + CHILD_SETUP_PROG_NAME); + if (socketpair(AF_UNIX, SOCK_STREAM, 0, fds) < 0) { + erl_exit(ERTS_ABORT_EXIT, + "Could not open unix domain socket in spawn_init: %d\n", + errno); + } + + forker_fd = fds[0]; + + unbind = erts_sched_bind_atfork_prepare(); + + i = fork(); + + if (i == 0) { + /* The child */ + char *cs_argv[FORKER_ARGV_NO_OF_ARGS] = + {CHILD_SETUP_PROG_NAME, NULL, NULL}; + char buff[128]; + + erts_sched_bind_atfork_child(unbind); + + snprintf(buff, 128, "%d", sys_max_files()); + cs_argv[FORKER_ARGV_MAX_FILES] = buff; + + /* We preallocate fd 3 for the uds fd */ + if (fds[1] != 3) { + dup2(fds[1], 3); + } + +#if defined(USE_SETPGRP_NOARGS) /* SysV */ + (void) setpgrp(); +#elif defined(USE_SETPGRP) /* BSD */ + (void) setpgrp(0, getpid()); +#else /* POSIX */ + (void) setsid(); +#endif + + execv(child_setup_prog, cs_argv); + _exit(1); + } + + erts_sched_bind_atfork_parent(unbind); + + erts_free(ERTS_ALC_T_CS_PROG_PATH, child_setup_prog); + + close(fds[1]); + + SET_NONBLOCKING(forker_fd); + + driver_select(port_num, forker_fd, ERL_DRV_READ|ERL_DRV_USE, 1); + + return (ErlDrvData)port_num; +} + +static void forker_stop(ErlDrvData e) +{ + /* we probably should do something here, + the port has been closed by the user. */ +} + +static void forker_ready_input(ErlDrvData e, ErlDrvEvent fd) +{ + int res; + ErtsSysForkerProto *proto; + + proto = erts_alloc(ERTS_ALC_T_DRV_CTRL_DATA, sizeof(*proto)); + + if ((res = read(fd, proto, sizeof(*proto))) < 0) { + if (errno == ERRNO_BLOCK) + return; + erl_exit(ERTS_DUMP_EXIT, "Failed to read from erl_child_setup: %d\n", errno); + } + + if (res == 0) + erl_exit(ERTS_DUMP_EXIT, "erl_child_setup closed\n"); + + ASSERT(res == sizeof(*proto)); + +#ifdef FORKER_PROTO_START_ACK + if (proto->action == ErtsSysForkerProtoAction_StartAck) { + /* Ideally we would like to not have to ack each Start + command being sent over the uds, but it would seem + that some operating systems (only observed on FreeBSD) + throw away data on the uds when the socket becomes full, + so we have to. + */ + ErlDrvPort port_num = (ErlDrvPort)e; + int vlen; + SysIOVec *iov = driver_peekq(port_num, &vlen); + ErtsSysForkerProto *proto = (ErtsSysForkerProto *)iov[0].iov_base; + + close(proto->u.start.fds[0]); + close(proto->u.start.fds[1]); + if (proto->u.start.fds[1] != proto->u.start.fds[2]) + close(proto->u.start.fds[2]); + + driver_deq(port_num, sizeof(*proto)); + + if (driver_sizeq(port_num) > 0) + driver_select(port_num, forker_fd, ERL_DRV_WRITE|ERL_DRV_USE, 1); + } else +#endif + { + ASSERT(proto->action == ErtsSysForkerProtoAction_SigChld); + + /* ideally this would be a port_command call, but as command is + already used by the spawn_driver, we use control instead. + Note that when using erl_drv_port_control it is an asynchronous + control. */ + erl_drv_port_control(proto->u.sigchld.port_id, 'S', + (char*)proto, sizeof(*proto)); + } + +} + +static void forker_ready_output(ErlDrvData e, ErlDrvEvent fd) +{ + ErlDrvPort port_num = (ErlDrvPort)e; + +#ifndef FORKER_PROTO_START_ACK + while (driver_sizeq(port_num) > 0) { +#endif + int vlen; + SysIOVec *iov = driver_peekq(port_num, &vlen); + ErtsSysForkerProto *proto = (ErtsSysForkerProto *)iov[0].iov_base; + ASSERT(iov[0].iov_len >= (sizeof(*proto))); + if (sys_uds_write(forker_fd, (char*)proto, sizeof(*proto), + proto->u.start.fds, 3, 0) < 0) { + if (errno == ERRNO_BLOCK) + return; + erl_exit(ERTS_DUMP_EXIT, "Failed to write to erl_child_setup: %d\n", errno); + } +#ifndef FORKER_PROTO_START_ACK + close(proto->u.start.fds[0]); + close(proto->u.start.fds[1]); + if (proto->u.start.fds[1] != proto->u.start.fds[2]) + close(proto->u.start.fds[2]); + driver_deq(port_num, sizeof(*proto)); + } +#endif + + driver_select(port_num, forker_fd, ERL_DRV_WRITE, 0); +} + +static ErlDrvSSizeT forker_control(ErlDrvData e, unsigned int cmd, char *buf, + ErlDrvSizeT len, char **rbuf, ErlDrvSizeT rlen) +{ + ErtsSysForkerProto *proto = (ErtsSysForkerProto *)buf; + ErlDrvPort port_num = (ErlDrvPort)e; + int res; + + driver_enq(port_num, buf, len); + if (driver_sizeq(port_num) > sizeof(*proto)) { + return 0; + } + + if ((res = sys_uds_write(forker_fd, (char*)proto, sizeof(*proto), + proto->u.start.fds, 3, 0)) < 0) { + if (errno == ERRNO_BLOCK) { + driver_select(port_num, forker_fd, ERL_DRV_WRITE|ERL_DRV_USE, 1); + return 0; + } + erl_exit(ERTS_DUMP_EXIT, "Failed to write to erl_child_setup: %d\n", errno); + } + +#ifndef FORKER_PROTO_START_ACK + ASSERT(res == sizeof(*proto)); + close(proto->u.start.fds[0]); + close(proto->u.start.fds[1]); + if (proto->u.start.fds[1] != proto->u.start.fds[2]) + close(proto->u.start.fds[2]); + driver_deq(port_num, sizeof(*proto)); +#endif + + return 0; +} diff --git a/erts/emulator/sys/unix/sys_uds.c b/erts/emulator/sys/unix/sys_uds.c new file mode 100644 index 0000000000..015d0346a1 --- /dev/null +++ b/erts/emulator/sys/unix/sys_uds.c @@ -0,0 +1,155 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2002-2009. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * %CopyrightEnd% + */ + +#include "sys_uds.h" + +int +sys_uds_readv(int fd, struct iovec *iov, size_t iov_len, + int *fds, int fd_count, int flags) { + struct msghdr msg; + struct cmsghdr *cmsg = NULL; + char ancillary_buff[256] = {0}; + int res, i = 0; + + /* setup a place to fill in message contents */ + memset(&msg, 0, sizeof(struct msghdr)); + msg.msg_iov = iov; + msg.msg_iovlen = iov_len; + + /* provide space for the ancillary data */ + msg.msg_control = ancillary_buff; + msg.msg_controllen = sizeof(ancillary_buff); + + if((res = recvmsg(fd, &msg, flags)) < 0) { +#if defined(__APPLE__) && defined(__MACH__) && !defined(__DARWIN__) + /* When some OS X versions run out of fd's + they give EMSGSIZE instead of EMFILE. + We remap this as we want the correct + error to appear for the user */ + if (errno == EMSGSIZE) + errno = EMFILE; +#endif + return res; + } + + if((msg.msg_flags & MSG_CTRUNC) == MSG_CTRUNC) + { + /* We assume that we have given enough space for any header + that are sent to us. So the only remaining reason to get + this flag set is if the caller has run out of file descriptors. + */ + errno = EMFILE; + return -1; + } + + for (cmsg = CMSG_FIRSTHDR(&msg); cmsg; cmsg = CMSG_NXTHDR(&msg, cmsg) ) { + if ((cmsg->cmsg_level == SOL_SOCKET) && + (cmsg->cmsg_type == SCM_RIGHTS)) { + int *cmsg_data = (int *)CMSG_DATA(cmsg); + while ((char*)cmsg_data < (char*)cmsg + cmsg->cmsg_len) { + if (i < fd_count) { + fds[i++] = *cmsg_data++; + } else { + /* for some strange reason, we have received more FD's + than we wanted... close them if we are not running + debug. */ + if(i >= fd_count) abort(); + close(*cmsg_data++); + } + } + } + } + + return res; +} + +int +sys_uds_read(int fd, char *buff, size_t len, + int *fds, int fd_count, int flags) { + struct iovec iov; + iov.iov_base = buff; + iov.iov_len = len; + return sys_uds_readv(fd, &iov, 1, fds, fd_count, flags); +} + + +int +sys_uds_writev(int fd, struct iovec *iov, size_t iov_len, + int *fds, int fd_count, int flags) { + + struct msghdr msg; + struct cmsghdr *cmsg = NULL; + int res, i; + + /* initialize socket message */ + memset(&msg, 0, sizeof(struct msghdr)); + + /* We flatten the iov if it is too long */ + if (iov_len > MAXIOV) { + int size = 0; + char *buff; + for (i = 0; i < iov_len; i++) + size += iov[i].iov_len; + buff = malloc(size); + + for (i = 0; i < iov_len; i++) { + memcpy(buff, iov[i].iov_base, iov[i].iov_len); + buff += iov[i].iov_len; + } + + iov[0].iov_base = buff - size; + iov[0].iov_len = size; + msg.msg_iov = iov; + msg.msg_iovlen = 1; + } else { + msg.msg_iov = iov; + msg.msg_iovlen = iov_len; + } + + /* initialize the ancillary data */ + msg.msg_control = calloc(1, CMSG_SPACE(sizeof(int) * fd_count)); + msg.msg_controllen = CMSG_SPACE(sizeof(int) * fd_count); + + /* copy the fd array into the ancillary data */ + cmsg = CMSG_FIRSTHDR(&msg); + if(!cmsg) abort(); + cmsg->cmsg_level = SOL_SOCKET; + cmsg->cmsg_type = SCM_RIGHTS; + cmsg->cmsg_len = CMSG_LEN(sizeof(int) * fd_count); + memcpy(CMSG_DATA(cmsg), fds, sizeof(int) * fd_count); + + res = sendmsg(fd, &msg, flags); + + if (iov_len > MAXIOV) + free(iov[0].iov_base); + + free(msg.msg_control); + + return res; +} + +int +sys_uds_write(int fd, char *buff, size_t len, + int *fds, int fd_count, int flags) { + struct iovec iov; + iov.iov_base = buff; + iov.iov_len = len; + return sys_uds_writev(fd, &iov, 1, fds, fd_count, flags); +} diff --git a/erts/emulator/sys/unix/sys_uds.h b/erts/emulator/sys/unix/sys_uds.h new file mode 100644 index 0000000000..844a2804d8 --- /dev/null +++ b/erts/emulator/sys/unix/sys_uds.h @@ -0,0 +1,57 @@ +/* + * %CopyrightBegin% + * + * Copyright Ericsson AB 2002-2009. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * %CopyrightEnd% + */ + +#ifndef _ERL_UNIX_UDS_H +#define _ERL_UNIX_UDS_H + +#ifdef HAVE_CONFIG_H +# include "config.h" +#endif + +#if defined(__sun__) && !defined(_XOPEN_SOURCE) +#define _XOPEN_SOURCE 500 +#endif + +#include <limits.h> + +#include <sys/types.h> +#include <sys/socket.h> +#include <sys/uio.h> + +#if defined IOV_MAX +#define MAXIOV IOV_MAX +#elif defined UIO_MAXIOV +#define MAXIOV UIO_MAXIOV +#else +#define MAXIOV 16 +#endif + +#include "sys.h" + +int sys_uds_readv(int fd, struct iovec *iov, size_t iov_len, + int *fds, int fd_count, int flags); +int sys_uds_read(int fd, char *buff, size_t len, + int *fds, int fd_count, int flags); +int sys_uds_writev(int fd, struct iovec *iov, size_t iov_len, + int *fds, int fd_count, int flags); +int sys_uds_write(int fd, char *buff, size_t len, + int *fds, int fd_count, int flags); + +#endif /* #ifndef _ERL_UNIX_UDS_H */ diff --git a/erts/emulator/sys/win32/sys.c b/erts/emulator/sys/win32/sys.c index fce76db28f..76ce25916a 100644 --- a/erts/emulator/sys/win32/sys.c +++ b/erts/emulator/sys/win32/sys.c @@ -1334,10 +1334,8 @@ spawn_start(ErlDrvPort port_num, char* utf8_name, SysDriverOpts* opts) retval = set_driver_data(dp, hFromChild, hToChild, opts->read_write, opts->exit_status); if (retval != ERL_DRV_ERROR_GENERAL && retval != ERL_DRV_ERROR_ERRNO) { - Port *prt = erts_drvport2port(port_num); - /* We assume that this cannot generate a negative number */ - ASSERT(prt != ERTS_INVALID_ERL_DRV_PORT); - prt->os_pid = (SWord) pid; + /* We assume that this cannot generate a negative number */ + erl_drv_set_os_pid(port_num, pid); } } @@ -1528,8 +1526,8 @@ create_child_process * Parse out the program name from the command line (it can be quoted and * contain spaces). */ - newcmdline = (wchar_t *) erts_alloc(ERTS_ALC_T_TMP, 2048*sizeof(wchar_t)); cmdlength = parse_command(origcmd); + newcmdline = (wchar_t *) erts_alloc(ERTS_ALC_T_TMP, (MAX_PATH+wcslen(origcmd)-cmdlength)*sizeof(wchar_t)); thecommand = (wchar_t *) erts_alloc(ERTS_ALC_T_TMP, (cmdlength+1)*sizeof(wchar_t)); wcsncpy(thecommand, origcmd, cmdlength); thecommand[cmdlength] = L'\0'; @@ -3273,6 +3271,12 @@ void erl_sys_init(void) } void +erl_sys_late_init(void) +{ + /* do nothing */ +} + +void erts_sys_schedule_interrupt(int set) { erts_check_io_interrupt(set); diff --git a/erts/emulator/test/driver_SUITE.erl b/erts/emulator/test/driver_SUITE.erl index 4fd7b36e6a..ce55fe3c52 100644 --- a/erts/emulator/test/driver_SUITE.erl +++ b/erts/emulator/test/driver_SUITE.erl @@ -2395,13 +2395,35 @@ z_test(Config) when is_list(Config) -> check_io_debug() -> get_stable_check_io_info(), - {NoErrorFds, NoUsedFds, NoDrvSelStructs, NoDrvEvStructs} + {NoErrorFds, NoUsedFds, NoDrvSelStructs, NoDrvEvStructs} = CheckIoDebug = erts_debug:get_internal_state(check_io_debug), + HasGetHost = has_gethost(), + ct:log("check_io_debug: ~p~n" + "HasGetHost: ~p",[CheckIoDebug, HasGetHost]), 0 = NoErrorFds, - NoUsedFds = NoDrvSelStructs, + if + NoUsedFds == NoDrvSelStructs -> + ok; + HasGetHost andalso (NoUsedFds == (NoDrvSelStructs - 1)) -> + %% If the inet_gethost port is alive, we may have + %% one extra used fd that is not selected on + ok + end, 0 = NoDrvEvStructs, ok. +has_gethost() -> + has_gethost(erlang:ports()). +has_gethost([P|T]) -> + case erlang:port_info(P, name) of + {name,"inet_gethost"++_} -> + true; + _ -> + has_gethost(T) + end; +has_gethost([]) -> + false. + %flush_msgs() -> % receive % M -> diff --git a/erts/emulator/test/port_SUITE.erl b/erts/emulator/test/port_SUITE.erl index b42e02a1e1..ff75ee86d6 100644 --- a/erts/emulator/test/port_SUITE.erl +++ b/erts/emulator/test/port_SUITE.erl @@ -82,10 +82,11 @@ mul_basic/1, mul_slow_writes/1, dying_port/1, port_program_with_path/1, open_input_file_port/1, open_output_file_port/1, + count_fds/1, iter_max_ports/1, eof/1, input_only/1, output_only/1, name1/1, t_binary/1, parallell/1, t_exit/1, - env/1, bad_env/1, cd/1, exit_status/1, + env/1, huge_env/1, bad_env/1, cd/1, exit_status/1, tps_16_bytes/1, tps_1K/1, line/1, stderr_to_stdout/1, otp_3906/1, otp_4389/1, win_massive/1, win_massive_client/1, mix_up_ports/1, otp_5112/1, otp_5119/1, otp_6224/1, @@ -111,8 +112,8 @@ all() -> bad_packet, bad_port_messages, {group, options}, {group, multiple_packets}, parallell, dying_port, port_program_with_path, open_input_file_port, - open_output_file_port, name1, env, bad_env, cd, - exit_status, iter_max_ports, t_exit, {group, tps}, line, + open_output_file_port, name1, env, huge_env, bad_env, cd, + exit_status, iter_max_ports, count_fds, t_exit, {group, tps}, line, stderr_to_stdout, otp_3906, otp_4389, win_massive, mix_up_ports, otp_5112, otp_5119, exit_status_multi_scheduling_block, ports, spawn_driver, @@ -385,27 +386,33 @@ input_only(Config) when is_list(Config) -> output_only(Config) when is_list(Config) -> Dog = test_server:timetrap(test_server:seconds(100)), Dir = ?config(priv_dir, Config), + + %% First we test that the port program gets the data Filename = filename:join(Dir, "output_only_stream"), - output_and_verify(Config, Filename, "-h0", - random_packet(35777, "echo")), + Data = random_packet(35777, "echo"), + output_and_verify(Config, ["-h0 -o", Filename], Data), + Wait_time = 500, + test_server:sleep(Wait_time), + {ok, Written} = file:read_file(Filename), + Data = binary_to_list(Written), + + %% Then we test that any writes to stdout from + %% the port program is not sent to erlang + output_and_verify(Config, ["-h0"], Data), + test_server:timetrap_cancel(Dog), ok. -output_and_verify(Config, Filename, Options, Data) -> +output_and_verify(Config, Options, Data) -> PortTest = port_test(Config), - Command = lists:concat([PortTest, " ", - Options, " -o", Filename]), + Command = lists:concat([PortTest, " " | Options]), Port = open_port({spawn, Command}, [out]), Port ! {self(), {command, Data}}, Port ! {self(), close}, receive - {Port, closed} -> ok - end, - Wait_time = 500, - test_server:sleep(Wait_time), - {ok, Written} = file:read_file(Filename), - Data = binary_to_list(Written), - ok. + {Port, closed} -> ok; + Msg -> ct:fail({received_unexpected_message, Msg}) + end. %% Test that receiving several packages written in the same %% write operation works. @@ -610,6 +617,38 @@ open_output_file_port(Config) when is_list(Config) -> test_server:timetrap_cancel(Dog), ok. +%% Tests that all appropriate fd's have been closed in the port program +count_fds(suite) -> []; +count_fds(Config) when is_list(Config) -> + case os:type() of + {unix, _} -> + PrivDir = proplists:get_value(priv_dir, Config), + Filename = filename:join(PrivDir, "my_fd_counter"), + + RunTest = fun(PortOpts) -> + PortTest = port_test(Config), + Command = lists:concat([PortTest, " -n -f -o", Filename]), + Port = open_port({spawn, Command}, PortOpts), + Port ! {self(), close}, + receive + {Port, closed} -> ok + end, + test_server:sleep(500), + {ok, Written} = file:read_file(Filename), + Written + end, + <<4:32/native>> = RunTest([out, nouse_stdio]), + <<4:32/native>> = RunTest([in, nouse_stdio]), + <<5:32/native>> = RunTest([in, out, nouse_stdio]), + <<3:32/native>> = RunTest([out, use_stdio]), + <<3:32/native>> = RunTest([in, use_stdio]), + <<3:32/native>> = RunTest([in, out, use_stdio]), + <<3:32/native>> = RunTest([in, out, use_stdio, stderr_to_stdout]), + <<3:32/native>> = RunTest([out, use_stdio, stderr_to_stdout]); + _ -> + {skip, "Skipped on windows"} + end. + %% %% Open as many ports as possible. Do this several times and check %% that we get the same number of ports every time. @@ -680,7 +719,16 @@ close_ports([]) -> ok. open_ports(Name, Settings) -> - test_server:sleep(5), + case os:type() of + {unix, freebsd} -> + %% FreeBsd has issues with sendmsg/recvmsg in fork + %% implementation and we therefor have to spawn + %% slower to make sure that we always hit the same + %% make roof. + test_server:sleep(10); + _ -> + test_server:sleep(5) + end, case catch open_port(Name, Settings) of P when is_port(P) -> [P| open_ports(Name, Settings)]; @@ -923,6 +971,40 @@ try_bad_env(Env) -> error:badarg -> ok end. +%% Test that we can handle a very very large environment gracefully. +huge_env(Config) when is_list(Config) -> + Vars = case os:type() of + {win32,_} -> 500; + _ -> + %% We create a huge environment, + %% 20000 variables is about 25MB + %% which seems to be the limit on Linux. + 20000 + end, + Env = [{[$a + I div (25*25*25*25) rem 25, + $a + I div (25*25*25) rem 25, + $a + I div (25*25) rem 25, + $a+I div 25 rem 25, $a+I rem 25], + lists:duplicate(100,$a+I rem 25)} + || I <- lists:seq(1,Vars)], + try erlang:open_port({spawn,"ls"},[exit_status, {env, Env}]) of + P -> + receive + {P, {exit_status,N}} = M -> + %% We test that the exit status is an integer, this means + %% that the child program has started. If we get an atom + %% something went wrong in the driver which is not ok. + ct:log("Got ~p",[M]), + true = is_integer(N) + end + catch E:R -> + %% Have to catch the error here, as printing the stackdump + %% in the ct log is way to heavy for some test machines. + ct:fail("Open port failed ~p:~p",[E,R]) + end. + + + %% 'cd' option %% (Can perhaps be made smaller by calling the other utility functions %% in this module.) @@ -1228,13 +1310,15 @@ otp_4389(Config) when is_list(Config) -> {P,{exit_status,_}} -> TCR ! {self(),ok}; {'EXIT',_,{R2,_}} when R2 == emfile; - R2 == eagain -> + R2 == eagain; + R2 == enomem -> TCR ! {self(),ok}; Err2 -> TCR ! {self(),{msg,Err2}} end; {'EXIT',{R1,_}} when R1 == emfile; - R1 == eagain -> + R1 == eagain; + R1 == enomem -> TCR ! {self(),ok}; Err1 -> TCR ! {self(), {open_port,Err1}} @@ -1840,10 +1924,12 @@ exit_status_msb_test(Config, SleepSecs) when is_list(Config) -> {Prt, erlang:system_info(scheduler_id)}; {'EXIT', {Err, _}} when Err == eagain; - Err == emfile -> + Err == emfile; + Err == enomem -> noop; {'EXIT', Err} when Err == eagain; - Err == emfile -> + Err == emfile; + Err == enomem -> noop; Error -> ?t:fail(Error) diff --git a/erts/emulator/test/port_SUITE_data/port_test.c b/erts/emulator/test/port_SUITE_data/port_test.c index 7abefab2e3..cc3ebdf0f8 100644 --- a/erts/emulator/test/port_SUITE_data/port_test.c +++ b/erts/emulator/test/port_SUITE_data/port_test.c @@ -13,6 +13,7 @@ #ifndef __WIN32__ #include <unistd.h> +#include <limits.h> #include <sys/time.h> @@ -48,6 +49,7 @@ typedef struct { * after reading the header for a packet * before reading the rest. */ + int fd_count; /* Count the number of open fds */ int break_mode; /* If set, this program will close standard * input, which should case broken pipe * error in the writer. @@ -107,7 +109,7 @@ MAIN(argc, argv) int argc; char *argv[]; { - int ret; + int ret, fd_count; if((port_data = (PORT_TEST_DATA *) malloc(sizeof(PORT_TEST_DATA))) == NULL) { fprintf(stderr, "Couldn't malloc for port_data"); exit(1); @@ -115,6 +117,7 @@ char *argv[]; port_data->header_size = 0; port_data->io_buf_size = 0; port_data->delay_mode = 0; + port_data->fd_count = 0; port_data->break_mode = 0; port_data->quit_mode = 0; port_data->slow_writes = 0; @@ -144,6 +147,9 @@ char *argv[]; case 'e': port_data->fd_to_erl = 2; break; + case 'f': + port_data->fd_count = 1; + break; case 'h': /* Header size for packets. */ switch (argv[1][2]) { case '0': port_data->header_size = 0; break; @@ -189,18 +195,31 @@ char *argv[]; /* XXX Add error printout here */ } + if (port_data->fd_count) { +#ifdef __WIN32__ + DWORD handles; + GetProcessHandleCount(GetCurrentProcess(), &handles); + fd_count = handles; +#else + int i; + for (i = 0, fd_count = 0; i < 1024; i++) + if (fcntl(i, F_GETFD) >= 0) { + fd_count++; + } +#endif + } + + if (port_data->output_file) + replace_stdout(port_data->output_file); + + if (port_data->fd_count) + reply(&fd_count, sizeof(fd_count)); + if (port_data->no_packet_loop){ free(port_data); exit(0); } - /* - * If an output file was given, let it replace standard output. - */ - - if (port_data->output_file) - replace_stdout(port_data->output_file); - ret = packet_loop(); if(port_data->io_buf_size > 0) free(port_data->io_buf); diff --git a/erts/emulator/test/z_SUITE.erl b/erts/emulator/test/z_SUITE.erl index f4d9030255..abc353fb01 100644 --- a/erts/emulator/test/z_SUITE.erl +++ b/erts/emulator/test/z_SUITE.erl @@ -249,6 +249,7 @@ pollset_size(Config) when is_list(Config) -> ?line io:format("Initial: ~p~nFinal: ~p~n", [InitChkIo, FinChkIo]), ?line InitPollsetSize = lists:keysearch(total_poll_set_size, 1, InitChkIo), ?line FinPollsetSize = lists:keysearch(total_poll_set_size, 1, FinChkIo), + HasGethost = case has_gethost() of true -> 1; _ -> 0 end, ?line case InitPollsetSize =:= FinPollsetSize of true -> case InitPollsetSize of @@ -269,7 +270,7 @@ pollset_size(Config) when is_list(Config) -> = InitPollsetSize, ?line {value, {total_poll_set_size, FinSize}} = FinPollsetSize, - ?line true = FinSize < InitSize, + ?line true = FinSize < (InitSize + HasGethost), ?line true = 2 =< FinSize, ?line {comment, "Start pollset size: " @@ -289,16 +290,39 @@ check_io_debug(Config) when is_list(Config) -> end. check_io_debug_test() -> - ?line erlang:display(get_check_io_info()), - ?line erts_debug:set_internal_state(available_internal_state, true), - ?line {NoErrorFds, NoUsedFds, NoDrvSelStructs, NoDrvEvStructs} + erlang:display(get_check_io_info()), + erts_debug:set_internal_state(available_internal_state, true), + {NoErrorFds, NoUsedFds, NoDrvSelStructs, NoDrvEvStructs} = CheckIoDebug = erts_debug:get_internal_state(check_io_debug), - ?line erts_debug:set_internal_state(available_internal_state, false), - ?line 0 = NoErrorFds, - ?line NoUsedFds = NoDrvSelStructs, + erts_debug:set_internal_state(available_internal_state, false), + HasGetHost = has_gethost(), + ct:log("check_io_debug: ~p~n" + "HasGetHost: ~p",[CheckIoDebug, HasGetHost]), + 0 = NoErrorFds, + if + NoUsedFds == NoDrvSelStructs -> + ok; + HasGetHost andalso (NoUsedFds == (NoDrvSelStructs - 1)) -> + %% If the inet_gethost port is alive, we may have + %% one extra used fd that is not selected on. + %% This happens when the initial setup of the + %% port returns an EAGAIN + ok + end, ?line 0 = NoDrvEvStructs, ?line ok. +has_gethost() -> + has_gethost(erlang:ports()). +has_gethost([P|T]) -> + case erlang:port_info(P, name) of + {name,"inet_gethost"++_} -> + true; + _ -> + has_gethost(T) + end; +has_gethost([]) -> + false. %% diff --git a/erts/preloaded/ebin/erl_prim_loader.beam b/erts/preloaded/ebin/erl_prim_loader.beam Binary files differindex e94a1ba796..8ccbd1e36b 100644 --- a/erts/preloaded/ebin/erl_prim_loader.beam +++ b/erts/preloaded/ebin/erl_prim_loader.beam diff --git a/erts/preloaded/ebin/erlang.beam b/erts/preloaded/ebin/erlang.beam Binary files differindex 632defdb46..87a48525bb 100644 --- a/erts/preloaded/ebin/erlang.beam +++ b/erts/preloaded/ebin/erlang.beam diff --git a/erts/preloaded/ebin/erts_internal.beam b/erts/preloaded/ebin/erts_internal.beam Binary files differindex fd0a502d2c..191dd332e2 100644 --- a/erts/preloaded/ebin/erts_internal.beam +++ b/erts/preloaded/ebin/erts_internal.beam diff --git a/erts/preloaded/ebin/init.beam b/erts/preloaded/ebin/init.beam Binary files differindex 60c08819eb..a78d9abc20 100644 --- a/erts/preloaded/ebin/init.beam +++ b/erts/preloaded/ebin/init.beam diff --git a/erts/preloaded/ebin/otp_ring0.beam b/erts/preloaded/ebin/otp_ring0.beam Binary files differindex 04814c091b..9cc91be343 100644 --- a/erts/preloaded/ebin/otp_ring0.beam +++ b/erts/preloaded/ebin/otp_ring0.beam diff --git a/erts/preloaded/ebin/prim_eval.beam b/erts/preloaded/ebin/prim_eval.beam Binary files differindex 7779c8374d..e84c8ffd2d 100644 --- a/erts/preloaded/ebin/prim_eval.beam +++ b/erts/preloaded/ebin/prim_eval.beam diff --git a/erts/preloaded/ebin/prim_file.beam b/erts/preloaded/ebin/prim_file.beam Binary files differindex 254b0e5b90..e1b2a1c8eb 100644 --- a/erts/preloaded/ebin/prim_file.beam +++ b/erts/preloaded/ebin/prim_file.beam diff --git a/erts/preloaded/ebin/prim_inet.beam b/erts/preloaded/ebin/prim_inet.beam Binary files differindex b7cfe26462..8853ae8bda 100644 --- a/erts/preloaded/ebin/prim_inet.beam +++ b/erts/preloaded/ebin/prim_inet.beam diff --git a/erts/preloaded/ebin/prim_zip.beam b/erts/preloaded/ebin/prim_zip.beam Binary files differindex 6b5c6195c8..563e604a1d 100644 --- a/erts/preloaded/ebin/prim_zip.beam +++ b/erts/preloaded/ebin/prim_zip.beam diff --git a/erts/preloaded/ebin/zlib.beam b/erts/preloaded/ebin/zlib.beam Binary files differindex 43d7b436be..304783ae37 100644 --- a/erts/preloaded/ebin/zlib.beam +++ b/erts/preloaded/ebin/zlib.beam diff --git a/erts/preloaded/src/erlang.erl b/erts/preloaded/src/erlang.erl index 8ebb92d5b2..9517acd78e 100644 --- a/erts/preloaded/src/erlang.erl +++ b/erts/preloaded/src/erlang.erl @@ -2034,8 +2034,14 @@ nodes(_Arg) -> | eof | {parallelism, Boolean :: boolean()} | hide. -open_port(_PortName,_PortSettings) -> - erlang:nif_error(undefined). +open_port(PortName, PortSettings) -> + case case erts_internal:open_port(PortName, PortSettings) of + Ref when erlang:is_reference(Ref) -> receive {Ref, Res} -> Res end; + Res -> Res + end of + Port when erlang:is_port(Port) -> Port; + Error -> erlang:error(Error, [PortName, PortSettings]) + end. -type priority_level() :: low | normal | high | max. diff --git a/erts/preloaded/src/erts_internal.erl b/erts/preloaded/src/erts_internal.erl index ce0a6a1d9e..426749264f 100644 --- a/erts/preloaded/src/erts_internal.erl +++ b/erts/preloaded/src/erts_internal.erl @@ -32,7 +32,7 @@ -export([await_port_send_result/3]). -export([cmp_term/2]). -export([map_to_tuple_keys/1, term_type/1, map_hashmap_children/1]). --export([port_command/3, port_connect/2, port_close/1, +-export([open_port/2, port_command/3, port_connect/2, port_close/1, port_control/3, port_call/3, port_info/1, port_info/2]). -export([request_system_task/3]). @@ -88,6 +88,13 @@ gather_io_bytes(Ref, No, InAcc, OutAcc) -> %% Statically linked port NIFs %% +-spec erts_internal:open_port(PortName, PortSettings) -> Result when + PortName :: tuple(), + PortSettings :: term(), + Result :: port() | reference() | atom(). +open_port(_PortName, _PortSettings) -> + erlang:nif_error(undefined). + -spec erts_internal:port_command(Port, Data, OptionList) -> Result when Port :: port() | atom(), Data :: iodata(), diff --git a/lib/asn1/doc/src/notes.xml b/lib/asn1/doc/src/notes.xml index 9b2d6bbde5..315c472465 100644 --- a/lib/asn1/doc/src/notes.xml +++ b/lib/asn1/doc/src/notes.xml @@ -32,6 +32,23 @@ <p>This document describes the changes made to the asn1 application.</p> +<section><title>Asn1 4.0.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Trying to encode an empty named BIT STRING in BER would + fail with a <c>function_clause</c> exception. (Thanks to + Svilen Ivanov for reporting this bug.)</p> + <p> + Own Id: OTP-13149</p> + </item> + </list> + </section> + +</section> + <section><title>Asn1 4.0</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/asn1/vsn.mk b/lib/asn1/vsn.mk index d4c46863a3..87a229424c 100644 --- a/lib/asn1/vsn.mk +++ b/lib/asn1/vsn.mk @@ -1 +1 @@ -ASN1_VSN = 4.0 +ASN1_VSN = 4.0.1 diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index 6972d18dfc..9906db2c90 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -33,6 +33,66 @@ <file>notes.xml</file> </header> +<section><title>Common_Test 1.11.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + When data from the netconf server was split into many ssh + packages, the netconf client performed really bad. This + is now improved.</p> + <p> + Own Id: OTP-13007</p> + </item> + <item> + <p> + In ct_netconfc, if a timer expired 'at the same time' as + the server sent the rpc-reply, the timeout message might + already be in the client's message queue when the client + removed the timer ref from its 'pending' list. This + caused a crash in the client since the timer ref could no + longer be found when handling the timeout message. This + problem is now fixed by always flushing the timeout + message from the message queue when canceling a timer.</p> + <p> + Own Id: OTP-13008</p> + </item> + <item> + <p> + The error logger handler ct_conn_log_h did not respect + the 'silent' option, and tried to print to an undefined + file descriptor. This has been corrected.</p> + <p> + Own Id: OTP-13035</p> + </item> + <item> + <p> + If the user would let the test run proceed after test + suite compilation failure, Common Test did not set the + exit status to indicate failure as expected. This has + been corrected. Also, the 'abort_if_missing_suites' + option now makes Common Test abort the test run without + asking the user if compilation fails, even if access to + stdin/stdout exists.</p> + <p> + Own Id: OTP-13173 Aux Id: seq12978 </p> + </item> + <item> + <p> + With the Common Test 'create_priv_dir' start option set + to 'auto_per_tc', the name of the priv directory for a + configuration function could clash with the name of the + priv directory for a test case, which would cause Test + Server failure. This error has been corrected.</p> + <p> + Own Id: OTP-13181</p> + </item> + </list> + </section> + +</section> + <section><title>Common_Test 1.11</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/common_test/vsn.mk b/lib/common_test/vsn.mk index ff2bd20ab3..f33b705dcb 100644 --- a/lib/common_test/vsn.mk +++ b/lib/common_test/vsn.mk @@ -1 +1 @@ -COMMON_TEST_VSN = 1.11 +COMMON_TEST_VSN = 1.11.1 diff --git a/lib/compiler/doc/src/notes.xml b/lib/compiler/doc/src/notes.xml index daf3bd3af9..3c06e4f98e 100644 --- a/lib/compiler/doc/src/notes.xml +++ b/lib/compiler/doc/src/notes.xml @@ -32,6 +32,29 @@ <p>This document describes the changes made to the Compiler application.</p> +<section><title>Compiler 6.0.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix cerl_trees:label/2 bug with map K/V swap</p> + <p> + Own Id: OTP-13091</p> + </item> + <item> + <p> + Warnings produced when the '<c>bin_opt_info</c>' option + was given could sometimes lack filenames and line + numbers. (Thanks to José Valim for reporting this bug.)</p> + <p> + Own Id: OTP-13113</p> + </item> + </list> + </section> + +</section> + <section><title>Compiler 6.0.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/compiler/vsn.mk b/lib/compiler/vsn.mk index 357b35e47b..c5089ff57e 100644 --- a/lib/compiler/vsn.mk +++ b/lib/compiler/vsn.mk @@ -1 +1 @@ -COMPILER_VSN = 6.0.1 +COMPILER_VSN = 6.0.2 diff --git a/lib/crypto/doc/src/notes.xml b/lib/crypto/doc/src/notes.xml index 54dd8872eb..f684b6f6eb 100644 --- a/lib/crypto/doc/src/notes.xml +++ b/lib/crypto/doc/src/notes.xml @@ -31,6 +31,21 @@ </header> <p>This document describes the changes made to the Crypto application.</p> +<section><title>Crypto 3.6.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Small documentation fixes</p> + <p> + Own Id: OTP-13017</p> + </item> + </list> + </section> + +</section> + <section><title>Crypto 3.6.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/crypto/vsn.mk b/lib/crypto/vsn.mk index c2166a8e75..de4329e612 100644 --- a/lib/crypto/vsn.mk +++ b/lib/crypto/vsn.mk @@ -1 +1 @@ -CRYPTO_VSN = 3.6.1 +CRYPTO_VSN = 3.6.2 diff --git a/lib/debugger/src/dbg_iload.erl b/lib/debugger/src/dbg_iload.erl index 7746a06fcb..369b456524 100644 --- a/lib/debugger/src/dbg_iload.erl +++ b/lib/debugger/src/dbg_iload.erl @@ -39,7 +39,7 @@ %% dbg_iserver. We are suspended until the module has been loaded. %%-------------------------------------------------------------------- -spec load_mod(Mod, file:filename(), binary(), ets:tid()) -> - {'ok', Mod} when is_subtype(Mod, atom()). + {'ok', Mod} when Mod :: atom(). load_mod(Mod, File, Binary, Db) -> Flag = process_flag(trap_exit, true), diff --git a/lib/dialyzer/doc/src/notes.xml b/lib/dialyzer/doc/src/notes.xml index aa29684697..27364ae06a 100644 --- a/lib/dialyzer/doc/src/notes.xml +++ b/lib/dialyzer/doc/src/notes.xml @@ -32,6 +32,22 @@ <p>This document describes the changes made to the Dialyzer application.</p> +<section><title>Dialyzer 2.8.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Reintroduce the <c>erlang:make_fun/3</c> BIF in + erl_bif_types.</p> + <p> + Own Id: OTP-13068</p> + </item> + </list> + </section> + +</section> + <section><title>Dialyzer 2.8.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/dialyzer/src/dialyzer_contracts.erl b/lib/dialyzer/src/dialyzer_contracts.erl index 7251de8b10..83b3ef72f2 100644 --- a/lib/dialyzer/src/dialyzer_contracts.erl +++ b/lib/dialyzer/src/dialyzer_contracts.erl @@ -126,13 +126,19 @@ butlast([H|T]) -> [H|butlast(T)]. constraints_to_string([]) -> ""; -constraints_to_string([{type, _, constraint, [{atom, _, What}, Types]}]) -> - atom_to_list(What) ++ "(" ++ - sequence([erl_types:t_form_to_string(T) || T <- Types], ",") ++ ")"; constraints_to_string([{type, _, constraint, [{atom, _, What}, Types]}|Rest]) -> - atom_to_list(What) ++ "(" ++ - sequence([erl_types:t_form_to_string(T) || T <- Types], ",") - ++ "), " ++ constraints_to_string(Rest). + S = constraint_to_string(What, Types), + case Rest of + [] -> S; + _ -> S ++ ", " ++ constraints_to_string(Rest) + end. + +constraint_to_string(is_subtype, [{var, _, Var}, T]) -> + atom_to_list(Var) ++ " :: " ++ erl_types:t_form_to_string(T); +constraint_to_string(What, Types) -> + atom_to_list(What) ++ "(" + ++ sequence([erl_types:t_form_to_string(T) || T <- Types], ",") + ++ ")". sequence([], _Delimiter) -> ""; sequence([H], _Delimiter) -> H; diff --git a/lib/dialyzer/test/r9c_SUITE_data/results/mnesia b/lib/dialyzer/test/r9c_SUITE_data/results/mnesia index 1dc5a105bf..220b18ca7a 100644 --- a/lib/dialyzer/test/r9c_SUITE_data/results/mnesia +++ b/lib/dialyzer/test/r9c_SUITE_data/results/mnesia @@ -6,7 +6,7 @@ mnesia_bup.erl:111: The created fun has no local return mnesia_bup.erl:574: Function fallback_receiver/2 has no local return mnesia_bup.erl:967: Function uninstall_fallback_master/2 has no local return mnesia_checkpoint.erl:1014: The variable Error can never match since previous clauses completely covered the type {'ok',#checkpoint_args{nodes::[any()],retainers::[any(),...]}} -mnesia_checkpoint.erl:894: The call sys:handle_system_msg(Msg::any(),From::any(),'no_parent','mnesia_checkpoint',[],Cp::#checkpoint_args{}) breaks the contract (Msg,From,Parent,Module,Debug,Misc) -> no_return() when is_subtype(Msg,term()), is_subtype(From,{pid(),Tag::_}), is_subtype(Parent,pid()), is_subtype(Module,module()), is_subtype(Debug,[dbg_opt()]), is_subtype(Misc,term()) +mnesia_checkpoint.erl:894: The call sys:handle_system_msg(Msg::any(),From::any(),'no_parent','mnesia_checkpoint',[],Cp::#checkpoint_args{}) breaks the contract (Msg,From,Parent,Module,Debug,Misc) -> no_return() when Msg :: term(), From :: {pid(),Tag::_}, Parent :: pid(), Module :: module(), Debug :: [dbg_opt()], Misc :: term() mnesia_controller.erl:1666: The variable Tab can never match since previous clauses completely covered the type [any()] mnesia_controller.erl:1679: The pattern {'stop', Reason, Reply, State2} can never match the type {'noreply',_} | {'reply',_,_} | {'stop','shutdown',#state{}} mnesia_controller.erl:1685: The pattern {'noreply', State2, _Timeout} can never match the type {'reply',_,_} diff --git a/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes b/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes index a9fbfb6068..d2a3ebb766 100644 --- a/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes +++ b/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes @@ -1,15 +1,15 @@ -contracts_with_subtypes.erl:106: The call contracts_with_subtypes:rec_arg({'a','b'}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,{'a',A} | {'b',B}), is_subtype(A,'a' | {'b',B}), is_subtype(B,'b' | {'a',A}) -contracts_with_subtypes.erl:107: The call contracts_with_subtypes:rec_arg({'b','a'}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,{'a',A} | {'b',B}), is_subtype(A,'a' | {'b',B}), is_subtype(B,'b' | {'a',A}) -contracts_with_subtypes.erl:109: The call contracts_with_subtypes:rec_arg({'b',{'a','b'}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,{'a',A} | {'b',B}), is_subtype(A,'a' | {'b',B}), is_subtype(B,'b' | {'a',A}) -contracts_with_subtypes.erl:135: The call contracts_with_subtypes:rec2({'a','b'}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) -contracts_with_subtypes.erl:136: The call contracts_with_subtypes:rec2({'b','a'}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) -contracts_with_subtypes.erl:137: The call contracts_with_subtypes:rec2({'a',{'b','a'}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) -contracts_with_subtypes.erl:138: The call contracts_with_subtypes:rec2({'b',{'a','b'}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) -contracts_with_subtypes.erl:139: The call contracts_with_subtypes:rec2({'a',{'b',{'a','b'}}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) -contracts_with_subtypes.erl:140: The call contracts_with_subtypes:rec2({'b',{'a',{'b','a'}}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) -contracts_with_subtypes.erl:141: The call contracts_with_subtypes:rec2({'a',{'b',{'a',{'b','a'}}}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) -contracts_with_subtypes.erl:142: The call contracts_with_subtypes:rec2({'b',{'a',{'b',{'a','b'}}}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,ab()) +contracts_with_subtypes.erl:106: The call contracts_with_subtypes:rec_arg({'a','b'}) breaks the contract (Arg) -> 'ok' when Arg :: {'a',A} | {'b',B}, A :: 'a' | {'b',B}, B :: 'b' | {'a',A} +contracts_with_subtypes.erl:107: The call contracts_with_subtypes:rec_arg({'b','a'}) breaks the contract (Arg) -> 'ok' when Arg :: {'a',A} | {'b',B}, A :: 'a' | {'b',B}, B :: 'b' | {'a',A} +contracts_with_subtypes.erl:109: The call contracts_with_subtypes:rec_arg({'b',{'a','b'}}) breaks the contract (Arg) -> 'ok' when Arg :: {'a',A} | {'b',B}, A :: 'a' | {'b',B}, B :: 'b' | {'a',A} +contracts_with_subtypes.erl:135: The call contracts_with_subtypes:rec2({'a','b'}) breaks the contract (Arg) -> 'ok' when Arg :: ab() +contracts_with_subtypes.erl:136: The call contracts_with_subtypes:rec2({'b','a'}) breaks the contract (Arg) -> 'ok' when Arg :: ab() +contracts_with_subtypes.erl:137: The call contracts_with_subtypes:rec2({'a',{'b','a'}}) breaks the contract (Arg) -> 'ok' when Arg :: ab() +contracts_with_subtypes.erl:138: The call contracts_with_subtypes:rec2({'b',{'a','b'}}) breaks the contract (Arg) -> 'ok' when Arg :: ab() +contracts_with_subtypes.erl:139: The call contracts_with_subtypes:rec2({'a',{'b',{'a','b'}}}) breaks the contract (Arg) -> 'ok' when Arg :: ab() +contracts_with_subtypes.erl:140: The call contracts_with_subtypes:rec2({'b',{'a',{'b','a'}}}) breaks the contract (Arg) -> 'ok' when Arg :: ab() +contracts_with_subtypes.erl:141: The call contracts_with_subtypes:rec2({'a',{'b',{'a',{'b','a'}}}}) breaks the contract (Arg) -> 'ok' when Arg :: ab() +contracts_with_subtypes.erl:142: The call contracts_with_subtypes:rec2({'b',{'a',{'b',{'a','b'}}}}) breaks the contract (Arg) -> 'ok' when Arg :: ab() contracts_with_subtypes.erl:175: The pattern 1 can never match the type string() contracts_with_subtypes.erl:178: The pattern 'alpha' can never match the type {'ok',_} | {'ok',_,string()} contracts_with_subtypes.erl:180: The pattern 42 can never match the type {'ok',_} | {'ok',_,string()} @@ -24,13 +24,13 @@ contracts_with_subtypes.erl:23: Invalid type specification for function contract contracts_with_subtypes.erl:240: The pattern {'ok', 42} can never match the type {'ok',_,string()} contracts_with_subtypes.erl:241: The pattern 42 can never match the type {'ok',_,string()} contracts_with_subtypes.erl:267: Function flat_ets_new_t/0 has no local return -contracts_with_subtypes.erl:268: The call contracts_with_subtypes:flat_ets_new(12,[]) breaks the contract (Name,Options) -> atom() when is_subtype(Name,atom()), is_subtype(Options,[Option]), is_subtype(Option,'set' | 'ordered_set' | 'bag' | 'duplicate_bag' | 'public' | 'protected' | 'private' | 'named_table' | {'keypos',integer()} | {'heir',pid(),term()} | {'heir','none'} | {'write_concurrency',boolean()} | {'read_concurrency',boolean()} | 'compressed') +contracts_with_subtypes.erl:268: The call contracts_with_subtypes:flat_ets_new(12,[]) breaks the contract (Name,Options) -> atom() when Name :: atom(), Options :: [Option], Option :: 'set' | 'ordered_set' | 'bag' | 'duplicate_bag' | 'public' | 'protected' | 'private' | 'named_table' | {'keypos',integer()} | {'heir',pid(),term()} | {'heir','none'} | {'write_concurrency',boolean()} | {'read_concurrency',boolean()} | 'compressed' contracts_with_subtypes.erl:294: Function factored_ets_new_t/0 has no local return -contracts_with_subtypes.erl:295: The call contracts_with_subtypes:factored_ets_new(12,[]) breaks the contract (Name,Options) -> atom() when is_subtype(Name,atom()), is_subtype(Options,[Option]), is_subtype(Option,Type | Access | 'named_table' | {'keypos',Pos} | {'heir',Pid::pid(),HeirData} | {'heir','none'} | Tweaks), is_subtype(Type,type()), is_subtype(Access,access()), is_subtype(Tweaks,{'write_concurrency',boolean()} | {'read_concurrency',boolean()} | 'compressed'), is_subtype(Pos,pos_integer()), is_subtype(HeirData,term()) -contracts_with_subtypes.erl:77: The call contracts_with_subtypes:foo1(5) breaks the contract (Arg1) -> Res when is_subtype(Arg1,atom()), is_subtype(Res,atom()) -contracts_with_subtypes.erl:78: The call contracts_with_subtypes:foo2(5) breaks the contract (Arg1) -> Res when is_subtype(Arg1,Arg2), is_subtype(Arg2,atom()), is_subtype(Res,atom()) -contracts_with_subtypes.erl:79: The call contracts_with_subtypes:foo3(5) breaks the contract (Arg1) -> Res when is_subtype(Arg2,atom()), is_subtype(Arg1,Arg2), is_subtype(Res,atom()) +contracts_with_subtypes.erl:295: The call contracts_with_subtypes:factored_ets_new(12,[]) breaks the contract (Name,Options) -> atom() when Name :: atom(), Options :: [Option], Option :: Type | Access | 'named_table' | {'keypos',Pos} | {'heir',Pid::pid(),HeirData} | {'heir','none'} | Tweaks, Type :: type(), Access :: access(), Tweaks :: {'write_concurrency',boolean()} | {'read_concurrency',boolean()} | 'compressed', Pos :: pos_integer(), HeirData :: term() +contracts_with_subtypes.erl:77: The call contracts_with_subtypes:foo1(5) breaks the contract (Arg1) -> Res when Arg1 :: atom(), Res :: atom() +contracts_with_subtypes.erl:78: The call contracts_with_subtypes:foo2(5) breaks the contract (Arg1) -> Res when Arg1 :: Arg2, Arg2 :: atom(), Res :: atom() +contracts_with_subtypes.erl:79: The call contracts_with_subtypes:foo3(5) breaks the contract (Arg1) -> Res when Arg2 :: atom(), Arg1 :: Arg2, Res :: atom() contracts_with_subtypes.erl:7: Invalid type specification for function contracts_with_subtypes:extract/0. The success typing is () -> 'something' -contracts_with_subtypes.erl:80: The call contracts_with_subtypes:foo4(5) breaks the contract (Type) -> Type when is_subtype(Type,atom()) +contracts_with_subtypes.erl:80: The call contracts_with_subtypes:foo4(5) breaks the contract (Type) -> Type when Type :: atom() contracts_with_subtypes.erl:81: The call contracts_with_subtypes:foo5(5) breaks the contract (Type::atom()) -> Type::atom() -contracts_with_subtypes.erl:82: The call contracts_with_subtypes:foo6(5) breaks the contract (Type) -> Type when is_subtype(Type,atom()) +contracts_with_subtypes.erl:82: The call contracts_with_subtypes:foo6(5) breaks the contract (Type) -> Type when Type :: atom() diff --git a/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes2 b/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes2 index 9f5433a13d..1a8aeb13d0 100644 --- a/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes2 +++ b/lib/dialyzer/test/small_SUITE_data/results/contracts_with_subtypes2 @@ -1,3 +1,3 @@ contracts_with_subtypes2.erl:18: Function t/0 has no local return -contracts_with_subtypes2.erl:19: The call contracts_with_subtypes2:t({'a',{'b',{'c',{'d',{'e',{'g',3}}}}}}) breaks the contract (Arg) -> 'ok' when is_subtype(Arg,{'a',A}), is_subtype(A,{'b',B}), is_subtype(B,{'c',C}), is_subtype(C,{'d',D}), is_subtype(D,{'e',E}), is_subtype(E,{'f',_}) +contracts_with_subtypes2.erl:19: The call contracts_with_subtypes2:t({'a',{'b',{'c',{'d',{'e',{'g',3}}}}}}) breaks the contract (Arg) -> 'ok' when Arg :: {'a',A}, A :: {'b',B}, B :: {'c',C}, C :: {'d',D}, D :: {'e',E}, E :: {'f',_} diff --git a/lib/dialyzer/test/small_SUITE_data/results/pretty_bitstring b/lib/dialyzer/test/small_SUITE_data/results/pretty_bitstring index 0ad6eee766..e148e5cf22 100644 --- a/lib/dialyzer/test/small_SUITE_data/results/pretty_bitstring +++ b/lib/dialyzer/test/small_SUITE_data/results/pretty_bitstring @@ -1,3 +1,3 @@ pretty_bitstring.erl:7: Function t/0 has no local return -pretty_bitstring.erl:8: The call binary:copy(#{#<1>(8, 1, 'integer', ['unsigned', 'big']), #<2>(8, 1, 'integer', ['unsigned', 'big']), #<3>(3, 1, 'integer', ['unsigned', 'big'])}#,2) breaks the contract (Subject,N) -> binary() when is_subtype(Subject,binary()), is_subtype(N,non_neg_integer()) +pretty_bitstring.erl:8: The call binary:copy(#{#<1>(8, 1, 'integer', ['unsigned', 'big']), #<2>(8, 1, 'integer', ['unsigned', 'big']), #<3>(3, 1, 'integer', ['unsigned', 'big'])}#,2) breaks the contract (Subject,N) -> binary() when Subject :: binary(), N :: non_neg_integer() diff --git a/lib/dialyzer/test/user_SUITE_data/results/wsp_pdu b/lib/dialyzer/test/user_SUITE_data/results/wsp_pdu index d1f8f4caf2..626e677524 100644 --- a/lib/dialyzer/test/user_SUITE_data/results/wsp_pdu +++ b/lib/dialyzer/test/user_SUITE_data/results/wsp_pdu @@ -6,7 +6,7 @@ wsp_pdu.erl:2403: The call wsp_pdu:d_date(Data1::binary()) will never return sin wsp_pdu.erl:2406: Guard test is_integer(Sec::{[byte()] | byte() | {'long',binary()} | {'short',binary()},binary()}) can never succeed wsp_pdu.erl:2408: The pattern {'short', Data2} can never match the type {[byte()] | byte() | {'long',binary()} | {'short',binary()},binary()} wsp_pdu.erl:2755: Function parse_push_flag/1 has no local return -wsp_pdu.erl:2756: The call erlang:integer_to_list(Value::[any()]) breaks the contract (Integer) -> string() when is_subtype(Integer,integer()) +wsp_pdu.erl:2756: The call erlang:integer_to_list(Value::[any()]) breaks the contract (Integer) -> string() when Integer :: integer() wsp_pdu.erl:2875: The call wsp_pdu:d_text_string(Data::byte()) will never return since it differs in the 1st argument from the success typing arguments: (binary()) wsp_pdu.erl:2976: The call wsp_pdu:d_q_value(QData::byte()) will never return since it differs in the 1st argument from the success typing arguments: (<<_:8,_:_*8>>) wsp_pdu.erl:3336: The call wsp_pdu:encode_typed_field(Ver::any(),'Q-value',ParamValue::any()) will never return since it differs in the 2nd argument from the success typing arguments: (any(),'Constrained-encoding' | 'Date-value' | 'No-value' | 'Short-integer' | 'Text-string' | 'Text-value' | 'Well-known-charset',any()) diff --git a/lib/diameter/doc/src/notes.xml b/lib/diameter/doc/src/notes.xml index 61bed37682..828ade4a71 100644 --- a/lib/diameter/doc/src/notes.xml +++ b/lib/diameter/doc/src/notes.xml @@ -43,6 +43,29 @@ first.</p> <!-- ===================================================================== --> +<section><title>diameter 1.11.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix request table leaks</p> + <p> + The End-to-End and Hop-by-Hop identifiers of outgoing + Diameter requests are stored in a table in order for the + caller to be located when the corresponding answer + message is received. Entries were orphaned if the handler + was terminated by an exit signal as a consequence of + actions taken by callback functions, or if callbacks + modified identifiers in retransmission cases.</p> + <p> + Own Id: OTP-13137</p> + </item> + </list> + </section> + +</section> + <section><title>diameter 1.11</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/erl_docgen/doc/src/notes.xml b/lib/erl_docgen/doc/src/notes.xml index ba1ad2f5e8..aa8bf14919 100644 --- a/lib/erl_docgen/doc/src/notes.xml +++ b/lib/erl_docgen/doc/src/notes.xml @@ -31,7 +31,24 @@ </header> <p>This document describes the changes made to the <em>erl_docgen</em> application.</p> - <section><title>Erl_Docgen 0.4</title> + <section><title>Erl_Docgen 0.4.1</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> Updated the xmllint target to just check the xml + files with real documentation content.<br/> Corrected + some errors and added some missing target in the DTD's. + </p> + <p> + Own Id: OTP-13026</p> + </item> + </list> + </section> + +</section> + +<section><title>Erl_Docgen 0.4</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/erl_interface/doc/src/notes.xml b/lib/erl_interface/doc/src/notes.xml index 00427ea2ee..6d951e895f 100644 --- a/lib/erl_interface/doc/src/notes.xml +++ b/lib/erl_interface/doc/src/notes.xml @@ -31,6 +31,22 @@ </header> <p>This document describes the changes made to the Erl_interface application.</p> +<section><title>Erl_Interface 3.8.1</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Fix the conditional selection of gethostbyname_r and + gethostbyaddr_r.</p> + <p> + Own Id: OTP-13188</p> + </item> + </list> + </section> + +</section> + <section><title>Erl_Interface 3.8</title> <section><title>Improvements and New Features</title> diff --git a/lib/erl_interface/vsn.mk b/lib/erl_interface/vsn.mk index 94cfd30cec..18ba9df41e 100644 --- a/lib/erl_interface/vsn.mk +++ b/lib/erl_interface/vsn.mk @@ -1,2 +1,2 @@ -EI_VSN = 3.8 +EI_VSN = 3.8.1 ERL_INTERFACE_VSN = $(EI_VSN) diff --git a/lib/eunit/doc/src/notes.xml b/lib/eunit/doc/src/notes.xml index 3760e396ee..cf0523d230 100644 --- a/lib/eunit/doc/src/notes.xml +++ b/lib/eunit/doc/src/notes.xml @@ -33,6 +33,21 @@ </header> <p>This document describes the changes made to the EUnit application.</p> +<section><title>Eunit 2.2.12</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Small documentation fixes</p> + <p> + Own Id: OTP-13017</p> + </item> + </list> + </section> + +</section> + <section><title>Eunit 2.2.11</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/eunit/vsn.mk b/lib/eunit/vsn.mk index 079520def2..00cbab5829 100644 --- a/lib/eunit/vsn.mk +++ b/lib/eunit/vsn.mk @@ -1 +1 @@ -EUNIT_VSN = 2.2.11 +EUNIT_VSN = 2.2.12 diff --git a/lib/hipe/doc/src/notes.xml b/lib/hipe/doc/src/notes.xml index e1aec698e4..b5b13948e9 100644 --- a/lib/hipe/doc/src/notes.xml +++ b/lib/hipe/doc/src/notes.xml @@ -31,6 +31,51 @@ </header> <p>This document describes the changes made to HiPE.</p> +<section><title>Hipe 3.14</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix hipe bug causing segfaults when native code + constructs binaries starting with a zero-length integer + field.</p> + <p> + Own Id: OTP-13048</p> + </item> + <item> + <p> + Reintroduce the <c>erlang:make_fun/3</c> BIF in + erl_bif_types.</p> + <p> + Own Id: OTP-13068</p> + </item> + <item> + <p> + In certain cases of matching with very big binaries, the + HiPE compiler generated code that would fail the match, + even in cases that the matching was successful. The + problem was more quite noticeable on 32-bit platforms.</p> + <p> + Own Id: OTP-13092</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + mikpe/hipe_x86_signal-musl-support</p> + <p> + Own Id: OTP-13159</p> + </item> + </list> + </section> + +</section> + <section><title>Hipe 3.13</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/hipe/icode/hipe_icode.erl b/lib/hipe/icode/hipe_icode.erl index 9692eebb10..07d230491d 100644 --- a/lib/hipe/icode/hipe_icode.erl +++ b/lib/hipe/icode/hipe_icode.erl @@ -1376,12 +1376,12 @@ remove_constants(L) -> %% Substitution: replace occurrences of X by Y if {X,Y} is in the %% Subst_list. --spec subst([{_,_}], I) -> I when is_subtype(I, icode_instr()). +-spec subst([{_,_}], I) -> I when I :: icode_instr(). subst(Subst, I) -> subst_defines(Subst, subst_uses(Subst, I)). --spec subst_uses([{_,_}], I) -> I when is_subtype(I, icode_instr()). +-spec subst_uses([{_,_}], I) -> I when I :: icode_instr(). subst_uses(Subst, I) -> case I of @@ -1405,7 +1405,7 @@ subst_uses(Subst, I) -> #icode_label{} -> I end. --spec subst_defines([{_,_}], I) -> I when is_subtype(I, icode_instr()). +-spec subst_defines([{_,_}], I) -> I when I :: icode_instr(). subst_defines(Subst, I) -> case I of diff --git a/lib/hipe/vsn.mk b/lib/hipe/vsn.mk index 3ec9d7ee45..123792e708 100644 --- a/lib/hipe/vsn.mk +++ b/lib/hipe/vsn.mk @@ -1 +1 @@ -HIPE_VSN = 3.13 +HIPE_VSN = 3.14 diff --git a/lib/inets/doc/src/notes.xml b/lib/inets/doc/src/notes.xml index 7ebb3ddffa..c98ec1a9dc 100644 --- a/lib/inets/doc/src/notes.xml +++ b/lib/inets/doc/src/notes.xml @@ -33,7 +33,72 @@ <file>notes.xml</file> </header> - <section><title>Inets 6.0.3</title> + <section><title>Inets 6.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Replace obs-folds with spaces instead of failing</p> + <p> + Own Id: OTP-13069</p> + </item> + <item> + <p> + Add validation fun for URI scheme to http_uri API</p> + <p> + Own Id: OTP-13071</p> + </item> + <item> + <p> + Handle stream bodies as documented.</p> + <p> + Own Id: OTP-13093</p> + </item> + <item> + <p> + Correct error handling of mod_esi generated chunks. Send + warning headers in chunk trailers instead of generating + an unexpected additional 500 request response, when + problems, such as a timeout occurs.</p> + <p> + Own Id: OTP-13110</p> + </item> + <item> + <p> + HTTP client terminates gracefully when an invalid chunked + length header is encountered.</p> + <p> + Own Id: OTP-13117</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add default for SNI (Server Name Indication) when running + https using the inets HTTP-client.</p> + <p> + Own Id: OTP-12985</p> + </item> + <item> + <p> + Be forgiving to chunked sizes that have trailing + whitespaces as prior implementation was. Also some legacy + embedded devices does actually have trailing whitespaces + even though this in not according to the spec.</p> + <p> + Own Id: OTP-13116</p> + </item> + </list> + </section> + +</section> + +<section><title>Inets 6.0.3</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/inets/src/http_server/Makefile b/lib/inets/src/http_server/Makefile index b9f2290289..1c05d454a5 100644 --- a/lib/inets/src/http_server/Makefile +++ b/lib/inets/src/http_server/Makefile @@ -137,7 +137,7 @@ release_spec: opt $(INSTALL_DIR) "$(RELSYSDIR)/src/http_server" $(INSTALL_DATA) $(HRL_FILES) $(ERL_FILES) "$(RELSYSDIR)/src/http_server" $(INSTALL_DIR) "$(RELSYSDIR)/ebin" - $(INSTALL_DATA) $(TARGET_FILES) "$(RELSYSDIR)/ebin" + $(INSTALL_DATA) $(TARGET_FILES) $(BEHAVIOUR_TARGET_FILES) "$(RELSYSDIR)/ebin" release_docs_spec: diff --git a/lib/inets/src/http_server/httpd_custom_api.erl b/lib/inets/src/http_server/httpd_custom_api.erl index 282f3a6ee6..d5a6fa8715 100644 --- a/lib/inets/src/http_server/httpd_custom_api.erl +++ b/lib/inets/src/http_server/httpd_custom_api.erl @@ -23,7 +23,8 @@ -callback response_default_headers() -> [{Key::string(), Value::string()}]. -callback response_header({Key::string(), Value::string()}) -> - {true, {Key::string(), Value::string()}} | false. + {true, {Key::string(), Value::string()}} | false | + {true, string()}. %% Used internally to avoid traversing headers twice -callback request_header({Key::string(), Value::string()}) -> {true, {Key::string(), Value::string()}} | false. diff --git a/lib/inets/src/inets_app/inets.app.src b/lib/inets/src/inets_app/inets.app.src index 2b9b8f5f32..883ba84e8e 100644 --- a/lib/inets/src/inets_app/inets.app.src +++ b/lib/inets/src/inets_app/inets.app.src @@ -65,6 +65,7 @@ httpd_connection_sup, httpd_conf, httpd_custom, + httpd_custom_api, httpd_esi, httpd_example, httpd_file, diff --git a/lib/inets/vsn.mk b/lib/inets/vsn.mk index 7cc95fa6d3..2717f5b110 100644 --- a/lib/inets/vsn.mk +++ b/lib/inets/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = inets -INETS_VSN = 6.0.3 +INETS_VSN = 6.1 PRE_VSN = APP_VSN = "$(APPLICATION)-$(INETS_VSN)$(PRE_VSN)" diff --git a/lib/jinterface/doc/src/notes.xml b/lib/jinterface/doc/src/notes.xml index d9f7ae9f92..e66bcda0c1 100644 --- a/lib/jinterface/doc/src/notes.xml +++ b/lib/jinterface/doc/src/notes.xml @@ -31,6 +31,28 @@ </header> <p>This document describes the changes made to the Jinterface application.</p> +<section><title>Jinterface 1.6.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Add missing Term tag matching switch statement that was + missing an external fun tag.</p> + <p> + Own Id: OTP-13106</p> + </item> + <item> + <p> + fixed writing small compressed values.</p> + <p> + Own Id: OTP-13165</p> + </item> + </list> + </section> + +</section> + <section><title>Jinterface 1.6</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/jinterface/vsn.mk b/lib/jinterface/vsn.mk index 4df01d1151..41e670528a 100644 --- a/lib/jinterface/vsn.mk +++ b/lib/jinterface/vsn.mk @@ -1 +1 @@ -JINTERFACE_VSN = 1.6 +JINTERFACE_VSN = 1.6.1 diff --git a/lib/kernel/doc/src/notes.xml b/lib/kernel/doc/src/notes.xml index 1049e72ffe..b625ddba5d 100644 --- a/lib/kernel/doc/src/notes.xml +++ b/lib/kernel/doc/src/notes.xml @@ -31,6 +31,70 @@ </header> <p>This document describes the changes made to the Kernel application.</p> +<section><title>Kernel 4.1.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Host name lookups though inet_res, the Erlang DNS + resolver, are now done case insensitively according to + RFC 4343. Patch by Holger Weiß.</p> + <p> + Own Id: OTP-12836</p> + </item> + <item> + <p> + IPv6 distribution handler has been updated to share code + with IPv4 so that all features are supported in IPv6 as + well. A bug when using an IPv4 address as hostname has + been fixed.</p> + <p> + Own Id: OTP-13040</p> + </item> + <item> + <p> + Caching of host names in the internal DNS resolver + inet_res has been made character case insensitive for + host names according to RFC 4343.</p> + <p> + Own Id: OTP-13083</p> + </item> + <item> + <p>Cooked file mode buffering has been fixed so + file:position/2 now works according to Posix on Posix + systems i.e. when file:position/2 returns an error the + file pointer is unaffected.</p> <p>The Windows system + documentation, however, is unclear on this point so the + documentation of file:position/2 still does not promise + anything.</p> <p>Cooked file mode file:pread/2,3 and + file:pwrite/2,3 have been corrected to honor character + encoding like the combination of file:position/2 and + file:read/2 or file:write/2 already does. This is + probably not very useful since the character + representation on the caller's side is latin1, + period.</p> + <p> + Own Id: OTP-13155 Aux Id: PR#646 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add {line_delim, byte()} option to inet:setopts/2 and + decode_packet/3</p> + <p> + Own Id: OTP-12837</p> + </item> + </list> + </section> + +</section> + <section><title>Kernel 4.1</title> <section><title>Improvements and New Features</title> diff --git a/lib/kernel/src/kernel.appup.src b/lib/kernel/src/kernel.appup.src index 3fda55d1a9..860d3640d0 100644 --- a/lib/kernel/src/kernel.appup.src +++ b/lib/kernel/src/kernel.appup.src @@ -18,9 +18,9 @@ %% %CopyrightEnd% {"%VSN%", %% Up from - max one major revision back - [{<<"4\\.0(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.0.* + [{<<"4\\.[0-1](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.* {<<"3\\.[0-2](\\.[0-9]+)*">>,[restart_new_emulator]}], % OTP-17 %% Down to - max one major revision back - [{<<"4\\.0(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.0.* + [{<<"4\\.[0-1](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.* {<<"3\\.[0-2](\\.[0-9]+)*">>,[restart_new_emulator]}] % OTP-17 }. diff --git a/lib/kernel/src/os.erl b/lib/kernel/src/os.erl index ffb899e5ca..0022959c11 100644 --- a/lib/kernel/src/os.erl +++ b/lib/kernel/src/os.erl @@ -212,174 +212,33 @@ extensions() -> Command :: atom() | io_lib:chars(). cmd(Cmd) -> validate(Cmd), - Bytes = case type() of - {unix, _} -> - unix_cmd(Cmd); - {win32, Wtype} -> - Command0 = case {os:getenv("COMSPEC"),Wtype} of - {false,windows} -> lists:concat(["command.com /c", Cmd]); - {false,_} -> lists:concat(["cmd /c", Cmd]); - {Cspec,_} -> lists:concat([Cspec," /c",Cmd]) - end, - %% open_port/2 awaits string() in Command, but io_lib:chars() can be - %% deep lists according to io_lib module description. - Command = lists:flatten(Command0), - Port = open_port({spawn, Command}, [stream, in, eof, hide]), - get_data(Port, []) - end, - String = unicode:characters_to_list(list_to_binary(Bytes)), + {SpawnCmd, SpawnOpts, SpawnInput} = mk_cmd(os:type(), Cmd), + Port = open_port({spawn, SpawnCmd}, [binary, stderr_to_stdout, + stream, in, eof, hide | SpawnOpts]), + true = port_command(Port, SpawnInput), + Bytes = get_data(Port, []), + String = unicode:characters_to_list(Bytes), if %% Convert to unicode list if possible otherwise return bytes is_list(String) -> String; - true -> Bytes + true -> binary_to_list(Bytes) end. -unix_cmd(Cmd) -> - Tag = make_ref(), - {Pid,Mref} = erlang:spawn_monitor( - fun() -> - process_flag(trap_exit, true), - Port = start_port(), - erlang:port_command(Port, mk_cmd(Cmd)), - exit({Tag,unix_get_data(Port)}) - end), - receive - {'DOWN',Mref,_,Pid,{Tag,Result}} -> - Result; - {'DOWN',Mref,_,Pid,Reason} -> - exit(Reason) - end. - -%% The -s flag implies that only the positional parameters are set, -%% and the commands are read from standard input. We set the -%% $1 parameter for easy identification of the resident shell. -%% --define(ROOT, "/"). --define(ROOT_ANDROID, "/system"). --define(SHELL, "bin/sh -s unix:cmd 2>&1"). --define(PORT_CREATOR_NAME, os_cmd_port_creator). - -%% -%% Serializing open_port through a process to avoid smp lock contention -%% when many concurrent os:cmd() want to do vfork (OTP-7890). -%% --spec start_port() -> port(). -start_port() -> - Ref = make_ref(), - Request = {Ref,self()}, - {Pid, Mon} = case whereis(?PORT_CREATOR_NAME) of - undefined -> - spawn_monitor(fun() -> - start_port_srv(Request) - end); - P -> - P ! Request, - M = erlang:monitor(process, P), - {P, M} - end, - receive - {Ref, Port} when is_port(Port) -> - erlang:demonitor(Mon, [flush]), - Port; - {Ref, Error} -> - erlang:demonitor(Mon, [flush]), - exit(Error); - {'DOWN', Mon, process, Pid, _Reason} -> - start_port() - end. - -start_port_srv(Request) -> - %% We don't want a group leader of some random application. Use - %% kernel_sup's group leader. - {group_leader, GL} = process_info(whereis(kernel_sup), - group_leader), - true = group_leader(GL, self()), - process_flag(trap_exit, true), - StayAlive = try register(?PORT_CREATOR_NAME, self()) - catch - error:_ -> false - end, - start_port_srv_handle(Request), - case StayAlive of - true -> start_port_srv_loop(); - false -> exiting - end. - -start_port_srv_handle({Ref,Client}) -> - Path = case lists:reverse(erlang:system_info(system_architecture)) of - % androideabi - "ibaediordna" ++ _ -> filename:join([?ROOT_ANDROID, ?SHELL]); - _ -> filename:join([?ROOT, ?SHELL]) - end, - Reply = try open_port({spawn, Path},[stream]) of - Port when is_port(Port) -> - (catch port_connect(Port, Client)), - unlink(Port), - Port - catch - error:Reason -> - {Reason,erlang:get_stacktrace()} - end, - Client ! {Ref,Reply}, - ok. - -start_port_srv_loop() -> - receive - {Ref, Client} = Request when is_reference(Ref), - is_pid(Client) -> - start_port_srv_handle(Request); - _Junk -> - ok - end, - start_port_srv_loop(). - -%% -%% unix_get_data(Port) -> Result -%% -unix_get_data(Port) -> - unix_get_data(Port, []). - -unix_get_data(Port, Sofar) -> - receive - {Port,{data, Bytes}} -> - case eot(Bytes) of - {done, Last} -> - lists:flatten([Sofar|Last]); - more -> - unix_get_data(Port, [Sofar|Bytes]) - end; - {'EXIT', Port, _} -> - lists:flatten(Sofar) - end. - -%% -%% eot(String) -> more | {done, Result} -%% -eot(Bs) -> - eot(Bs, []). - -eot([4| _Bs], As) -> - {done, lists:reverse(As)}; -eot([B| Bs], As) -> - eot(Bs, [B| As]); -eot([], _As) -> - more. - -%% -%% mk_cmd(Cmd) -> {ok, ShellCommandString} | {error, ErrorString} -%% -%% We do not allow any input to Cmd (hence commands that want -%% to read from standard input will return immediately). -%% Standard error is redirected to standard output. -%% -%% We use ^D (= EOT = 4) to mark the end of the stream. -%% -mk_cmd(Cmd) when is_atom(Cmd) -> % backward comp. - mk_cmd(atom_to_list(Cmd)); -mk_cmd(Cmd) -> - %% We insert a new line after the command, in case the command - %% contains a comment character. - [$(, unicode:characters_to_binary(Cmd), "\n) </dev/null; echo \"\^D\"\n"]. - +mk_cmd({win32,Wtype}, Cmd) -> + Command = case {os:getenv("COMSPEC"),Wtype} of + {false,windows} -> lists:concat(["command.com /c", Cmd]); + {false,_} -> lists:concat(["cmd /c", Cmd]); + {Cspec,_} -> lists:concat([Cspec," /c",Cmd]) + end, + {Command, [], []}; +mk_cmd(OsType,Cmd) when is_atom(Cmd) -> + mk_cmd(OsType, atom_to_list(Cmd)); +mk_cmd(_,Cmd) -> + %% Have to send command in like this in order to make sh commands like + %% cd and ulimit available + {"/bin/sh -s unix:cmd", [out], + %% We insert a new line after the command, in case the command + %% contains a comment character. + ["(", unicode:characters_to_binary(Cmd), "\n); exit\n"]}. validate(Atom) when is_atom(Atom) -> ok; @@ -397,7 +256,7 @@ validate1([]) -> get_data(Port, Sofar) -> receive {Port, {data, Bytes}} -> - get_data(Port, [Sofar|Bytes]); + get_data(Port, [Sofar,Bytes]); {Port, eof} -> Port ! {self(), close}, receive @@ -410,5 +269,5 @@ get_data(Port, Sofar) -> after 1 -> % force context switch ok end, - lists:flatten(Sofar) + iolist_to_binary(Sofar) end. diff --git a/lib/kernel/test/os_SUITE.erl b/lib/kernel/test/os_SUITE.erl index 29d8d10262..83a95019e7 100644 --- a/lib/kernel/test/os_SUITE.erl +++ b/lib/kernel/test/os_SUITE.erl @@ -22,7 +22,8 @@ -export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, init_per_group/2,end_per_group/2]). -export([space_in_cwd/1, quoting/1, cmd_unicode/1, space_in_name/1, bad_command/1, - find_executable/1, unix_comment_in_command/1, deep_list_command/1, evil/1]). + find_executable/1, unix_comment_in_command/1, deep_list_command/1, + large_output_command/1]). -include_lib("test_server/include/test_server.hrl"). @@ -30,7 +31,8 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [space_in_cwd, quoting, cmd_unicode, space_in_name, bad_command, - find_executable, unix_comment_in_command, deep_list_command, evil]. + find_executable, unix_comment_in_command, deep_list_command, + large_output_command]. groups() -> []. @@ -267,50 +269,14 @@ deep_list_command(Config) when is_list(Config) -> %% FYI: [$e, $c, "ho"] =:= io_lib:format("ec~s", ["ho"]) ok. - --define(EVIL_PROCS, 100). --define(EVIL_LOOPS, 100). --define(PORT_CREATOR, os_cmd_port_creator). -evil(Config) when is_list(Config) -> - Dog = test_server:timetrap(test_server:minutes(5)), - Parent = self(), - Ps = lists:map(fun (N) -> - spawn_link(fun () -> - evil_loop(Parent, ?EVIL_LOOPS,N) - end) - end, lists:seq(1, ?EVIL_PROCS)), - Devil = spawn_link(fun () -> devil(hd(Ps), hd(lists:reverse(Ps))) end), - lists:foreach(fun (P) -> receive {P, done} -> ok end end, Ps), - unlink(Devil), - exit(Devil, kill), - test_server:timetrap_cancel(Dog), - ok. - -devil(P1, P2) -> - erlang:display({?PORT_CREATOR, whereis(?PORT_CREATOR)}), - (catch ?PORT_CREATOR ! lists:seq(1,1000000)), - (catch ?PORT_CREATOR ! lists:seq(1,666)), - (catch ?PORT_CREATOR ! grrrrrrrrrrrrrrrr), - (catch ?PORT_CREATOR ! {'EXIT', P1, buhuuu}), - (catch ?PORT_CREATOR ! {'EXIT', hd(erlang:ports()), buhuuu}), - (catch ?PORT_CREATOR ! {'EXIT', P2, arggggggg}), - receive after 500 -> ok end, - (catch exit(whereis(?PORT_CREATOR), kill)), - (catch ?PORT_CREATOR ! ">8|"), - receive after 500 -> ok end, - (catch exit(whereis(?PORT_CREATOR), diiiiiiiiiiiiiiiiiiiie)), - receive after 100 -> ok end, - devil(P1, P2). - -evil_loop(Parent, Loops, N) -> - Res = integer_to_list(N), - evil_loop(Parent, Loops, Res, "echo " ++ Res). - -evil_loop(Parent, 0, _Res, _Cmd) -> - Parent ! {self(), done}; -evil_loop(Parent, Loops, Res, Cmd) -> - comp(Res, os:cmd(Cmd)), - evil_loop(Parent, Loops-1, Res, Cmd). +large_output_command(doc) -> + "Test to take sure that the correct data is" + "received when doing large commands"; +large_output_command(suite) -> []; +large_output_command(Config) when is_list(Config) -> + %% Maximum allowed on windows is 8192, so we test well below that + AAA = lists:duplicate(7000, $a), + comp(AAA,os:cmd("echo " ++ AAA)). comp(Expected, Got) -> case strip_nl(Got) of diff --git a/lib/kernel/vsn.mk b/lib/kernel/vsn.mk index d549033302..703075634b 100644 --- a/lib/kernel/vsn.mk +++ b/lib/kernel/vsn.mk @@ -1 +1 @@ -KERNEL_VSN = 4.1 +KERNEL_VSN = 4.1.1 diff --git a/lib/observer/doc/src/notes.xml b/lib/observer/doc/src/notes.xml index f0c87d865e..e2eeffc667 100644 --- a/lib/observer/doc/src/notes.xml +++ b/lib/observer/doc/src/notes.xml @@ -32,6 +32,36 @@ <p>This document describes the changes made to the Observer application.</p> +<section><title>Observer 2.1.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Show ets owner pid in crashdump viewers popup window, + thanks Leo Liu.</p> + <p> + Own Id: OTP-13030</p> + </item> + <item> + <p> + Several initialisms (eg, ERTS, ETS, SMP) are used as + headings. They were being capitalized incorrectly.</p> + <p> + Own Id: OTP-13044</p> + </item> + <item> + <p> + Fixed a crash in crashdump viewer when dump contained a + truncated binary.</p> + <p> + Own Id: OTP-13163</p> + </item> + </list> + </section> + +</section> + <section><title>Observer 2.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/observer/vsn.mk b/lib/observer/vsn.mk index 7e7e32099b..bd89977e28 100644 --- a/lib/observer/vsn.mk +++ b/lib/observer/vsn.mk @@ -1 +1 @@ -OBSERVER_VSN = 2.1 +OBSERVER_VSN = 2.1.1 diff --git a/lib/parsetools/doc/src/notes.xml b/lib/parsetools/doc/src/notes.xml index 7a9770e667..43840a3bc7 100644 --- a/lib/parsetools/doc/src/notes.xml +++ b/lib/parsetools/doc/src/notes.xml @@ -31,6 +31,22 @@ </header> <p>This document describes the changes made to the Parsetools application.</p> +<section><title>Parsetools 2.1.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Correct the documentation of the error tuple returned by + Yecc and Leex.</p> + <p> + Own Id: OTP-13031</p> + </item> + </list> + </section> + +</section> + <section><title>Parsetools 2.1</title> <section><title>Improvements and New Features</title> diff --git a/lib/parsetools/vsn.mk b/lib/parsetools/vsn.mk index b99b3bb713..de3da23c8a 100644 --- a/lib/parsetools/vsn.mk +++ b/lib/parsetools/vsn.mk @@ -1 +1 @@ -PARSETOOLS_VSN = 2.1 +PARSETOOLS_VSN = 2.1.1 diff --git a/lib/public_key/doc/src/notes.xml b/lib/public_key/doc/src/notes.xml index 8034d7fade..8eb56f7354 100644 --- a/lib/public_key/doc/src/notes.xml +++ b/lib/public_key/doc/src/notes.xml @@ -35,6 +35,46 @@ <file>notes.xml</file> </header> +<section><title>Public_Key 1.1</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + The 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384' and + 'ecdsa-sha2-nistp521' signature algorithms for ssh are + implemented. See RFC 5656.</p> + <p> + Own Id: OTP-12936</p> + </item> + <item> + <p> + There is now a file (public_key/priv/moduli) which lists + size-generator-modulus triples. The purpose is to give + servers the possibility to select the crypto primes + randomly among a list of pregenerated triples. This + reduces the risk for some attacks on diffie-hellman + negotiation.</p> + <p> + See the reference manual for public_key:dh_gex_group/4 + where the handling of this is described.</p> + <p> + The ssh server (ssh:daemon) uses this.</p> + <p> + Own Id: OTP-13054 Aux Id: OTP-13052 </p> + </item> + <item> + <p> + Add different upper bounds for diffrent string types as + suggested by comment in PKIX1Explicit88.</p> + <p> + Own Id: OTP-13132</p> + </item> + </list> + </section> + +</section> + <section><title>Public_Key 1.0.1</title> <section><title>Improvements and New Features</title> diff --git a/lib/runtime_tools/doc/src/notes.xml b/lib/runtime_tools/doc/src/notes.xml index 5aebea98ce..e92f0e02ad 100644 --- a/lib/runtime_tools/doc/src/notes.xml +++ b/lib/runtime_tools/doc/src/notes.xml @@ -32,6 +32,21 @@ <p>This document describes the changes made to the Runtime_Tools application.</p> +<section><title>Runtime_Tools 1.9.2</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Clarified dbg:stop documentation</p> + <p> + Own Id: OTP-13078</p> + </item> + </list> + </section> + +</section> + <section><title>Runtime_Tools 1.9.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/runtime_tools/vsn.mk b/lib/runtime_tools/vsn.mk index 83e3612561..3dd7df9f2e 100644 --- a/lib/runtime_tools/vsn.mk +++ b/lib/runtime_tools/vsn.mk @@ -1 +1 @@ -RUNTIME_TOOLS_VSN = 1.9.1 +RUNTIME_TOOLS_VSN = 1.9.2 diff --git a/lib/sasl/doc/src/notes.xml b/lib/sasl/doc/src/notes.xml index 5945ef6490..537511a865 100644 --- a/lib/sasl/doc/src/notes.xml +++ b/lib/sasl/doc/src/notes.xml @@ -31,6 +31,21 @@ </header> <p>This document describes the changes made to the SASL application.</p> +<section><title>SASL 2.6.1</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Documentation improvements</p> + <p> + Own Id: OTP-13000</p> + </item> + </list> + </section> + +</section> + <section><title>SASL 2.6</title> <section><title>Improvements and New Features</title> diff --git a/lib/sasl/src/sasl.appup.src b/lib/sasl/src/sasl.appup.src index 2c8812f566..8faa0afbd4 100644 --- a/lib/sasl/src/sasl.appup.src +++ b/lib/sasl/src/sasl.appup.src @@ -18,9 +18,9 @@ %% %CopyrightEnd% {"%VSN%", %% Up from - max one major revision back - [{<<"2\\.5(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.0.* - {<<"2\\.4(\\.[0-9]+)*">>,[restart_new_emulator]}], % OTP-17 + [{<<"2\\.[5-6](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.* + {<<"2\\.4(\\.[0-9]+)*">>,[restart_new_emulator]}], % OTP-17 %% Down to - max one major revision back - [{<<"2\\.5(\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.0.* - {<<"2\\.4(\\.[0-9]+)*">>,[restart_new_emulator]}] % OTP-17 + [{<<"2\\.[5-6](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.* + {<<"2\\.4(\\.[0-9]+)*">>,[restart_new_emulator]}] % OTP-17 }. diff --git a/lib/sasl/vsn.mk b/lib/sasl/vsn.mk index 959d9c88d5..cb454d5331 100644 --- a/lib/sasl/vsn.mk +++ b/lib/sasl/vsn.mk @@ -1 +1 @@ -SASL_VSN = 2.6 +SASL_VSN = 2.6.1 diff --git a/lib/snmp/doc/src/notes.xml b/lib/snmp/doc/src/notes.xml index 94e73ddfca..9e7f13e126 100644 --- a/lib/snmp/doc/src/notes.xml +++ b/lib/snmp/doc/src/notes.xml @@ -34,7 +34,35 @@ </header> - <section><title>SNMP 5.2</title> + <section><title>SNMP 5.2.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Small documentation fixes</p> + <p> + Own Id: OTP-13017</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Update configuration check of imask ( list of ones and + zeros) to allow the empty list.</p> + <p> + Own Id: OTP-13101</p> + </item> + </list> + </section> + +</section> + +<section><title>SNMP 5.2</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/snmp/vsn.mk b/lib/snmp/vsn.mk index bf8e87fa0c..61a7d2207a 100644 --- a/lib/snmp/vsn.mk +++ b/lib/snmp/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = snmp -SNMP_VSN = 5.2 +SNMP_VSN = 5.2.1 PRE_VSN = APP_VSN = "$(APPLICATION)-$(SNMP_VSN)$(PRE_VSN)" diff --git a/lib/ssh/doc/src/notes.xml b/lib/ssh/doc/src/notes.xml index 010b1b15c7..8fb689fdd5 100644 --- a/lib/ssh/doc/src/notes.xml +++ b/lib/ssh/doc/src/notes.xml @@ -30,6 +30,191 @@ <file>notes.xml</file> </header> +<section><title>Ssh 4.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Better error handling in ssh_file. There was some rare + errors when a NFS-mounted file was opened by ssh_file and + then remotely deleted during reading. That caused an + endless loop. </p> + <p> + That bug is now fixed.</p> + <p> + Own Id: OTP-12699 Aux Id: OTP-11688 </p> + </item> + <item> + <p> + Fixed a bug in the compression algorithm + [email protected].</p> + <p> + Own Id: OTP-12759</p> + </item> + <item> + <p> + It is now possible to start more than one daemon with a + file descriptor given in option fd. Each daemon must of + course have a unique file descriptor.</p> + <p> + Own Id: OTP-12966 Aux Id: seq12945 </p> + </item> + <item> + <p> + Fixed a bug that caused the option <c>dh_gex_limit</c> to + be ignored.</p> + <p> + Own Id: OTP-13029</p> + </item> + <item> + <p> + A problem is fixed with the <c>ssh:connect</c> option + <c>pref_public_key_algs</c> specifying user keys.</p> + <p> + Own Id: OTP-13158</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Document updates in the ssh reference manual: app doc + file and ssh_connection.</p> + <p> + Own Id: OTP-12003</p> + </item> + <item> + <p> + The authorization phase is made stateful to prevent ssh + acting on messages sent in wrong order.</p> + <p> + Own Id: OTP-12787</p> + </item> + <item> + <p> + Testcases for bad message lengths and for bad subfield + lengths added.</p> + <p> + Own Id: OTP-12792 Aux Id: Codenomicon #5214, 6166 </p> + </item> + <item> + <p> + The 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384' and + 'ecdsa-sha2-nistp521' signature algorithms for ssh are + implemented. See RFC 5656.</p> + <p> + Own Id: OTP-12936</p> + </item> + <item> + <p> + The crypto algorithms 'aes192-ctr' and 'aes256-ctr' are + implemented. See RFC 4344.</p> + <p> + Own Id: OTP-12939</p> + </item> + <item> + <p> + The ciphers and macs AEAD_AES_128_GCM and + AEAD_AES_256_GCM are implemented but not enabled per + default. See the SSH App Reference Manual and RFC5647 for + details.</p> + <p> + The ciphers [email protected] and + [email protected] are also implemented and available + in the default configuration.</p> + <p> + Own Id: OTP-13018</p> + </item> + <item> + <p> + The ssh:daemon option dh_gex_groups is extended to read a + user provided ssh moduli file with generator-modulus + pairs. The file is in openssh format.</p> + <p> + Own Id: OTP-13052 Aux Id: OTP-13054 </p> + </item> + <item> + <p> + There is now a file (public_key/priv/moduli) which lists + size-generator-modulus triples. The purpose is to give + servers the possibility to select the crypto primes + randomly among a list of pregenerated triples. This + reduces the risk for some attacks on diffie-hellman + negotiation.</p> + <p> + See the reference manual for public_key:dh_gex_group/4 + where the handling of this is described.</p> + <p> + The ssh server (ssh:daemon) uses this.</p> + <p> + Own Id: OTP-13054 Aux Id: OTP-13052 </p> + </item> + <item> + <p> + The ssh:daemon option pwdfun now also takes a fun/4. This + enables the user to 1) check userid-password in another + way than the builtin algorithm, 2) implement rate + limiting per user or source IP or IP+Port, and 3) + implement blocking of missbehaving peers.</p> + <p> + The old fun/2 still works as previously.</p> + <p> + Own Id: OTP-13055 Aux Id: OTP-13053 </p> + </item> + <item> + <p> + There is now a new option to make the server limit the + size range of moduli available for the diffie-hellman + group exchange negotiation. See option <c> + {dh_gex_limits,{Min,Max}}</c> in ssh:daemon/3.</p> + <p> + Own Id: OTP-13066</p> + </item> + <item> + <p> + Ecdh key exchange now validates compressed and + uncompressed keys as defined in rfc5656</p> + <p> + Own Id: OTP-13067</p> + </item> + <item> + <p> + Search order for the .ssh directory are changed so + <c>$HOME</c> is tried before + <c>init:get_argument(home)</c>.</p> + <p> + Own Id: OTP-13109</p> + </item> + <item> + <p> + The sftp receive window handling is optimized so it will + not update the remote end too often. This makes "sftp + mget" considerable faster.</p> + <p> + Own Id: OTP-13130</p> + </item> + <item> + <p> + The option <c>key_cb</c> is extended to take an optional + list that is passed to the callback module as an option. + With this it is possible to have different keys depending + on which host that is connected. Another possibility is + to write a callback module that fetches keys etc from a + database.</p> + <p> + Thanks to Vipin Nair.</p> + <p> + Own Id: OTP-13156</p> + </item> + </list> + </section> + +</section> + <section><title>Ssh 4.1.3</title> <section><title>Known Bugs and Problems</title> diff --git a/lib/ssl/doc/src/notes.xml b/lib/ssl/doc/src/notes.xml index 4d4a219b4f..61d1c8355a 100644 --- a/lib/ssl/doc/src/notes.xml +++ b/lib/ssl/doc/src/notes.xml @@ -28,6 +28,94 @@ <p>This document describes the changes made to the SSL application.</p> +<section><title>SSL 7.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Honor distribution port range options</p> + <p> + Own Id: OTP-12838</p> + </item> + <item> + <p> + Correct supervisor specification in TLS distribution.</p> + <p> + Own Id: OTP-13134</p> + </item> + <item> + <p> + Correct cache timeout</p> + <p> + Own Id: OTP-13141</p> + </item> + <item> + <p> + Avoid crash and restart of ssl process when key file does + not exist.</p> + <p> + Own Id: OTP-13144</p> + </item> + <item> + <p> + Enable passing of raw socket options on the format + {raw,_,_,_} to the underlying socket.</p> + <p> + Own Id: OTP-13166</p> + </item> + <item> + <p> + Hibernation with small or a zero timeout will now work as + expected</p> + <p> + Own Id: OTP-13189</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add upper limit for session cache, configurable on ssl + application level.</p> + <p> + If upper limit is reached, invalidate the current cache + entries, e.i the session lifetime is the max time a + session will be keept, but it may be invalidated earlier + if the max limit for the table is reached. This will keep + the ssl manager process well behaved, not exhusting + memeory. Invalidating the entries will incrementally + empty the cache to make room for fresh sessions entries.</p> + <p> + Own Id: OTP-12392</p> + </item> + <item> + <p> + Use new time functions to measure passed time.</p> + <p> + Own Id: OTP-12457</p> + </item> + <item> + <p> + Improved error handling in TLS distribution</p> + <p> + Own Id: OTP-13142</p> + </item> + <item> + <p> + Distribution over TLS now honors the nodelay distribution + flag</p> + <p> + Own Id: OTP-13143</p> + </item> + </list> + </section> + +</section> + <section><title>SSL 7.1</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/ssl/test/make_certs.erl b/lib/ssl/test/make_certs.erl index 7215a59823..5eebf773a7 100644 --- a/lib/ssl/test/make_certs.erl +++ b/lib/ssl/test/make_certs.erl @@ -116,16 +116,16 @@ do_append_files([F|Fs], RF) -> do_append_files(Fs, RF). rootCA(Root, Name, C) -> - create_ca_dir(Root, Name, ca_cnf(C#config{commonName = Name})), - create_self_signed_cert(Root, Name, req_cnf(C#config{commonName = Name}), C), + create_ca_dir(Root, Name, ca_cnf(Root, C#config{commonName = Name})), + create_self_signed_cert(Root, Name, req_cnf(Root, C#config{commonName = Name}), C), file:copy(filename:join([Root, Name, "cert.pem"]), filename:join([Root, Name, "cacerts.pem"])), gencrl(Root, Name, C). intermediateCA(Root, CA, ParentCA, C) -> - create_ca_dir(Root, CA, ca_cnf(C#config{commonName = CA})), + create_ca_dir(Root, CA, ca_cnf(Root, C#config{commonName = CA})), CARoot = filename:join([Root, CA]), CnfFile = filename:join([CARoot, "req.cnf"]), - file:write_file(CnfFile, req_cnf(C#config{commonName = CA})), + file:write_file(CnfFile, req_cnf(Root, C#config{commonName = CA})), KeyFile = filename:join([CARoot, "private", "key.pem"]), ReqFile = filename:join([CARoot, "req.pem"]), create_req(Root, CnfFile, KeyFile, ReqFile, C), @@ -147,7 +147,7 @@ enduser(Root, CA, User, C) -> UsrRoot = filename:join([Root, User]), file:make_dir(UsrRoot), CnfFile = filename:join([UsrRoot, "req.cnf"]), - file:write_file(CnfFile, req_cnf(C#config{commonName = User})), + file:write_file(CnfFile, req_cnf(Root, C#config{commonName = User})), KeyFile = filename:join([UsrRoot, "key.pem"]), ReqFile = filename:join([UsrRoot, "req.pem"]), create_req(Root, CnfFile, KeyFile, ReqFile, C), @@ -337,10 +337,10 @@ eval_cmd(Port, Cmd) -> %% Contents of configuration files %% -req_cnf(C) -> +req_cnf(Root, C) -> ["# Purpose: Configuration for requests (end users and CAs)." "\n" - "ROOTDIR = $ENV::ROOTDIR\n" + "ROOTDIR = " ++ Root ++ "\n" "\n" "[req]\n" @@ -371,10 +371,10 @@ req_cnf(C) -> "subjectKeyIdentifier = hash\n" "subjectAltName = email:copy\n"]. -ca_cnf(C = #config{issuing_distribution_point = true}) -> +ca_cnf(Root, C = #config{issuing_distribution_point = true}) -> ["# Purpose: Configuration for CAs.\n" "\n" - "ROOTDIR = $ENV::ROOTDIR\n" + "ROOTDIR = " ++ Root ++ "\n" "default_ca = ca\n" "\n" @@ -450,10 +450,10 @@ ca_cnf(C = #config{issuing_distribution_point = true}) -> "crlDistributionPoints=@crl_section\n" ]; -ca_cnf(C = #config{issuing_distribution_point = false}) -> +ca_cnf(Root, C = #config{issuing_distribution_point = false}) -> ["# Purpose: Configuration for CAs.\n" "\n" - "ROOTDIR = $ENV::ROOTDIR\n" + "ROOTDIR = " ++ Root ++ "\n" "default_ca = ca\n" "\n" diff --git a/lib/ssl/test/ssl_to_openssl_SUITE.erl b/lib/ssl/test/ssl_to_openssl_SUITE.erl index 119baf1072..13523730b0 100644 --- a/lib/ssl/test/ssl_to_openssl_SUITE.erl +++ b/lib/ssl/test/ssl_to_openssl_SUITE.erl @@ -112,6 +112,7 @@ init_per_suite(Config0) -> false -> {skip, "Openssl not found"}; _ -> + ct:pal("Version: ~p", [os:cmd("openssl version")]), catch crypto:stop(), try crypto:start() of ok -> @@ -753,11 +754,9 @@ erlang_server_openssl_client_client_cert(Config) when is_list(Config) -> "-CAfile", CaCertFile, "-key", KeyFile,"-connect", "localhost:" ++ integer_to_list(Port), ssl_test_lib:version_flag(Version)], - OpenSslPort = ssl_test_lib:portable_open_port(Exe, Args), - true = port_command(OpenSslPort, Data), - + true = port_command(OpenSslPort, Data), ssl_test_lib:check_result(Server, ok), %% Clean close down! Server needs to be closed first !! @@ -841,7 +840,6 @@ erlang_client_bad_openssl_server(Config) when is_list(Config) -> Exe = "openssl", Args = ["s_server", "-accept", integer_to_list(Port), ssl_test_lib:version_flag(Version), "-cert", CertFile, "-key", KeyFile], - OpensslPort = ssl_test_lib:portable_open_port(Exe, Args), ssl_test_lib:wait_for_openssl_server(Port), @@ -1003,7 +1001,7 @@ erlang_client_alpn_openssl_server(Config) when is_list(Config) -> Data = "From openssl to erlang", start_erlang_client_and_openssl_server_with_opts(Config, [{alpn_advertised_protocols, [<<"spdy/2">>]}], - "", + [], Data, fun(Server, OpensslPort) -> true = port_command(OpensslPort, Data), ssl_test_lib:check_result(Server, ok) @@ -1016,7 +1014,7 @@ erlang_client_openssl_server_alpn(Config) when is_list(Config) -> Data = "From openssl to erlang", start_erlang_client_and_openssl_server_with_opts(Config, [], - "-alpn spdy/2", + ["-alpn", "spdy/2"], Data, fun(Server, OpensslPort) -> true = port_command(OpensslPort, Data), ssl_test_lib:check_result(Server, ok) @@ -1153,7 +1151,7 @@ erlang_server_openssl_client_npn_renegotiate(Config) when is_list(Config) -> erlang_client_openssl_server_npn_only_server(Config) when is_list(Config) -> Data = "From openssl to erlang", start_erlang_client_and_openssl_server_with_opts(Config, [], - "-nextprotoneg spdy/2", Data, fun(Server, OpensslPort) -> + ["-nextprotoneg", "spdy/2"], Data, fun(Server, OpensslPort) -> true = port_command(OpensslPort, Data), ssl_test_lib:check_result(Server, ok) end), @@ -1165,7 +1163,7 @@ erlang_client_openssl_server_npn_only_client(Config) when is_list(Config) -> Data = "From openssl to erlang", start_erlang_client_and_openssl_server_with_opts(Config, [{client_preferred_next_protocols, - {client, [<<"spdy/2">>], <<"http/1.1">>}}], "", + {client, [<<"spdy/2">>], <<"http/1.1">>}}], [], Data, fun(Server, OpensslPort) -> true = port_command(OpensslPort, Data), ssl_test_lib:check_result(Server, ok) @@ -1276,14 +1274,14 @@ erlang_server_openssl_client_sni_test(Config, SNIHostname, ExpectedSNIHostname, {from, self()}, {mfa, {?MODULE, send_and_hostname, []}}, {options, ServerOptions}]), Port = ssl_test_lib:inet_port(Server), - ClientCommand = case SNIHostname of + Exe = "openssl", + ClientArgs = case SNIHostname of undefined -> - "openssl s_client -connect " ++ Hostname ++ ":" ++ integer_to_list(Port); + ["s_client", "-connect", Hostname ++ ":" ++ integer_to_list(Port)]; _ -> - "openssl s_client -connect " ++ Hostname ++ ":" ++ integer_to_list(Port) ++ " -servername " ++ SNIHostname - end, - ct:log("Options: ~p", [[ServerOptions, ClientCommand]]), - ClientPort = open_port({spawn, ClientCommand}, [stderr_to_stdout]), + ["s_client", "-connect", Hostname ++ ":" ++ integer_to_list(Port), "-servername", SNIHostname] + end, + ClientPort = ssl_test_lib:portable_open_port(Exe, ClientArgs), %% Client check needs to be done befor server check, %% or server check might consume client messages @@ -1305,14 +1303,14 @@ erlang_server_openssl_client_sni_test_sni_fun(Config, SNIHostname, ExpectedSNIHo {from, self()}, {mfa, {?MODULE, send_and_hostname, []}}, {options, ServerOptions}]), Port = ssl_test_lib:inet_port(Server), - ClientCommand = case SNIHostname of + Exe = "openssl", + ClientArgs = case SNIHostname of undefined -> - "openssl s_client -connect " ++ Hostname ++ ":" ++ integer_to_list(Port); + ["s_client", "-connect", Hostname ++ ":" ++ integer_to_list(Port)]; _ -> - "openssl s_client -connect " ++ Hostname ++ ":" ++ integer_to_list(Port) ++ " -servername " ++ SNIHostname + ["s_client", "-connect", Hostname ++ ":" ++ integer_to_list(Port), "-servername", SNIHostname] end, - ct:log("Options: ~p", [[ServerOptions, ClientCommand]]), - ClientPort = open_port({spawn, ClientCommand}, [stderr_to_stdout]), + ClientPort = ssl_test_lib:portable_open_port(Exe, ClientArgs), %% Client check needs to be done befor server check, %% or server check might consume client messages @@ -1332,12 +1330,11 @@ cipher(CipherSuite, Version, Config, ClientOpts, ServerOpts) -> CertFile = proplists:get_value(certfile, ServerOpts), KeyFile = proplists:get_value(keyfile, ServerOpts), - Cmd = "openssl s_server -accept " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ - " -cert " ++ CertFile ++ " -key " ++ KeyFile ++ "", - - ct:log("openssl cmd: ~p~n", [Cmd]), + Exe = "openssl", + Args = ["s_server", "-accept", integer_to_list(Port), ssl_test_lib:version_flag(Version), + "-cert", CertFile, "-key", KeyFile], - OpenSslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + OpenSslPort = ssl_test_lib:portable_open_port(Exe, Args), ssl_test_lib:wait_for_openssl_server(Port), @@ -1395,13 +1392,19 @@ start_erlang_client_and_openssl_server_with_opts(Config, ErlangClientOpts, Opens KeyFile = proplists:get_value(keyfile, ServerOpts), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), - Cmd = "openssl s_server " ++ OpensslServerOpts ++ " -accept " ++ - integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ - " -cert " ++ CertFile ++ " -key " ++ KeyFile, - - ct:log("openssl cmd: ~p~n", [Cmd]), - - OpensslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + Exe = "openssl", + Args = case OpensslServerOpts of + [] -> + ["s_server", "-accept", + integer_to_list(Port), ssl_test_lib:version_flag(Version), + "-cert", CertFile,"-key", KeyFile]; + [Opt, Value] -> + ["s_server", Opt, Value, "-accept", + integer_to_list(Port), ssl_test_lib:version_flag(Version), + "-cert", CertFile,"-key", KeyFile] + end, + + OpensslPort = ssl_test_lib:portable_open_port(Exe, Args), ssl_test_lib:wait_for_openssl_server(Port), @@ -1435,13 +1438,10 @@ start_erlang_client_and_openssl_server_for_alpn_negotiation(Config, Data, Callba KeyFile = proplists:get_value(keyfile, ServerOpts), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), - Cmd = "openssl s_server -msg -alpn http/1.1,spdy/2 -accept " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ - " -cert " ++ CertFile ++ " -key " ++ KeyFile, - - ct:log("openssl cmd: ~p~n", [Cmd]), - - OpensslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), - + Exe = "openssl", + Args = ["s_server", "-msg", "-alpn", "http/1.1,spdy/2", "-accept", integer_to_list(Port), ssl_test_lib:version_flag(Version), + "-cert", CertFile, "-key" ++ KeyFile], + OpensslPort = ssl_test_lib:portable_open_port(Exe, Args), ssl_test_lib:wait_for_openssl_server(Port), Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, @@ -1473,12 +1473,13 @@ start_erlang_server_and_openssl_client_for_alpn_negotiation(Config, Data, Callba {options, ServerOpts}]), Port = ssl_test_lib:inet_port(Server), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), - Cmd = "openssl s_client -alpn http/1.0,spdy/2 -msg -port " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ - " -host localhost", - ct:log("openssl cmd: ~p~n", [Cmd]), + Exe = "openssl", + Args = ["s_client", "-alpn", "http/1.0,spdy/2" "-msg" "-port", + integer_to_list(Port), ssl_test_lib:version_flag(Version), + "-host", "localhost"], - OpenSslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + OpenSslPort = ssl_test_lib:portable_open_port(Exe, Args), Callback(Server, OpenSslPort), @@ -1503,12 +1504,12 @@ start_erlang_client_and_openssl_server_for_alpn_npn_negotiation(Config, Data, Ca KeyFile = proplists:get_value(keyfile, ServerOpts), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), - Cmd = "openssl s_server -msg -alpn http/1.1,spdy/2 -nextprotoneg spdy/3 -accept " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ - " -cert " ++ CertFile ++ " -key " ++ KeyFile, - - ct:log("openssl cmd: ~p~n", [Cmd]), + Exe = "openssl", + Args = ["s_server", "-msg", "-alpn", "http/1.1,spdy/2", "-nextprotoneg", + "spdy/3", "-accept", integer_to_list(Port), ssl_test_lib:version_flag(Version), + "-cert" ++ CertFile ++ "-key" ++ KeyFile], - OpensslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + OpensslPort = ssl_test_lib:portable_open_port(Exe, Args), ssl_test_lib:wait_for_openssl_server(Port), @@ -1542,17 +1543,15 @@ start_erlang_server_and_openssl_client_for_alpn_npn_negotiation(Config, Data, Ca {options, ServerOpts}]), Port = ssl_test_lib:inet_port(Server), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), - Cmd = "openssl s_client -alpn http/1.1,spdy/2 -nextprotoneg spdy/3 -msg -port " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ - " -host localhost", - - ct:log("openssl cmd: ~p~n", [Cmd]), - - OpenSslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + Exe = "openssl", + Args = ["s_client", "-alpn", "http/1.1,spdy/2", "-nextprotoneg", "spdy/3", + "-msg", "-port", integer_to_list(Port), ssl_test_lib:version_flag(Version), + "-host", "localhost"], + OpenSslPort = ssl_test_lib:portable_open_port(Exe, Args), Callback(Server, OpenSslPort), ssl_test_lib:close(Server), - ssl_test_lib:close_port(OpenSslPort), process_flag(trap_exit, false). @@ -1570,13 +1569,12 @@ start_erlang_client_and_openssl_server_for_npn_negotiation(Config, Data, Callbac CertFile = proplists:get_value(certfile, ServerOpts), KeyFile = proplists:get_value(keyfile, ServerOpts), Version = tls_record:protocol_version(tls_record:highest_protocol_version([])), - - Cmd = "openssl s_server -msg -nextprotoneg http/1.1,spdy/2 -accept " ++ integer_to_list(Port) ++ ssl_test_lib:version_flag(Version) ++ - " -cert " ++ CertFile ++ " -key " ++ KeyFile, - - ct:log("openssl cmd: ~p~n", [Cmd]), - - OpensslPort = open_port({spawn, Cmd}, [stderr_to_stdout]), + + Exe = "openssl", + Args = ["s_server", "-msg", "-nextprotoneg", "http/1.1,spdy/2", "-accept", integer_to_list(Port), + ssl_test_lib:version_flag(Version), + "-cert", CertFile, "-key", KeyFile], + OpensslPort = ssl_test_lib:portable_open_port(Exe, Args), ssl_test_lib:wait_for_openssl_server(Port), @@ -1759,7 +1757,9 @@ check_sane_openssl_renegotaite(Config) -> end. check_sane_openssl_sslv2(Config) -> - Port = open_port({spawn, "openssl s_client -ssl2 "}, [stderr_to_stdout]), + Exe = "openssl", + Args = ["s_client", "-ssl2"], + Port = ssl_test_lib:portable_open_port(Exe, Args), case supports_sslv2(Port) of true -> Config; diff --git a/lib/stdlib/doc/src/notes.xml b/lib/stdlib/doc/src/notes.xml index c84ca9c8ad..267a993a1b 100644 --- a/lib/stdlib/doc/src/notes.xml +++ b/lib/stdlib/doc/src/notes.xml @@ -31,6 +31,67 @@ </header> <p>This document describes the changes made to the STDLIB application.</p> +<section><title>STDLIB 2.7</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>The Erlang Pretty Printer uses <c>::</c> for function + type constraints.</p> <p>A bug concerning pretty printing + of annotated type union elements in map pair types has + been fixed.</p> <p>Some minor issues regarding the + documentation of types and specs have been corrected.</p> + <p> + Own Id: OTP-13084</p> + </item> + <item> + <p> The shell command <c>rp</c> prints strings as lists + of integers if pretty printing of lists is set to + <c>false</c>. </p> + <p> + Own Id: OTP-13145</p> + </item> + <item> + <p> + The shell would crash if a bit syntax expression with + conflicting types were given (e.g. if a field type was + given as '<c>integer-binary</c>'). (Thanks to Aleksei + Magusev for reporting this bug.)</p> + <p> + Own Id: OTP-13157</p> + </item> + <item> + <p>The <c>rand:export_seed/0</c> would never return + '<c>undefined</c>' even if no seed has previously been + created. Fixed to return '<c>undefined</c>' if there is + no seed in the process dictionary.</p> + <p> + Own Id: OTP-13162</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add support for the Delete, Home and End keys in the + Erlang shell.</p> + <p> + Own Id: OTP-13032</p> + </item> + <item> + <p><c>beam_lib:all_chunks/1</c> and + <c>beam_lib:build_module/1</c> have been documented.</p> + <p> + Own Id: OTP-13063</p> + </item> + </list> + </section> + +</section> + <section><title>STDLIB 2.6</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/stdlib/src/erl_lint.erl b/lib/stdlib/src/erl_lint.erl index 4a4019b8bd..3ce6abe752 100644 --- a/lib/stdlib/src/erl_lint.erl +++ b/lib/stdlib/src/erl_lint.erl @@ -373,9 +373,9 @@ format_error({spec_fun_undefined, {F, A}}) -> format_error({missing_spec, {F,A}}) -> io_lib:format("missing specification for function ~w/~w", [F, A]); format_error(spec_wrong_arity) -> - "spec has the wrong arity"; + "spec has wrong arity"; format_error(callback_wrong_arity) -> - "callback has the wrong arity"; + "callback has wrong arity"; format_error({deprecated_builtin_type, {Name, Arity}, Replacement, Rel}) -> UseS = case Replacement of @@ -2878,7 +2878,7 @@ spec_decl(Line, MFA0, TypeSpecs, St0 = #lint{specs = Specs, module = Mod}) -> St1 = St0#lint{specs = dict:store(MFA, Line, Specs)}, case dict:is_key(MFA, Specs) of true -> add_error(Line, {redefine_spec, MFA0}, St1); - false -> check_specs(TypeSpecs, Arity, St1) + false -> check_specs(TypeSpecs, spec_wrong_arity, Arity, St1) end. %% callback_decl(Line, Fun, Types, State) -> State. @@ -2892,7 +2892,8 @@ callback_decl(Line, MFA0, TypeSpecs, St1 = St0#lint{callbacks = dict:store(MFA, Line, Callbacks)}, case dict:is_key(MFA, Callbacks) of true -> add_error(Line, {redefine_callback, MFA0}, St1); - false -> check_specs(TypeSpecs, Arity, St1) + false -> check_specs(TypeSpecs, callback_wrong_arity, + Arity, St1) end end. @@ -2929,7 +2930,7 @@ is_fa({FuncName, Arity}) when is_atom(FuncName), is_integer(Arity), Arity >= 0 -> true; is_fa(_) -> false. -check_specs([FunType|Left], Arity, St0) -> +check_specs([FunType|Left], ETag, Arity, St0) -> {FunType1, CTypes} = case FunType of {type, _, bounded_fun, [FT = {type, _, 'fun', _}, Cs]} -> @@ -2937,18 +2938,16 @@ check_specs([FunType|Left], Arity, St0) -> {FT, lists:append(Types0)}; {type, _, 'fun', _} = FT -> {FT, []} end, - SpecArity = - case FunType1 of - {type, L, 'fun', [any, _]} -> any; - {type, L, 'fun', [{type, _, product, D}, _]} -> length(D) - end, + {type, L, 'fun', [{type, _, product, D}, _]} = FunType1, + SpecArity = length(D), St1 = case Arity =:= SpecArity of true -> St0; - false -> add_error(L, spec_wrong_arity, St0) + false -> %% Cannot happen if called from the compiler. + add_error(L, ETag, St0) end, St2 = check_type({type, nowarn(), product, [FunType1|CTypes]}, St1), - check_specs(Left, Arity, St2); -check_specs([], _Arity, St) -> + check_specs(Left, ETag, Arity, St2); +check_specs([], _ETag, _Arity, St) -> St. nowarn() -> diff --git a/lib/stdlib/src/erl_parse.yrl b/lib/stdlib/src/erl_parse.yrl index ae42a8f0b1..e07ab2efc2 100644 --- a/lib/stdlib/src/erl_parse.yrl +++ b/lib/stdlib/src/erl_parse.yrl @@ -85,10 +85,6 @@ type_spec -> '(' spec_fun type_sigs ')' : {'$2', '$3'}. spec_fun -> atom : '$1'. spec_fun -> atom ':' atom : {'$1', '$3'}. -%% The following two are retained only for backwards compatibility; -%% they are not part of the EEP syntax and should be removed. -spec_fun -> atom '/' integer '::' : {'$1', '$3'}. -spec_fun -> atom ':' atom '/' integer '::' : {'$1', '$3', '$5'}. typed_attr_val -> expr ',' typed_record_fields : {typed_record, '$1', '$3'}. typed_attr_val -> expr '::' top_type : {type_def, '$1', '$3'}. @@ -634,14 +630,8 @@ build_type_spec({Kind,Aa}, {SpecFun, TypeSpecs}) {atom, _, Fun} -> {Fun, find_arity_from_specs(TypeSpecs)}; {{atom,_, Mod}, {atom,_, Fun}} -> - {Mod,Fun,find_arity_from_specs(TypeSpecs)}; - {{atom, _, Fun}, {integer, _, Arity}} -> - %% Old style spec. Allow this for now. - {Fun,Arity}; - {{atom,_, Mod}, {atom, _, Fun}, {integer, _, Arity}} -> - %% Old style spec. Allow this for now. - {Mod,Fun,Arity} - end, + {Mod,Fun,find_arity_from_specs(TypeSpecs)} + end, {attribute,Aa,Kind,{NewSpecFun, TypeSpecs}}. find_arity_from_specs([Spec|_]) -> diff --git a/lib/stdlib/src/stdlib.appup.src b/lib/stdlib/src/stdlib.appup.src index 5f61752655..04cdf31ada 100644 --- a/lib/stdlib/src/stdlib.appup.src +++ b/lib/stdlib/src/stdlib.appup.src @@ -18,9 +18,9 @@ %% %CopyrightEnd% {"%VSN%", %% Up from - max one major revision back - [{<<"2\\.5(\\.[0-9]+)*">>,[restart_new_emulator]}, %% OTP-18.0.* - {<<"2\\.[0-4](\\.[0-9]+)*">>,[restart_new_emulator]}], %% 17.0-17.5 + [{<<"2\\.[5-7](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.* + {<<"2\\.[0-4](\\.[0-9]+)*">>,[restart_new_emulator]}], % 17.0-17.5 %% Down to - max one major revision back - [{<<"2\\.5(\\.[0-9]+)*">>,[restart_new_emulator]}, %% OTP-18.0.* - {<<"2\\.[0-4](\\.[0-9]+)*">>,[restart_new_emulator]}] %% 17.0-17.5 + [{<<"2\\.[5-7](\\.[0-9]+)*">>,[restart_new_emulator]}, % OTP-18.* + {<<"2\\.[0-4](\\.[0-9]+)*">>,[restart_new_emulator]}] % 17.0-17.5 }. diff --git a/lib/stdlib/test/erl_lint_SUITE.erl b/lib/stdlib/test/erl_lint_SUITE.erl index 1d35cc71a5..3c746a13d7 100644 --- a/lib/stdlib/test/erl_lint_SUITE.erl +++ b/lib/stdlib/test/erl_lint_SUITE.erl @@ -65,7 +65,7 @@ too_many_arguments/1, basic_errors/1,bin_syntax_errors/1, predef/1, - maps/1,maps_type/1,otp_11851/1 + maps/1,maps_type/1,otp_11851/1,otp_11879/1 ]). % Default timetrap timeout (set in init_per_testcase). @@ -94,7 +94,7 @@ all() -> bif_clash, behaviour_basic, behaviour_multiple, otp_11861, otp_7550, otp_8051, format_warn, {group, on_load}, too_many_arguments, basic_errors, bin_syntax_errors, predef, - maps, maps_type, otp_11851]. + maps, maps_type, otp_11851, otp_11879]. groups() -> [{unused_vars_warn, [], @@ -3849,6 +3849,29 @@ otp_11851(Config) when is_list(Config) -> [] = run(Config, Ts), ok. +otp_11879(doc) -> + "OTP-11879: The -spec f/a :: (As) -> B; syntax removed, " + "and is_subtype/2 deprecated"; +otp_11879(_Config) -> + Fs = [{attribute,0,file,{"file.erl",0}}, + {attribute,0,module,m}, + {attribute,1,spec, + {{f,1}, + [{type,2,'fun',[{type,3,product,[{var,4,'V1'}, + {var,5,'V1'}]}, + {type,6,integer,[]}]}]}}, + {attribute,20,callback, + {{cb,21}, + [{type,22,'fun',[{type,23,product,[{var,24,'V1'}, + {var,25,'V1'}]}, + {type,6,integer,[]}]}]}}], + {error,[{"file.erl", + [{1,erl_lint,{spec_fun_undefined,{f,1}}}, + {2,erl_lint,spec_wrong_arity}, + {22,erl_lint,callback_wrong_arity}]}], + []} = compile:forms(Fs, [return,report]), + ok. + run(Config, Tests) -> F = fun({N,P,Ws,E}, BadL) -> case catch run_test(Config, P, Ws) of diff --git a/lib/stdlib/test/erl_pp_SUITE.erl b/lib/stdlib/test/erl_pp_SUITE.erl index 8cdd2ceca9..8a128b3815 100644 --- a/lib/stdlib/test/erl_pp_SUITE.erl +++ b/lib/stdlib/test/erl_pp_SUITE.erl @@ -876,6 +876,9 @@ type_examples() -> {ex30,<<"-type t99() ::" "{t2(),'\\'t::4'(),t5(),t6(),t7(),t8(),t10(),t14()," "t15(),t20(),t21(), t22(),t25()}. ">>}, + %% Writing constraints as is_subtype(V, T) is not supported since + %% Erlang/OTP 19.0, but as long as the parser recognizes the + %% is_subtype(V, T) syntax, we need a few examples of the syntax. {ex31,<<"-spec t1(FooBar :: t99()) -> t99();" "(t2()) -> t2();" "('\\'t::4'()) -> '\\'t::4'() when is_subtype('\\'t::4'(), t24);" @@ -1000,18 +1003,10 @@ otp_8567(Config) when is_list(Config) -> "t() ->\n" " 3.\n" "\n" - "-spec(t1/1 :: (ot()) -> ot1()).\n" - "t1(A) ->\n" - " A.\n" - "\n" "-spec(t2 (ot()) -> ot1()).\n" "t2(A) ->\n" " A.\n" "\n" - "-spec(otp_8567:t3/1 :: (ot()) -> ot1()).\n" - "t3(A) ->\n" - " A.\n" - "\n" "-spec(otp_8567:t4 (ot()) -> ot1()).\n" "t4(A) ->\n" " A.\n">>, diff --git a/lib/stdlib/vsn.mk b/lib/stdlib/vsn.mk index 3387d74e57..39b44c9104 100644 --- a/lib/stdlib/vsn.mk +++ b/lib/stdlib/vsn.mk @@ -1 +1 @@ -STDLIB_VSN = 2.6 +STDLIB_VSN = 2.7 diff --git a/lib/test_server/doc/src/notes.xml b/lib/test_server/doc/src/notes.xml index da956de9ef..b48bda94d0 100644 --- a/lib/test_server/doc/src/notes.xml +++ b/lib/test_server/doc/src/notes.xml @@ -33,6 +33,35 @@ <file>notes.xml</file> </header> +<section><title>Test_Server 3.9.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + When generating Makefile from Makefile.src, + ts_lib:get_arg/4 earlier removed all spaces in the + extracted argument. The code was probably meant for + removing leading and trailing spaces only, and is now + corrected to do so.</p> + <p> + Own Id: OTP-13015</p> + </item> + <item> + <p> + With the Common Test 'create_priv_dir' start option set + to 'auto_per_tc', the name of the priv directory for a + configuration function could clash with the name of the + priv directory for a test case, which would cause Test + Server failure. This error has been corrected.</p> + <p> + Own Id: OTP-13181</p> + </item> + </list> + </section> + +</section> + <section><title>Test_Server 3.9</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/test_server/vsn.mk b/lib/test_server/vsn.mk index fd9e4e6d74..3a3815c557 100644 --- a/lib/test_server/vsn.mk +++ b/lib/test_server/vsn.mk @@ -1 +1 @@ -TEST_SERVER_VSN = 3.9 +TEST_SERVER_VSN = 3.9.1 diff --git a/lib/tools/doc/src/notes.xml b/lib/tools/doc/src/notes.xml index bf27d2a3e5..985207a39b 100644 --- a/lib/tools/doc/src/notes.xml +++ b/lib/tools/doc/src/notes.xml @@ -31,6 +31,22 @@ </header> <p>This document describes the changes made to the Tools application.</p> +<section><title>Tools 2.8.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The emacs mode does not add a newline after the arrow on + -callback lines anymore.</p> + <p> + Own Id: OTP-13042</p> + </item> + </list> + </section> + +</section> + <section><title>Tools 2.8.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/tools/vsn.mk b/lib/tools/vsn.mk index e4eda213ba..3efe89d9f9 100644 --- a/lib/tools/vsn.mk +++ b/lib/tools/vsn.mk @@ -1 +1 @@ -TOOLS_VSN = 2.8.1 +TOOLS_VSN = 2.8.2 diff --git a/lib/typer/doc/src/notes.xml b/lib/typer/doc/src/notes.xml index 044873ec94..21a2a6d597 100644 --- a/lib/typer/doc/src/notes.xml +++ b/lib/typer/doc/src/notes.xml @@ -31,6 +31,21 @@ </header> <p>This document describes the changes made to TypEr.</p> +<section><title>TypEr 0.9.10</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>Fix a bug that could result in a crash when printing + warnings onto standard error. </p> + <p> + Own Id: OTP-13010</p> + </item> + </list> + </section> + +</section> + <section><title>TypEr 0.9.9</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/typer/vsn.mk b/lib/typer/vsn.mk index 74c0ccfc59..507593ef56 100644 --- a/lib/typer/vsn.mk +++ b/lib/typer/vsn.mk @@ -1 +1 @@ -TYPER_VSN = 0.9.9 +TYPER_VSN = 0.9.10 diff --git a/lib/wx/doc/src/notes.xml b/lib/wx/doc/src/notes.xml index 6a0dd898e3..0bbeeaafab 100644 --- a/lib/wx/doc/src/notes.xml +++ b/lib/wx/doc/src/notes.xml @@ -32,6 +32,24 @@ <p>This document describes the changes made to the wxErlang application.</p> +<section><title>Wx 1.6</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p>Add wxOverlay and make wxPostScripDC optional to make + it easier to build on windows.</p> <p>Correct some + function specifications.</p> <p>The driver implementation + have been optimized and now invokes commands after events + have been sent to erlang.</p> + <p> + Own Id: OTP-13160</p> + </item> + </list> + </section> + +</section> + <section><title>Wx 1.5</title> <section><title>Improvements and New Features</title> diff --git a/lib/wx/vsn.mk b/lib/wx/vsn.mk index 7608bb3014..a1bacb5d66 100644 --- a/lib/wx/vsn.mk +++ b/lib/wx/vsn.mk @@ -1 +1 @@ -WX_VSN = 1.5 +WX_VSN = 1.6 diff --git a/lib/xmerl/doc/src/notes.xml b/lib/xmerl/doc/src/notes.xml index a1558b224f..19274e95ae 100644 --- a/lib/xmerl/doc/src/notes.xml +++ b/lib/xmerl/doc/src/notes.xml @@ -32,6 +32,22 @@ <p>This document describes the changes made to the Xmerl application.</p> +<section><title>Xmerl 1.3.9</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Removed the built-in definitions of xml.xsd from the + xmerl_xsd module.</p> + <p> + Own Id: OTP-13070</p> + </item> + </list> + </section> + +</section> + <section><title>Xmerl 1.3.8</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/xmerl/vsn.mk b/lib/xmerl/vsn.mk index 1ed230316f..0d6082e023 100644 --- a/lib/xmerl/vsn.mk +++ b/lib/xmerl/vsn.mk @@ -1 +1 @@ -XMERL_VSN = 1.3.8 +XMERL_VSN = 1.3.9 diff --git a/otp_versions.table b/otp_versions.table index 587748895f..b0f8efeb4c 100644 --- a/otp_versions.table +++ b/otp_versions.table @@ -1,3 +1,4 @@ +OTP-18.2 : asn1-4.0.1 common_test-1.11.1 compiler-6.0.2 crypto-3.6.2 dialyzer-2.8.2 diameter-1.11.1 erl_docgen-0.4.1 erl_interface-3.8.1 erts-7.2 eunit-2.2.12 hipe-3.14 inets-6.1 jinterface-1.6.1 kernel-4.1.1 observer-2.1.1 parsetools-2.1.1 public_key-1.1 runtime_tools-1.9.2 sasl-2.6.1 snmp-5.2.1 ssh-4.2 ssl-7.2 stdlib-2.7 test_server-3.9.1 tools-2.8.2 typer-0.9.10 wx-1.6 xmerl-1.3.9 # cosEvent-2.2 cosEventDomain-1.2 cosFileTransfer-1.2 cosNotification-1.2 cosProperty-1.2 cosTime-1.2 cosTransactions-1.3 debugger-4.1.1 edoc-0.7.17 eldap-1.2 et-1.5.1 gs-1.6 ic-4.4 megaco-3.18 mnesia-4.13.2 odbc-2.11.1 orber-3.8 os_mon-2.4 ose-1.1 otp_mibs-1.1 percept-0.8.11 reltool-0.7 syntax_tools-1.7 webtool-0.9 : OTP-18.1.5 : ssh-4.1.3 # asn1-4.0 common_test-1.11 compiler-6.0.1 cosEvent-2.2 cosEventDomain-1.2 cosFileTransfer-1.2 cosNotification-1.2 cosProperty-1.2 cosTime-1.2 cosTransactions-1.3 crypto-3.6.1 debugger-4.1.1 dialyzer-2.8.1 diameter-1.11 edoc-0.7.17 eldap-1.2 erl_docgen-0.4 erl_interface-3.8 erts-7.1 et-1.5.1 eunit-2.2.11 gs-1.6 hipe-3.13 ic-4.4 inets-6.0.3 jinterface-1.6 kernel-4.1 megaco-3.18 mnesia-4.13.2 observer-2.1 odbc-2.11.1 orber-3.8 os_mon-2.4 ose-1.1 otp_mibs-1.1 parsetools-2.1 percept-0.8.11 public_key-1.0.1 reltool-0.7 runtime_tools-1.9.1 sasl-2.6 snmp-5.2 ssl-7.1 stdlib-2.6 syntax_tools-1.7 test_server-3.9 tools-2.8.1 typer-0.9.9 webtool-0.9 wx-1.5 xmerl-1.3.8 : OTP-18.1.4 : inets-6.0.3 # asn1-4.0 common_test-1.11 compiler-6.0.1 cosEvent-2.2 cosEventDomain-1.2 cosFileTransfer-1.2 cosNotification-1.2 cosProperty-1.2 cosTime-1.2 cosTransactions-1.3 crypto-3.6.1 debugger-4.1.1 dialyzer-2.8.1 diameter-1.11 edoc-0.7.17 eldap-1.2 erl_docgen-0.4 erl_interface-3.8 erts-7.1 et-1.5.1 eunit-2.2.11 gs-1.6 hipe-3.13 ic-4.4 jinterface-1.6 kernel-4.1 megaco-3.18 mnesia-4.13.2 observer-2.1 odbc-2.11.1 orber-3.8 os_mon-2.4 ose-1.1 otp_mibs-1.1 parsetools-2.1 percept-0.8.11 public_key-1.0.1 reltool-0.7 runtime_tools-1.9.1 sasl-2.6 snmp-5.2 ssh-4.1.2 ssl-7.1 stdlib-2.6 syntax_tools-1.7 test_server-3.9 tools-2.8.1 typer-0.9.9 webtool-0.9 wx-1.5 xmerl-1.3.8 : OTP-18.1.3 : ssh-4.1.2 # asn1-4.0 common_test-1.11 compiler-6.0.1 cosEvent-2.2 cosEventDomain-1.2 cosFileTransfer-1.2 cosNotification-1.2 cosProperty-1.2 cosTime-1.2 cosTransactions-1.3 crypto-3.6.1 debugger-4.1.1 dialyzer-2.8.1 diameter-1.11 edoc-0.7.17 eldap-1.2 erl_docgen-0.4 erl_interface-3.8 erts-7.1 et-1.5.1 eunit-2.2.11 gs-1.6 hipe-3.13 ic-4.4 inets-6.0.2 jinterface-1.6 kernel-4.1 megaco-3.18 mnesia-4.13.2 observer-2.1 odbc-2.11.1 orber-3.8 os_mon-2.4 ose-1.1 otp_mibs-1.1 parsetools-2.1 percept-0.8.11 public_key-1.0.1 reltool-0.7 runtime_tools-1.9.1 sasl-2.6 snmp-5.2 ssl-7.1 stdlib-2.6 syntax_tools-1.7 test_server-3.9 tools-2.8.1 typer-0.9.9 webtool-0.9 wx-1.5 xmerl-1.3.8 : diff --git a/system/doc/reference_manual/typespec.xml b/system/doc/reference_manual/typespec.xml index 22627058c1..a238e3135a 100644 --- a/system/doc/reference_manual/typespec.xml +++ b/system/doc/reference_manual/typespec.xml @@ -495,7 +495,8 @@ </p> <pre> -spec id(X) -> X when X :: tuple().</pre> <p> - Currently, the <c>::</c> constraint (read as <c>is_subtype</c>) is + Currently, the <c>::</c> constraint + (read as «is a subtype of») is the only guard constraint that can be used in the <c>'when'</c> part of a <c>'-spec'</c> attribute. </p> @@ -529,16 +530,6 @@ <pre> -spec foo({X, integer()}) -> X when X :: atom() ; ([Y]) -> Y when Y :: number().</pre> - <note> - <p> - For backwards compatibility the following form is also allowed: - </p> - <pre> -spec id(X) -> X when is_subtype(X, tuple()).</pre> - <p> - but its use is discouraged. It will be removed in a future - Erlang/OTP release. - </p> - </note> <p> Some functions in Erlang are not meant to return; either because they define servers or because they are used to |