| Age | Commit message (Collapse) | Author |
|---|
|
* fdm/file-allocate/OTP-10680:
Update preloaded prim_file.beam
erts: Fix xcomp configure for fallocate
Add file:allocate/3 operation
|
|
This operation allows pre-allocation of space for files.
It succeeds only on systems that support such operation.
The POSIX standard defines the optional system call
posix_fallocate() to implement this feature. However,
some systems implement more specific functions to
accomplish the same operation.
On Linux, if the more specific function fallocate() is
implemented, it is used instead of posix_fallocate(),
falling back to posix_fallocate() if the fallocate()
call failed (it's only supported for the ext4, ocfs2,
xfs and btrfs file systems at the moment).
On Mac OS X it uses the specific fcntl() operation
F_PREALLOCATE, falling back to posix_fallocate() if
it's available (at the moment Mac OS X doesn't provide
posix_fallocate()).
On any other UNIX system, it uses posix_fallocate() if it's
available. Any other system not providing this system call
or any function to pre-allocate space for files, this operation
always fails with the ENOTSUP POSIX error.
|
|
When using the async thread pool, if an erlang process
asks to open a file and it gets shutdown/killed while
the file:open/2 call hasn't returned, it's possible to
leak a file descriptor against the target file.
This happens because when the file driver is stopped
(file_stop() function is called), an async thread is
executing, about to execute, or executed already the
invoke_open() function. After file_stop() is called,
the file_async_ready() function will not run, and this
function is responsible for setting desc->fd with the
file descriptor that invoke_open() got. The file_stop()
call closes desc->fd if it refers to a valid file
descriptor, which is not the case here, because this
function was called before file_async_ready() could
run.
This leak is easily reproducile in a GNU/Linux system
using the following test code:
-module(t).
-export([t/1]).
t(N) ->
Pid = spawn_link(fun() ->
process_flag(trap_exit, true),
loop(N)
end),
Ref = erlang:monitor(process, Pid),
receive {'DOWN', Ref, _, _, _} ->
ok
end.
loop(0) ->
ok;
loop(N) ->
Name = integer_to_list(N),
Server = self(),
Pid = spawn(fun() ->
Server ! continue,
{ok, FdW} = file:open(Name, [raw, write]),
{ok, FdR} = file:open(Name, [raw, read]),
% Optional close calls, with or without them
% it makes no difference.
%ok = file:close(FdW),
%ok = file:close(FdR),
ok
end),
receive continue -> ok end,
exit(Pid, shutdown),
loop(N - 1).
Running this code with a few iterations is enough to very often
notice, with the lsof command, that the beam.smp process is holding
forever file descriptors open. This issue doesn't happen if the
async thread pool is not used.
Example:
$ erl +A 4
Erlang R15B03 (erts-5.9.3) [source] [64-bit] [smp:4:4] [async-threads:4] [hipe] [kernel-poll:false]
Eshell V5.9.3 (abort with ^G)
1> c(t).
{ok,t}
2> os:getpid().
"31975"
3> t:t(20).
ok
In a separate shell:
$ lsof -p 31975
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
beam.smp 31975 fdmanana cwd DIR 8,18 22736896 32563204 /home/fdmanana/git/hub/otp/tmp
beam.smp 31975 fdmanana rtd DIR 8,1 4096 2 /
beam.smp 31975 fdmanana txt REG 8,1 7600263 1835126 /opt/r15b03/lib/erlang/erts-5.9.3/bin/beam.smp
beam.smp 31975 fdmanana mem REG 8,1 7220736 2497283 /usr/lib/locale/locale-archive
beam.smp 31975 fdmanana mem REG 8,1 10280 2505021 /usr/lib/libsctp.so.1.0.11
beam.smp 31975 fdmanana mem REG 8,1 1811128 917795 /lib/x86_64-linux-gnu/libc-2.15.so
beam.smp 31975 fdmanana mem REG 8,1 31752 917803 /lib/x86_64-linux-gnu/librt-2.15.so
beam.smp 31975 fdmanana mem REG 8,1 135366 917799 /lib/x86_64-linux-gnu/libpthread-2.15.so
beam.smp 31975 fdmanana mem REG 8,1 159200 921249 /lib/x86_64-linux-gnu/libtinfo.so.5.9
beam.smp 31975 fdmanana mem REG 8,1 1030512 917962 /lib/x86_64-linux-gnu/libm-2.15.so
beam.smp 31975 fdmanana mem REG 8,1 14768 917702 /lib/x86_64-linux-gnu/libdl-2.15.so
beam.smp 31975 fdmanana mem REG 8,1 149280 917974 /lib/x86_64-linux-gnu/ld-2.15.so
beam.smp 31975 fdmanana 0u CHR 136,1 4 /dev/pts/1
beam.smp 31975 fdmanana 1u CHR 136,1 4 /dev/pts/1
beam.smp 31975 fdmanana 2u CHR 136,1 4 /dev/pts/1
beam.smp 31975 fdmanana 3r FIFO 0,8 1298297 pipe
beam.smp 31975 fdmanana 4w FIFO 0,8 1298297 pipe
beam.smp 31975 fdmanana 5r FIFO 0,8 1298298 pipe
beam.smp 31975 fdmanana 6w FIFO 0,8 1298298 pipe
beam.smp 31975 fdmanana 7w REG 8,18 0 32564173 /home/fdmanana/git/hub/otp/tmp/20
beam.smp 31975 fdmanana 8w REG 8,18 0 32564176 /home/fdmanana/git/hub/otp/tmp/16
beam.smp 31975 fdmanana 9w REG 8,18 0 32564177 /home/fdmanana/git/hub/otp/tmp/15
beam.smp 31975 fdmanana 10w REG 8,18 0 32564179 /home/fdmanana/git/hub/otp/tmp/12
beam.smp 31975 fdmanana 11w REG 8,18 0 32564180 /home/fdmanana/git/hub/otp/tmp/11
beam.smp 31975 fdmanana 12w REG 8,18 0 32564205 /home/fdmanana/git/hub/otp/tmp/10
beam.smp 31975 fdmanana 13w REG 8,18 0 32564182 /home/fdmanana/git/hub/otp/tmp/8
beam.smp 31975 fdmanana 14w REG 8,18 0 32564183 /home/fdmanana/git/hub/otp/tmp/7
beam.smp 31975 fdmanana 15w REG 8,18 0 32564186 /home/fdmanana/git/hub/otp/tmp/3
|
|
rickard/r16/port-optimizations/OTP-10336
* rickard/port-optimizations/OTP-10336:
Change annotate level for emacs-22 in cerl
Update etp-commands
Add documentation on communication in Erlang
Add support for busy port message queue
Add driver callback epilogue
Implement true asynchronous signaling between processes and ports
Add erl_drv_[send|output]_term
Move busy port flag
Use rwlock for driver list
Optimize management of port tasks
Improve configuration of process and port tables
Remove R9 compatibility features
Use ptab functionality also for ports
Prepare for use of ptab functionality also for ports
Atomic port state
Generalize process table implementation
Implement functionality for delaying thread progress from unmanaged threads
Conflicts:
erts/doc/src/erl_driver.xml
erts/doc/src/erlang.xml
erts/emulator/beam/beam_bif_load.c
erts/emulator/beam/beam_bp.c
erts/emulator/beam/beam_emu.c
erts/emulator/beam/bif.c
erts/emulator/beam/copy.c
erts/emulator/beam/erl_alloc.c
erts/emulator/beam/erl_alloc.types
erts/emulator/beam/erl_bif_info.c
erts/emulator/beam/erl_bif_port.c
erts/emulator/beam/erl_bif_trace.c
erts/emulator/beam/erl_init.c
erts/emulator/beam/erl_message.c
erts/emulator/beam/erl_port_task.c
erts/emulator/beam/erl_process.c
erts/emulator/beam/erl_process.h
erts/emulator/beam/erl_process_lock.c
erts/emulator/beam/erl_trace.c
erts/emulator/beam/export.h
erts/emulator/beam/global.h
erts/emulator/beam/io.c
erts/emulator/sys/unix/sys.c
erts/emulator/sys/vxworks/sys.c
erts/emulator/test/port_SUITE.erl
erts/etc/unix/cerl.src
erts/preloaded/ebin/erlang.beam
erts/preloaded/ebin/prim_inet.beam
erts/preloaded/src/prim_inet.erl
lib/hipe/cerl/erl_bif_types.erl
lib/kernel/doc/src/inet.xml
lib/kernel/src/inet.erl
|
|
|
|
|
|
* raimo/IPV6_V6ONLY/OTP-8928:
kernel: Document socket option ipv6_v6only
kernel: Add test cases for socket option ipv6_v6only
erts,kernel: Implement socket option ipv6_v6only in erlang code
erts: Implement socket option IPV6_V6ONLY
erts: Add configure test for IPV6_V6ONLY
|
|
|
|
Peeloff feature of SCTP association creates a new socket which is not
set to nonblocking.
Function for receving data is shared with udp which has a default
loop when reading packets which is set to 5.
Calling the receive function more then once, is fine as long as
there are more data to receive or socket are nonblocking.
Set new peeled off socket to be nonblocking to prevent a erlang vm hangup.
|
|
|
|
Long input paths (longer than MAX_PATH) would get copied
into a buffer of size MAX_PATH for read_link and altname
in efile_drv.
Also fixed misuse of size_t parameter as wchar_t *
string length in win_efile:efile_readlink.
|
|
|
|
|
|
* maint:
Bumped version nr
ssl & public_key: Workaround that some certificates encode countryname as utf8 and close down gracefully if other ASN-1 errors occur.
Add more cross reference links to ct docs
Remove config option from common_test args
Update user config to use nested tuple keys
Allow mixed IPv4 and IPv6 addresses to sctp_bindx
Add checks for in6addr_any and in6addr_loopback
Fix SCTP multihoming
observer: fix app file (Noticed-by: Motiejus Jakstys)
Fix lib/src/test/ssh_basic_SUITE.erl to fix IPv6 option typos
Prevent index from being corrupted if a nonexistent item is deleted
Add tests showing that trying to delete non-existing object may corrupt the table index
Fix Table Viewer search crash on new|changed|deleted rows
Escape control characters in Table Viewer
Fix Table Viewer crash after a 'Found' -> 'Not found' search sequence
inet_drv.c: Set sockaddr lengths in inet_set_[f]address
Conflicts:
erts/preloaded/ebin/prim_inet.beam
|
|
* tab/fix-sctp-multihoming-IPv6/OTP-10217:
Allow mixed IPv4 and IPv6 addresses to sctp_bindx
Add checks for in6addr_any and in6addr_loopback
Fix SCTP multihoming
inet_drv.c: Set sockaddr lengths in inet_set_[f]address
|
|
Also allow mixed address families to bind, since the first address on
a multihomed sctp socket must be bound with bind, while the rest are
to be bound using sctp_bindx.
At least Linux supports adding address of mixing families.
Make inet_set_faddress function available also when HAVE_SCTP is not
defined, since we use it to find an address for bind to be able to mix
ipv4 and ipv6 addresses.
|
|
These variables are normally declared by <netinet/in.h>,
but for instance not on Windows 7, SDK 7.1.
Work around that by using IN6ADDR_ANY_INIT and IN6ADDR_LOOPBACK_INIT
if present, fallback to using :: and ::1.
|
|
Setting several ip addresses for an SCTP socket worked only for IPv4
on Linux. For IPv6 and for other for instance Solaris and FreeBSD, it
failed with badarg for both IPv4 and IPv6.
For the first address specified to gen_sctp:open, bind is now called,
while for any following addresses, sctp_bindx is called, repeatedly,
with one address at a time. Previously, sctp_bindx was called for all
addresses in one go, with the addresses in reverse order, and bind was
not called at all if more than one address was specified. Both
Solaris and FreeBSD requires bind to have been called before calling
sctp_bindx, and FreeBSD additionally allows at most one address at a
time in the call to sctp_bindx.
For some versions of Linux, for instance SuSE 10, the port can be 0
only for the call to bind but not for subsequent calls to sctp_bindx,
so replace with the port number assigned by the operating system.
|
|
|
|
Conflicts:
erts/preloaded/ebin/erl_prim_loader.beam
lib/kernel/src/code.erl
|
|
Set appropriate values for the sockaddr length fields---sai.sin_len
and sai6.sin6_len---if those fields exist; rely on the NO_SA_LEN
configure check to see if they exist. The length field exists on at
least FreeBSD, and there it needs to have a proper value for instance
in the call to sctp_bindx, or else that will fail with EINVAL.
|
|
Ensure displayed sizes are not negative.
|
|
|
|
This significantly reduces latency for tcp servers
with high load, as we need not go into poll to get
the next message. Maximum throughput may increase
between 4 to 6 times compared to R15B.
|
|
|
|
|
|
|
|
Conflicts:
erts/emulator/beam/beam_emu.c
erts/emulator/beam/bif.tab
erts/preloaded/ebin/prim_file.beam
lib/hipe/cerl/erl_bif_types.erl
|
|
|
|
|
|
User tags in a dynamic trace enabled VM are spread throughout the system
in the same way as seq_trace tokens. This is used by the file module
and various other modules to get hold of the tag from the user process
without changing the protocol.
|
|
|
|
Add probes to (mostly) the efile_drv.c driver and other
file I/O-related source files.
|
|
Conflicts:
erts/preloaded/ebin/zlib.beam
Rebuilt zlib.beam as part of this merge commit
|
|
* sverk/zlib_port_leak:
Fix port leaks in zlib
OTP-9981
|
|
|
|
* ps/inet6-less-build-fix:
Add missing HAVE_IN6 && AF_INET6 ifdef
OTP-9996
|
|
* maint:
Fix reselecting bug on OS X
Fix bug when sending long files using select
Fix ifdef to check if we are on OS X
Add test case for sending multiple small files on same connection
Fix memory leak when sendfile process crashes
Extend timeout for windows
Skip sendfile suite if solaris 8
|
|
* lukas/kernel/sendfile_fixes:
Fix reselecting bug on OS X
Fix bug when sending long files using select
Fix ifdef to check if we are on OS X
Add test case for sending multiple small files on same connection
Fix memory leak when sendfile process crashes
Extend timeout for windows
Skip sendfile suite if solaris 8
|
|
Since stop_select is called at an arbitrary point in the future
it would sometime not be alled before the tcp driver started
selecting on the fd. So now ERL_DRV_USE_NO_CALLBACK is used
so that the stop_select call is never made.
This seems to only have happened OS X.
|
|
The return value from efile_sendfile was not consistent
inbetween platforms. The API should now be working as it
was intended.
OTP-9994
|
|
|
|
We use the fact that file_flush is called when there is
data in the driver queue when a port is closed to ensure
that all data is cleaned up as it should.
OTP-9993
|
|
Conflicts:
erts/preloaded/ebin/prim_inet.beam
|
|
* ph/fix-enfile/OTP-9990:
Update preloaded prim_inet.beam
gen_tcp: Use correct specs for accept and listen
prim_inet: Catch system_limit in open_port
Fix returned error from gen_tcp:accept/1,2 when running out of ports
|
|
zip,unzip,gzip,gunzip,compress and uncompress did not close the port
when given invalid data argument. Fixed by putting close in "try-after".
A closed port could in same cases get stuck in limbo due to data in
io-queue. Fixed by adding a flush callback.
|
|
This prevents building for operating systems that do not yet have IPv6
compatible socket API. Error can be reproduced by (cross)building for
a os that doesn't have these headers. This will abort with a error
about incomplete type 'struct in6_addr'. All other uses of IPv6 API
in inet_drv.c are conditionalized like in the patch.
|
|
|
|
* raimo/64-bit-efile_drv/OTP-9820:
file_SUITE: Assume 64-bit windows has enough memory
erts: Badarg if port output overflows iov_len
prim_file_SUITE: large_file success depends on wordsize
file_SUITE: Increase timeout for large_file
file_SITE,prim_file_SUITE: Accept old OpenBSD pecularity
prim_file_SUITE: large_write - meaner test data and check result content
file_SUITE: large_write - check mem size before creating huge binary
erts: rewrite efile_writev to handle partial writes correctly
erts: Bugfix - driver_deq freed wrong length due to short type (int)
prim_file_SUITE: Add large_write/1
file_SUITE: Add large_write/1
file_SUITE: Refactor large_file/1
file_SUITE: Fix unix_free/1
|
|
The {error, enfile} return value is badly misleading and confusing for
this case, since the Posix ENFILE errno value has a well-defined meaning
that has nothing to do with Erlang ports. The fix changes the return
value to {error, system_limit}, which is consistent with e.g. various
file(3) functions. inet:format_error/1 has also been updated to support
system_limit in the same manner as file:format_error/1.
|
