| Age | Commit message (Collapse) | Author |
|---|
|
* raimo/type-posix-0-overhaul/ERL_550/OTP-14019:
Update types for posix error codes
|
|
I have read the man pages for most socket and file operations
on recent Linux, FreeBSD, OpenBSD and Solaris 10 and noted
the possible error codes.
Which error codes that are possible for file operations have
been updated in file:posix/0. Error codes for socket operations
in inet:posix/0. The latter refers to the former so it is
a superset, assuming that e.g sendfile and AF_UNIX socket
operations could cause socket operations to return any file
error code. That is not entirely true, but could be,
especially in the future.
Added to file:posix/0 are:
ebadmsg edeadlk edeadlock eftype emultihop enobufs enolck enolink
enosr enostr enosys eopnotsupp eoverflow erange etxtbsy
Added to inet:posix/0 are all but:
exbadport exbadseq file:posix()
These are still possible according to erl_posix_str.c,
but are not in file:posix/0 nor in inet:posix/0,
and many of them are not file nor inet related, but some might be:
e2big eadv ealign ebade ebadfd ebadr ebadrpc ebadrqc
ebadslt ebfont echild echrng ecomm edirty
edom edotdot eduppkg eidrm einit eisnam
elbin el2hlt el2nsync el3hlt el3rst
elibacc elibbad elibexec elibmax elibscn elnrng
enavail enet enoano enocsi enodata enoexec
enonet enosym enotempty enotnam enotuniq
eproclim eprocunavail eprogmismatch eprogunavail
erefused eremchg eremdev eremote eremoteio
eremoterelease erpcmismatch erremote eshutdown
esrmnt esuccesss etime etoomanyrefs
euclean eunatch eusers eversion exfull
sysnotready vernotsupported ediscon enomore
ecancelled einvalidproctable einvalidprovider eproviderfailedinit
syscallfailure service_not_found type_not_found e_no_more
e_cancelled unknown
|
|
|
|
|
|
This improves the latency of file operations as dirty schedulers
are a bit more eager to run jobs than async threads, and use a
single global queue rather than per-thread queues, eliminating the
risk of a job stalling behind a long-running job on the same thread
while other async threads sit idle.
There's no such thing as a free lunch though; the lowered latency
comes at the cost of increased busy-waiting which may have an
adverse effect on some applications. This behavior can be tweaked
with the +sbwt flag, but unfortunately it affects all types of
schedulers and not just dirty ones. We plan to add type-specific
flags at a later stage.
sendfile has been moved to inet_drv to lessen the effect of a nasty
race; the cooperation between inet_drv and efile has never been
airtight and the socket dying at the wrong time (Regardless of
reason) could result in fd aliasing. Moving it to the inet driver
makes it impossible to trigger this by closing the socket in the
middle of a sendfile operation, while still allowing it to be
aborted -- something that can't be done if it stays in the file
driver.
The race still occurs if the controlling process dies in the short
window between dispatching the sendfile operation and the dup(2)
call in the driver, but it's much less likely to happen now.
A proper fix is in the works.
--
Notable functional differences:
* The use_threads option for file:sendfile/5 no longer has any
effect.
* The file-specific DTrace probes have been removed. The same
effect can be achieved with normal tracing together with the
nif__entry/nif__return probes to track scheduling.
--
OTP-14256
|
|
|
|
* maint:
Store messages for 'rex' and 'error_logger' off heap
file: match enoent and enotdir in path_open
|
|
|
|
Previously, this function would turn any input into a single binary
before writing. This meant it could not take advantage of the `writev`
system call if it was given a list of binaries and told to write with
`raw` mode.
To see this, start an erlang shell on the parent commit, and also
start this dtrace script:
https://github.com/evanmiller/tracewrite
like this:
sudo dtrace -s tracewrite.d -p $(pgrep beam)
In the erlang shell, run the following:
file:write_file("/tmp/tmp.txt", [<<97,98>>, <<98,97>>], [raw]).
In the dtrace output, you will see that the system call used is `write`.
Now repeat with this commit, and you will see that `writev` is used.
|
|
* henrik/update-copyrightyear:
update copyright-year
|
|
This is necessary for windows to perform correctly when writing
large files.
|
|
|
|
|
|
* maint:
kernel: Do not check unsync:ed file size
Introduce new option 'raw' in file_info functions
|
|
* nox/read_file_info-raw/OTP-12325:
kernel: Do not check unsync:ed file size
Introduce new option 'raw' in file_info functions
|
|
* nox/rm-raw_files/OTP-12276:
Remove untested option 'raw_files' from file module
|
|
This undocumented option makes an application:get_env/2 call every time a raw
file is opened.
|
|
This option allows the caller not to go through the file server for information
about files guaranteed to be local.
|
|
file:file_info/1 was removed a long time ago, but its
documentation was not removed.
|
|
* josevalim/set_cwd-typespec:
Correct file:set_cwd/1 typespec
OTP-11787
|
|
file:set_cwd/1 accepts binaries as arguments, however the binaries
must be properly encoded as per file:native_name_encoding/0.
Also update the note under no_translation error to refer that passing
a ISO-latin-1 encoded binary under any unicode file name encoding.
|
|
This has been done because a slow client attack is possible if the
async thread pool is used. The scenario is:
Client does a request for a file and then slowly receives the file one
byte at a time. This will eventually fill the async thread pool with blocking
sendfile operations and thus starving the vm of all file operations.
If you still want to use the async threads pool for sendfile an option to
enable it has been introduced.
|
|
* sv/file-osync/OTP-11498:
Add sync option to file:open/2
Conflicts:
erts/preloaded/ebin/prim_file.beam
|
|
The sync option adds the POSIX O_SYNC flag to the open system call on
platforms that support the flag or its equivalent, e.g.,
FILE_FLAG_WRITE_THROUGH on Windows. For platforms that don't support it,
file:open/2 returns {error, enotsup} if the sync option is passed in.
The semantics of O_SYNC are platform-specific. For example, not all
platforms guarantee that all file metadata are written to the disk along
with the file data when the flag is in effect. This issue is noted in the
documentation this commit adds for the sync option.
Add a test for the sync option. Note however that the underlying OS
semantics for O_SYNC can't be tested automatically in any practical way, so
the test assumes the OS does the right thing with the flag when
present. For manual verification, dtruss on OS X and strace on Linux were
both run against beam processes to watch calls to open(), and file:open/2
was called in Erlang shells to open files for writing, both with and
without the sync option. Both the dtruss output and the strace output
showed that the O_SYNC flag was present in the open() calls when sync was
specified and was clear when sync was not specified.
|
|
|
|
The file module communicates with a file io server with the following
protocol for file operations:
> {file_request,From,ReplyAs,Request}
< {file_reply,ReplyAs,Reply}
The ReplyAs value is sent by the client side to match against when
receiving the reply and is otherwise left untouched and passed as is by
the server.
This commit enables receive optimizations by using the reference of the
server monitor, changing the protocol to:
> {file_request,From,MonitorRef,Request}
< {file_reply,MonitorRef,Reply}
As the shape of the messages is not changed, backwards compatibility is
not a concern.
|
|
|
|
The recommended type for filenames is a list of characters (which
may be Unicode characters greater than 255). Change the
file:filename() to reflect that.
For the filename module we still need a type that can be either
a string or a binary, so we need to introduce the type
file:filename_all().
|
|
* pan/unicode_filename_warnings:
Add file:list_dir_all/1 and file:read_link_all/1
prim_file: Add list_dir_all() and read_link_all()
Teach prim_file:set_cwd() to avoid entering non-translatable directories
Make prim_file skip invalid filenames in unicode mode
prim_file: Refactor functions that return filenames
prim_file: Refactor handling of responses
prim_file: Always open non-file ports in binary mode
Test that list_dir("non-existing-dir") fails with the correct error
|
|
|
|
|
|
|
|
This operation allows pre-allocation of space for files.
It succeeds only on systems that support such operation.
The POSIX standard defines the optional system call
posix_fallocate() to implement this feature. However,
some systems implement more specific functions to
accomplish the same operation.
On Linux, if the more specific function fallocate() is
implemented, it is used instead of posix_fallocate(),
falling back to posix_fallocate() if the fallocate()
call failed (it's only supported for the ext4, ocfs2,
xfs and btrfs file systems at the moment).
On Mac OS X it uses the specific fcntl() operation
F_PREALLOCATE, falling back to posix_fallocate() if
it's available (at the moment Mac OS X doesn't provide
posix_fallocate()).
On any other UNIX system, it uses posix_fallocate() if it's
available. Any other system not providing this system call
or any function to pre-allocate space for files, this operation
always fails with the ENOTSUP POSIX error.
|
|
Expect modifications, additions and corrections.
There is a kludge in file_io_server and
erl_scan:continuation_location() that's not so pleasing.
|
|
The mode() type is used in path_open and copy, two functions that
do _not_ accept the ram option (or is at least not supposed to),
why I moved the option 'ram' from the type down to the spec for
open itself. That also makes the option visible directly under open/2
in the documentation.
|
|
|
|
* maint:
Fix chunk usage check
|
|
|
|
|
|
Conflicts:
erts/emulator/beam/beam_emu.c
erts/emulator/beam/bif.tab
erts/preloaded/ebin/prim_file.beam
lib/hipe/cerl/erl_bif_types.erl
|
|
|
|
|
|
User tags in a dynamic trace enabled VM are spread throughout the system
in the same way as seq_trace tokens. This is used by the file module
and various other modules to get hold of the tag from the user process
without changing the protocol.
|
|
|
|
Add probes to (mostly) the efile_drv.c driver and other
file I/O-related source files.
|
|
|
|
* ta/sendfile/OTP-9240:
Do not use async threads on DARWIN
Fix cleanup when sendfile process crashes
Return {error,closed} from sendfile if closed
Do not use SFV_NOWAIT as it does not exist on all solaris
Clarify some code comments
Make solaris use sendfilev
|
|
* egil/file-info-opt-utc/OTP-7687: (39 commits)
Remove time_t specific test in prim_file_SUITE
Update prim_file.beam and prim_zip.beam
Add types for posixtime_to_universaltime and the reverse
Set BASEYEAR to 1902
Set lower limit of years handled to 1601
Emulate localtime, gmtime and mktime to enable negative time_t
Document file:*_file_info/2
Fix compiler warning in unix_efile.c
Change name of bif universaltime_to_seconds/1
Change options to prim_file:*_file_info/*
Remove dead code
Catch errors from prim_file:*_file_info
Testcase for utc <-> seconds conversion
Fix negative time in seconds_to_universaltime/1
Remove OS taint from datetime conversion
Add utc <-> seconds conversions bifs
Let prim_file validate ctime in file_info
Teach #file_info spec unix epochs for file times
Add file_info_opt tests in prim_file_SUITE
unix_efile: Zero is a valid number in utime
...
Conflicts:
erts/emulator/beam/erl_time_sup.c
erts/emulator/sys/win32/erl_win_sys.h
erts/emulator/sys/win32/sys_time.c
|
|
* file:read_file_info/2
* file:write_file_info/3
Document options and time behavior.
|
|
|
