| Age | Commit message (Collapse) | Author |
|---|
|
This is to align the timestamps with external logs.
|
|
|
|
Closing files in these callbacks could block scheduler progress
and cause major system instability. We now defer these operations
to a dedicated process instead.
This process may in turn block forever and prevent further orphaned
files from being closed, but it will keep the emulator itself from
misbehaving.
|
|
We said reverse/2 but used reverse/1 which is unsafe to use in
preloaded modules. This didn't have any effect in practice as the
affected functions weren't used before the code server was started,
but it's still an error.
|
|
|
|
|
|
|
|
|
|
It wasn't possible to change group/owner separately, and our test
suite lacked coverage for that.
ERL-589
|
|
This improves the latency of file operations as dirty schedulers
are a bit more eager to run jobs than async threads, and use a
single global queue rather than per-thread queues, eliminating the
risk of a job stalling behind a long-running job on the same thread
while other async threads sit idle.
There's no such thing as a free lunch though; the lowered latency
comes at the cost of increased busy-waiting which may have an
adverse effect on some applications. This behavior can be tweaked
with the +sbwt flag, but unfortunately it affects all types of
schedulers and not just dirty ones. We plan to add type-specific
flags at a later stage.
sendfile has been moved to inet_drv to lessen the effect of a nasty
race; the cooperation between inet_drv and efile has never been
airtight and the socket dying at the wrong time (Regardless of
reason) could result in fd aliasing. Moving it to the inet driver
makes it impossible to trigger this by closing the socket in the
middle of a sendfile operation, while still allowing it to be
aborted -- something that can't be done if it stays in the file
driver.
The race still occurs if the controlling process dies in the short
window between dispatching the sendfile operation and the dup(2)
call in the driver, but it's much less likely to happen now.
A proper fix is in the works.
--
Notable functional differences:
* The use_threads option for file:sendfile/5 no longer has any
effect.
* The file-specific DTrace probes have been removed. The same
effect can be achieved with normal tracing together with the
nif__entry/nif__return probes to track scheduling.
--
OTP-14256
|
|
|
|
|
|
This has been done because a slow client attack is possible if the
async thread pool is used. The scenario is:
Client does a request for a file and then slowly receives the file one
byte at a time. This will eventually fill the async thread pool with blocking
sendfile operations and thus starving the vm of all file operations.
If you still want to use the async threads pool for sendfile an option to
enable it has been introduced.
|
|
The sync option adds the POSIX O_SYNC flag to the open system call on
platforms that support the flag or its equivalent, e.g.,
FILE_FLAG_WRITE_THROUGH on Windows. For platforms that don't support it,
file:open/2 returns {error, enotsup} if the sync option is passed in.
The semantics of O_SYNC are platform-specific. For example, not all
platforms guarantee that all file metadata are written to the disk along
with the file data when the flag is in effect. This issue is noted in the
documentation this commit adds for the sync option.
Add a test for the sync option. Note however that the underlying OS
semantics for O_SYNC can't be tested automatically in any practical way, so
the test assumes the OS does the right thing with the flag when
present. For manual verification, dtruss on OS X and strace on Linux were
both run against beam processes to watch calls to open(), and file:open/2
was called in Erlang shells to open files for writing, both with and
without the sync option. Both the dtruss output and the strace output
showed that the O_SYNC flag was present in the open() calls when sync was
specified and was clear when sync was not specified.
|
|
When the run-time system was started with +fnue, the error tuple
indicating a non-translatable filename was added as a non-proper
list tail inside an {ok,Files} term.
|
|
6d516de001dde82c02fe050db8e3aab47914fa90 added prim_file:list_dir_all/1.
Unfortunately, only the first element in the list would be handled
as intended.
|
|
|
|
We have decided that we don't want to deal with the compilations
of prim_file:get_cwd() returning a binary when the current
directory name cannot be translated losslessly to a list (i.e.
when the run-time system was started with +fnu and the current directory
name contains bytes that are not part of a valid UTF-8 sequence).
Therefore, if prim_file:set_cwd() is given a binary as the pathname,
we will need to check the binary to make sure it can be translated
to a list. We will introduce a new BIF, called prim_file:is_translatable/1,
which will check both filename encoding mode, and if it is one of
Unicode modes, the binary as well.
We don't need to do anything special if prim_file:set_cwd() is passed
a list.
|
|
The fix affects list_dir and read_link. Raw filenames are now
never produced, just consumed even if +fnu or +fna is used on
Linux etc.
This also adds the options to get error return or error handler
warning messages with +fn{u|a}{i|w|e} as an option to erl.
This is still not documented and there needs to be other versions
of read_dir and read_link to facilitate reading of all types
of filenames and links.
A check that we will not change to an invalid directory is also needed.
|
|
|
|
Currently, the format of the return value from drv_command/3 is
determined solely by the efile driver's response. In a future
commit, we will need to produce different return values that
also dependend on which function in prim_file that was called;
thus, we will need some way to pass down some sort of state
to drv_get_response/2.
As a preparation for that, allow the third argument of drv_command/3
to be a fun. That also allows us to remove the convoluted special
case handling of the list_dir operation.
|
|
Ports for operations that did not directly operate on a file (such
as listing the files in a directory) was always opened in a binary
mode, but there was still code that supported such port opened in
non-binary mode.
Since we are about to update the code reading directories, and we
don't want to bother we supporting non-binary ports, make sure that
we force the use of binary mode.
|
|
The code related to the introduction of unicode_string() and
unicode_char() has been removed. The types char() and string() have
been extended to include Unicode characters.
In fact char() was changed some time ago; this commit is about
cleaning up the documentation and introduce better names for some
functions.
|
|
This operation allows pre-allocation of space for files.
It succeeds only on systems that support such operation.
The POSIX standard defines the optional system call
posix_fallocate() to implement this feature. However,
some systems implement more specific functions to
accomplish the same operation.
On Linux, if the more specific function fallocate() is
implemented, it is used instead of posix_fallocate(),
falling back to posix_fallocate() if the fallocate()
call failed (it's only supported for the ext4, ocfs2,
xfs and btrfs file systems at the moment).
On Mac OS X it uses the specific fcntl() operation
F_PREALLOCATE, falling back to posix_fallocate() if
it's available (at the moment Mac OS X doesn't provide
posix_fallocate()).
On any other UNIX system, it uses posix_fallocate() if it's
available. Any other system not providing this system call
or any function to pre-allocate space for files, this operation
always fails with the ENOTSUP POSIX error.
|
|
|
|
|
|
|
|
|
|
Conflicts:
erts/emulator/beam/beam_emu.c
erts/emulator/beam/bif.tab
erts/preloaded/ebin/prim_file.beam
lib/hipe/cerl/erl_bif_types.erl
|
|
|
|
|
|
User tags in a dynamic trace enabled VM are spread throughout the system
in the same way as seq_trace tokens. This is used by the file module
and various other modules to get hold of the tag from the user process
without changing the protocol.
|
|
|
|
Add probes to (mostly) the efile_drv.c driver and other
file I/O-related source files.
|
|
|
|
* erlang:universaltime_to_seconds/1 changed to
erlang:universaltime_to_posixtime/1
* erlang:seconds_to_universaltime/1 changed to
erlang:posixtime_to_universaltime/1
Let prim_file.erl reflect these changes.
|
|
* 'utc' changed to 'universal'
* 'epoch' changed to 'posix'
This change conforms to other naming already in OTP, e.g.
erlang:universaltime_to_localtime/1.
|
|
We do not want to crash the file server.
|
|
|
|
|
|
|
|
|
|
|
|
First stage in utc-time for prim_file.
|
|
Since the API for headers/trailers seem to be very awkward to
work with when using non-blocking io the feature is dropped
for now. See unix_efile.c for more details.
|
|
|
|
Have to figure out how to represent progress in header writing when
using non-blocking, not sure how to do this.
|
|
Move sendfile data to invoke data instead of file_descr.
Remove usage of ready_output when doing a send.
If told to send 0 bytes, file_sendfile now sends the entire file
for linux.
|
|
Move the command handling to outputv in preparation for
header and trailer inclusion in the sendfile api.
Use the standard efile communication functions for sendfile.
|
|
Created erlang fallback for sendfile in gen_tcp and
moved sendfile from file to gen_tcp. Also created testcases
for testing all different options to sendfile.
For info about how sendfile should work see the BSD man pages
as they contain a more complete API than other *nixes.
|
