| Age | Commit message (Collapse) | Author |
|---|
|
This improves the latency of file operations as dirty schedulers
are a bit more eager to run jobs than async threads, and use a
single global queue rather than per-thread queues, eliminating the
risk of a job stalling behind a long-running job on the same thread
while other async threads sit idle.
There's no such thing as a free lunch though; the lowered latency
comes at the cost of increased busy-waiting which may have an
adverse effect on some applications. This behavior can be tweaked
with the +sbwt flag, but unfortunately it affects all types of
schedulers and not just dirty ones. We plan to add type-specific
flags at a later stage.
sendfile has been moved to inet_drv to lessen the effect of a nasty
race; the cooperation between inet_drv and efile has never been
airtight and the socket dying at the wrong time (Regardless of
reason) could result in fd aliasing. Moving it to the inet driver
makes it impossible to trigger this by closing the socket in the
middle of a sendfile operation, while still allowing it to be
aborted -- something that can't be done if it stays in the file
driver.
The race still occurs if the controlling process dies in the short
window between dispatching the sendfile operation and the dup(2)
call in the driver, but it's much less likely to happen now.
A proper fix is in the works.
--
Notable functional differences:
* The use_threads option for file:sendfile/5 no longer has any
effect.
* The file-specific DTrace probes have been removed. The same
effect can be achieved with normal tracing together with the
nif__entry/nif__return probes to track scheduling.
--
OTP-14256
|
|
|
|
|
|
|
|
* sverk/win-long-filenames/OTP-11813:
erts: Cleanup debug tracing in win_efile.c
erts: Fix file:list_dir for windows paths 258 or 259 chars long
erts: Increase MAXPATHLEN to 4096 for windows
erts: Fix bug in efile_readlink for long win paths
kernel: Fix failed tests in prim_file_SUITE for windows
erts: Fix compiler warning in win_efile.c
erts: Skip tests of paths longer than 255 characters as atoms
erts: Skip tests of file:set_cwd for too long path on Windows
erts: Make file:make_symlink/2 return {error,eperm} on Windows
erts: Revert file:set_cwd impl for windows
erts: Ignore reduntant slashes in windows paths
fix file_SUITE:cur_dir_0 for long windows paths
erts: Fix file_SUITE:make_del_dir for long paths
erts: Fix long windows paths for compressed files
erts: Use GetFullPathNameW to construct abs paths from relative ones
erts: Fix file driver to handle long paths on windows
Conflicts:
erts/emulator/drivers/win32/win_efile.c
|
|
|
|
Appending wildcard "\*" made the path too long (>= 260 chars).
|
|
|
|
and some improved debug tracing
|
|
if the user has not the privilege SE_CREATE_SYMBOLIC_LINK_NAME
|
|
No need to even try as CWD can not bee a long path anyway.
|
|
|
|
|
|
|
|
|
|
|
|
The sync option adds the POSIX O_SYNC flag to the open system call on
platforms that support the flag or its equivalent, e.g.,
FILE_FLAG_WRITE_THROUGH on Windows. For platforms that don't support it,
file:open/2 returns {error, enotsup} if the sync option is passed in.
The semantics of O_SYNC are platform-specific. For example, not all
platforms guarantee that all file metadata are written to the disk along
with the file data when the flag is in effect. This issue is noted in the
documentation this commit adds for the sync option.
Add a test for the sync option. Note however that the underlying OS
semantics for O_SYNC can't be tested automatically in any practical way, so
the test assumes the OS does the right thing with the flag when
present. For manual verification, dtruss on OS X and strace on Linux were
both run against beam processes to watch calls to open(), and file:open/2
was called in Erlang shells to open files for writing, both with and
without the sync option. Both the dtruss output and the strace output
showed that the O_SYNC flag was present in the open() calls when sync was
specified and was clear when sync was not specified.
|
|
files as delimiters.
While working on a tool that processes Erlang code and testing it against this repo,
I found out about those little sneaky 0xff. I thought it may be of help to other
people build such tools to remove non-conforming-to-standard characters.
|
|
|
|
|
|
Using a 64bit type for bytes read will not always clear the higher bits.
|
|
Some file operations provided by the Erlang file module
didn't open the target file with all the file share flags.
This made some concurrent file operations against the
same file fail on Windows, while on other platforms such
as GNU/Linux or Mac OS X they succeed. The operations will
fail only if they're performed concurrently by different
threads (async IO threads or scheduler threads).
For example, one Erlang process does a file:delete/1 call
while another Erlang process is doing a filelib:file_size/1
call. This made the former process get an eacces error from
the file:delete/1 call. On GNU/Linux or Mac OS X the call would
succeed. Another example is if one Erlang process attempts to
open a file for reading while another one is in the middle of
a file:read_file_info/1 call (after it opened the file and
before it closed the file).
It's easy to verify that if a file is not open with all the
share flags, it's impossible for other threads (even if they
belong to the same OS process) to open the file while the
file is not closed by the first thread.
The following test program shows this:
#include <windows.h>
#include <iostream>
// Must be an existing file
//#define SHARE_FLAGS (FILE_SHARE_READ)
static DWORD WINAPI MyThreadFunction(LPVOID lpParam);
static char *lastError();
int main(int argc, char *argv[])
{
DWORD threadId;
HANDLE threadHandle, hFile;
hFile = CreateFile(FILENAME, GENERIC_READ, SHARE_FLAGS, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (hFile == INVALID_HANDLE_VALUE) {
std::cerr << "File open error from main: " <<
lastError() << std::endl;
return 1;
}
std::cout << "Main thread opened file successfully" << std::endl;
threadHandle = CreateThread(NULL, 0, MyThreadFunction, NULL, 0, &threadId);
if (threadHandle == INVALID_HANDLE_VALUE) {
std::cerr << "Thread create error from main: " <<
lastError() << std::endl;
return 1;
}
WaitForSingleObject(threadHandle, INFINITE);
CloseHandle(threadHandle);
CloseHandle(hFile);
return 0;
}
static DWORD WINAPI MyThreadFunction( LPVOID lpParam )
{
HANDLE hFile;
hFile = CreateFile(FILENAME, GENERIC_READ, SHARE_FLAGS, NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (hFile == INVALID_HANDLE_VALUE) {
std::cerr << "File open error from second thread: " <<
lastError() << std::endl;
return 1;
}
std::cout << "Second thread opened file successfully" << std::endl;
CloseHandle(hFile);
return 0;
}
static char *lastError()
{
static char *buf = NULL;
DWORD dw = GetLastError();
if (buf != NULL) {
LocalFree((LPTSTR) &buf);
}
FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM |
FORMAT_MESSAGE_IGNORE_INSERTS,
NULL, dw, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
(LPTSTR) &buf, 0, NULL);
return buf;
}
Rnning this program with SHARE_FLAGS set to 0 (as efile_fileinfo()
does for e.g.), shows that the second thread is unable to open the
file:
C:\cygwin\home\fdmanana\tmp>touch foo.bar
C:\cygwin\home\fdmanana\tmp>threads_fopen_test.exe
Main thread opened file successfully
File open error from second thread: The process cannot access the
file because it is being used by another process.
Changing the program's SHARE_FLAGS to FILE_SHARE_READ, shows that
both threads are able to open the file:
C:\cygwin\home\fdmanana\tmp>touch foo.bar
C:\cygwin\home\fdmanana\tmp>threads_test.exe
Main thread opened file successfully
Second thread opened file successfully
Same logic applies to opening files for writing or deleting and
renaming files while they're open by some other thread that didn't
specify the flags FILE_SHARE_WRITE and FILE_SHARE_DELETE.
|
|
This operation allows pre-allocation of space for files.
It succeeds only on systems that support such operation.
The POSIX standard defines the optional system call
posix_fallocate() to implement this feature. However,
some systems implement more specific functions to
accomplish the same operation.
On Linux, if the more specific function fallocate() is
implemented, it is used instead of posix_fallocate(),
falling back to posix_fallocate() if the fallocate()
call failed (it's only supported for the ext4, ocfs2,
xfs and btrfs file systems at the moment).
On Mac OS X it uses the specific fcntl() operation
F_PREALLOCATE, falling back to posix_fallocate() if
it's available (at the moment Mac OS X doesn't provide
posix_fallocate()).
On any other UNIX system, it uses posix_fallocate() if it's
available. Any other system not providing this system call
or any function to pre-allocate space for files, this operation
always fails with the ENOTSUP POSIX error.
|
|
|
|
Long input paths (longer than MAX_PATH) would get copied
into a buffer of size MAX_PATH for read_link and altname
in efile_drv.
Also fixed misuse of size_t parameter as wchar_t *
string length in win_efile:efile_readlink.
|
|
|
|
|
|
* ctime were never defined for invalid file handles
* fix epoch <-> fileinfo conversions
|
|
|
|
Conflicts:
erts/emulator/drivers/win32/win_efile.c
|
|
Still does not run, just compiles.
|
|
|
|
|
|
|
|
When setting file_info it will now correctly set access and
modified time. Previously these entities were swapped.
|
|
|
|
Also close find-handles in altname and other minor corrections to patch
|
|
|
|
* jr/windows-file-append:
Fix appending to large files (>4GB) on Windows
OTP-8958
|
|
Hard linking requires XP or later and a NTFS formatted drive.
Symbolic linking requires Windows Vista, Windows Server 2008,
Windows 7 or later. Symbolic linking on earlier versions of
Windows will return {error, enotsup}. file:make_link/2,
file:make_sym_link/2 and file:read_link/1 are all implemented.
file:read_file_info/1 does NOT return the number of hard
links on the file as this results in performance problems
for read_file_info/1, however this can be enabled by uncommenting
a few lines of code.
The original path was submitted on 9/2/2009 by Alex Tearse-Doyle.
|
|
Append mode doesn't work for files larger that 4GB on Windows.
Caused by incorrect usage of SetFilePointer in win_efile.c.
Fix uses OVERLAPPED structure to specify write position (EOF).
http://msdn.microsoft.com/en-us/library/aa365747.aspx
Opening file in append mode was also considered, but rejected, because
it might cause backwards incompatibility by limiting applicable
operations on the descriptor.
SetFilePointerEx was not used because it would caused more system calls
than using OVERLAPPED structure.
|
|
Add an option that atomically tests for the existence of a file and
creates it if the file does not exist, by passing the O_EXCL flag
to open() on Unix and CREATE_NEW flag on Windows. Support for O_EXCL
varies across platforms and filesystems.
{ok, Fd} = file:open("/tmp/foo", [write,exclusive]),
{error, eexist} = file:open("/tmp/foo", [write,exclusive]).
|
|
See dwShareMode on
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
|
|
* fm/file-operations:
Update preloaded modules
Add file:advise/4 - a wrapper to the POSIX syscall posix_fadvise
Add file:datasync/1 for syncing file contents only
sys.h: Correct the get_int64() macro
OTP-8637 fm/file-operations
The functions file:advise/4 and file:datasync/1 have been added. (Thanks to
Filipe David Manana.)
|
|
Useful for informing the Operating System about the access pattern
for a file's data, so that it can adapt the caching strategy to
maximize disk IO performance.
|
|
file:datasync/1 invokes the POSIX system call "int fdatasync(int fd)".
This system call is similar to "fsync" but, unlike fsync, it does not
update the metadata associated with the file (like the access time for
example). It's used by many DBMSs (MySQL and SQLite of example) to
increase disk IO performance, as it avoids disk seeks and disk write
operations compared to fsync.
More details on it at:
http://linux.die.net/man/2/fdatasync
An example, from the MySQL source:
http://bazaar.launchpad.net/~mysql/mysql-server/mysql-5.1-telco-6.1/annotate/head%3A/mysys/my_sync.c#L61
This new function just calls fsync on systems not implementing fdatasync.
|
|
|
