aboutsummaryrefslogtreecommitdiffstats
path: root/lib/kernel/src/file.erl
AgeCommit message (Collapse)Author
2018-03-26Merge branch 'raimo/type-posix-0-overhaul/ERL_550/OTP-14019'Raimo Niskanen
* raimo/type-posix-0-overhaul/ERL_550/OTP-14019: Update types for posix error codes
2018-03-26Update types for posix error codesRaimo Niskanen
I have read the man pages for most socket and file operations on recent Linux, FreeBSD, OpenBSD and Solaris 10 and noted the possible error codes. Which error codes that are possible for file operations have been updated in file:posix/0. Error codes for socket operations in inet:posix/0. The latter refers to the former so it is a superset, assuming that e.g sendfile and AF_UNIX socket operations could cause socket operations to return any file error code. That is not entirely true, but could be, especially in the future. Added to file:posix/0 are: ebadmsg edeadlk edeadlock eftype emultihop enobufs enolck enolink enosr enostr enosys eopnotsupp eoverflow erange etxtbsy Added to inet:posix/0 are all but: exbadport exbadseq file:posix() These are still possible according to erl_posix_str.c, but are not in file:posix/0 nor in inet:posix/0, and many of them are not file nor inet related, but some might be: e2big eadv ealign ebade ebadfd ebadr ebadrpc ebadrqc ebadslt ebfont echild echrng ecomm edirty edom edotdot eduppkg eidrm einit eisnam elbin el2hlt el2nsync el3hlt el3rst elibacc elibbad elibexec elibmax elibscn elnrng enavail enet enoano enocsi enodata enoexec enonet enosym enotempty enotnam enotuniq eproclim eprocunavail eprogmismatch eprogunavail erefused eremchg eremdev eremote eremoteio eremoterelease erpcmismatch erremote eshutdown esrmnt esuccesss etime etoomanyrefs euclean eunatch eusers eversion exfull sysnotready vernotsupported ediscon enomore ecancelled einvalidproctable einvalidprovider eproviderfailedinit syscallfailure service_not_found type_not_found e_no_more e_cancelled unknown
2018-02-05kernel: Do not call erlang:get_stacktrace()Hans Bolinder
2017-11-30Make file:sendfile/5 follow its documented error behaviorJohn Högberg
2017-11-30Reimplement efile_drv as a dirty NIFJohn Högberg
This improves the latency of file operations as dirty schedulers are a bit more eager to run jobs than async threads, and use a single global queue rather than per-thread queues, eliminating the risk of a job stalling behind a long-running job on the same thread while other async threads sit idle. There's no such thing as a free lunch though; the lowered latency comes at the cost of increased busy-waiting which may have an adverse effect on some applications. This behavior can be tweaked with the +sbwt flag, but unfortunately it affects all types of schedulers and not just dirty ones. We plan to add type-specific flags at a later stage. sendfile has been moved to inet_drv to lessen the effect of a nasty race; the cooperation between inet_drv and efile has never been airtight and the socket dying at the wrong time (Regardless of reason) could result in fd aliasing. Moving it to the inet driver makes it impossible to trigger this by closing the socket in the middle of a sendfile operation, while still allowing it to be aborted -- something that can't be done if it stays in the file driver. The race still occurs if the controlling process dies in the short window between dispatching the sendfile operation and the dup(2) call in the driver, but it's much less likely to happen now. A proper fix is in the works. -- Notable functional differences: * The use_threads option for file:sendfile/5 no longer has any effect. * The file-specific DTrace probes have been removed. The same effect can be achieved with normal tracing together with the nif__entry/nif__return probes to track scheduling. -- OTP-14256
2017-05-04Update copyright yearRaimo Niskanen
2017-02-06Merge branch 'maint'Björn Gustavsson
* maint: Store messages for 'rex' and 'error_logger' off heap file: match enoent and enotdir in path_open
2017-02-01file: match enoent and enotdir in path_openpulitta
2016-09-24Let file:write_file/3 use writevNathan Long
Previously, this function would turn any input into a single binary before writing. This meant it could not take advantage of the `writev` system call if it was given a list of binaries and told to write with `raw` mode. To see this, start an erlang shell on the parent commit, and also start this dtrace script: https://github.com/evanmiller/tracewrite like this: sudo dtrace -s tracewrite.d -p $(pgrep beam) In the erlang shell, run the following: file:write_file("/tmp/tmp.txt", [<<97,98>>, <<98,97>>], [raw]). In the dtrace output, you will see that the system call used is `write`. Now repeat with this commit, and you will see that `writev` is used.
2016-04-13Merge branch 'henrik/update-copyrightyear'Henrik Nord
* henrik/update-copyrightyear: update copyright-year
2016-03-18kernel: Reduce sendfile fallback chunk sizeLukas Larsson
This is necessary for windows to perform correctly when writing large files.
2016-03-15update copyright-yearHenrik Nord
2015-06-18Change license text to APLv2Bruce Yinhe
2014-11-26Merge branch 'maint'Lukas Larsson
* maint: kernel: Do not check unsync:ed file size Introduce new option 'raw' in file_info functions
2014-11-26Merge branch 'nox/read_file_info-raw/OTP-12325' into maintLukas Larsson
* nox/read_file_info-raw/OTP-12325: kernel: Do not check unsync:ed file size Introduce new option 'raw' in file_info functions
2014-10-30Merge branch 'nox/rm-raw_files/OTP-12276'Marcus Arendt
* nox/rm-raw_files/OTP-12276: Remove untested option 'raw_files' from file module
2014-06-30Remove untested option 'raw_files' from file moduleAnthony Ramine
This undocumented option makes an application:get_env/2 call every time a raw file is opened.
2014-06-28Introduce new option 'raw' in file_info functionsAnthony Ramine
This option allows the caller not to go through the file server for information about files guaranteed to be local.
2014-06-13Remove documentation and stub for non-exist file:file_info/1Björn Gustavsson
file:file_info/1 was removed a long time ago, but its documentation was not removed.
2014-03-19Merge branch 'josevalim/set_cwd-typespec'Henrik Nord
* josevalim/set_cwd-typespec: Correct file:set_cwd/1 typespec OTP-11787
2014-03-10Correct file:set_cwd/1 typespecJosé Valim
file:set_cwd/1 accepts binaries as arguments, however the binaries must be properly encoded as per file:native_name_encoding/0. Also update the note under no_translation error to refer that passing a ISO-latin-1 encoded binary under any unicode file name encoding.
2014-01-27erts/kernel: sendfile no longer uses async threadsLukas Larsson
This has been done because a slow client attack is possible if the async thread pool is used. The scenario is: Client does a request for a file and then slowly receives the file one byte at a time. This will eventually fill the async thread pool with blocking sendfile operations and thus starving the vm of all file operations. If you still want to use the async threads pool for sendfile an option to enable it has been introduced.
2013-11-29Merge branch 'sv/file-osync/OTP-11498'Lukas Larsson
* sv/file-osync/OTP-11498: Add sync option to file:open/2 Conflicts: erts/preloaded/ebin/prim_file.beam
2013-11-15Add sync option to file:open/2Joseph Blomstedt
The sync option adds the POSIX O_SYNC flag to the open system call on platforms that support the flag or its equivalent, e.g., FILE_FLAG_WRITE_THROUGH on Windows. For platforms that don't support it, file:open/2 returns {error, enotsup} if the sync option is passed in. The semantics of O_SYNC are platform-specific. For example, not all platforms guarantee that all file metadata are written to the disk along with the file data when the flag is in effect. This issue is noted in the documentation this commit adds for the sync option. Add a test for the sync option. Note however that the underlying OS semantics for O_SYNC can't be tested automatically in any practical way, so the test assumes the OS does the right thing with the flag when present. For manual verification, dtruss on OS X and strace on Linux were both run against beam processes to watch calls to open(), and file:open/2 was called in Erlang shells to open files for writing, both with and without the sync option. Both the dtruss output and the strace output showed that the O_SYNC flag was present in the open() calls when sync was specified and was clear when sync was not specified.
2013-05-06Fix unmatched_returns warnings in STDLIB and KernelHans Bolinder
2013-04-11Optimize communication with file io serverAnthony Ramine
The file module communicates with a file io server with the following protocol for file operations: > {file_request,From,ReplyAs,Request} < {file_reply,ReplyAs,Reply} The ReplyAs value is sent by the client side to match against when receiving the reply and is otherwise left untouched and passed as is by the server. This commit enables receive optimizations by using the reference of the server monitor, changing the protocol to: > {file_request,From,MonitorRef,Request} < {file_reply,MonitorRef,Reply} As the shape of the messages is not changed, backwards compatibility is not a concern.
2013-02-21Introduce new type file:name_all()Hans Bolinder
2013-02-13Make the file:filename() type more restrictiveBjörn Gustavsson
The recommended type for filenames is a list of characters (which may be Unicode characters greater than 255). Change the file:filename() to reflect that. For the filename module we still need a type that can be either a string or a binary, so we need to introduce the type file:filename_all().
2013-02-12Merge branch 'pan/unicode_filename_warnings'Björn Gustavsson
* pan/unicode_filename_warnings: Add file:list_dir_all/1 and file:read_link_all/1 prim_file: Add list_dir_all() and read_link_all() Teach prim_file:set_cwd() to avoid entering non-translatable directories Make prim_file skip invalid filenames in unicode mode prim_file: Refactor functions that return filenames prim_file: Refactor handling of responses prim_file: Always open non-file ports in binary mode Test that list_dir("non-existing-dir") fails with the correct error
2013-02-11Add file:list_dir_all/1 and file:read_link_all/1Björn Gustavsson
2013-02-11Make Unicode correctionsHans Bolinder
2013-01-25Make adjustments for UnicodeHans Bolinder
2013-01-09Add file:allocate/3 operationFilipe David Manana
This operation allows pre-allocation of space for files. It succeeds only on systems that support such operation. The POSIX standard defines the optional system call posix_fallocate() to implement this feature. However, some systems implement more specific functions to accomplish the same operation. On Linux, if the more specific function fallocate() is implemented, it is used instead of posix_fallocate(), falling back to posix_fallocate() if the fallocate() call failed (it's only supported for the ext4, ocfs2, xfs and btrfs file systems at the moment). On Mac OS X it uses the specific fcntl() operation F_PREALLOCATE, falling back to posix_fallocate() if it's available (at the moment Mac OS X doesn't provide posix_fallocate()). On any other UNIX system, it uses posix_fallocate() if it's available. Any other system not providing this system call or any function to pre-allocate space for files, this operation always fails with the ENOTSUP POSIX error.
2013-01-02[stdlib, kernel] Introduce Unicode support for Erlang source filesHans Bolinder
Expect modifications, additions and corrections. There is a kludge in file_io_server and erl_scan:continuation_location() that's not so pleasing.
2012-12-20Move ram option from mode() type into open/2 specPatrik Nyblom
The mode() type is used in path_open and copy, two functions that do _not_ accept the ram option (or is at least not supposed to), why I moved the option 'ram' from the type down to the spec for open itself. That also makes the option visible directly under open/2 in the documentation.
2012-12-20Basic documentation of the 'ram' mode in the 'file' moduleJan Daniel Bothma
2012-05-30Merge branch 'maint'Lukas Larsson
* maint: Fix chunk usage check
2012-05-30Fix chunk usage checkLukas Larsson
2012-03-30Update copyright yearsBjörn-Egil Dahlberg
2012-03-22Merge branch 'maint'Patrik Nyblom
Conflicts: erts/emulator/beam/beam_emu.c erts/emulator/beam/bif.tab erts/preloaded/ebin/prim_file.beam lib/hipe/cerl/erl_bif_types.erl
2012-03-22Rename dyntrace BIFs to more suiting namesPatrik Nyblom
2012-03-22Correct some errors in the user tag spreadingPatrik Nyblom
2012-03-22Add user tag spreading functionality to VM and use in filePatrik Nyblom
User tags in a dynamic trace enabled VM are spread throughout the system in the same way as seq_trace tokens. This is used by the file module and various other modules to get hold of the tag from the user process without changing the protocol.
2012-03-22Update dtrace for changes in R15Björn-Egil Dahlberg
2012-03-22Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 4/4Scott Lystig Fritchie
Add probes to (mostly) the efile_drv.c driver and other file I/O-related source files.
2012-01-27Move types and specs from erl_bif_types.erl to modulesHans Bolinder
2011-12-08Merge branch 'ta/sendfile/OTP-9240'Lukas Larsson
* ta/sendfile/OTP-9240: Do not use async threads on DARWIN Fix cleanup when sendfile process crashes Return {error,closed} from sendfile if closed Do not use SFV_NOWAIT as it does not exist on all solaris Clarify some code comments Make solaris use sendfilev
2011-12-08Merge branch 'egil/file-info-opt-utc/OTP-7687'Björn-Egil Dahlberg
* egil/file-info-opt-utc/OTP-7687: (39 commits) Remove time_t specific test in prim_file_SUITE Update prim_file.beam and prim_zip.beam Add types for posixtime_to_universaltime and the reverse Set BASEYEAR to 1902 Set lower limit of years handled to 1601 Emulate localtime, gmtime and mktime to enable negative time_t Document file:*_file_info/2 Fix compiler warning in unix_efile.c Change name of bif universaltime_to_seconds/1 Change options to prim_file:*_file_info/* Remove dead code Catch errors from prim_file:*_file_info Testcase for utc <-> seconds conversion Fix negative time in seconds_to_universaltime/1 Remove OS taint from datetime conversion Add utc <-> seconds conversions bifs Let prim_file validate ctime in file_info Teach #file_info spec unix epochs for file times Add file_info_opt tests in prim_file_SUITE unix_efile: Zero is a valid number in utime ... Conflicts: erts/emulator/beam/erl_time_sup.c erts/emulator/sys/win32/erl_win_sys.h erts/emulator/sys/win32/sys_time.c
2011-12-08Document file:*_file_info/2Björn-Egil Dahlberg
* file:read_file_info/2 * file:write_file_info/3 Document options and time behavior.
2011-12-08Remove dead codeBjörn-Egil Dahlberg