From c1c03ae4ee50e58b7669ea88ec4d29c6b2b67c7b Mon Sep 17 00:00:00 2001 From: Steve Vinoski Date: Thu, 9 Jan 2014 21:22:45 -0500 Subject: initial support for dirty schedulers and dirty NIFs Add initial support for dirty schedulers. There are two types of dirty schedulers: CPU schedulers and I/O schedulers. By default, there are as many dirty CPU schedulers as there are normal schedulers and as many dirty CPU schedulers online as normal schedulers online. There are 10 dirty I/O schedulers (similar to the choice of 10 as the default for async threads). By default, dirty schedulers are disabled and conditionally compiled out. To enable them, you must pass --enable-dirty-schedulers to the top-level configure script when building Erlang/OTP. Current dirty scheduler support requires the emulator to be built with SMP support. This restriction will be lifted in the future. You can specify the number of dirty schedulers with the command-line options +SDcpu (for dirty CPU schedulers) and +SDio (for dirty I/O schedulers). The +SDcpu option is similar to the +S option in that it takes two numbers separated by a colon: C1:C2, where C1 specifies the number of dirty schedulers available and C2 specifies the number of dirty schedulers online. The +SDPcpu option allows numbers of dirty CPU schedulers available and dirty CPU schedulers online to be specified as percentages, similar to the existing +SP option for normal schedulers. The number of dirty CPU schedulers created and dirty CPU schedulers online may not exceed the number of normal schedulers created and normal schedulers online, respectively. The +SDio option takes only a single number specifying the number of dirty I/O schedulers available and online. There is no support yet for programmatically changing at run time the number of dirty CPU schedulers online via erlang:system_flag/2. Also, changing the number of normal schedulers online via erlang:system_flag(schedulers_online, NewSchedulersOnline) should ensure that there are no more dirty CPU schedulers than normal schedulers, but this is not yet implemented. You can retrieve the number of dirty schedulers by passing dirty_cpu_schedulers, dirty_cpu_schedulers_online, or dirty_io_schedulers to erlang:system_info/1. Currently only NIFs are able to access dirty scheduler functionality. Neither drivers nor BIFs currently support dirty schedulers. This restriction will be addressed in the future. If dirty scheduler support is present in the runtime, the initial status line Erlang prints before presenting its interactive prompt will include the indicator "[ds:C1:C2:I]" where "ds" indicates "dirty schedulers", "C1" indicates the number of dirty CPU schedulers available, "C2" indicates the number of dirty CPU schedulers online, and "I" indicates the number of dirty I/O schedulers. Document The dirty NIF API in the erl_nif man page. The API closely follows Rickard Green's presentation slides from his talk "Future Extensions to the Native Interface", presented at the 2011 Erlang Factory held in the San Francisco Bay Area. Rickard's slides are available online at http://bit.ly/1m34UHB . Document the new erl command-line options, the additions to erlang:system_info/1, and also add the erlang:system_flag/2 dirty scheduler documentation even though it's not yet implemented. To determine whether the dirty NIF API is available, native code can check to see whether the C preprocessor macro ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined. To check if dirty schedulers are available at run time, native code can call the boolean enif_have_dirty_schedulers() function, and Erlang code can call erlang:system_info(dirty_cpu_schedulers), which raises badarg if no dirty scheduler support is available. Add a simple dirty NIF test to the emulator NIF suite. --- erts/doc/src/erl.xml | 48 +++++++++++++++++++ erts/doc/src/erl_nif.xml | 122 ++++++++++++++++++++++++++++++++++++++++++++++- erts/doc/src/erlang.xml | 100 +++++++++++++++++++++++++++++++++++--- 3 files changed, 262 insertions(+), 8 deletions(-) (limited to 'erts/doc/src') diff --git a/erts/doc/src/erl.xml b/erts/doc/src/erl.xml index e737727941..27a23174d5 100644 --- a/erts/doc/src/erl.xml +++ b/erts/doc/src/erl.xml @@ -792,6 +792,54 @@ SMP support enabled (see the -smp flag).

+ + +

Sets the number of dirty CPU scheduler threads to create and dirty + CPU scheduler threads to set online when threading support has been + enabled. The maximum for both values is 1024, and each value is further + limited by the settings for normal schedulers: the number of dirty CPU + scheduler threads created cannot exceed the number of normal scheduler + threads created, and the number of dirty CPU scheduler threads online + cannot exceed the number of normal scheduler threads online (see the + +S and +SP + flags for more details). By default, the number of dirty CPU scheduler + threads created equals the number of normal scheduler threads created, + and the number of dirty CPU scheduler threads online equals the number + of normal scheduler threads online. DirtyCPUSchedulers may be + omitted if :DirtyCPUSchedulersOnline is not and vice versa. The + number of dirty CPU schedulers online can be changed at run time via + erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline). +

+

This option is ignored if the emulator doesn't have threading support + enabled. Currently, this option is experimental and is supported only + if the emulator was configured and built with support for dirty schedulers + enabled (it's disabled by default). +

+ + + +

Similar to +SDcpu but uses percentages to set the + number of dirty CPU scheduler threads to create and number of dirty CPU scheduler threads + to set online when threading support has been enabled. Specified values must be greater + than 0. For example, +SDPcpu 50:25 sets the number of dirty CPU scheduler threads + to 50% of the logical processors configured and the number of dirty CPU scheduler threads + online to 25% of the logical processors available. DirtyCPUSchedulersPercentage may + be omitted if :DirtyCPUSchedulersOnlinePercentage is not and vice versa. The + number of dirty CPU schedulers online can be changed at run time via + erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline). +

+

This option interacts with +SDcpu settings. + For example, on a system with 8 logical cores configured and 8 logical cores available, + the combination of the options +SDcpu 4:4 +SDPcpu 50:25 (in either order) results + in 2 dirty CPU scheduler threads (50% of 4) and 1 dirty CPU scheduler thread online (25% of 4). +

+

This option is ignored if the emulator doesn't have threading support + enabled. Currently, this option is experimental and is supported only + if the emulator was configured and built with support for dirty schedulers + enabled (it's disabled by default). +

+ +

Scheduling specific flags.

diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml index 7ac8181d47..8b19725c02 100644 --- a/erts/doc/src/erl_nif.xml +++ b/erts/doc/src/erl_nif.xml @@ -181,7 +181,11 @@ ok to dispatch the work to another thread, return from the native function, and wait for the result. The thread can send the result back to the calling thread using message passing. Information - about thread primitives can be found below.

+ about thread primitives can be found below. If you have built your system + with the currently experimental support for dirty schedulers, + you may want to try out this functionality by dispatching the work to a + dirty NIF, + which does not have the same duration restriction as a normal NIF.

FUNCTIONALITY @@ -312,6 +316,38 @@ ok

The library initialization callbacks load, reload and upgrade are all thread-safe even for shared state data.

+ Dirty NIFs +

Note that the dirty NIF functionality + is experimental and that you have to enable support for dirty + schedulers when building OTP in order to try the functionality out. Native functions + + must normally run quickly, as explained earlier in this document. They + generally should execute for no more than a millisecond. But not all native functions + can execute so quickly; for example, functions that encrypt large blocks of data or + perform lengthy file system operations can often run for tens of seconds or more.

+

A NIF that cannot execute in a millisecond or less is called a "dirty NIF" since + it performs work that the Erlang runtime cannot handle cleanly. Applications + that make use of such functions must indicate to the runtime that the functions are + dirty so they can be handled specially. To schedule a dirty NIF for execution, the + application calls enif_schedule_dirty_nif, + passing to it a pointer to the dirty NIF to be executed and indicating with a flag + argument whether it expects the operation to be CPU-bound or I/O-bound.

+

All dirty NIFs must ultimately invoke the + enif_schedule_dirty_nif_finalizer as their final action, passing to it the + result they wish to return to the original caller. A finalizer function can either + receive the result and return it directly, or it can return a different value instead. + For convenience, the NIF API provides the + enif_dirty_nif_finalizer function that applications can use as a finalizer; + it simply returns its result argument.

+

Dirty NIF support is available only when the emulator is configured with dirty + schedulers enabled. This feature is currently disabled by default. To determine whether + the dirty NIF API is available, native code can check to see if the C preprocessor macro + ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined. Also, if the Erlang runtime was built + without threading support, dirty schedulers are disabled. To check at runtime for the presence + of dirty scheduler threads, code can call the + enif_have_dirty_schedulers() API function, which returns true if dirty + scheduler threads are present, false otherwise.

 +
@@ -610,6 +646,18 @@ typedef enum { See also the warning text at the beginning of this document.

+ ERL_NIF_TERMenif_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result) + Simple dirty NIF result finalizer + +

A convenience function that a dirty NIF can use as a finalizer that simply + return its result argument as its return value. This function is provided + for dirty NIFs with results that should be returned directly to the original caller.

+

This function is available only when the emulator is configured with dirty + schedulers enabled. This feature is currently disabled by default. To determine whether + the dirty NIF API is available, native code can check to see if the C preprocessor macro + ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined.

 + + intenif_equal_tids(ErlNifTid tid1, ErlNifTid tid2)

Same as erl_drv_equal_tids. @@ -730,6 +778,22 @@ typedef enum { and return true, or return false if term is not an unsigned integer or is outside the bounds of type unsigned long.

 + intenif_have_dirty_schedulers() + Runtime check for the presence of dirty scheduler threads + +

Check at runtime for the presence of dirty scheduler threads. If the emulator is + built with threading support, dirty scheduler threads are available and + enif_have_dirty_schedulers() returns true. If the emulator was built without + threading support, enif_have_dirty_schedulers() returns false.

+

If dirty scheduler threads are not available in the emulator, calls to + enif_schedule_dirty_nif and enif_schedule_dirty_nif_finalizer result in + the NIF and finalizer functions being called directly within the calling thread.

+

This function is available only when the emulator is configured with dirty + schedulers enabled. This feature is currently disabled by default. To determine whether + the dirty NIF API is available, native code can check to see if the C preprocessor macro + ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined.

 + + intenif_inspect_binary(ErlNifEnv* env, ERL_NIF_TERM bin_term, ErlNifBinary* bin) Inspect the content of a binary

Initialize the structure pointed to by bin with @@ -777,6 +841,20 @@ typedef enum { Erlang operators =:= and =/=.

 + intenif_is_on_dirty_scheduler(ErlNifEnv* env) + Check to see if executing on a dirty scheduler thread + +

Check to see if the current NIF is executing on a dirty scheduler thread. If the + emulator is built with threading support, calling enif_is_on_dirty_scheduler + from within a dirty NIF returns true. It returns false when the calling NIF is a regular + NIF or a NIF finalizer, both of which run on normal scheduler threads, or when the emulator + is built without threading support.

+

This function is available only when the emulator is configured with dirty + schedulers enabled. This feature is currently disabled by default. To determine whether + the dirty NIF API is available, native code can check to see if the C preprocessor macro + ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined.

 + + intenif_is_pid(ErlNifEnv* env, ERL_NIF_TERM term) Determine if a term is a pid

Return true if term is a pid.

 @@ -1141,6 +1219,48 @@ typedef enum {

Same as erl_drv_rwlock_tryrwlock.

+ ERL_NIF_TERMenif_schedule_dirty_nif(ErlNifEnv* env, int flags, ERL_NIF_TERM (*fp)(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]), int argc, const ERL_NIF_TERM argv[]) + Schedule a dirty NIF for execution + +

Schedule dirty NIF fp to execute a long-running operation. The flags + argument must be set to either ERL_NIF_DIRTY_JOB_CPU_BOUND if the job is expected to + be primarily CPU-bound, or ERL_NIF_DIRTY_JOB_IO_BOUND for jobs that will be + I/O-bound. The argc and argv arguments can either be the originals passed + into the calling NIF, or they can be values created by the calling NIF. The calling + NIF must use the return value of enif_schedule_dirty_nif as its own return value.

+

Be aware that enif_schedule_dirty_nif, as its name implies, only schedules the + dirty NIF for future execution. The calling NIF does not block waiting for the dirty NIF to + execute and return, which means that the calling NIF can't expect to receive the dirty NIF + return value and use it for further operations.

+

A dirty NIF may not invoke the enif_make_badarg + to raise an exception. If it wishes to return an exception, the dirty NIF should pass a + regular result indicating the exception details to its finalizer, and allow the finalizer + to raise the exception on its behalf.

+

This function is available only when the emulator is configured with dirty schedulers + enabled. This feature is currently disabled by default. To determine whether the dirty NIF API + is available, native code can check to see if the C preprocessor macro + ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined.

 + + + ERL_NIF_TERMenif_schedule_dirty_nif_finalizer(ErlNifEnv* env, ERL_NIF_TERM result, ERL_NIF_TERM (*fp)(ErlNifEnv* env, ERL_NIF_TERM result)) + Schedule a dirty NIF finalizer + +

When a dirty NIF finishes executing, it must schedule a finalizer function to return + its result to the original NIF caller. The dirty NIF passes result as the value it + wants the finalizer to use as the return value. The fp argument is a pointer to the + finalizer function. The NIF API provides the + enif_dirty_nif_finalizer function that can be used as a finalizer that simply + returns its result argument. You are also free to write your own custom finalizer + that uses result to derive a different return value, or ignores result + entirely and returns a completely different value.

+

Without exception, all dirty NIFs must invoke enif_schedule_dirty_nif_finalizer + to complete their execution.

+

This function is available only when the emulator is configured with dirty + schedulers enabled. This feature is currently disabled by default. To determine whether + the dirty NIF API is available, native code can check to see if the C preprocessor macro + ERL_NIF_DIRTY_SCHEDULER_SUPPORT is defined.

 + + ErlNifPid *enif_self(ErlNifEnv* caller_env, ErlNifPid* pid) Get the pid of the calling process.

Initialize the pid variable *pid to represent the diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index ea753cfaaf..4cf5631727 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -5194,6 +5194,27 @@ ok + Set system flag dirty CPU schedulers online + +

+ Sets the amount of dirty CPU schedulers online. Valid range is + where N is the + lesser of the return values of erlang:system_info(dirty_cpu_schedulers) and + erlang:system_info(schedulers_online). +

+

Returns the old value of the flag.

+

Note that the dirty schedulers functionality is experimental, and + that you have to enable support for dirty schedulers when building OTP in + order to try the functionality out.

+

For more information see + erlang:system_info(dirty_cpu_schedulers) + and + erlang:system_info(dirty_cpu_schedulers_online). +

+ + + + Set system flag fullsweep_after

Number is a non-negative integer which indicates @@ -5211,7 +5232,7 @@ ok - + Set system flag min_heap_size

Sets the default minimum heap size for processes. The @@ -5226,7 +5247,7 @@ ok - + Set system flag min_bin_vheap_size

Sets the default minimum binary virtual heap size for processes. The @@ -5241,7 +5262,7 @@ ok - + Set system flag multi_scheduling

@@ -5279,7 +5300,7 @@ ok - + Set system flag scheduler_bind_type @@ -5399,7 +5420,7 @@ ok - + Set system flag scheduler_wall_time

Turns on/off scheduler wall time measurements.

@@ -5409,7 +5430,7 @@ ok - + Set system flag schedulers_online

@@ -5425,7 +5446,7 @@ ok - + Set system flag trace_control_word

Sets the value of the node's trace control word to @@ -5785,6 +5806,71 @@ ok compiled; otherwise, false.

+ dirty_cpu_schedulers + +

Returns the number of dirty CPU scheduler threads used by + the emulator. Dirty CPU schedulers execute CPU-bound + native functions such as NIFs, linked-in driver code, and BIFs + that cannot be managed cleanly by the emulator's normal schedulers. +

+

The number of dirty CPU scheduler threads is determined at emulator + boot time and cannot be changed after that. The number of dirty CPU + scheduler threads online can however be changed at any time. The number of + dirty CPU schedulers can be set on startup by passing + the +SDcpu command line flag, see + erl(1). +

+

Note that the dirty schedulers functionality is experimental, and + that you have to enable support for dirty schedulers when building OTP in + order to try the functionality out.

+

See also erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline), + erlang:system_info(dirty_cpu_schedulers_online), + erlang:system_info(dirty_io_schedulers), + erlang:system_info(schedulers), + erlang:system_info(schedulers_online), and + erlang:system_flag(schedulers_online, SchedulersOnline).

+ + dirty_cpu_schedulers_online + +

Returns the number of dirty CPU schedulers online. The return value + satisfies the following relationship: + , where N is + the lesser of the return values of erlang:system_info(dirty_cpu_schedulers) and + erlang:system_info(schedulers_online). +

+

The number of dirty CPU schedulers online can be set on startup by passing + the +SDcpu command line flag, see + erl(1). +

+

Note that the dirty schedulers functionality is experimental, and + that you have to enable support for dirty schedulers when building OTP in + order to try the functionality out.

+

For more information, see + erlang:system_info(dirty_cpu_schedulers), + erlang:system_info(dirty_io_schedulers), + erlang:system_info(schedulers_online), and + erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline). +

+ + dirty_io_schedulers + +

Returns the number of dirty I/O schedulers as an integer. Dirty I/O schedulers + execute I/O-bound native functions such as NIFs and linked-in driver code that + cannot be managed cleanly by the emulator's normal schedulers. +

+

This value can be set on startup by passing + the +SDio command line flag, see + erl(1). +

+

Note that the dirty schedulers functionality is experimental, and + that you have to enable support for dirty schedulers when building OTP in + order to try the functionality out.

+

For more information, see + erlang:system_info(dirty_cpu_schedulers), + erlang:system_info(dirty_cpu_schedulers_online), and + erlang:system_flag(dirty_cpu_schedulers_online, DirtyCPUSchedulersOnline). +

+ dist

Returns a binary containing a string of distribution -- cgit v1.2.3