Improved scheduler suspend functionality

- The calling process is now suspended while synchronizing
  scheduler suspends via erlang:system_flag(schedulers_online, _)
  and erlang:system_flag(multi_scheduling, _), instead of blocking
  the scheduler thread in the BIF call waiting for the operation
  to synchronize. Besides releasing the scheduler for other work
  (or immediate suspend) it also makes it possible to abort the
  operation by killing the process.
- erlang:system_flag(schedulers_online, _) now only wait for normal
  schedulers to complete before it returns. This since it may take
  a very long time before all dirty schedulers suspends.
- erlang:system_flag(multi_scheduling, block_normal|unblock_normal)
  which only operate on normal schedulers has been introduced. This
  since there are use cases where suspend of dirty schedulers are
  not of interest (hipe loader).
- erlang:system_flag(multi_scheduling, block) still blocks all
  dirty schedulers as well as all normal schedulers except one since
  it is hard to redefine what multi scheduling block means.
- The three operations:
  - changing amount of schedulers online
  - blocking/unblocking normal multi scheduling
  - blocking/unblocking full multi scheduling
  can now be done in parallel. This is important since otherwise
  a full multi scheduling block would potentially delay the other
  operations for a very long time.

2 files changed, 5 insertions, 4 deletions

diff --git a/erts/preloaded/ebin/erlang.beam b/erts/preloaded/ebin/erlang.beam
index db17c53ff3..2ea2de4c70 100644
--- a/erts/preloaded/ebin/erlang.beam
+++ b/erts/preloaded/ebin/erlang.beam
diff --git a/erts/preloaded/src/erlang.erl b/erts/preloaded/src/erlang.erl
index 9bf8d13fde..dfbd116d6e 100644
--- a/erts/preloaded/src/erlang.erl
+++ b/erts/preloaded/src/erlang.erl
@@ -2300,8 +2300,8 @@ subtract(_,_) ->
       MinBinVHeapSize :: non_neg_integer(),
       OldMinBinVHeapSize :: non_neg_integer();
                         (multi_scheduling, BlockState) -> OldBlockState when
-      BlockState :: block | unblock,
-      OldBlockState :: block | unblock | enabled;
+      BlockState :: block | unblock | block_normal | unblock_normal,
+      OldBlockState :: blocked | disabled | enabled;
                         (scheduler_bind_type, How) -> OldBindType when
       How :: scheduler_bind_type() | default_bind,
       OldBindType :: scheduler_bind_type();
@@ -2446,14 +2446,15 @@ tuple_to_list(_Tuple) ->
           logical_processors_available |
           logical_processors_online) -> unknown | pos_integer();
          (machine) -> string();
+         (message_queue_data) -> message_queue_data();
          (min_heap_size) -> {min_heap_size, MinHeapSize :: pos_integer()};
          (min_bin_vheap_size) -> {min_bin_vheap_size,
                                   MinBinVHeapSize :: pos_integer()};
          (modified_timing_level) -> integer() | undefined;
-         (multi_scheduling) -> disabled | blocked | enabled;
+         (multi_scheduling) -> disabled | blocked | blocked_normal | enabled;
          (multi_scheduling_blockers) -> [Pid :: pid()];
          (nif_version) -> string();
-         (message_queue_data) -> message_queue_data();
+         (normal_multi_scheduling_blockers) -> [Pid :: pid()];
          (otp_release) -> string();
          (os_monotonic_time_source) -> [{atom(),term()}];
          (os_system_time_source) -> [{atom(),term()}];