diff options
Diffstat (limited to 'lib')
172 files changed, 3838 insertions, 3509 deletions
diff --git a/lib/asn1/doc/src/notes.xml b/lib/asn1/doc/src/notes.xml index c5074418cd..499a7e40c3 100644 --- a/lib/asn1/doc/src/notes.xml +++ b/lib/asn1/doc/src/notes.xml @@ -32,55 +32,6 @@ <p>This document describes the changes made to the asn1 application.</p> -<section><title>Asn1 5.0</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Add compile option <c>-compile(no_native)</c> in modules - with <c>on_load</c> directive which is not yet supported - by HiPE.</p> - <p> - Own Id: OTP-14316 Aux Id: PR-1390 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p>The <c>error</c> tuple returned from the <c>encode</c> - and <c>decode</c> functions will now include the stack - backtrace to make it easier to understand what went - wrong.</p> - <p> - Own Id: OTP-13961</p> - </item> - <item> - <p>The deprecated module <c>asn1rt</c> has been removed. - The deprecated functions <c>asn1ct:encode/3</c> and - <c>asn1ct:decode/3</c> have been removed. The - undocumented function <c>asn1ct:encode/2</c> has been - removed.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14146</p> - </item> - <item> - <p>The new '<c>maps</c>' option changes the - representation of the types <c>SEQUENCE</c> and - <c>SET</c> to be maps (instead of records).</p> - <p> - Own Id: OTP-14219</p> - </item> - </list> - </section> - -</section> - <section><title>Asn1 4.0.4</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/asn1/vsn.mk b/lib/asn1/vsn.mk index 7329a9f879..e4bf3e2236 100644 --- a/lib/asn1/vsn.mk +++ b/lib/asn1/vsn.mk @@ -1 +1 @@ -ASN1_VSN = 5.0 +ASN1_VSN = 4.0.4 diff --git a/lib/common_test/doc/specs/.gitignore b/lib/common_test/doc/specs/.gitignore new file mode 100644 index 0000000000..322eebcb06 --- /dev/null +++ b/lib/common_test/doc/specs/.gitignore @@ -0,0 +1 @@ +specs_*.xml diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile index 152ece5d25..e3e478ab7f 100644 --- a/lib/common_test/doc/src/Makefile +++ b/lib/common_test/doc/src/Makefile @@ -106,11 +106,17 @@ HTML_REF_MAN_FILE = $(HTMLDIR)/index.html TOP_PDF_FILE = $(PDFDIR)/$(APPLICATION)-$(VSN).pdf +SPECS_FILES = $(XML_REF3_FILES:%.xml=$(SPECDIR)/specs_%.xml) + +TOP_SPECS_FILE = specs.xml + # ---------------------------------------------------- # FLAGS # ---------------------------------------------------- XML_FLAGS += DVIPS_FLAGS += +SPECS_FLAGS = -I../../include -I../../../snmp/include \ + -I../../../kernel/include # ---------------------------------------------------- # Targets @@ -119,7 +125,7 @@ DVIPS_FLAGS += $(HTMLDIR)/%.gif: %.gif $(INSTALL_DATA) $< $@ -docs: pdf html man +docs: man pdf html $(TOP_PDF_FILE): $(XML_FILES) @@ -140,6 +146,7 @@ clean clean_docs: rm -f $(MAN3DIR)/* rm -f $(MAN6DIR)/* rm -f $(TOP_PDF_FILE) $(TOP_PDF_FILE:%.pdf=%.fo) + rm -f $(SPECDIR)/* rm -f errs core *~ # ---------------------------------------------------- diff --git a/lib/common_test/doc/src/basics_chapter.xml b/lib/common_test/doc/src/basics_chapter.xml index b349d93813..9be71ae5df 100644 --- a/lib/common_test/doc/src/basics_chapter.xml +++ b/lib/common_test/doc/src/basics_chapter.xml @@ -75,7 +75,7 @@ <p><c>Common Test</c> is also a very useful tool for white-box testing Erlang code (for example, module testing), as the test programs can call exported Erlang - functions directly. there is very little overhead required for + functions directly. There is very little overhead required for implementing basic test suites and executing simple tests. For black-box testing Erlang software, Erlang RPC and standard O&M interfaces can be used for example. diff --git a/lib/common_test/doc/src/ct.xml b/lib/common_test/doc/src/ct.xml index 82fcd80375..1a3cfdb0c5 100644 --- a/lib/common_test/doc/src/ct.xml +++ b/lib/common_test/doc/src/ct.xml @@ -70,17 +70,69 @@ <marker id="types"/> <taglist> - <tag><c>handle() = pid()</c></tag> - <item><marker id="type-handle"/> - <p>The identity (handle) of a connection.</p></item> - - <tag><c>target_name() = atom()</c></tag> - <item><marker id="type-target_name"/> - <p>A name and association to configuration data introduced - through a require statement, or a call to - <seealso marker="#require-2"><c>ct:require/2</c></seealso>, - for example, - <c>ct:require(mynodename,{node,[telnet]})</c>.</p></item> + <tag> + <marker id="type-handle"/> + <c>handle() = pid()</c> + </tag> + <item> + <p>The identity (handle) of a connection.</p> + </item> + + <tag> + <marker id="type-config_key"/> + <c>config_key() = atom()</c> + </tag> + <item> + <p>A configuration key which exists in a configuration file</p> + </item> + + <tag> + <marker id="type-target_name"/><c>target_name() = atom()</c> + </tag> + <item> + <p>A name and association to configuration data introduced + through a require statement, or a call to + <seealso marker="#require-2"><c>ct:require/2</c></seealso>, + for example, + <c>ct:require(mynodename,{node,[telnet]})</c>.</p> + </item> + + <tag> + <marker id="type-key_or_name"/> + <c>key_or_name() = config_key() | target_name()</c> + </tag> + <item/> + + <tag> + <marker id="type-conn_log_options"/> + <c>conn_log_options() = [conn_log_option()]</c> + </tag> + <item> + <p>Options that can be given to the <c>cth_conn_log</c> hook, + which is used for logging of NETCONF and Telnet + connections. See + <seealso marker="ct_netconfc#Logging">ct_netconfc</seealso> + or <seealso marker="ct_telnet#Logging">ct_telnet</seealso> + for description and examples of how to use this hook.</p> + </item> + + <tag> + <marker id="type-conn_log_option"/> + <c>conn_log_option() = {log_type,conn_log_type()} | {hosts,[key_or_name()]}</c> + </tag> + <item/> + + <tag> + <marker id="type-conn_log_type"/> + <c>conn_log_type() = raw | pretty | html | silent</c> + </tag> + <item/> + + <tag> + <marker id="type-conn_log_mod"/> + <c>conn_log_mod() = ct_netconfc | ct_telnet</c> + </tag> + <item/> </taglist> </section> diff --git a/lib/common_test/doc/src/ct_netconfc.xml b/lib/common_test/doc/src/ct_netconfc.xml index e6930b30d5..ab83f50733 100644 --- a/lib/common_test/doc/src/ct_netconfc.xml +++ b/lib/common_test/doc/src/ct_netconfc.xml @@ -35,13 +35,35 @@ <module>ct_netconfc</module> <modulesummary>NETCONF client module.</modulesummary> -<description> + <description> <p>NETCONF client module.</p> <p>The NETCONF client is compliant with RFC 4741 NETCONF Configuration Protocol and RFC 4742 Using the NETCONF Configuration Protocol over - Secure SHell (SSH)..</p> + Secure SHell (SSH).</p> + + <marker id="Connecting"/> + <p><em>Connecting to a NETCONF server</em></p> + + <p>NETCONF sessions can either be opened by a single call + to <seealso marker="#open-1"><c>open/1,2</c></seealso> or by a call + to <seealso marker="#connect-1"><c>connect/1,2</c></seealso> followed + by one or more calls to + <seealso marker="#session-1"><c>session/1,2,3</c></seealso>.</p> + + <p>The properties of the sessions will be exactly the same, except + that when + using <seealso marker="#connect-1"><c>connect/1,2</c></seealso>, you + may start multiple sessions over the same SSH connection. Each + session is implemented as an SSH channel.</p> + + <p><seealso marker="#open-1"><c>open/1,2</c></seealso> will establish one + SSH connection with one SSH channel implementing one NETCONF + session. You may start mutiple sessions by + calling <seealso marker="#open-1"><c>open/1,2</c></seealso> multiple + times, but then a new SSH connection will be established for each + session.</p> <p>For each server to test against, the following entry can be added to a configuration file:</p> @@ -49,23 +71,21 @@ <pre> {server_id(),options()}.</pre> - <p>The <c>server_id()</c> or an associated <c>target_name()</c> (see - module <seealso marker="ct"><c>ct</c></seealso>) must then be used - in calls to - <seealso marker="#open-2"><c>ct_netconfc:open/2</c></seealso>.</p> + <p>The <seealso marker="#type-server_id"><c>server_id()</c></seealso> + or an associated + <seealso marker="ct#type-target_name"><c>ct:target_name()</c></seealso> + must then be used in calls to + <seealso marker="#connect-2"><c>connect/2</c></seealso> + or <seealso marker="#open-2"><c>open/2</c></seealso>.</p> - <p>If no configuration exists for a server, a session can still be - opened by calling - <seealso marker="#open-2"><c>ct_netconfc:open/2</c></seealso> with - all necessary options specified in the call. The first argument to - <seealso marker="#open-2"><c>ct_netconfc:open/2</c></seealso> can - then be any atom.</p> + <p>If no configuration exists for a server, + use <seealso marker="#connect-1"><c>connect/1</c></seealso> + or <seealso marker="#open-1"><c>open/1</c></seealso> instead, + and specify all necessary options in the <c>Options</c> parameter.</p> - </description> + <marker id="Logging"/> + <p><em>Logging</em></p> - <section> - <marker id="Logging"/> - <title>Logging</title> <p>The NETCONF server uses <c>error_logger</c> for logging of NETCONF traffic. A special purpose error handler is implemented in <c>ct_conn_log_h</c>. To use this error handler, add the @@ -73,9 +93,9 @@ <pre> suite() -> - [{ct_hooks, [{cth_conn_log, [{conn_mod(),hook_options()}]}]}].</pre> + [{ct_hooks, [{cth_conn_log, [{<seealso marker="ct#type-conn_log_mod"><c>ct:conn_log_mod()</c></seealso>,<seealso marker="ct#type-conn_log_options"><c>ct:conn_log_options()</c></seealso>}]}]}].</pre> - <p><c>conn_mod()</c> is the name of the <c>Common Test</c> module + <p><c>conn_log_mod()</c> is the name of the <c>Common Test</c> module implementing the connection protocol, for example, <c>ct_netconfc</c>.</p> <p>Hook option <c>log_type</c> specifies the type of logging:</p> @@ -84,7 +104,7 @@ <tag><c>raw</c></tag> <item><p>The sent and received NETCONF data is logged to a separate text file "as is" without any formatting. A link to the file is - added to the test case HTML log.</p>.</item> + added to the test case HTML log.</p></item> <tag><c>pretty</c></tag> <item><p>The sent and received NETCONF data is logged to a separate @@ -104,7 +124,7 @@ To do this, use hook option <c>hosts</c> and list the names of the servers/connections to be used in the suite. The connections must be named for this to work, that is, they must be opened with - <seealso marker="#open-2"><c>ct_netconfc:open/2</c></seealso>.</p> + <seealso marker="#open-2"><c>open/2</c></seealso>.</p> <p>Option <c>hosts</c> has no effect if <c>log_type</c> is set to <c>html</c> or <c>silent</c>.</p> @@ -113,13 +133,13 @@ configuration variable <c>ct_conn_log</c>:</p> <pre> - {ct_conn_log,[{conn_mod(),hook_options()}]}.</pre> + {ct_conn_log,[{<seealso marker="ct#type-conn_log_mod"><c>ct:conn_log_mod()</c></seealso>,<seealso marker="ct#type-conn_log_options"><c>ct:conn_log_options()</c></seealso>}]}.</pre> <p>For example:</p> <pre> {ct_conn_log,[{ct_netconfc,[{log_type,pretty}, - {hosts,[key_or_name()]}]}]}</pre> + {hosts,[<seealso marker="ct#type-key_or_name"><c>ct:key_or_name()</c></seealso>]}]}]}</pre> <note> <p>Hook options specified in a configuration file overwrite the @@ -164,173 +184,149 @@ <p>The same <c>ct_hooks</c> statement without the configuration file would cause HTML logging of all NETCONF connections in to the test case HTML log.</p> - </section> - <section> - <marker id="Notifications"/> - <title>Notifications</title> + <marker id="Notifications"/> + <p><em>Notifications</em></p> <p>The NETCONF client is also compliant with RFC 5277 NETCONF Event Notifications, which defines a mechanism for an asynchronous message notification delivery service for the NETCONF protocol.</p> <p>Specific functions to support this are - <seealso marker="#create_subscription-6"><c>ct_netconfc:create_subscription/6</c></seealso> + <seealso marker="#create_subscription-1"><c>create_subscription/1-6</c></seealso> and - <seealso marker="#get_event_streams-3"><c>ct_netconfc:get_event_streams/3</c></seealso>. - (The functions also exist with other arities.)</p> - </section> - - <section> - <title>Data Types</title> - <marker id="types"/> - <taglist> - <tag><c>client() = handle() | key_or_name()</c></tag> - <item><marker id="type-client"/> - <p>For <c>handle()</c>, see module - <seealso marker="ct"><c>ct</c></seealso>.</p></item> - - <tag><c>error_reason() = term()</c></tag> - <item><marker id="type-error_reason"/> </item> - <tag><c>event_time() = {eventTime, xml_attributes(), [xs_datetime()]}</c></tag> - <item><marker id="type-event_time"/> </item> - - <tag><c>handle() = term()</c></tag> - <item><marker id="type-handle"/> - <p>Opaque reference for a connection (NETCONF session). For more - information, see module <seealso marker="ct"><c>ct</c></seealso>.</p> - </item> - - <tag><c>host() = </c><seealso marker="kernel:inet#type-hostname"><c>inet:hostname()</c></seealso> - <c> | </c><seealso marker="kernel:inet#type-ip_address"><c>inet:ip_address()</c></seealso></tag> - <item><marker id="type-host"/></item> - - <tag><c>key_or_name() = server_id() | target_name()</c></tag> - <item><marker id="type-key_or_name"/> - <p>For <c>target_name</c>, see module - <seealso marker="ct"><c>ct</c></seealso>.</p></item> - - <tag><c>netconf_db() = running | startup | candidate</c></tag> - <item><marker id="type-netconf_db"/> </item> + <seealso marker="#get_event_streams-1"><c>get_event_streams/1-3</c></seealso>.</p> - <tag><c>notification() = {notification, xml_attributes(), notification_content()}</c></tag> - <item><marker id="type-notification"/> </item> + <marker id="Default_timeout"/> + <p><em>Default Timeout</em></p> - <tag><c>notification_content() = [event_time() | simple_xml()]</c></tag> - <item><marker id="type-notification_content"/> </item> + <p>Most of the functions in this module have one variant with + a <c>Timeout</c> parameter, and one without. If nothing else is + specified, the default value <c>infinity</c> is used when + the <c>Timeout</c> parameter is not given.</p> - <tag><c>option() = {ssh, host()} | {port, </c> - <seealso marker="kernel:inet#type-port_number"><c>inet:port_number()</c></seealso> - <c>} | {timeout, timeout()} | SshConnectOption</c></tag> - <item><marker id="type-option"/> + </description> + <datatypes> + <datatype> + <name name="client"/> + </datatype> + <datatype> + <name name="error_reason"/> + </datatype> + <datatype> + <name name="event_time"/> + </datatype> + <datatype> + <name name="handle"/> + <desc> + <p>Opaque reference for a connection to a NETCONF server or a + NETCONF session.</p> + </desc> + </datatype> + <datatype> + <name name="host"/> + </datatype> + <datatype> + <name name="netconf_db"/> + </datatype> + <datatype> + <name name="notification"/> + </datatype> + <datatype> + <name name="notification_content"/> + </datatype> + <datatype> + <name name="option"/> + <desc> <p><c>SshConnectOption</c> is any valid option to <seealso marker="ssh:ssh#connect-3"><c>ssh:connect/3,4</c></seealso>. Common options used are <c>user</c>, <c>password</c> and <c>user_dir</c>. The <c>SshConnectOptions</c> are - verfied by the SSH application.</p></item> - - <tag><c>options() = [option()]</c></tag> - <item><marker id="type-options"/> - <p>Options used for setting up an SSH connection to a NETCONF - server.</p></item> - - <tag><c>server_id() = atom()</c></tag> - <item><marker id="type-server_id"/> + verfied by the SSH application.</p> + </desc> + </datatype> + <datatype> + <name name="options"/> + <desc> + <p>Options used for setting up an SSH connection to a NETCONF + server.</p> + </desc> + </datatype> + <datatype> + <name name="server_id"/> + <desc> <p>The identity of a server, specified in a configuration - file.</p></item> - - <tag><c>simple_xml() = {xml_tag(), xml_attributes(), xml_content()} | {xml_tag(), xml_content()} | xml_tag()</c></tag> - <item><marker id="type-simple_xml"/> - <p>This type is further described in application - <seealso marker="xmerl:index"><c>xmerl</c></seealso>.</p></item> - - <tag><c>stream_data() = {description, string()} | {replaySupport, string()} | {replayLogCreationTime, string()} | {replayLogAgedTime, string()}</c></tag> - <item><marker id="type-stream_data"/> - <p>For details about the data format for the string values, see - "XML Schema for Event Notifications" in RFC 5277.</p></item> - - <tag><c>stream_name() = string()</c></tag> - <item><marker id="type-stream_name"/> </item> - - <tag><c>streams() = [{stream_name(), [stream_data()]}]</c></tag> - <item><marker id="type-streams"/> </item> - - <tag><c>xml_attribute_tag() = atom()</c></tag> - <item><marker id="type-xml_attribute_tag"/> </item> - - <tag><c>xml_attribute_value() = string()</c></tag> - <item><marker id="type-xml_attribute_value"/> </item> - - <tag><c>xml_attributes() = [{xml_attribute_tag(), xml_attribute_value()}]</c></tag> - <item><marker id="type-xml_attributes"/> </item> - - <tag><c>xml_content() = [simple_xml() | iolist()]</c></tag> - <item><marker id="type-xml_content"/> </item> - - <tag><c>xml_tag() = atom()</c></tag> - <item><marker id="type-xml_tag"/> </item> - - <tag><c>xpath() = {xpath, string()}</c></tag> - <item><marker id="type-xpath"/> </item> - - <tag><c>xs_datetime() = string()</c></tag> - <item><marker id="type-xs_datetime"/> - <p>This date and time identifier has the same format as the XML type + file.</p> + </desc> + </datatype> + <datatype> + <name name="simple_xml"/> + <desc> + <p>This type is further described in application + <seealso marker="xmerl:index"><c>xmerl</c></seealso>.</p> + </desc> + </datatype> + <datatype> + <name name="stream_data"/> + <desc> + <p>For details about the data format for the string values, see + "XML Schema for Event Notifications" in RFC 5277.</p> + </desc> + </datatype> + <datatype> + <name name="stream_name"/> + </datatype> + <datatype> + <name name="streams"/> + </datatype> + <datatype> + <name name="xml_attribute_tag"/> + </datatype> + <datatype> + <name name="xml_attribute_value"/> + </datatype> + <datatype> + <name name="xml_attributes"/> + </datatype> + <datatype> + <name name="xml_content"/> + </datatype> + <datatype> + <name name="xml_tag"/> + </datatype> + <datatype> + <name name="xpath"/> + </datatype> + <datatype> + <name name="xs_datetime"/> + <desc> + <p>This date and time identifier has the same format as the XML type <c>dateTime</c> and is compliant with RFC 3339 Date and Time on the Internet Timestamps. The format is as follows:</p> - <pre> + <pre> [-]CCYY-MM-DDThh:mm:ss[.s][Z|(+|-)hh:mm]</pre> - </item> - </taglist> - </section> - - <funcs> - <func> - <name>action(Client, Action) -> Result</name> - <fsummary>Equivalent to action(Client, Action, infinity).</fsummary> - <desc><marker id="action-2"/> - <p>Equivalent to - <seealso marker="#action-3"><c>ct_netconfc:action(Client, Action, - infinity)</c></seealso>.</p> </desc> - </func> + </datatype> + </datatypes> + <funcs> <func> - <name>action(Client, Action, Timeout) -> Result</name> + <name name="action" arity="2"/> + <name name="action" arity="3"/> <fsummary>Executes an action.</fsummary> - <type> - <v>Client = client()</v> - <v>Action = simple_xml()</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {ok, [simple_xml()]} | {error, error_reason()}</v> - </type> - <desc><marker id="action-3"/> + <desc> <p>Executes an action. If the return type is void, <c>ok</c> is returned instead of <c>{ok,[simple_xml()]}</c>.</p> </desc> </func> <func> - <name>close_session(Client) -> Result</name> - <fsummary>Equivalent to close_session(Client, infinity).</fsummary> - <desc><marker id="close_session-1"/> - <p>Equivalent to - <seealso marker="#close_session-2"><c>ct_netconfc:close_session(Client, - infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>close_session(Client, Timeout) -> Result</name> + <name name="close_session" arity="1"/> + <name name="close_session" arity="2"/> <fsummary>Requests graceful termination of the session associated with the client.</fsummary> - <type> - <v>Client = client()</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="close_session-2"/> + <desc> <p>Requests graceful termination of the session associated with the client.</p> @@ -343,115 +339,148 @@ </func> <func> - <name>copy_config(Client, Source, Target) -> Result</name> - <fsummary>Equivalent to copy_config(Client, Source, Target, - infinity).</fsummary> - <desc><marker id="copy_config-3"/> - <p>Equivalent to - <seealso marker="#copy_config-4"><c>ct_netconfc:copy_config(Client, - Source, Target, infinity)</c></seealso>.</p> - </desc> - </func> + <name name="connect" arity="1"/> + <fsummary>Opens an SSH connection to a NETCONF server.</fsummary> + <desc> + <p>Opens an SSH connection to a NETCONF server.</p> - <func> - <name>copy_config(Client, Target, Source, Timeout) -> Result</name> - <fsummary>Copies configuration data.</fsummary> - <type> - <v>Client = client()</v> - <v>Target = netconf_db()</v> - <v>Source = netconf_db()</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="copy_config-4"/> - <p>Copies configuration data.</p> + <p>If the server options are specified in a configuration file, use + <seealso marker="#connect-2"><c>connect/2</c></seealso> + instead.</p> - <p>Which source and target options that can be issued depends on the - capabilities supported by the server. That is, <c>:candidate</c> - and/or <c>:startup</c> are required.</p> + <p>The opaque <seealso marker="#type-handle"><c>handle()</c></seealso> + reference returned from this + function is required as connection identifier when opening + sessions over this connection, see + <seealso marker="#session-1"><c>session/1,2,3</c></seealso>.</p> + + <p>Option <c>timeout</c> (milliseconds) is used when setting up the + SSH connection. It is not used for any other purposes during the + lifetime of the connection.</p> </desc> </func> <func> - <name>create_subscription(Client) -> term()</name> - <fsummary>Creates a subscription for event notifications.</fsummary> - <desc><marker id="create_subscription-1"/></desc> - </func> + <name name="connect" arity="2"/> + <fsummary>Opens an SSH connection to a named NETCONF server.</fsummary> + <desc> + <p>Open an SSH connection to a named NETCONF server.</p> - <func> - <name>create_subscription(Client, Timeout) -> term()</name> - <fsummary>Creates a subscription for event notifications.</fsummary> - <desc><marker id="create_subscription-2"/></desc> - </func> + <p>If <c><anno>KeyOrName</anno></c> is a + configured <c>server_id()</c> or a + <c>target_name()</c> associated with such an Id, then the options + for this server are fetched from the configuration file.</p> - <func> - <name>create_subscription(Client, Stream, Timeout) -> term()</name> - <fsummary>Creates a subscription for event notifications.</fsummary> - <desc><marker id="create_subscription-3"/></desc> - </func> + <p>Argument <c><anno>ExtraOptions</anno></c> is added to the + options found in the configuration file. If the same options + are specified, the values from the configuration file + overwrite <c><anno>ExtraOptions</anno></c>.</p> - <func> - <name>create_subscription(Client, StartTime, StopTime, Timeout) -> term()</name> - <fsummary>Creates a subscription for event notifications.</fsummary> - <desc><marker id="create_subscription-4"/></desc> + <p>If the server is not specified in a configuration file, use + <seealso marker="#connect-1"><c>connect/1</c></seealso> + instead.</p> + + <p>The opaque <seealso marker="#type-handle"><c>handle()</c></seealso> + reference returned from this + function can be used as connection identifier when opening + sessions over this connection, see + <seealso marker="#session-1"><c>session/1,2,3</c></seealso>. + However, if <c><anno>KeyOrName</anno></c> is a + <c>target_name()</c>, that is, if the server is named through a + call to <seealso marker="ct#require-2"><c>ct:require/2</c></seealso> + or a <c>require</c> statement in the test suite, then this name can + be used instead of + <seealso marker="#type-handle"><c>handle()</c></seealso>.</p> + + <p>Option <c>timeout</c> (milliseconds) is used when setting up the + SSH connection. It is not used for any other purposes during the + lifetime of the connection.</p> + </desc> </func> <func> - <name>create_subscription(Client, Stream, StartTime, StopTime, Timeout) -> term()</name> - <fsummary>Creates a subscription for event notifications.</fsummary> - <desc><marker id="create_subscription-5"/></desc> + <name name="copy_config" arity="3"/> + <name name="copy_config" arity="4"/> + <fsummary>Copies configuration data.</fsummary> + <desc> + <p>Copies configuration data.</p> + + <p>Which source and target options that can be issued depends on the + capabilities supported by the server. That is, <c>:candidate</c> + and/or <c>:startup</c> are required.</p> + </desc> </func> <func> - <name>create_subscription(Client, Stream, Filter, StartTime, StopTime, Timeout) -> Result</name> + <name>create_subscription(Client) -> Result</name> + <name>create_subscription(Client, Stream) -> Result</name> + <name>create_subscription(Client, Stream, Filter) -> Result</name> + <name>create_subscription(Client, Stream, Filter, Timeout) -> Result</name> + <name name="create_subscription" arity="5" clause_i="2"/> + <name name="create_subscription" arity="6"/> <fsummary>Creates a subscription for event notifications.</fsummary> - <type> - <v>Client = client()</v> - <v>Stream = stream_name()</v> - <v>Filter = simple_xml() | [simple_xml()]</v> - <v>StartTime = xs_datetime()</v> - <v>StopTime = xs_datetime()</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="create_subscription-6"/> + <desc> <p>Creates a subscription for event notifications.</p> <p>This function sets up a subscription for NETCONF event notifications of the specified stream type, matching the specified filter. The calling process receives notifications as messages of - type <c>notification()</c>.</p> + type <seealso marker="#type-notification"><c>notification()</c></seealso>.</p> + + <p>Only a subset of the function clauses are show above. The + full set of valid combinations of input parameters is as + follows:</p> + +<pre>create_subscription(Client) + +create_subscription(Client, Timeout) +create_subscription(Client, Stream) +create_subscription(Client, Filter) + +create_subscription(Client, Stream, Timeout) +create_subscription(Client, Filter, Timeout) +create_subscription(Client, Stream, Filter) +create_subscription(Client, StartTime, StopTime) + +create_subscription(Client, Stream, Filter, Timeout) +create_subscription(Client, StartTime, StopTime, Timeout) +create_subscription(Client, Stream, StartTime, StopTime) +create_subscription(Client, Filter, StartTime, StopTime) + +create_subscription(Client, Stream, StartTime, StopTime, Timeout) +create_subscription(Client, Stream, Filter, StartTime, StopTime) +create_subscription(Client, Stream, Filter, StartTime, StopTime, Timeout)</pre> <taglist> - <tag><c>Stream</c></tag> + <tag><c><anno>Stream</anno></c></tag> <item><p>Optional parameter that indicates which stream of event is of interest. If not present, events in the default NETCONF stream are sent.</p></item> - <tag><c>Filter</c></tag> + <tag><c><anno>Filter</anno></c></tag> <item><p>Optional parameter that indicates which subset of all possible events is of interest. The parameter format is the same as that of the filter parameter in the NETCONF protocol operations. If not present, all events not precluded by other parameters are sent.</p></item> - <tag><c>StartTime</c></tag> + <tag><c><anno>StartTime</anno></c></tag> <item><p>Optional parameter used to trigger the replay feature and indicate that the replay is to start at the time specified. - If <c>StartTime</c> is not present, this is not a replay - subscription.</p> + If <c><anno>StartTime</anno></c> is not present, this is not a + replay subscription.</p> <p>It is not valid to specify start times that are later than - the current time. If <c>StartTime</c> is specified earlier - than the log can support, the replay begins with the earliest - available notification.</p> + the current time. If <c><anno>StartTime</anno></c> is specified + earlier than the log can support, the replay begins with the + earliest available notification.</p> <p>This parameter is of type <c>dateTime</c> and compliant to RFC 3339. Implementations must support time zones.</p></item> - <tag><c>StopTime</c></tag> + <tag><c><anno>StopTime</anno></c></tag> <item><p>Optional parameter used with the optional replay feature to indicate the newest notifications of interest. If - <c>StopTime</c> is not present, the notifications continues - until the subscription is terminated.</p> + <c><anno>StopTime</anno></c> is not present, the notifications + continues until the subscription is terminated.</p> <p>Must be used with and be later than <c>StartTime</c>. Values - of <c>StopTime</c> in the future are valid. This parameter is - of type <c>dateTime</c> and compliant to RFC 3339. + of <c><anno>StopTime</anno></c> in the future are valid. This + parameter is of type <c>dateTime</c> and compliant to RFC 3339. Implementations must support time zones.</p></item> </taglist> @@ -461,25 +490,10 @@ </func> <func> - <name>delete_config(Client, Target) -> Result</name> - <fsummary>Equivalent to delete_config(Client, Target, - infinity).</fsummary> - <desc><marker id="delete_config-2"/> - <p>Equivalent to - <seealso marker="#delete_config-3"><c>ct_netconfc:delete_config(Client, Target, infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>delete_config(Client, Target, Timeout) -> Result</name> + <name name="delete_config" arity="2"/> + <name name="delete_config" arity="3"/> <fsummary>Deletes configuration data.</fsummary> - <type> - <v>Client = client()</v> - <v>Target = startup | candidate</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="delete_config-3"/> + <desc> <p>Deletes configuration data.</p> <p>The running configuration cannot be deleted and <c>:candidate</c> @@ -487,54 +501,25 @@ </desc> </func> - <func> - <name>edit_config(Client, Target, Config) -> Result</name> - <fsummary>Equivalent to edit_config(Client, Target, Config, [], - infinity).</fsummary> - <desc><marker id="edit_config-3"/> - <p>Equivalent to - <seealso marker="#edit_config-5"><c>ct_netconfc:edit_config(Client, - Target, Config, [], infinity)</c></seealso>.</p> - </desc> - </func> + <func> + <name name="disconnect" arity="1"/> + <fsummary>Closes the given SSH connection.</fsummary> + <desc> + <p>Closes the given SSH connection.</p> - <func> - <name>edit_config(Client, Target, Config, OptParamsOrTimeout) -> Result</name> - <fsummary>If OptParamsOrTimeout is a time-out value, this function is - equivalent to ct_netconfc:edit_config(Client, Target, Config, [], - Timeout).</fsummary> - <type> - <v>Client = client()</v> - <v>Target = netconf_db()</v> - <v>Config = simple_xml()</v> - <v>OptParamsOrTimeout = [simple_xml()] | timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="edit_config-4"/> - <p>If <c>OptParamsOrTimeout</c> is a time-out value, this function is - equivalent to - <seealso marker="#edit_config-5"><c>ct_netconfc:edit_config(Client, - Target, Config, [], Timeout)</c></seealso>.</p> - - <p>If <c>OptParamsOrTimeout</c> is a list of simple XML, this - function is equivalent to - <seealso marker="#edit_config-5"><c>ct_netconfc:edit_config(Client, - Target, Config, OptParams, infinity)</c></seealso>.</p> + <p>If there are open NETCONF sessions on the connection, these + will be brutally aborted. To avoid this, close each session + with <seealso marker="#close_session-1"><c>close_session/1,2</c></seealso></p> </desc> </func> <func> - <name>edit_config(Client, Target, Config, OptParams, Timeout) -> Result</name> + <name name="edit_config" arity="3"/> + <name name="edit_config" arity="4" clause_i="1"/> + <name name="edit_config" arity="4" clause_i="2"/> + <name name="edit_config" arity="5"/> <fsummary>Edits configuration data.</fsummary> - <type> - <v>Client = client()</v> - <v>Target = netconf_db()</v> - <v>Config = simple_xml()</v> - <v>OptParams = [simple_xml()]</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="edit_config-5"/> + <desc> <p>Edits configuration data.</p> <p>By default only the running target is available, unless the server @@ -550,29 +535,17 @@ <pre> [{'default-operation', ["none"]}, {'error-option', ["rollback-on-error"]}]</pre> - </desc> - </func> - <func> - <name>get(Client, Filter) -> Result</name> - <fsummary>Equivalent to get(Client, Filter, infinity).</fsummary> - <desc><marker id="get-2"/> - <p>Equivalent to - <seealso marker="#get-3"><c>ct_netconfc:get(Client, Filter, - infinity)</c></seealso>.</p> + <p>If <c><anno>OptParams</anno></c> is not given, the default + value <c>[]</c> is used.</p> </desc> </func> <func> - <name>get(Client, Filter, Timeout) -> Result</name> + <name name="get" arity="2"/> + <name name="get" arity="3"/> <fsummary>Gets data.</fsummary> - <type> - <v>Client = client()</v> - <v>Filter = simple_xml() | xpath()</v> - <v>Timeout = timeout()</v> - <v>Result = {ok, [simple_xml()]} | {error, error_reason()}</v> - </type> - <desc><marker id="get-3"/> + <desc> <p>Gets data.</p> <p>This operation returns both configuration and state data from the @@ -584,24 +557,10 @@ </func> <func> - <name>get_capabilities(Client) -> Result</name> - <fsummary>Equivalent to get_capabilities(Client, infinity).</fsummary> - <desc><marker id="get_capabilities-1"/> - <p>Equivalent to - <seealso marker="#get_capabilities-2"><c>ct_netconfc:get_capabilities(Client, - infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>get_capabilities(Client, Timeout) -> Result</name> + <name name="get_capabilities" arity="1"/> + <name name="get_capabilities" arity="2"/> <fsummary>Returns the server side capabilities.</fsummary> - <type> - <v>Client = client()</v> - <v>Timeout = timeout()</v> - <v>Result = [string()] | {error, error_reason()}</v> - </type> - <desc><marker id="get_capabilities-2"/> + <desc> <p>Returns the server side capabilities.</p> <p>The following capability identifiers, defined in RFC 4741 NETCONF @@ -623,26 +582,10 @@ </func> <func> - <name>get_config(Client, Source, Filter) -> Result</name> - <fsummary>Equivalent to get_config(Client, Source, Filter, - infinity).</fsummary> - <desc><marker id="get_config-3"/> - <p>Equivalent to - <seealso marker="#get_config-4"><c>ct_netconfc:get_config(Client, Source, Filter, infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>get_config(Client, Source, Filter, Timeout) -> Result</name> + <name name="get_config" arity="3"/> + <name name="get_config" arity="4"/> <fsummary>Gets configuration data.</fsummary> - <type> - <v>Client = client()</v> - <v>Source = netconf_db()</v> - <v>Filter = simple_xml() | xpath()</v> - <v>Timeout = timeout()</v> - <v>Result = {ok, [simple_xml()]} | {error, error_reason()}</v> - </type> - <desc><marker id="get_config-4"/> + <desc> <p>Gets configuration data.</p> <p>To be able to access another source than <c>running</c>, the @@ -654,25 +597,12 @@ </func> <func> - <name>get_event_streams(Client, Timeout) -> Result</name> - <fsummary>Equivalent to get_event_streams(Client, [], Timeout).</fsummary> - <desc><marker id="get_event_streams-2"/> - <p>Equivalent to - <seealso marker="#get_event_streams-3"><c>ct_netconfc:get_event_streams(Client, - [], Timeout)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>get_event_streams(Client, Streams, Timeout) -> Result</name> + <name name="get_event_streams" arity="1"/> + <name name="get_event_streams" arity="2" clause_i="1"/> + <name name="get_event_streams" arity="2" clause_i="2"/> + <name name="get_event_streams" arity="3"/> <fsummary>Sends a request to get the specified event streams.</fsummary> - <type> - <v>Client = client()</v> - <v>Streams = [stream_name()]</v> - <v>Timeout = timeout()</v> - <v>Result = {ok, streams()} | {error, error_reason()}</v> - </type> - <desc><marker id="get_event_streams-3"/> + <desc> <p>Sends a request to get the specified event streams.</p> <p><c>Streams</c> is a list of stream names. The following filter is @@ -700,67 +630,28 @@ </netconf></pre> <p>If more complex filtering is needed, use - <seealso marker="#get-2"><c>ct_netconfc:get/2</c></seealso> or - <seealso marker="#get-3"><c>ct_netconfc:get/3</c></seealso> and + <seealso marker="#get-2"><c>ct_netconfc:get/2,3</c></seealso> and specify the exact filter according to "XML Schema for Event Notifications" in RFC 5277.</p> </desc> </func> <func> - <name>get_session_id(Client) -> Result</name> - <fsummary>Equivalent to get_session_id(Client, infinity).</fsummary> - <desc><marker id="get_session_id-1"/> - <p>Equivalent to - <seealso marker="#get_session_id-2"><c>ct_netconfc:get_session_id(Client, - infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>get_session_id(Client, Timeout) -> Result</name> + <name name="get_session_id" arity="1"/> + <name name="get_session_id" arity="2"/> <fsummary>Returns the session Id associated with the specified client.</fsummary> - <type> - <v>Client = client()</v> - <v>Timeout = timeout()</v> - <v>Result = pos_integer() | {error, error_reason()}</v> - </type> - <desc><marker id="get_session_id-2"/> + <desc> <p>Returns the session Id associated with the specified client.</p> </desc> </func> <func> - <name>hello(Client) -> Result</name> - <fsummary>Equivalent to hello(Client, [], infinity).</fsummary> - <desc><marker id="hello-1"/> - <p>Equivalent to - <seealso marker="#hello-3"><c>ct_netconfc:hello(Client, [], - infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>hello(Client, Timeout) -> Result</name> - <fsummary>Equivalent to hello(Client, [], Timeout).</fsummary> - <desc><marker id="hello-2"/> - <p>Equivalent to - <seealso marker="#hello-3"><c>ct_netconfc:hello(Client, [], - Timeout)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>hello(Client, Options, Timeout) -> Result</name> + <name name="hello" arity="1"/> + <name name="hello" arity="2"/> + <name name="hello" arity="3"/> <fsummary>Exchanges hello messages with the server.</fsummary> - <type> - <v>Client = handle()</v> - <v>Options = [{capability, [string()]}]</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="hello-3"/> + <desc> <p>Exchanges <c>hello</c> messages with the server.</p> <p>Adds optional capabilities and sends a <c>hello</c> message to the @@ -769,27 +660,11 @@ </func> <func> - <name>kill_session(Client, SessionId) -> Result</name> - <fsummary>Equivalent to kill_session(Client, SessionId, - infinity).</fsummary> - <desc><marker id="kill_session-2"/> - <p>Equivalent to - <seealso marker="#kill_session-3"><c>ct_netconfc:kill_session(Client, -SessionId, infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>kill_session(Client, SessionId, Timeout) -> Result</name> + <name name="kill_session" arity="2"/> + <name name="kill_session" arity="3"/> <fsummary>Forces termination of the session associated with the supplied session Id.</fsummary> - <type> - <v>Client = client()</v> - <v>SessionId = pos_integer()</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="kill_session-3"/> + <desc> <p>Forces termination of the session associated with the supplied session Id.</p> @@ -807,26 +682,11 @@ SessionId, infinity)</c></seealso>.</p> </func> <func> - <name>lock(Client, Target) -> Result</name> - <fsummary>Equivalent to lock(Client, Target, infinity).</fsummary> - <desc><marker id="lock-2"/> - <p>Equivalent to - <seealso marker="#lock-3"><c>ct_netconfc:lock(Client, Target, - infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>lock(Client, Target, Timeout) -> Result</name> - <fsummary>Unlocks the configuration target.</fsummary> - <type> - <v>Client = client()</v> - <v>Target = netconf_db()</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="lock-3"/> - <p>Unlocks the configuration target.</p> + <name name="lock" arity="2"/> + <name name="lock" arity="3"/> + <fsummary>Locks the configuration target.</fsummary> + <desc> + <p>Locks the configuration target.</p> <p>Which target parameters that can be used depends on if <c>:candidate</c> and/or <c>:startup</c> are supported by the @@ -835,9 +695,7 @@ SessionId, infinity)</c></seealso>.</p> Locks are intended to be short-lived.</p> <p>Operation - <seealso marker="#kill_session-2"><c>ct_netconfc:kill_session/2</c></seealso> - or - <seealso marker="#kill_session-3"><c>ct_netconfc:kill_session/3</c></seealso> + <seealso marker="#kill_session-2"><c>kill_session/2,3</c></seealso> can be used to force the release of a lock owned by another NETCONF session. How this is achieved by the server side is implementation-specific.</p> @@ -845,54 +703,41 @@ SessionId, infinity)</c></seealso>.</p> </func> <func> - <name>only_open(Options) -> Result</name> + <name name="only_open" arity="1"/> <fsummary>Opens a NETCONF session, but does not send hello.</fsummary> - <type> - <v>Options = options()</v> - <v>Result = {ok, handle()} | {error, error_reason()}</v> - </type> - <desc><marker id="only_open-1"/> + <desc> <p>Opens a NETCONF session, but does not send <c>hello</c>.</p> - <p>As <seealso marker="#open-1"><c>ct_netconfc:open/1</c></seealso>, - but does not send a <c>hello</c> message.</p> + <p>As <seealso marker="#open-1"><c>open/1</c></seealso>, but + does not send a <c>hello</c> message.</p> </desc> </func> <func> - <name>only_open(KeyOrName, ExtraOptions) -> Result</name> - <fsummary>Opens a name NETCONF session, but does not send - hello.</fsummary> - <type> - <v>KeyOrName = key_or_name()</v> - <v>ExtraOptions = options()</v> - <v>Result = {ok, handle()} | {error, error_reason()}</v> - </type> - <desc><marker id="only_open-2"/> - <p>Opens a name NETCONF session, but does not send <c>hello</c>.</p> - - <p>As <seealso marker="#open-2"><c>ct_netconfc:open/2</c></seealso>, - but does not send a <c>hello</c> message.</p> + <name name="only_open" arity="2"/> + <fsummary>Opens a named NETCONF session, but does not send hello.</fsummary> + <desc> + <p>Opens a named NETCONF session, but does not send <c>hello</c>.</p> + + <p>As <seealso marker="#open-2"><c>open/2</c></seealso>, but + does not send a <c>hello</c> message.</p> </desc> </func> <func> - <name>open(Options) -> Result</name> - <fsummary>Opens a NETCONF session and exchanges hello messages.</fsummary> - <type> - <v>Options = options()</v> - <v>Result = {ok, handle()} | {error, error_reason()}</v> - </type> - <desc><marker id="open-1"/> + <name name="open" arity="1"/> + <fsummary>Opens a NETCONF session and exchanges hello messages.</fsummary> + <desc> <p>Opens a NETCONF session and exchanges <c>hello</c> messages.</p> <p>If the server options are specified in a configuration file, or if a named client is needed for logging purposes (see section <seealso marker="#Logging">Logging</seealso> in this module), use - <seealso marker="#open-2"><c>ct_netconfc:open/2</c></seealso> + <seealso marker="#open-2"><c>open/2</c></seealso> instead.</p> - <p>The opaque <c>handle()</c> reference returned from this + <p>The opaque <seealso marker="#type-handle"><c>handle()</c></seealso> + reference returned from this function is required as client identifier when calling any other function in this module.</p> @@ -904,37 +749,36 @@ SessionId, infinity)</c></seealso>.</p> </func> <func> - <name>open(KeyOrName, ExtraOptions) -> Result</name> + <name name="open" arity="2"/> <fsummary>Opens a named NETCONF session and exchanges hello messages.</fsummary> - <type> - <v>KeyOrName = key_or_name()</v> - <v>ExtraOptions = options()</v> - <v>Result = {ok, handle()} | {error, error_reason()}</v> - </type> - <desc><marker id="open-2"/> + <desc> <p>Opens a named NETCONF session and exchanges <c>hello</c> messages.</p> - <p>If <c>KeyOrName</c> is a configured <c>server_id()</c> or a + <p>If <c><anno>KeyOrName</anno></c> is a + configured <c>server_id()</c> or a <c>target_name()</c> associated with such an Id, then the options for this server are fetched from the configuration file.</p> - <p>Argument <c>ExtraOptions</c> is added to the options found in the - configuration file. If the same options are specified, the values - from the configuration file overwrite <c>ExtraOptions</c>.</p> + <p>Argument <c><anno>ExtraOptions</anno></c> is added to the + options found in the configuration file. If the same + options are specified, the values from the configuration + file overwrite <c><anno>ExtraOptions</anno></c>.</p> <p>If the server is not specified in a configuration file, use - <seealso marker="#open-1"><c>ct_netconfc:open/1</c></seealso> + <seealso marker="#open-1"><c>open/1</c></seealso> instead.</p> - <p>The opaque <c>handle()</c> reference returned from this + <p>The opaque <seealso marker="#type-handle"><c>handle()</c></seealso> + reference returned from this function can be used as client identifier when calling any other - function in this module. However, if <c>KeyOrName</c> is a + function in this module. However, if <c><anno>KeyOrName</anno></c> is a <c>target_name()</c>, that is, if the server is named through a call to <seealso marker="ct#require-2"><c>ct:require/2</c></seealso> or a <c>require</c> statement in the test suite, then this name can - be used instead of <c>handle()</c>.</p> + be used instead of + <seealso marker="#type-handle"><c>handle()</c></seealso>.</p> <p>Option <c>timeout</c> (milliseconds) is used when setting up the SSH connection and when waiting for the <c>hello</c> message from @@ -947,25 +791,10 @@ SessionId, infinity)</c></seealso>.</p> </func> <func> - <name>send(Client, SimpleXml) -> Result</name> - <fsummary>Equivalent to send(Client, SimpleXml, infinity).</fsummary> - <desc><marker id="send-2"/> - <p>Equivalent to - <seealso marker="#send-3"><c>ct_netconfc:send(Client, SimpleXml, - infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>send(Client, SimpleXml, Timeout) -> Result</name> + <name name="send" arity="2"/> + <name name="send" arity="3"/> <fsummary>Sends an XML document to the server.</fsummary> - <type> - <v>Client = client()</v> - <v>SimpleXml = simple_xml()</v> - <v>Timeout = timeout()</v> - <v>Result = simple_xml() | {error, error_reason()}</v> - </type> - <desc><marker id="send-3"/> + <desc> <p>Sends an XML document to the server.</p> <p>The specified XML document is sent "as is" to the server. This @@ -975,25 +804,10 @@ SessionId, infinity)</c></seealso>.</p> </func> <func> - <name>send_rpc(Client, SimpleXml) -> Result</name> - <fsummary>Equivalent to send_rpc(Client, SimpleXml, infinity).</fsummary> - <desc><marker id="send_rpc-2"/> - <p>Equivalent to - <seealso marker="#send_rpc-3"><c>ct_netconfc:send_rpc(Client, - SimpleXml, infinity)</c></seealso>.</p> - </desc> - </func> - - <func> - <name>send_rpc(Client, SimpleXml, Timeout) -> Result</name> + <name name="send_rpc" arity="2"/> + <name name="send_rpc" arity="3"/> <fsummary>Sends a NETCONF rpc request to the server.</fsummary> - <type> - <v>Client = client()</v> - <v>SimpleXml = simple_xml()</v> - <v>Timeout = timeout()</v> - <v>Result = [simple_xml()] | {error, error_reason()}</v> - </type> - <desc><marker id="send_rpc-3"/> + <desc> <p>Sends a NETCONF <c>rpc</c> request to the server.</p> <p>The specified XML document is wrapped in a valid NETCONF <c>rpc</c> @@ -1006,30 +820,42 @@ SessionId, infinity)</c></seealso>.</p> </func> <func> - <name>unlock(Client, Target) -> Result</name> - <fsummary>Equivalent to unlock(Client, Target, infinity).</fsummary> - <desc><marker id="unlock-2"/> - <p>Equivalent to - <seealso marker="#unlock-3"><c>ct_netconfc:unlock(Client, Target, - infinity)</c></seealso>.</p> + <name name="session" arity="1"/> + <name name="session" arity="2" clause_i="1"/> + <name name="session" arity="2" clause_i="2"/> + <name name="session" arity="3"/> + <fsummary>Opens a NETCONF session as a channel on the given SSH + connection, and exchanges hello messages with the + server.</fsummary> + <type name="session_options"/> + <type name="session_option"/> + <desc> + <p>Opens a NETCONF session as a channel on the given SSH + connection, and exchanges hello messages with the server.</p> + + <p>The opaque <seealso marker="#type-handle"><c>handle()</c></seealso> + reference returned from this + function can be used as client identifier when calling any + other function in this module. However, if <c><anno>KeyOrName</anno></c> + is used and it is a <c>target_name()</c>, that is, if the + server is named through a call + to <seealso marker="ct#require-2"><c>ct:require/2</c></seealso> + or a <c>require</c> statement in the test suite, then this + name can be used instead of + <seealso marker="#type-handle"><c>handle()</c></seealso>.</p> + </desc> </func> <func> - <name>unlock(Client, Target, Timeout) -> Result</name> + <name name="unlock" arity="2"/> + <name name="unlock" arity="3"/> <fsummary>Unlocks the configuration target.</fsummary> - <type> - <v>Client = client()</v> - <v>Target = netconf_db()</v> - <v>Timeout = timeout()</v> - <v>Result = ok | {error, error_reason()}</v> - </type> - <desc><marker id="unlock-3"/> + <desc> <p>Unlocks the configuration target.</p> <p>If the client earlier has acquired a lock through - <seealso marker="#lock-2"><c>ct_netconfc:lock/2</c></seealso> or - <seealso marker="#lock-3"><c>ct_netconfc:lock/3</c></seealso>, this + <seealso marker="#lock-2"><c>lock/2,3</c></seealso>, this operation releases the associated lock. To access another target than <c>running</c>, the server must support <c>:candidate</c> and/or <c>:startup</c>.</p> diff --git a/lib/common_test/doc/src/ct_ssh.xml b/lib/common_test/doc/src/ct_ssh.xml index 137e4c3f1d..63627d8840 100644 --- a/lib/common_test/doc/src/ct_ssh.xml +++ b/lib/common_test/doc/src/ct_ssh.xml @@ -1034,6 +1034,33 @@ ChannelId, 0, Data, End, Timeout)</c></seealso>.</p> </func> <func> + <name>shell(SSH, ChannelId) -> ok | {error, Reason}</name> + <fsummary>Equivalent to shell(SSH, ChannelId, DefaultTimeout).</fsummary> + <desc><marker id="shell-2"/> + <p>Equivalent to + <seealso marker="#shell-3"><c>ct_ssh:shell(SSH, ChannelId, + DefaultTimeout)</c></seealso>.</p> + </desc> + </func> + + <func> + <name>shell(SSH, ChannelId, Timeout) -> ok | {error, Reason}</name> + <fsummary>Requests that the user default shell is executed at the + server end.</fsummary> + <type> + <v>SSH = connection()</v> + <v>ChannelId = integer()</v> + <v>Timeout = integer()</v> + <v>Reason = term()</v> + </type> + <desc><marker id="shell-3"/> + <p>Requests that the user default shell (typically defined in + <c>/etc/passwd</c> in Unix systems) is executed at the + server end.</p> + </desc> + </func> + + <func> <name>subsystem(SSH, ChannelId, Subsystem) -> Status | {error, Reason}</name> <fsummary>Equivalent to subsystem(SSH, ChannelId, Subsystem, DefaultTimeout).</fsummary> diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index 9f2286d7f0..a0079fd0c0 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -33,45 +33,6 @@ <file>notes.xml</file> </header> -<section><title>Common_Test 1.15</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Errors in the documentation for user HTML stylesheets - have been corrected.</p> - <p> - Own Id: OTP-14332 Aux Id: seq13299 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p>The <c>ct_slave</c> modules now handle nodenames in - the same way as nodenames passed to <c>-sname</c>. That - means <c>ct_slave:start('[email protected]').</c> will now - work.</p> - <p> - Own Id: OTP-13806</p> - </item> - <item> - <p> - Added the new option, <c>keep_logs</c>. If setting the - value for this option to an integer, N, common_test will - remove all ct_run.* directories in the current log - directory, except the N newest.</p> - <p> - Own Id: OTP-14179</p> - </item> - </list> - </section> - -</section> - <section><title>Common_Test 1.14</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/common_test/doc/src/specs.xml b/lib/common_test/doc/src/specs.xml new file mode 100644 index 0000000000..7e40e8351d --- /dev/null +++ b/lib/common_test/doc/src/specs.xml @@ -0,0 +1,5 @@ +<?xml version="1.0" encoding="utf-8" ?> +<specs xmlns:xi="http://www.w3.org/2001/XInclude"> + <xi:include href="../specs/specs_ct_netconfc.xml"/> + <xi:include href="../specs/specs_ct.xml"/> +</specs> diff --git a/lib/common_test/src/ct.erl b/lib/common_test/src/ct.erl index 43abb91819..662732d475 100644 --- a/lib/common_test/src/ct.erl +++ b/lib/common_test/src/ct.erl @@ -89,6 +89,36 @@ -export([get_target_name/1]). -export([parse_table/1, listenv/1]). +%%---------------------------------------------------------------------- +%% Exported types +%%---------------------------------------------------------------------- +%% For ct_gen_conn +-export_type([config_key/0, + target_name/0, + key_or_name/0]). + +%% For cth_conn_log +-export_type([conn_log_options/0, + conn_log_type/0, + conn_log_mod/0]). + +%%------------------------------------------------------------------ +%% Type declarations +%% ------------------------------------------------------------------ +-type config_key() :: atom(). % Config key which exists in a config file +-type target_name() :: atom().% Name associated to a config_key() though 'require' +-type key_or_name() :: config_key() | target_name(). + +%% Types used when logging connections with the 'cth_conn_log' hook +-type conn_log_options() :: [conn_log_option()]. +-type conn_log_option() :: {log_type,conn_log_type()} | + {hosts,[key_or_name()]}. +-type conn_log_type() :: raw | pretty | html | silent. +-type conn_log_mod() :: ct_netconfc | ct_telnet. +%%---------------------------------------------------------------------- + + + %%%----------------------------------------------------------------- %%% @spec install(Opts) -> ok | {error,Reason} %%% Opts = [Opt] diff --git a/lib/common_test/src/ct_conn_log_h.erl b/lib/common_test/src/ct_conn_log_h.erl index 93e64c65fe..6c1e46925f 100644 --- a/lib/common_test/src/ct_conn_log_h.erl +++ b/lib/common_test/src/ct_conn_log_h.erl @@ -238,6 +238,8 @@ actionstr(#conn_log{action=cmd}) -> "----->"; actionstr(#conn_log{action=recv}) -> "<-----"; actionstr(#conn_log{action=open}) -> "opened session to"; actionstr(#conn_log{action=close}) -> "closed session to"; +actionstr(#conn_log{action=connect}) -> "connected to"; +actionstr(#conn_log{action=disconnect}) -> "disconnected from"; actionstr(_) -> "<---->". serverstr(#conn_log{name=undefined,address={undefined,_}}) -> diff --git a/lib/common_test/src/ct_gen_conn.erl b/lib/common_test/src/ct_gen_conn.erl index 8b59d3ab23..88b05cf955 100644 --- a/lib/common_test/src/ct_gen_conn.erl +++ b/lib/common_test/src/ct_gen_conn.erl @@ -29,13 +29,6 @@ -export([call/2, call/3, return/2, do_within_time/2]). -export([log/3, start_log/1, cont_log/2, cont_log_no_timestamp/2, end_log/0]). -%%---------------------------------------------------------------------- -%% Exported types -%%---------------------------------------------------------------------- --export_type([server_id/0, - target_name/0, - key_or_name/0]). - -ifdef(debug). -define(dbg,true). -else. @@ -54,18 +47,6 @@ cb_state, ct_util_server}). -%%------------------------------------------------------------------ -%% Type declarations -%%------------------------------------------------------------------ --type server_id() :: atom(). -%% A `ServerId' which exists in a configuration file. --type target_name() :: atom(). -%% A name which is associated to a `server_id()' via a -%% `require' statement or a call to {@link ct:require/2} in the -%% test suite. --type key_or_name() :: server_id() | target_name(). - - %%%----------------------------------------------------------------- %%% @spec start(Address,InitData,CallbackMod,Opts) -> %%% {ok,Handle} | {error,Reason} @@ -373,8 +354,9 @@ loop(Opts) -> end; {stop, From} -> ct_util:unregister_connection(self()), - (Opts#gen_opts.callback):terminate(Opts#gen_opts.conn_pid, - Opts#gen_opts.cb_state), + ConnPid = Opts#gen_opts.conn_pid, + unlink(ConnPid), + (Opts#gen_opts.callback):terminate(ConnPid,Opts#gen_opts.cb_state), return(From,ok), ok; {{retry,{Error,_Name,CPid,_Msg}}, From} when @@ -411,8 +393,9 @@ loop(Opts) -> loop(Opts#gen_opts{cb_state=NewState}); {stop,Reply,NewState} -> ct_util:unregister_connection(self()), - (Opts#gen_opts.callback):terminate(Opts#gen_opts.conn_pid, - NewState), + ConnPid = Opts#gen_opts.conn_pid, + unlink(ConnPid), + (Opts#gen_opts.callback):terminate(ConnPid,NewState), return(From,Reply) end; Msg when Opts#gen_opts.forward==true -> @@ -422,8 +405,9 @@ loop(Opts) -> loop(Opts#gen_opts{cb_state=NewState}); {stop,NewState} -> ct_util:unregister_connection(self()), - (Opts#gen_opts.callback):terminate(Opts#gen_opts.conn_pid, - NewState) + ConnPid = Opts#gen_opts.conn_pid, + unlink(ConnPid), + (Opts#gen_opts.callback):terminate(ConnPid,NewState) end end. diff --git a/lib/common_test/src/ct_netconfc.erl b/lib/common_test/src/ct_netconfc.erl index ff45407fe0..b77570c121 100644 --- a/lib/common_test/src/ct_netconfc.erl +++ b/lib/common_test/src/ct_netconfc.erl @@ -23,24 +23,15 @@ %% Description: %% This file contains the Netconf client interface %% -%% @author Support +%% Netconf servers can be configured by adding the following statement +%% to a configuration file: %% -%% @doc Netconf client module. +%% {server_id(),options()}. %% -%% <p>The Netconf client is compliant with RFC4741 and RFC4742.</p> +%% The server_id() or an associated ct:target_name() shall then be +%% used in calls to open/2 connect/2. %% -%% <p> For each server to test against, the following entry can be -%% added to a configuration file:</p> -%% -%% <p>`{server_id(),options()}.'</p> -%% -%% <p> The `server_id()' or an associated `target_name()' (see -%% {@link ct}) shall then be used in calls to {@link open/2}.</p> -%% -%% <p>If no configuration exists for a server, a session can still be -%% opened by calling {@link open/2} with all necessary options given -%% in the call. The first argument to {@link open/2} can then be any -%% atom.</p> +%% If no configuration exists for a server, use open/1 and connect/1. %% %% == Logging == %% @@ -49,102 +40,15 @@ %% `ct_conn_log_h'. To use this error handler, add the `cth_conn_log' %% hook in your test suite, e.g. %% -%% ``` %% suite() -> -%% [{ct_hooks, [{cth_conn_log, [{conn_mod(),hook_options()}]}]}]. -%%''' -%% -%% The `conn_mod()' is the name of the common_test module implementing -%% the connection protocol, e.g. `ct_netconfc'. -%% -%% The hook option `log_type' specifies the type of logging: -%% -%% <dl> -%% <dt>`raw'</dt> -%% <dd>The sent and received netconf data is logged to a separate -%% text file as is without any formatting. A link to the file is -%% added to the test case HTML log.</dd> -%% -%% <dt>`pretty'</dt> -%% <dd>The sent and received netconf data is logged to a separate -%% text file with XML data nicely indented. A link to the file is -%% added to the test case HTML log.</dd> -%% -%% <dt>`html (default)'</dt> -%% <dd>The sent and received netconf traffic is pretty printed -%% directly in the test case HTML log.</dd> -%% -%% <dt>`silent'</dt> -%% <dd>Netconf traffic is not logged.</dd> -%% </dl> -%% -%% By default, all netconf traffic is logged in one single log -%% file. However, it is possible to have different connections logged -%% in separate files. To do this, use the hook option `hosts' and -%% list the names of the servers/connections that will be used in the -%% suite. Note that the connections must be named for this to work, -%% i.e. they must be opened with {@link open/2}. -%% -%% The `hosts' option has no effect if `log_type' is set to `html' or -%% `silent'. -%% -%% The hook options can also be specified in a configuration file with -%% the configuration variable `ct_conn_log': -%% -%% ``` -%% {ct_conn_log,[{conn_mod(),hook_options()}]}. -%% ''' +%% [{ct_hooks, [{cth_conn_log, [{ct:conn_log_mod(),ct:conn_log_options()}]}]}]. %% %% For example: %% -%% ``` -%% {ct_conn_log,[{ct_netconfc,[{log_type,pretty}, -%% {hosts,[key_or_name()]}]}]} -%% ''' -%% -%% <b>Note</b> that hook options specified in a configuration file -%% will overwrite the hardcoded hook options in the test suite. -%% -%% === Logging example 1 === -%% -%% The following `ct_hooks' statement will cause pretty printing of -%% netconf traffic to separate logs for the connections named -%% `nc_server1' and `nc_server2'. Any other connections will be logged -%% to default netconf log. -%% -%% ``` %% suite() -> -%% [{ct_hooks, [{cth_conn_log, [{ct_netconfc,[{log_type,pretty}}, -%% {hosts,[nc_server1,nc_server2]}]} -%% ]}]}]. -%%''' -%% -%% Connections must be opened like this: -%% -%% ``` -%% open(nc_server1,[...]), -%% open(nc_server2,[...]). -%% ''' -%% -%% === Logging example 2 === -%% -%% The following configuration file will cause raw logging of all -%% netconf traffic into one single text file. -%% -%% ``` -%% {ct_conn_log,[{ct_netconfc,[{log_type,raw}]}]}. -%% ''' -%% -%% The `ct_hooks' statement must look like this: -%% -%% ``` -%% suite() -> -%% [{ct_hooks, [{cth_conn_log, []}]}]. -%% ''' -%% -%% The same `ct_hooks' statement without the configuration file would -%% cause HTML logging of all netconf connections into the test case -%% HTML log. +%% [{ct_hooks, +%% [{cth_conn_log,[{ct_netconfc,[{log_type,pretty}, +%% {hosts,[my_configured_server]}]}]} %% %% == Notifications == %% @@ -152,11 +56,9 @@ %% Notifications, which defines a mechanism for an asynchronous %% message notification delivery service for the netconf protocol. %% -%% Specific functions to support this are {@link -%% create_subscription/6} and {@link get_event_streams/3}. (The -%% functions also exist with other arities.) +%% Specific functions to support this are create_subscription/6 +%% get_event_streams/3. (The functions also exist with other arities.) %% -%% @end %%---------------------------------------------------------------------- -module(ct_netconfc). @@ -167,7 +69,13 @@ %%---------------------------------------------------------------------- %% External exports %%---------------------------------------------------------------------- --export([open/1, +-export([connect/1, + connect/2, + disconnect/1, + session/1, + session/2, + session/3, + open/1, open/2, only_open/1, only_open/2, @@ -205,6 +113,7 @@ create_subscription/4, create_subscription/5, create_subscription/6, + get_event_streams/1, get_event_streams/2, get_event_streams/3, get_capabilities/1, @@ -215,7 +124,9 @@ %%---------------------------------------------------------------------- %% Exported types %%---------------------------------------------------------------------- --export_type([notification/0]). +-export_type([client/0, + handle/0, + notification/0]). %%---------------------------------------------------------------------- %% Internal exports @@ -273,13 +184,15 @@ host, port = ?DEFAULT_PORT, timeout = ?DEFAULT_TIMEOUT, - name}). + name, + type}). %% Connection reference -record(connection, {reference, % {CM,Ch} host, port, - name}). + name, + type}). %% Pending replies from server -record(pending, {tref, % timer ref (returned from timer:xxx) @@ -291,17 +204,17 @@ %%---------------------------------------------------------------------- %% Type declarations %%---------------------------------------------------------------------- --type client() :: handle() | ct_gen_conn:server_id() | ct_gen_conn:target_name(). --type handle() :: term(). -%% An opaque reference for a connection (netconf session). See {@link -%% ct} for more information. +-type client() :: handle() | server_id() | ct:target_name(). +-opaque handle() :: pid(). -type options() :: [option()]. -%% Options used for setting up ssh connection to a netconf server. - -type option() :: {ssh,host()} | {port,inet:port_number()} | {user,string()} | {password,string()} | {user_dir,string()} | {timeout,timeout()}. + +-type session_options() :: [session_option()]. +-type session_option() :: {timeout,timeout()}. + -type host() :: inet:hostname() | inet:ip_address(). -type notification() :: {notification, xml_attributes(), notification_content()}. @@ -317,14 +230,13 @@ %% See XML Schema for Event Notifications found in RFC5277 for further %% detail about the data format for the string values. -%-type error_handler() :: module(). -type error_reason() :: term(). +-type server_id() :: atom(). + -type simple_xml() :: {xml_tag(), xml_attributes(), xml_content()} | {xml_tag(), xml_content()} | xml_tag(). -%% <p>This type is further described in the documentation for the -%% <tt>Xmerl</tt> application.</p> -type xml_tag() :: atom(). -type xml_attributes() :: [{xml_attribute_tag(),xml_attribute_value()}]. -type xml_attribute_tag() :: atom(). @@ -336,69 +248,130 @@ -type xs_datetime() :: string(). %% This date and time identifyer has the same format as the XML type %% dateTime and compliant to RFC3339. The format is -%% ```[-]CCYY-MM-DDThh:mm:ss[.s][Z|(+|-)hh:mm]''' +%% "[-]CCYY-MM-DDThh:mm:ss[.s][Z|(+|-)hh:mm]" %%---------------------------------------------------------------------- %% External interface functions %%---------------------------------------------------------------------- %%---------------------------------------------------------------------- --spec open(Options) -> Result when +%% Open an SSH connection to a Netconf server +%% If the server options are specified in a configuration file, use +%% open/2. +-spec connect(Options) -> Result when Options :: options(), Result :: {ok,handle()} | {error,error_reason()}. -%% @doc Open a netconf session and exchange `hello' messages. -%% -%% If the server options are specified in a configuration file, or if -%% a named client is needed for logging purposes (see {@section -%% Logging}) use {@link open/2} instead. -%% -%% The opaque `handler()' reference which is returned from this -%% function is required as client identifier when calling any other -%% function in this module. -%% -%% The `timeout' option (milli seconds) is used when setting up -%% the ssh connection and when waiting for the hello message from the -%% server. It is not used for any other purposes during the lifetime -%% of the connection. -%% -%% @end +connect(Options) -> + do_connect(Options, #options{type=connection},[]). + +-spec connect(KeyOrName,ExtraOptions) -> Result when + KeyOrName :: ct:key_or_name(), + ExtraOptions :: options(), + Result :: {ok,handle()} | {error,error_reason()}. +connect(KeyOrName, ExtraOptions) -> + SortedExtra = lists:keysort(1,ExtraOptions), + SortedConfig = lists:keysort(1,ct:get_config(KeyOrName,[])), + AllOpts = lists:ukeymerge(1,SortedConfig,SortedExtra), + do_connect(AllOpts,#options{name=KeyOrName,type=connection},[{name,KeyOrName}]). + +do_connect(OptList,InitOptRec,NameOpt) -> + case check_options(OptList,InitOptRec) of + {Host,Port,Options} -> + ct_gen_conn:start({Host,Port},Options,?MODULE, + NameOpt ++ [{reconnect,false}, + {use_existing_connection,false}, + {forward_messages,false}]); + Error -> + Error + end. + %%---------------------------------------------------------------------- -open(Options) -> - open(Options,#options{},[],true). +%% Close the given SSH connection. +-spec disconnect(Conn) -> ok | {error,error_reason()} when + Conn :: handle(). +disconnect(Conn) -> + case call(Conn,get_ssh_connection) of + {ok,_} -> + ct_gen_conn:stop(Conn); + Error -> + Error + end. %%---------------------------------------------------------------------- +%% Open a netconf session as a channel on the given SSH connection, +%% and exchange `hello' messages. +-spec session(Conn) -> Result when + Conn :: handle(), + Result :: {ok,handle()} | {error,error_reason()}. +session(Conn) -> + do_session(Conn,[],#options{type=channel},[]). + +-spec session(Conn,Options) -> Result when + Conn :: handle(), + Options :: session_options(), + Result :: {ok,handle()} | {error,error_reason()}; + (KeyOrName,Conn) -> Result when + KeyOrName :: ct:key_or_name(), + Conn :: handle(), + Result :: {ok,handle()} | {error,error_reason()}. +session(Conn,Options) when is_list(Options) -> + do_session(Conn,Options,#options{type=channel},[]); +session(KeyOrName,Conn) -> + do_session(Conn,[],#options{name=KeyOrName,type=channel},[{name,KeyOrName}]). + +-spec session(KeyOrName,Conn,Options) -> Result when + Conn :: handle(), + Options :: session_options(), + KeyOrName :: ct:key_or_name(), + Result :: {ok,handle()} | {error,error_reason()}. +session(KeyOrName,Conn,ExtraOptions) -> + SortedExtra = lists:keysort(1,ExtraOptions), + SortedConfig = lists:keysort(1,ct:get_config(KeyOrName,[])), + AllOpts = lists:ukeymerge(1,SortedConfig,SortedExtra), + do_session(Conn,AllOpts,#options{name=KeyOrName,type=channel}, + [{name,KeyOrName}]). + +do_session(Conn,OptList,InitOptRec,NameOpt) -> + case call(Conn,get_ssh_connection) of + {ok,SshConn} -> + case check_session_options(OptList,InitOptRec) of + {ok,Options} -> + case ct_gen_conn:start(SshConn,Options,?MODULE, + NameOpt ++ + [{reconnect,false}, + {use_existing_connection,false}, + {forward_messages,true}]) of + {ok,Client} -> + case hello(Client,Options#options.timeout) of + ok -> + {ok,Client}; + Error -> + Error + end; + Error -> + Error + end; + Error -> + Error + end; + Error -> + Error + end. + +%%---------------------------------------------------------------------- +%% Open a netconf session and exchange 'hello' messages. +%% If the server options are specified in a configuration file, use +%% open/2. +-spec open(Options) -> Result when + Options :: options(), + Result :: {ok,handle()} | {error,error_reason()}. +open(Options) -> + open(Options,#options{type=connection_and_channel},[],true). + -spec open(KeyOrName, ExtraOptions) -> Result when - KeyOrName :: ct_gen_conn:key_or_name(), + KeyOrName :: ct:key_or_name(), ExtraOptions :: options(), Result :: {ok,handle()} | {error,error_reason()}. -%% @doc Open a named netconf session and exchange `hello' messages. -%% -%% If `KeyOrName' is a configured `server_id()' or a -%% `target_name()' associated with such an ID, then the options -%% for this server will be fetched from the configuration file. -% -%% The `ExtraOptions' argument will be added to the options found in -%% the configuration file. If the same options are given, the values -%% from the configuration file will overwrite `ExtraOptions'. -%% -%% If the server is not specified in a configuration file, use {@link -%% open/1} instead. -%% -%% The opaque `handle()' reference which is returned from this -%% function can be used as client identifier when calling any other -%% function in this module. However, if `KeyOrName' is a -%% `target_name()', i.e. if the server is named via a call to -%% `ct:require/2' or a `require' statement in the test -%% suite, then this name may be used instead of the `handle()'. -%% -%% The `timeout' option (milli seconds) is used when setting up -%% the ssh connection and when waiting for the hello message from the -%% server. It is not used for any other purposes during the lifetime -%% of the connection. -%% -%% @see ct:require/2 -%% @end -%%---------------------------------------------------------------------- open(KeyOrName, ExtraOpts) -> open(KeyOrName, ExtraOpts, true). @@ -406,10 +379,11 @@ open(KeyOrName, ExtraOpts, Hello) -> SortedExtra = lists:keysort(1,ExtraOpts), SortedConfig = lists:keysort(1,ct:get_config(KeyOrName,[])), AllOpts = lists:ukeymerge(1,SortedConfig,SortedExtra), - open(AllOpts,#options{name=KeyOrName},[{name,KeyOrName}],Hello). + open(AllOpts,#options{name=KeyOrName,type=connection_and_channel}, + [{name,KeyOrName}],Hello). open(OptList,InitOptRec,NameOpt,Hello) -> - case check_options(OptList,undefined,undefined,InitOptRec) of + case check_options(OptList,InitOptRec) of {Host,Port,Options} -> case ct_gen_conn:start({Host,Port},Options,?MODULE, NameOpt ++ [{reconnect,false}, @@ -431,296 +405,206 @@ open(OptList,InitOptRec,NameOpt,Hello) -> %%---------------------------------------------------------------------- +%% As open/1,2, except no 'hello' message is sent. -spec only_open(Options) -> Result when Options :: options(), Result :: {ok,handle()} | {error,error_reason()}. -%% @doc Open a netconf session, but don't send `hello'. -%% -%% As {@link open/1} but does not send a `hello' message. -%% -%% @end -%%---------------------------------------------------------------------- only_open(Options) -> - open(Options,#options{},[],false). + open(Options,#options{type=connection_and_channel},[],false). -%%---------------------------------------------------------------------- -spec only_open(KeyOrName,ExtraOptions) -> Result when - KeyOrName :: ct_gen_conn:key_or_name(), + KeyOrName :: ct:key_or_name(), ExtraOptions :: options(), Result :: {ok,handle()} | {error,error_reason()}. -%% @doc Open a name netconf session, but don't send `hello'. -%% -%% As {@link open/2} but does not send a `hello' message. -%% -%% @end -%%---------------------------------------------------------------------- only_open(KeyOrName, ExtraOpts) -> open(KeyOrName, ExtraOpts, false). %%---------------------------------------------------------------------- -%% @spec hello(Client) -> Result -%% @equiv hello(Client, [], infinity) +%% Send a 'hello' message. +-spec hello(Client) -> Result when + Client :: handle(), + Result :: ok | {error,error_reason()}. hello(Client) -> hello(Client,[],?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec hello(Client,Timeout) -> Result when Client :: handle(), Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @spec hello(Client, Timeout) -> Result -%% @equiv hello(Client, [], Timeout) hello(Client,Timeout) -> hello(Client,[],Timeout). -%%---------------------------------------------------------------------- -spec hello(Client,Options,Timeout) -> Result when Client :: handle(), Options :: [{capability, [string()]}], Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Exchange `hello' messages with the server. -%% -%% Adds optional capabilities and sends a `hello' message to the -%% server and waits for the return. -%% @end -%%---------------------------------------------------------------------- hello(Client,Options,Timeout) -> call(Client, {hello, Options, Timeout}). %%---------------------------------------------------------------------- -%% @spec get_session_id(Client) -> Result -%% @equiv get_session_id(Client, infinity) +%% Get the session id for the session specified by Client. +-spec get_session_id(Client) -> Result when + Client :: client(), + Result :: pos_integer() | {error,error_reason()}. get_session_id(Client) -> get_session_id(Client, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec get_session_id(Client, Timeout) -> Result when Client :: client(), Timeout :: timeout(), Result :: pos_integer() | {error,error_reason()}. -%% @doc Returns the session id associated with the given client. -%% -%% @end -%%---------------------------------------------------------------------- get_session_id(Client, Timeout) -> call(Client, get_session_id, Timeout). %%---------------------------------------------------------------------- -%% @spec get_capabilities(Client) -> Result -%% @equiv get_capabilities(Client, infinity) +%% Get the server side capabilities. +-spec get_capabilities(Client) -> Result when + Client :: client(), + Result :: [string()] | {error,error_reason()}. get_capabilities(Client) -> get_capabilities(Client, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec get_capabilities(Client, Timeout) -> Result when Client :: client(), Timeout :: timeout(), Result :: [string()] | {error,error_reason()}. -%% @doc Returns the server side capabilities -%% -%% The following capability identifiers, defined in RFC 4741, can be returned: -%% -%% <ul> -%% <li>`"urn:ietf:params:netconf:base:1.0"'</li> -%% <li>`"urn:ietf:params:netconf:capability:writable-running:1.0"'</li> -%% <li>`"urn:ietf:params:netconf:capability:candidate:1.0"'</li> -%% <li>`"urn:ietf:params:netconf:capability:confirmed-commit:1.0"'</li> -%% <li>`"urn:ietf:params:netconf:capability:rollback-on-error:1.0"'</li> -%% <li>`"urn:ietf:params:netconf:capability:startup:1.0"'</li> -%% <li>`"urn:ietf:params:netconf:capability:url:1.0"'</li> -%% <li>`"urn:ietf:params:netconf:capability:xpath:1.0"'</li> -%% </ul> -%% -%% Note, additional identifiers may exist, e.g. server side namespace. -%% -%% @end -%%---------------------------------------------------------------------- get_capabilities(Client, Timeout) -> call(Client, get_capabilities, Timeout). %%---------------------------------------------------------------------- -%% @spec send(Client, SimpleXml) -> Result -%% @equiv send(Client, SimpleXml, infinity) +%% Send an XML document to the server. +-spec send(Client, SimpleXml) -> Result when + Client :: client(), + SimpleXml :: simple_xml(), + Result :: simple_xml() | {error,error_reason()}. send(Client, SimpleXml) -> send(Client, SimpleXml, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec send(Client, SimpleXml, Timeout) -> Result when Client :: client(), SimpleXml :: simple_xml(), Timeout :: timeout(), Result :: simple_xml() | {error,error_reason()}. -%% @doc Send an XML document to the server. -%% -%% The given XML document is sent as is to the server. This function -%% can be used for sending XML documents that can not be expressed by -%% other interface functions in this module. send(Client, SimpleXml, Timeout) -> call(Client,{send, Timeout, SimpleXml}). %%---------------------------------------------------------------------- -%% @spec send_rpc(Client, SimpleXml) -> Result -%% @equiv send_rpc(Client, SimpleXml, infinity) +%% Wrap the given XML document in a valid netconf 'rpc' request and +%% send to the server. +-spec send_rpc(Client, SimpleXml) -> Result when + Client :: client(), + SimpleXml :: simple_xml(), + Result :: [simple_xml()] | {error,error_reason()}. send_rpc(Client, SimpleXml) -> send_rpc(Client, SimpleXml, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec send_rpc(Client, SimpleXml, Timeout) -> Result when Client :: client(), SimpleXml :: simple_xml(), Timeout :: timeout(), Result :: [simple_xml()] | {error,error_reason()}. -%% @doc Send a Netconf <code>rpc</code> request to the server. -%% -%% The given XML document is wrapped in a valid Netconf -%% <code>rpc</code> request and sent to the server. The -%% <code>message-id</code> and namespace attributes are added to the -%% <code>rpc</code> element. -%% -%% This function can be used for sending <code>rpc</code> requests -%% that can not be expressed by other interface functions in this -%% module. send_rpc(Client, SimpleXml, Timeout) -> call(Client,{send_rpc, SimpleXml, Timeout}). %%---------------------------------------------------------------------- -%% @spec lock(Client, Target) -> Result -%% @equiv lock(Client, Target, infinity) +%% Send a 'lock' request. +-spec lock(Client, Target) -> Result when + Client :: client(), + Target :: netconf_db(), + Result :: ok | {error,error_reason()}. lock(Client, Target) -> lock(Client, Target,?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec lock(Client, Target, Timeout) -> Result when Client :: client(), Target :: netconf_db(), Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Unlock configuration target. -%% -%% Which target parameters that can be used depends on if -%% `:candidate' and/or `:startup' are supported by the -%% server. If successfull, the configuration system of the device is -%% not available to other clients (Netconf, CORBA, SNMP etc). Locks -%% are intended to be short-lived. -%% -%% The operations {@link kill_session/2} or {@link kill_session/3} can -%% be used to force the release of a lock owned by another Netconf -%% session. How this is achieved by the server side is implementation -%% specific. -%% -%% @end -%%---------------------------------------------------------------------- lock(Client, Target, Timeout) -> call(Client,{send_rpc_op,lock,[Target],Timeout}). %%---------------------------------------------------------------------- -%% @spec unlock(Client, Target) -> Result -%% @equiv unlock(Client, Target, infinity) +%% Send a 'unlock' request. +-spec unlock(Client, Target) -> Result when + Client :: client(), + Target :: netconf_db(), + Result :: ok | {error,error_reason()}. unlock(Client, Target) -> unlock(Client, Target,?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec unlock(Client, Target, Timeout) -> Result when Client :: client(), Target :: netconf_db(), Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Unlock configuration target. -%% -%% If the client earlier has aquired a lock, via {@link lock/2} or -%% {@link lock/3}, this operation release the associated lock. To be -%% able to access another target than `running', the server must -%% support `:candidate' and/or `:startup'. -%% -%% @end -%%---------------------------------------------------------------------- unlock(Client, Target, Timeout) -> call(Client, {send_rpc_op, unlock, [Target], Timeout}). %%---------------------------------------------------------------------- -%% @spec get(Client, Filter) -> Result -%% @equiv get(Client, Filter, infinity) +%% Send a 'get' request. +-spec get(Client, Filter) -> Result when + Client :: client(), + Filter :: simple_xml() | xpath(), + Result :: {ok,[simple_xml()]} | {error,error_reason()}. get(Client, Filter) -> get(Client, Filter, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec get(Client, Filter, Timeout) -> Result when Client :: client(), Filter :: simple_xml() | xpath(), Timeout :: timeout(), Result :: {ok,[simple_xml()]} | {error,error_reason()}. -%% @doc Get data. -%% -%% This operation returns both configuration and state data from the -%% server. -%% -%% Filter type `xpath' can only be used if the server supports -%% `:xpath'. -%% -%% @end -%%---------------------------------------------------------------------- get(Client, Filter, Timeout) -> call(Client,{send_rpc_op, get, [Filter], Timeout}). %%---------------------------------------------------------------------- -%% @spec get_config(Client, Source, Filter) -> Result -%% @equiv get_config(Client, Source, Filter, infinity) +%% Send a 'get-config' request. +-spec get_config(Client, Source, Filter) -> Result when + Client :: client(), + Source :: netconf_db(), + Filter :: simple_xml() | xpath(), + Result :: {ok,[simple_xml()]} | {error,error_reason()}. get_config(Client, Source, Filter) -> get_config(Client, Source, Filter, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec get_config(Client, Source, Filter, Timeout) -> Result when Client :: client(), Source :: netconf_db(), Filter :: simple_xml() | xpath(), Timeout :: timeout(), Result :: {ok,[simple_xml()]} | {error,error_reason()}. -%% @doc Get configuration data. -%% -%% To be able to access another source than `running', the server -%% must advertise `:candidate' and/or `:startup'. -%% -%% Filter type `xpath' can only be used if the server supports -%% `:xpath'. -%% -%% -%% @end -%%---------------------------------------------------------------------- get_config(Client, Source, Filter, Timeout) -> call(Client, {send_rpc_op, get_config, [Source, Filter], Timeout}). %%---------------------------------------------------------------------- -%% @spec edit_config(Client, Target, Config) -> Result -%% @equiv edit_config(Client, Target, Config, [], infinity) +%% Send a 'edit-config' request. +-spec edit_config(Client, Target, Config) -> Result when + Client :: client(), + Target :: netconf_db(), + Config :: simple_xml(), + Result :: ok | {error,error_reason()}. edit_config(Client, Target, Config) -> edit_config(Client, Target, Config, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- --spec edit_config(Client, Target, Config, OptParamsOrTimeout) -> Result when +-spec edit_config(Client, Target, Config, OptParams) -> Result when + Client :: client(), + Target :: netconf_db(), + Config :: simple_xml(), + OptParams :: [simple_xml()], + Result :: ok | {error,error_reason()}; + (Client, Target, Config, Timeout) -> Result when Client :: client(), Target :: netconf_db(), Config :: simple_xml(), - OptParamsOrTimeout :: [simple_xml()] | timeout(), + Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc -%% -%% If `OptParamsOrTimeout' is a timeout value, then this is -%% equivalent to {@link edit_config/5. edit_config(Client, Target, -%% Config, [], Timeout)}. -%% -%% If `OptParamsOrTimeout' is a list of simple XML, then this is -%% equivalent to {@link edit_config/5. edit_config(Client, Target, -%% Config, OptParams, infinity)}. -%% -%% @end edit_config(Client, Target, Config, Timeout) when ?is_timeout(Timeout) -> edit_config(Client, Target, Config, [], Timeout); edit_config(Client, Target, Config, OptParams) when is_list(OptParams) -> edit_config(Client, Target, Config, OptParams, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec edit_config(Client, Target, Config, OptParams, Timeout) -> Result when Client :: client(), Target :: netconf_db(), @@ -728,99 +612,79 @@ edit_config(Client, Target, Config, OptParams) when is_list(OptParams) -> OptParams :: [simple_xml()], Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Edit configuration data. -%% -%% Per default only the running target is available, unless the server -%% include `:candidate' or `:startup' in its list of -%% capabilities. -%% -%% `OptParams' can be used for specifying optional parameters -%% (`default-operation', `test-option' or `error-option') that will be -%% added to the `edit-config' request. The value must be a list -%% containing valid simple XML, for example -%% -%% ``` -%% [{'default-operation', ["none"]}, -%% {'error-option', ["rollback-on-error"]}] -%%''' -%% -%% @end -%%---------------------------------------------------------------------- edit_config(Client, Target, Config, OptParams, Timeout) -> call(Client, {send_rpc_op, edit_config, [Target,Config,OptParams], Timeout}). %%---------------------------------------------------------------------- -%% @spec delete_config(Client, Target) -> Result -%% @equiv delete_config(Client, Target, infinity) +%% Send a 'delete-config' request. +-spec delete_config(Client, Target) -> Result when + Client :: client(), + Target :: startup | candidate, + Result :: ok | {error,error_reason()}. delete_config(Client, Target) -> delete_config(Client, Target, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec delete_config(Client, Target, Timeout) -> Result when Client :: client(), Target :: startup | candidate, Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Delete configuration data. -%% -%% The running configuration cannot be deleted and `:candidate' -%% or `:startup' must be advertised by the server. -%% -%% @end -%%---------------------------------------------------------------------- delete_config(Client, Target, Timeout) when Target == startup; Target == candidate -> call(Client,{send_rpc_op, delete_config, [Target], Timeout}). %%---------------------------------------------------------------------- -%% @spec copy_config(Client, Source, Target) -> Result -%% @equiv copy_config(Client, Source, Target, infinity) +%% Send a 'copy-config' request. +-spec copy_config(Client, Target, Source) -> Result when + Client :: client(), + Target :: netconf_db(), + Source :: netconf_db(), + Result :: ok | {error,error_reason()}. copy_config(Client, Source, Target) -> copy_config(Client, Source, Target, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec copy_config(Client, Target, Source, Timeout) -> Result when Client :: client(), Target :: netconf_db(), Source :: netconf_db(), Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Copy configuration data. -%% -%% Which source and target options that can be issued depends on the -%% capabilities supported by the server. I.e. `:candidate' and/or -%% `:startup' are required. -%% -%% @end -%%---------------------------------------------------------------------- copy_config(Client, Target, Source, Timeout) -> call(Client,{send_rpc_op, copy_config, [Target, Source], Timeout}). %%---------------------------------------------------------------------- -%% @spec action(Client, Action) -> Result -%% @equiv action(Client, Action, infinity) +%% Execute an action. +-spec action(Client, Action) -> Result when + Client :: client(), + Action :: simple_xml(), + Result :: ok | {ok,[simple_xml()]} | {error,error_reason()}. action(Client,Action) -> action(Client,Action,?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec action(Client, Action, Timeout) -> Result when Client :: client(), Action :: simple_xml(), Timeout :: timeout(), Result :: ok | {ok,[simple_xml()]} | {error,error_reason()}. -%% @doc Execute an action. If the return type is void, <c>ok</c> will -%% be returned instead of <c>{ok,[simple_xml()]}</c>. -%% -%% @end -%%---------------------------------------------------------------------- action(Client,Action,Timeout) -> call(Client,{send_rpc_op, action, [Action], Timeout}). %%---------------------------------------------------------------------- +%% Send a 'create-subscription' request +%% See RFC5277, NETCONF Event Notifications +-spec create_subscription(Client) -> Result when + Client :: client(), + Result :: ok | {error,error_reason()}. create_subscription(Client) -> create_subscription(Client,?DEFAULT_STREAM,?DEFAULT_TIMEOUT). +-spec create_subscription(Client, Stream | Filter | Timeout) -> Result when + Client :: client(), + Stream :: stream_name(), + Filter :: simple_xml() | [simple_xml()], + Timeout :: timeout(), + Result :: ok | {error,error_reason()}. create_subscription(Client,Timeout) when ?is_timeout(Timeout) -> create_subscription(Client,?DEFAULT_STREAM,Timeout); @@ -876,6 +740,22 @@ create_subscription(Client,Stream,Filter,Timeout) [Stream,Filter,undefined,undefined], Timeout}). +-spec create_subscription(Client, Stream, StartTime, StopTime, Timeout) -> + Result when + Client :: client(), + Stream :: stream_name(), + StartTime :: xs_datetime(), + StopTime :: xs_datetime(), + Timeout :: timeout(), + Result :: ok | {error,error_reason()}; + (Client, Stream, Filter,StartTime, StopTime) -> + Result when + Client :: client(), + Stream :: stream_name(), + Filter :: simple_xml() | [simple_xml()], + StartTime :: xs_datetime(), + StopTime :: xs_datetime(), + Result :: ok | {error,error_reason()}. create_subscription(Client,Stream,StartTime,StopTime,Timeout) when ?is_string(Stream) andalso ?is_string(StartTime) andalso @@ -891,7 +771,6 @@ create_subscription(Client,Stream,Filter,StartTime,StopTime) ?is_string(StopTime) -> create_subscription(Client,Stream,Filter,StartTime,StopTime,?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec create_subscription(Client, Stream, Filter,StartTime, StopTime, Timeout) -> Result when Client :: client(), @@ -901,168 +780,75 @@ create_subscription(Client,Stream,Filter,StartTime,StopTime) StopTime :: xs_datetime(), Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Create a subscription for event notifications. -%% -%% This function sets up a subscription for netconf event -%% notifications of the given stream type, matching the given -%% filter. The calling process will receive notifications as messages -%% of type `notification()'. -%% -%% <dl> -%% <dt>Stream:</dt> -%% <dd> An optional parameter that indicates which stream of events -%% is of interest. If not present, events in the default NETCONF -%% stream will be sent.</dd> -%% -%% <dt>Filter:</dt> -%% <dd>An optional parameter that indicates which subset of all -%% possible events is of interest. The format of this parameter is -%% the same as that of the filter parameter in the NETCONF protocol -%% operations. If not present, all events not precluded by other -%% parameters will be sent.</dd> -%% -%% <dt>StartTime:</dt> -%% <dd>An optional parameter used to trigger the replay feature and -%% indicate that the replay should start at the time specified. If -%% `StartTime' is not present, this is not a replay subscription. -%% It is not valid to specify start times that are later than the -%% current time. If the `StartTime' specified is earlier than the -%% log can support, the replay will begin with the earliest -%% available notification. This parameter is of type dateTime and -%% compliant to [RFC3339]. Implementations must support time -%% zones.</dd> -%% -%% <dt>StopTime:</dt> -%% <dd>An optional parameter used with the optional replay feature -%% to indicate the newest notifications of interest. If `StopTime' -%% is not present, the notifications will continue until the -%% subscription is terminated. Must be used with and be later than -%% `StartTime'. Values of `StopTime' in the future are valid. This -%% parameter is of type dateTime and compliant to [RFC3339]. -%% Implementations must support time zones.</dd> -%% </dl> -%% -%% See RFC5277 for further details about the event notification -%% mechanism. -%% -%% @end -%%---------------------------------------------------------------------- create_subscription(Client,Stream,Filter,StartTime,StopTime,Timeout) -> call(Client,{send_rpc_op,{create_subscription, self()}, [Stream,Filter,StartTime,StopTime], Timeout}). %%---------------------------------------------------------------------- -%% @spec get_event_streams(Client, Timeout) -> Result -%% @equiv get_event_streams(Client, [], Timeout) +%% Send a request to get the given event streams +%% See RFC5277, NETCONF Event Notifications +-spec get_event_streams(Client) + -> Result when + Client :: client(), + Result :: {ok,streams()} | {error,error_reason()}. +get_event_streams(Client) -> + get_event_streams(Client,[],?DEFAULT_TIMEOUT). + +-spec get_event_streams(Client, Timeout) + -> Result when + Client :: client(), + Timeout :: timeout(), + Result :: {ok,streams()} | {error,error_reason()}; + (Client, Streams) -> Result when + Client :: client(), + Streams :: [stream_name()], + Result :: {ok,streams()} | {error,error_reason()}. get_event_streams(Client,Timeout) when is_integer(Timeout); Timeout==infinity -> get_event_streams(Client,[],Timeout); - -%%---------------------------------------------------------------------- -%% @spec get_event_streams(Client, Streams) -> Result -%% @equiv get_event_streams(Client, Streams, infinity) get_event_streams(Client,Streams) when is_list(Streams) -> get_event_streams(Client,Streams,?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec get_event_streams(Client, Streams, Timeout) -> Result when Client :: client(), Streams :: [stream_name()], Timeout :: timeout(), Result :: {ok,streams()} | {error,error_reason()}. -%% @doc Send a request to get the given event streams. -%% -%% `Streams' is a list of stream names. The following filter will -%% be sent to the netconf server in a `get' request: -%% -%% ``` -%% <netconf xmlns="urn:ietf:params:xml:ns:netmod:notification"> -%% <streams> -%% <stream> -%% <name>StreamName1</name> -%% </stream> -%% <stream> -%% <name>StreamName2</name> -%% </stream> -%% ... -%% </streams> -%% </netconf> -%% ''' -%% -%% If `Streams' is an empty list, ALL streams will be requested -%% by sending the following filter: -%% -%% ``` -%% <netconf xmlns="urn:ietf:params:xml:ns:netmod:notification"> -%% <streams/> -%% </netconf> -%% ''' -%% -%% If more complex filtering is needed, a use {@link get/2} or {@link -%% get/3} and specify the exact filter according to XML Schema for -%% Event Notifications found in RFC5277. -%% -%% @end -%%---------------------------------------------------------------------- get_event_streams(Client,Streams,Timeout) -> call(Client,{get_event_streams,Streams,Timeout}). %%---------------------------------------------------------------------- -%% @spec close_session(Client) -> Result -%% @equiv close_session(Client, infinity) +%% Send a 'close-session' request +-spec close_session(Client) -> Result when + Client :: client(), + Result :: ok | {error,error_reason()}. close_session(Client) -> close_session(Client, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec close_session(Client, Timeout) -> Result when Client :: client(), Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Request graceful termination of the session associated with the client. -%% -%% When a netconf server receives a `close-session' request, it -%% will gracefully close the session. The server will release any -%% locks and resources associated with the session and gracefully -%% close any associated connections. Any NETCONF requests received -%% after a `close-session' request will be ignored. -%% -%% @end -%%---------------------------------------------------------------------- close_session(Client, Timeout) -> call(Client,{send_rpc_op, close_session, [], Timeout}, true). %%---------------------------------------------------------------------- -%% @spec kill_session(Client, SessionId) -> Result -%% @equiv kill_session(Client, SessionId, infinity) +%% Send a 'kill-session' request +-spec kill_session(Client, SessionId) -> Result when + Client :: client(), + SessionId :: pos_integer(), + Result :: ok | {error,error_reason()}. kill_session(Client, SessionId) -> kill_session(Client, SessionId, ?DEFAULT_TIMEOUT). -%%---------------------------------------------------------------------- -spec kill_session(Client, SessionId, Timeout) -> Result when Client :: client(), SessionId :: pos_integer(), Timeout :: timeout(), Result :: ok | {error,error_reason()}. -%% @doc Force termination of the session associated with the supplied -%% session id. -%% -%% The server side shall abort any operations currently in process, -%% release any locks and resources associated with the session, and -%% close any associated connections. -%% -%% Only if the server is in the confirmed commit phase, the -%% configuration will be restored to its state before entering the -%% confirmed commit phase. Otherwise, no configuration roll back will -%% be performed. -%% -%% If the given `SessionId' is equal to the current session id, -%% an error will be returned. -%% -%% @end -%% ---------------------------------------------------------------------- kill_session(Client, SessionId, Timeout) -> call(Client,{send_rpc_op, kill_session, [SessionId], Timeout}). @@ -1071,24 +857,35 @@ kill_session(Client, SessionId, Timeout) -> %% Callback functions %%---------------------------------------------------------------------- -%% @private +init(_KeyOrName,{CM,{Host,Port}},Options) -> + case ssh_channel(#connection{reference=CM,host=Host,port=Port},Options) of + {ok,Connection} -> + {ok, CM, #state{connection = Connection}}; + {error,Reason}-> + {error,Reason} + end; +init(_KeyOrName,{_Host,_Port},Options) when Options#options.type==connection -> + case ssh_connect(Options) of + {ok, Connection} -> + ConnPid = Connection#connection.reference, + {ok, ConnPid, #state{connection = Connection}}; + Error -> + Error + end; init(_KeyOrName,{_Host,_Port},Options) -> case ssh_open(Options) of {ok, Connection} -> - log(Connection,open), {ConnPid,_} = Connection#connection.reference, {ok, ConnPid, #state{connection = Connection}}; {error,Reason}-> {error,Reason} end. -%% @private + terminate(_, #state{connection=Connection}) -> ssh_close(Connection), - log(Connection,close), ok. -%% @private handle_msg({hello, Options, Timeout}, From, #state{connection=Connection,hello_status=HelloStatus} = State) -> case do_send(Connection, client_hello(Options)) of @@ -1107,6 +904,14 @@ handle_msg({hello, Options, Timeout}, From, Error -> {stop, Error, State} end; +handle_msg(get_ssh_connection, _From, #state{connection=Connection}=State) -> + Reply = + case Connection#connection.reference of + {_,_} -> {error,not_an_ssh_connection}; + CM -> {ok,{CM,{Connection#connection.host, + Connection#connection.port}}} + end, + {reply, Reply, State}; handle_msg(_, _From, #state{session_id=undefined} = State) -> %% Hello is not yet excanged - this shall never happen {reply,{error,waiting_for_hello},State}; @@ -1136,7 +941,6 @@ handle_msg({get_event_streams=Op,Streams,Timeout}, From, State) -> SimpleXml = encode_rpc_operation(get,[Filter]), do_send_rpc(Op, SimpleXml, Timeout, From, State). -%% @private handle_msg({ssh_cm, CM, {data, Ch, _Type, Data}}, State) -> ssh_connection:adjust_window(CM,Ch,size(Data)), handle_data(Data, State); @@ -1172,7 +976,6 @@ handle_msg({Ref,timeout},#state{pending=Pending} = State) -> %% the implementation before this patch {R,State#state{pending=Pending1, no_end_tag_buff= <<>>, buff= <<>>}}. -%% @private %% Called by ct_util_server to close registered connections before terminate. close(Client) -> case get_handle(Client) of @@ -1243,15 +1046,18 @@ get_handle(Client) -> Error end. +check_options(OptList,Options) -> + check_options(OptList,undefined,undefined,Options). + check_options([], undefined, _Port, _Options) -> {error, no_host_address}; check_options([], _Host, undefined, _Options) -> {error, no_port}; check_options([], Host, Port, Options) -> {Host,Port,Options}; -check_options([{ssh, Host}|T], _, Port, #options{} = Options) -> +check_options([{ssh, Host}|T], _, Port, Options) -> check_options(T, Host, Port, Options#options{host=Host}); -check_options([{port,Port}|T], Host, _, #options{} = Options) -> +check_options([{port,Port}|T], Host, _, Options) -> check_options(T, Host, Port, Options#options{port=Port}); check_options([{timeout, Timeout}|T], Host, Port, Options) when is_integer(Timeout); Timeout==infinity -> @@ -1262,6 +1068,15 @@ check_options([Opt|T], Host, Port, #options{ssh=SshOpts}=Options) -> %% Option verified by ssh check_options(T, Host, Port, Options#options{ssh=[Opt|SshOpts]}). +check_session_options([],Options) -> + {ok,Options}; +check_session_options([{timeout, Timeout}|T], Options) + when is_integer(Timeout); Timeout==infinity -> + check_session_options(T, Options#options{timeout = Timeout}); +check_session_options([Opt|_T], _Options) -> + {error, {invalid_option, Opt}}. + + %%%----------------------------------------------------------------- set_request_timer(infinity) -> {undefined,undefined}; @@ -1356,7 +1171,6 @@ do_send_rpc(Connection, MsgId, SimpleXml) -> do_send(Connection, SimpleXml) -> Xml=to_xml_doc(SimpleXml), - log(Connection,send,Xml), ssh_send(Connection, Xml). to_xml_doc(Simple) -> @@ -1766,9 +1580,14 @@ decode_streams([]) -> log(Connection,Action) -> log(Connection,Action,<<>>). -log(#connection{host=Host,port=Port,name=Name},Action,Data) -> +log(#connection{reference=Ref,host=Host,port=Port,name=Name},Action,Data) -> + Address = + case Ref of + {_,Ch} -> {Host,Port,Ch}; + _ -> {Host,Port} + end, error_logger:info_report(#conn_log{client=self(), - address={Host,Port}, + address=Address, name=Name, action=Action, module=?MODULE}, @@ -1776,7 +1595,6 @@ log(#connection{host=Host,port=Port,name=Name},Action,Data) -> %% Log callback - called from the error handler process -%% @private format_data(How,Data) -> %% Assuming that the data is encoded as UTF-8. If it is not, then %% the printout might be wrong, but the format function will not @@ -1915,42 +1733,84 @@ get_tag([]) -> %%%----------------------------------------------------------------- %%% SSH stuff - -ssh_open(#options{host=Host,timeout=Timeout,port=Port,ssh=SshOpts,name=Name}) -> +ssh_connect(#options{host=Host,timeout=Timeout,port=Port, + ssh=SshOpts,name=Name,type=Type}) -> case ssh:connect(Host, Port, [{user_interaction,false}, - {silently_accept_hosts, true}|SshOpts]) of + {silently_accept_hosts, true}|SshOpts], + Timeout) of {ok,CM} -> - case ssh_connection:session_channel(CM, Timeout) of - {ok,Ch} -> - case ssh_connection:subsystem(CM, Ch, "netconf", Timeout) of - success -> - {ok, #connection{reference = {CM,Ch}, - host = Host, - port = Port, - name = Name}}; - failure -> - ssh:close(CM), - {error,{ssh,could_not_execute_netconf_subsystem}}; - {error,timeout} -> - {error,{ssh,could_not_execute_netconf_subsystem,timeout}} - end; - {error, Reason} -> - ssh:close(CM), - {error,{ssh,could_not_open_channel,Reason}} - end; + Connection = #connection{reference = CM, + host = Host, + port = Port, + name = Name, + type = Type}, + log(Connection,connect), + {ok,Connection}; {error,Reason} -> {error,{ssh,could_not_connect_to_server,Reason}} end. -ssh_send(#connection{reference = {CM,Ch}}, Data) -> +ssh_channel(#connection{reference=CM}=Connection0, + #options{timeout=Timeout,name=Name,type=Type}) -> + case ssh_connection:session_channel(CM, Timeout) of + {ok,Ch} -> + case ssh_connection:subsystem(CM, Ch, "netconf", Timeout) of + success -> + Connection = Connection0#connection{reference = {CM,Ch}, + name = Name, + type = Type}, + log(Connection,open), + {ok, Connection}; + failure -> + ssh_connection:close(CM,Ch), + {error,{ssh,could_not_execute_netconf_subsystem}}; + {error,timeout} -> + ssh_connection:close(CM,Ch), + {error,{ssh,could_not_execute_netconf_subsystem,timeout}} + end; + {error, Reason} -> + {error,{ssh,could_not_open_channel,Reason}} + end. + + +ssh_open(Options) -> + case ssh_connect(Options) of + {ok,Connection} -> + case ssh_channel(Connection,Options) of + {ok,_} = Ok -> + Ok; + Error -> + ssh_close(Connection), + Error + end; + Error -> + Error + end. + +ssh_send(#connection{reference = {CM,Ch}}=Connection, Data) -> case ssh_connection:send(CM, Ch, Data) of - ok -> ok; - {error,Reason} -> {error,{ssh,failed_to_send_data,Reason}} + ok -> + log(Connection,send,Data), + ok; + {error,Reason} -> + {error,{ssh,failed_to_send_data,Reason}} end. -ssh_close(#connection{reference = {CM,_Ch}}) -> - ssh:close(CM). +ssh_close(Connection=#connection{reference = {CM,Ch}, type = Type}) -> + _ = ssh_connection:close(CM,Ch), + log(Connection,close), + case Type of + connection_and_channel -> + ssh_close(Connection#connection{reference = CM}); + _ -> + ok + end, + ok; +ssh_close(Connection=#connection{reference = CM}) -> + _ = ssh:close(CM), + log(Connection,disconnect), + ok. %%---------------------------------------------------------------------- diff --git a/lib/common_test/src/ct_run.erl b/lib/common_test/src/ct_run.erl index bbace18a28..ce30babc0d 100644 --- a/lib/common_test/src/ct_run.erl +++ b/lib/common_test/src/ct_run.erl @@ -436,13 +436,17 @@ script_start2(Opts = #opts{vts = undefined, Specs1 = get_start_opt(join_specs, [Specs], Specs, Args), %% using testspec as input for test Relaxed = get_start_opt(allow_user_terms, true, false, Args), - case catch ct_testspec:collect_tests_from_file(Specs1, Relaxed) of - {E,Reason} when E == error ; E == 'EXIT' -> + try ct_testspec:collect_tests_from_file(Specs1, Relaxed) of + TestSpecData -> + execute_all_specs(TestSpecData, Opts, Args, []) + catch + throw:{error,Reason} -> StackTrace = erlang:get_stacktrace(), {error,{invalid_testspec,{Reason,StackTrace}}}; - TestSpecData -> - execute_all_specs(TestSpecData, Opts, Args, []) - end; + _:Reason -> + StackTrace = erlang:get_stacktrace(), + {error,{invalid_testspec,{Reason,StackTrace}}} + end; [] -> {error,no_testspec_specified}; _ -> % no testspec used @@ -1198,12 +1202,16 @@ run_spec_file(Relaxed, end, AbsSpecs = lists:map(fun(SF) -> ?abs(SF) end, Specs1), AbsSpecs1 = get_start_opt(join_specs, [AbsSpecs], AbsSpecs, StartOpts), - case catch ct_testspec:collect_tests_from_file(AbsSpecs1, Relaxed) of - {Error,CTReason} when Error == error ; Error == 'EXIT' -> - StackTrace = erlang:get_stacktrace(), - exit({error,{invalid_testspec,{CTReason,StackTrace}}}); + try ct_testspec:collect_tests_from_file(AbsSpecs1, Relaxed) of TestSpecData -> run_all_specs(TestSpecData, Opts, StartOpts, []) + catch + throw:{error,CTReason} -> + StackTrace = erlang:get_stacktrace(), + exit({error,{invalid_testspec,{CTReason,StackTrace}}}); + _:CTReason -> + StackTrace = erlang:get_stacktrace(), + exit({error,{invalid_testspec,{CTReason,StackTrace}}}) end. run_all_specs([], _, _, TotResult) -> diff --git a/lib/common_test/src/ct_ssh.erl b/lib/common_test/src/ct_ssh.erl index 6ab3bf036c..5ac91f396b 100644 --- a/lib/common_test/src/ct_ssh.erl +++ b/lib/common_test/src/ct_ssh.erl @@ -68,7 +68,8 @@ send_and_receive/3, send_and_receive/4, send_and_receive/5, send_and_receive/6, exec/2, exec/3, exec/4, - subsystem/3, subsystem/4]). + subsystem/3, subsystem/4, + shell/2, shell/3]). %% STFP Functions -export([sftp_connect/1, @@ -94,6 +95,7 @@ -record(state, {ssh_ref, conn_type, target}). +-type handle() :: pid(). %%%----------------------------------------------------------------- %%%------------------------ SSH COMMANDS --------------------------- @@ -490,6 +492,22 @@ subsystem(SSH, ChannelId, Subsystem, Timeout) -> call(SSH, {subsystem,ChannelId,Subsystem,Timeout}). +-spec shell(SSH, ChannelId) -> Result when + SSH :: handle() | ct:target_name(), + ChannelId :: ssh:ssh_channel_id(), + Result :: ok | {error,term()}. +shell(SSH, ChannelId) -> + shell(SSH, ChannelId, ?DEFAULT_TIMEOUT). + +-spec shell(SSH, ChannelId, Timeout) -> Result when + SSH :: handle() | ct:target_name(), + ChannelId :: ssh:ssh_channel_id(), + Timeout :: timeout(), + Result :: ok | {error,term()}. +shell(SSH, ChannelId, Timeout) -> + call(SSH, {shell,ChannelId,Timeout}). + + %%%----------------------------------------------------------------- %%%------------------------ SFTP COMMANDS -------------------------- @@ -1067,6 +1085,14 @@ handle_msg({subsystem,Chn,Subsystem,TO}, State) -> Result = ssh_connection:subsystem(SSHRef, Chn, Subsystem, TO), {Result,State}; +handle_msg({shell,Chn,TO}, State) -> + #state{ssh_ref=SSHRef, target=Target} = State, + try_log(heading(shell,Target), + "SSH Ref: ~p, Chn: ~p, Timeout: ~p", + [SSHRef,Chn,TO]), + Result = ssh_connection:shell(SSHRef, Chn), + {Result,State}; + %% --- SFTP Commands --- handle_msg({read_file,Srv,File}=Cmd, S=#state{ssh_ref=SSHRef}) -> diff --git a/lib/common_test/src/ct_util.erl b/lib/common_test/src/ct_util.erl index 4d3a2ae7e3..802e9be97c 100644 --- a/lib/common_test/src/ct_util.erl +++ b/lib/common_test/src/ct_util.erl @@ -201,14 +201,7 @@ do_start(Parent, Mode, LogDir, Verbosity) -> ok -> Parent ! {self(),started}; {fail,CTHReason} -> - ErrorInfo = if is_atom(CTHReason) -> - io_lib:format("{~p,~p}", - [CTHReason, - erlang:get_stacktrace()]); - true -> - CTHReason - end, - ct_logs:tc_print('Suite Callback',ErrorInfo,[]), + ct_logs:tc_print('Suite Callback',CTHReason,[]), self() ! {{stop,{self(),{user_error,CTHReason}}}, {Parent,make_ref()}} catch diff --git a/lib/common_test/src/cth_conn_log.erl b/lib/common_test/src/cth_conn_log.erl index ef92532969..faff150ac9 100644 --- a/lib/common_test/src/cth_conn_log.erl +++ b/lib/common_test/src/cth_conn_log.erl @@ -24,11 +24,11 @@ %% %% suite() -> %% [{ct_hooks, [{cth_conn_log, -%% [{ct_netconfc:conn_mod(),ct_netconfc:hook_options()}]}]}]. +%% [{conn_mod(),hook_options()}]}]}]. %% %% or specified in a configuration file: %% -%% {ct_conn_log,[{ct_netconfc:conn_mod(),ct_netconfc:hook_options()}]}. +%% {ct_conn_log,[{conn_mod(),hook_options()}]}. %% %% The conn_mod() is the common test module implementing the protocol, %% e.g. ct_netconfc, ct_telnet, etc. This module must log by calling @@ -58,28 +58,17 @@ post_end_per_testcase/5]). %%---------------------------------------------------------------------- -%% Exported types -%%---------------------------------------------------------------------- --export_type([hook_options/0, - log_type/0, - conn_mod/0]). - -%%---------------------------------------------------------------------- %% Type declarations %%---------------------------------------------------------------------- --type hook_options() :: [hook_option()]. -%% Options that can be given to `cth_conn_log' in the `ct_hook' statement. --type hook_option() :: {log_type,log_type()} | - {hosts,[ct_gen_conn:key_or_name()]}. --type log_type() :: raw | pretty | html | silent. --type conn_mod() :: ct_netconfc | ct_telnet. +-type hook_options() :: ct:conn_log_options(). +-type log_type() :: ct:conn_log_type(). +-type conn_mod() :: ct:conn_log_mod(). %%---------------------------------------------------------------------- -spec init(Id, HookOpts) -> Result when Id :: term(), HookOpts :: hook_options(), - Result :: {ok,[{conn_mod(), - {log_type(),[ct_gen_conn:key_or_name()]}}]}. + Result :: {ok,[{conn_mod(),{log_type(),[ct:key_or_name()]}}]}. init(_Id, HookOpts) -> ConfOpts = ct:get_config(ct_conn_log,[]), {ok,merge_log_info(ConfOpts,HookOpts)}. diff --git a/lib/common_test/test/ct_keep_logs_SUITE.erl b/lib/common_test/test/ct_keep_logs_SUITE.erl index f0c85a88a2..6b7aaa57ac 100644 --- a/lib/common_test/test/ct_keep_logs_SUITE.erl +++ b/lib/common_test/test/ct_keep_logs_SUITE.erl @@ -70,8 +70,7 @@ keep_logs(Config) -> Opts = [{suite,Suite},{label,keep_logs} | Opts0], LogDir=?config(logdir,Opts), - KeepLogsDir = filename:join(LogDir,unique_name("keep_logs-")), - ok = file:make_dir(KeepLogsDir), + KeepLogsDir = create_dir(filename:join(LogDir,"keep_logs-")), Opts1 = lists:keyreplace(logdir,1,Opts,{logdir,KeepLogsDir}), ct:log("New LogDir = ~s", [KeepLogsDir]), @@ -121,7 +120,11 @@ keep_logs(Config) -> 9 = length(L10), 0 = ct_test_support:run_ct_script_start([{keep_logs,10}|Opts1], Config), L11 = filelib:wildcard(WC), - 10 = length(L11). + 10 = length(L11), + + {ok,Content} = file:list_dir(KeepLogsDir), + ct:log("Deleting dir: ~p~nContent: ~p~n",[KeepLogsDir,Content]), + ct_test_support:rm_dir(KeepLogsDir). %% Test the keep_logs option togwther with the refresh_logs option refresh_logs(Config) -> @@ -129,8 +132,7 @@ refresh_logs(Config) -> Suite = filename:join(DataDir, "keep_logs_SUITE"), Opts0 = ct_test_support:get_opts(Config), LogDir=?config(logdir,Opts0), - KeepLogsDir = filename:join(LogDir,unique_name("refresh_logs-")), - ok = file:make_dir(KeepLogsDir), + KeepLogsDir = create_dir(filename:join(LogDir,"refresh_logs-")), Opts1 = lists:keyreplace(logdir,1,Opts0,{logdir,KeepLogsDir}), ct:log("New LogDir = ~s", [KeepLogsDir]), @@ -178,9 +180,20 @@ refresh_logs(Config) -> L10 = filelib:wildcard(WC), 4 = length(L10), - ok. + {ok,Content} = file:list_dir(KeepLogsDir), + ct:log("Deleting dir: ~p~nContent: ~p~n",[KeepLogsDir,Content]), + ct_test_support:rm_dir(KeepLogsDir). + %%%----------------------------------------------------------------- %%% Internal -unique_name(Prefix) -> +create_dir(Prefix) -> I = erlang:unique_integer([positive]), - Prefix ++ integer_to_list(I). + Dir = Prefix ++ integer_to_list(I), + case filelib:is_dir(Dir) of + true -> + %% Try again + create_dir(Prefix); + false -> + ok = file:make_dir(Dir), + Dir + end. diff --git a/lib/common_test/test/ct_netconfc_SUITE.erl b/lib/common_test/test/ct_netconfc_SUITE.erl index 8932f930d1..05edb45fe8 100644 --- a/lib/common_test/test/ct_netconfc_SUITE.erl +++ b/lib/common_test/test/ct_netconfc_SUITE.erl @@ -52,9 +52,8 @@ init_per_suite(Config) -> end. check_crypto_and_ssh() -> - (catch code:load_file(crypto)), - case code:is_loaded(crypto) of - {file,_} -> + case code:ensure_loaded(crypto) of + {module,_} -> case catch ssh:start() of Ok when Ok==ok; Ok=={error,{already_started,ssh}} -> ct:log("ssh started",[]), diff --git a/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl b/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl index 2aa6c4d354..6a41f0a04c 100644 --- a/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl +++ b/lib/common_test/test/ct_netconfc_SUITE_data/netconfc1_SUITE.erl @@ -102,7 +102,9 @@ all() -> receive_one_event, receive_multiple_events, receive_event_and_rpc, - receive_event_and_rpc_in_chunks + receive_event_and_rpc_in_chunks, + multiple_channels, + kill_session_same_connection ] end. @@ -124,7 +126,7 @@ end_per_testcase(_Case, _Config) -> ok. init_per_suite(Config) -> - (catch code:load_file(crypto)), + code:ensure_loaded(crypto), case {ssh:start(),code:is_loaded(crypto)} of {Ok,{file,_}} when Ok==ok; Ok=={error,{already_started,ssh}} -> ct:log("ssh started",[]), @@ -498,10 +500,11 @@ kill_session(Config) -> ?NS:hello(2), ?NS:expect(2,hello), - {ok,_OtherClient} = open(SshDir), + {ok,OtherClient} = open(SshDir), ?NS:expect_do_reply('kill-session',{kill,2},ok), ?ok = ct_netconfc:kill_session(Client,2), + {error,_}=ct_netconfc:get(OtherClient,{server,[{xmlns,"myns"}],[]}), ?NS:expect_do_reply('close-session',close,ok), ?ok = ct_netconfc:close_session(Client), @@ -1179,13 +1182,73 @@ receive_event_and_rpc_in_chunks(Config) -> ?ok = ct_netconfc:close_session(Client), ok. +multiple_channels(Config) -> + SshDir = ?config(ssh_dir,Config), + SshOpts = ?DEFAULT_SSH_OPTS(SshDir), + {ok,Conn} = ct_netconfc:connect(SshOpts), + ?NS:hello(1), + ?NS:expect(hello), + {ok,Client1} = ct_netconfc:session(Conn), + ?NS:hello(2), + ?NS:expect(2,hello), + {ok,Client2} = ct_netconfc:session(Conn), + ?NS:hello(3), + ?NS:expect(3,hello), + {ok,Client3} = ct_netconfc:session(Conn), + + Data = [{server,[{xmlns,"myns"}],[{name,[],["myserver"]}]}], + ?NS:expect_reply(1,'get',{data,Data}), + {ok,Data} = ct_netconfc:get(Client1,{server,[{xmlns,"myns"}],[]}), + ?NS:expect_reply(2,'get',{data,Data}), + {ok,Data} = ct_netconfc:get(Client2,{server,[{xmlns,"myns"}],[]}), + ?NS:expect_reply(3,'get',{data,Data}), + {ok,Data} = ct_netconfc:get(Client3,{server,[{xmlns,"myns"}],[]}), + + ?NS:expect_do_reply(2,'close-session',close,ok), + ?ok = ct_netconfc:close_session(Client2), + + ?NS:expect_reply(1,'get',{data,Data}), + {ok,Data} = ct_netconfc:get(Client1,{server,[{xmlns,"myns"}],[]}), + {error,no_such_client}=ct_netconfc:get(Client2,{server,[{xmlns,"myns"}],[]}), + ?NS:expect_reply(3,'get',{data,Data}), + {ok,Data} = ct_netconfc:get(Client3,{server,[{xmlns,"myns"}],[]}), + + ?NS:expect_do_reply(1,'close-session',close,ok), + ?ok = ct_netconfc:close_session(Client1), + ?NS:expect_do_reply(3,'close-session',close,ok), + ?ok = ct_netconfc:close_session(Client3), + + ?ok = ct_netconfc:disconnect(Conn), + ok. + +kill_session_same_connection(Config) -> + SshDir = ?config(ssh_dir,Config), + SshOpts = ?DEFAULT_SSH_OPTS(SshDir), + {ok,Conn} = ct_netconfc:connect(SshOpts), + ?NS:hello(1), + ?NS:expect(hello), + {ok,Client1} = ct_netconfc:session(Conn), + ?NS:hello(2), + ?NS:expect(2,hello), + {ok,Client2} = ct_netconfc:session(Conn), + + ?NS:expect_do_reply('kill-session',{kill,2},ok), + ?ok = ct_netconfc:kill_session(Client1,2), + timer:sleep(1000), + {error,no_such_client}=ct_netconfc:get(Client2,{server,[{xmlns,"myns"}],[]}), + + ?NS:expect_do_reply('close-session',close,ok), + ?ok = ct_netconfc:close_session(Client1), + + ok. + %%%----------------------------------------------------------------- break(_Config) -> - test_server:break("break test case"). + ct:break("break test case"). br() -> - test_server:break(""). + ct:break(""). %%%----------------------------------------------------------------- %% Open a netconf session which is not specified in a config file diff --git a/lib/common_test/test/ct_netconfc_SUITE_data/netconfc_remote_SUITE.erl b/lib/common_test/test/ct_netconfc_SUITE_data/netconfc_remote_SUITE.erl index f2580ad8e9..3ce2d18c66 100644 --- a/lib/common_test/test/ct_netconfc_SUITE_data/netconfc_remote_SUITE.erl +++ b/lib/common_test/test/ct_netconfc_SUITE_data/netconfc_remote_SUITE.erl @@ -62,7 +62,7 @@ stop_node(Case) -> init_per_suite(Config) -> - (catch code:load_file(crypto)), + code:ensure_loaded(crypto), case {ssh:start(),code:is_loaded(crypto)} of {Ok,{file,_}} when Ok==ok; Ok=={error,{already_started,ssh}} -> ct:log("SSH started locally",[]), diff --git a/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl b/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl index 2412ea6aba..c40bf9e2cc 100644 --- a/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl +++ b/lib/common_test/test/ct_netconfc_SUITE_data/ns.erl @@ -254,7 +254,7 @@ data(Data, State = #session{connection = ConnRef, end. stop_channel(CM, Ch, State) -> - ssh:close(CM), + ssh_connection:close(CM,Ch), {stop, Ch, State}. @@ -290,8 +290,8 @@ send_frag({CM,Ch},Data) -> %%% Kill ssh connection -kill({CM,_Ch}) -> - ssh:close(CM). +kill({CM,Ch}) -> + ssh_connection:close(CM,Ch). add_expect(SessionId,Add) -> table_trans(fun do_add_expect/2,[SessionId,Add]). diff --git a/lib/common_test/test/ct_test_support.erl b/lib/common_test/test/ct_test_support.erl index 935d032667..ba7aadfeec 100644 --- a/lib/common_test/test/ct_test_support.erl +++ b/lib/common_test/test/ct_test_support.erl @@ -45,6 +45,8 @@ -export([unique_timestamp/0]). +-export([rm_dir/1]). + -include_lib("kernel/include/file.hrl"). %%%----------------------------------------------------------------- diff --git a/lib/common_test/vsn.mk b/lib/common_test/vsn.mk index a219aa4736..e6ae8b2e7a 100644 --- a/lib/common_test/vsn.mk +++ b/lib/common_test/vsn.mk @@ -1 +1 @@ -COMMON_TEST_VSN = 1.15 +COMMON_TEST_VSN = 1.14 diff --git a/lib/compiler/doc/src/compile.xml b/lib/compiler/doc/src/compile.xml index a82befd057..10164890f2 100644 --- a/lib/compiler/doc/src/compile.xml +++ b/lib/compiler/doc/src/compile.xml @@ -600,9 +600,9 @@ module.beam: module.erl \ to be deprecated.</p> </item> - <tag><c>warn_obsolete_guard</c></tag> + <tag><c>nowarn_obsolete_guard</c></tag> <item> - <p>Emits warnings for calls to old type testing BIFs, + <p>Turns off warnings for calls to old type testing BIFs, such as <c>pid/1</c> and <c>list/1</c>. See the <seealso marker="doc/reference_manual:expressions#guards">Erlang Reference Manual</seealso> for a complete list of type testing BIFs and their old @@ -629,6 +629,14 @@ module.beam: module.erl \ <p>Turns off warnings for unused record types. Default is to emit warnings for unused locally defined record types.</p> </item> + + <tag><c>nowarn_get_stacktrace</c></tag> + <item> + <p>Turns off warnings for using <c>get_stacktrace/0</c> in a context + where it will probably not work in a future release. For example, + by default there will be a warning if <c>get_stacktrace/0</c> is + used following a <c>catch</c> expression.</p> + </item> </taglist> <p>Another class of warnings is generated by the compiler diff --git a/lib/compiler/doc/src/notes.xml b/lib/compiler/doc/src/notes.xml index f3da04120a..1dc0c808e7 100644 --- a/lib/compiler/doc/src/notes.xml +++ b/lib/compiler/doc/src/notes.xml @@ -32,156 +32,6 @@ <p>This document describes the changes made to the Compiler application.</p> -<section><title>Compiler 7.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p>For many releases, it has been legal to override a BIF - with a local function having the same name. However, - calling a local function with the same name as guard BIF - as filter in a list comprehension was not allowed.</p> - <p> - Own Id: OTP-13690</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Optimized test for tuples with an atom as first element.</p> - <p> - Own Id: OTP-12148</p> - </item> - <item> - <p> - Compilation of modules with huge literal binary strings - is now much faster.</p> - <p> - Own Id: OTP-13794</p> - </item> - <item> - <p>Replaced usage of deprecated symbolic <seealso - marker="erts:erlang#type-time_unit"><c>time - unit</c></seealso> representations.</p> - <p> - Own Id: OTP-13831 Aux Id: OTP-13735 </p> - </item> - <item> - <p>The undocumented and unsupported module - <c>sys_pre_expand</c> has been removed. As a partical - replacement for the functionality, there is a new - function <c>erl_internal:add_predefined_functions/1</c> - and <c>erl_expand_records</c> will now add a module - prefix to calls to BIFs and imported functions.</p> - <p> - Own Id: OTP-13856</p> - </item> - <item> - <p>The internal compiler passes now start all generated - variables with "@" to avoid any conflicts with variables - in languages such as Elixir or LFE.</p> - <p> - Own Id: OTP-13924</p> - </item> - <item> - <p>The function <c>fmod/2</c> has been added to the - <c>math</c> module.</p> - <p> - Own Id: OTP-14000</p> - </item> - <item> - <p>Code generation for complicated guards have been - improved.</p> - <p> - Own Id: OTP-14042</p> - </item> - <item> - <p> - The compiler has new warnings for repeated identical map - keys.</p> - <p> - A map expression such as,</p> - <p> - <c> #{'a' => 1, 'b' => 2, 'a' => 3}.</c></p> - <p> - will produce a warning for the repeated key 'a'.</p> - <p> - Own Id: OTP-14058</p> - </item> - <item> - <p>By default, there will now be a warning when - <c>export_all</c> is used. The warning can be disabled - using <c>nowarn_export_all</c>.</p> - <p> - Own Id: OTP-14071</p> - </item> - <item> - <p> - Optimize maps pattern matching by only examining the - common keys in each clause first instead of all keys. - This will reduce the number of lookups of each key in - maps pattern matching.</p> - <p> - Own Id: OTP-14072</p> - </item> - <item> - <p>There is a new '<c>deterministc</c>' option to omit - '<c>source</c>' and '<c>options</c>' tuples in the BEAM - file.</p> - <p> - Own Id: OTP-14087</p> - </item> - <item> - <p> - Analyzing modules with binary construction with huge - strings is now much faster. The compiler also compiles - such modules slightly faster.</p> - <p> - Own Id: OTP-14125 Aux Id: ERL-308 </p> - </item> - <item> - <p>Atoms may now contain arbitrary Unicode - characters.</p> - <p> - Own Id: OTP-14178</p> - </item> - <item> - <p><c>compile:file/2</c> now accepts the option - <c>extra_chunks</c> to include extra chunks in the BEAM - file.</p> - <p> - Own Id: OTP-14221</p> - </item> - <item> - <p>The format of debug information that is stored in BEAM - files (when <c>debug_info</c> is used) has been changed. - The purpose of the change is to better support other - BEAM-based languages such as Elixir or LFE.</p> - <p>All tools included in OTP (dialyzer, debugger, cover, - and so on) will handle both the new format and the - previous format. Tools that retrieve the debug - information using <c>beam_lib:chunk(Beam, - [abstract_code])</c> will continue to work with both the - new and old format. Tools that call - <c>beam_lib:chunk(Beam, ["Abst"])</c> will not work with - the new format.</p> - <p>For more information, see the description of - <c>debug_info</c> in the documentation for - <c>beam_lib</c> and the description of the - <c>{debug_info,{Backend,Data}}</c> option in the - documentation for <c>compile</c>.</p> - <p> - Own Id: OTP-14369 Aux Id: PR-1367 </p> - </item> - </list> - </section> - -</section> - <section><title>Compiler 7.0.4</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/compiler/src/beam_asm.erl b/lib/compiler/src/beam_asm.erl index 6d5071271c..c35efdfc9d 100644 --- a/lib/compiler/src/beam_asm.erl +++ b/lib/compiler/src/beam_asm.erl @@ -194,7 +194,7 @@ build_file(Code, Attr, Dict, NumLabels, NumFuncs, ExtraChunks, SourceFile, Opts, Chunks = case member(slim, Opts) of true -> - [Essentials,AttrChunk,CheckedChunks]; + [Essentials,AttrChunk]; false -> [Essentials,LocChunk,AttrChunk, CompileChunk,CheckedChunks,LineChunk] diff --git a/lib/compiler/src/erl_bifs.erl b/lib/compiler/src/erl_bifs.erl index 35a12d7010..bafa9d75b7 100644 --- a/lib/compiler/src/erl_bifs.erl +++ b/lib/compiler/src/erl_bifs.erl @@ -81,7 +81,6 @@ is_pure(erlang, float, 1) -> true; is_pure(erlang, float_to_list, 1) -> true; is_pure(erlang, float_to_binary, 1) -> true; is_pure(erlang, floor, 1) -> true; -is_pure(erlang, hash, 2) -> false; is_pure(erlang, hd, 1) -> true; is_pure(erlang, integer_to_binary, 1) -> true; is_pure(erlang, integer_to_list, 1) -> true; @@ -108,14 +107,11 @@ is_pure(erlang, list_to_binary, 1) -> true; is_pure(erlang, list_to_float, 1) -> true; is_pure(erlang, list_to_integer, 1) -> true; is_pure(erlang, list_to_pid, 1) -> true; -is_pure(erlang, list_to_port, 1) -> true; -is_pure(erlang, list_to_ref, 1) -> true; is_pure(erlang, list_to_tuple, 1) -> true; is_pure(erlang, max, 2) -> true; is_pure(erlang, min, 2) -> true; is_pure(erlang, phash, 2) -> false; is_pure(erlang, pid_to_list, 1) -> true; -is_pure(erlang, port_to_list, 1) -> true; is_pure(erlang, round, 1) -> true; is_pure(erlang, setelement, 3) -> true; is_pure(erlang, size, 1) -> true; diff --git a/lib/compiler/src/genop.tab b/lib/compiler/src/genop.tab index 5e0c2b3ebf..9efb9ad6f8 100755 --- a/lib/compiler/src/genop.tab +++ b/lib/compiler/src/genop.tab @@ -538,6 +538,8 @@ BEAM_FORMAT_NUMBER=0 157: has_map_fields/3 158: get_map_elements/3 +# OTP 20 + ## @spec is_tagged_tuple Lbl Reg N Atom ## @doc Test the type of Reg and jumps to Lbl if it is not a tuple. ## Test the arity of Reg and jumps to Lbl if it is not N. diff --git a/lib/compiler/src/v3_kernel_pp.erl b/lib/compiler/src/v3_kernel_pp.erl index 187e69a22c..716280a95c 100644 --- a/lib/compiler/src/v3_kernel_pp.erl +++ b/lib/compiler/src/v3_kernel_pp.erl @@ -145,7 +145,7 @@ format_1(#k_local{name=N,arity=A}, Ctxt) -> "local " ++ format_fa_pair({N,A}, Ctxt); format_1(#k_remote{mod=M,name=N,arity=A}, _Ctxt) -> %% This is for our internal translator. - io_lib:format("remote ~s:~s/~w", [format(M),format(N),A]); + io_lib:format("remote ~ts:~ts/~w", [format(M),format(N),A]); format_1(#k_internal{name=N,arity=A}, Ctxt) -> "internal " ++ format_fa_pair({N,A}, Ctxt); format_1(#k_seq{arg=A,body=B}, Ctxt) -> diff --git a/lib/compiler/test/bs_construct_SUITE.erl b/lib/compiler/test/bs_construct_SUITE.erl index ff18f8b41c..fd97eea4cb 100644 --- a/lib/compiler/test/bs_construct_SUITE.erl +++ b/lib/compiler/test/bs_construct_SUITE.erl @@ -29,7 +29,8 @@ init_per_testcase/2,end_per_testcase/2, two/1,test1/1,fail/1,float_bin/1,in_guard/1,in_catch/1, nasty_literals/1,coerce_to_float/1,side_effect/1, - opt/1,otp_7556/1,float_arith/1,otp_8054/1]). + opt/1,otp_7556/1,float_arith/1,otp_8054/1, + cover/1]). -include_lib("common_test/include/ct.hrl"). @@ -45,7 +46,7 @@ groups() -> [{p,[parallel], [two,test1,fail,float_bin,in_guard,in_catch, nasty_literals,side_effect,opt,otp_7556,float_arith, - otp_8054]}]. + otp_8054,cover]}]. init_per_suite(Config) -> @@ -552,3 +553,19 @@ otp_8054_1([H|T], Bin) -> end, otp_8054_1(T, Bin); otp_8054_1([], Bin) -> Bin. + +-define(LONG_STRING, + "3lz7Q4au2i3DJWNlNhWuzmvA7gYWGXG+LAPtgtlEO2VGSxRqL2WOoHW" + "QxORTQfJw17mNEU8i87UKvEPbo9YY8ppiM7vfaG88TTyfEzgUMTgY3I" + "vsikMBELPz2AayVz5aaMh9PBFTZ4DkBIFxURBUKHho4Vgt7IzYnWNgn" + "3ON5D9VS89TPANK5/PwSUoMQYZ2fk5VLbq7D1ExlnCScvTDnF/WHMQ3" + "m2GUcQWb+ajfOf3bnP7EX4f1Q3d/1Soe6lEpf1KN/5S7A/ugjMhy4+H" + "Zuo1J1J6CCwEVZ/wDc79OpDPPj/qOGhDK73F8DaMcynZ91El+01vfTn" + "uUxNFUHLpuoQ=="). + +cover(Config) -> + %% Cover handling of a huge partially literal string. + L = length(Config), + Bin = id(<<L:32,?LONG_STRING>>), + <<L:32,?LONG_STRING>> = Bin, + ok. diff --git a/lib/compiler/test/compile_SUITE.erl b/lib/compiler/test/compile_SUITE.erl index e9e595084a..f5e904a50a 100644 --- a/lib/compiler/test/compile_SUITE.erl +++ b/lib/compiler/test/compile_SUITE.erl @@ -31,11 +31,12 @@ file_1/1, forms_2/1, module_mismatch/1, big_file/1, outdir/1, binary/1, makedep/1, cond_and_ifdef/1, listings/1, listings_big/1, other_output/1, kernel_listing/1, encrypted_abstr/1, - strict_record/1, utf8_atoms/1, extra_chunks/1, - cover/1, env/1, core/1, + strict_record/1, utf8_atoms/1, utf8_functions/1, extra_chunks/1, + cover/1, env/1, core_pp/1, core_roundtrip/1, asm/1, optimized_guards/1, sys_pre_attributes/1, dialyzer/1, - warnings/1, pre_load_check/1, env_compiler_options/1 + warnings/1, pre_load_check/1, env_compiler_options/1, + bc_options/1 ]). suite() -> [{ct_hooks,[ts_install_cth]}]. @@ -49,10 +50,10 @@ all() -> [app_test, appup_test, file_1, forms_2, module_mismatch, big_file, outdir, binary, makedep, cond_and_ifdef, listings, listings_big, other_output, kernel_listing, encrypted_abstr, - strict_record, utf8_atoms, extra_chunks, - cover, env, core, core_roundtrip, asm, optimized_guards, + strict_record, utf8_atoms, utf8_functions, extra_chunks, + cover, env, core_pp, core_roundtrip, asm, optimized_guards, sys_pre_attributes, dialyzer, warnings, pre_load_check, - env_compiler_options, custom_debug_info]. + env_compiler_options, custom_debug_info, bc_options]. groups() -> []. @@ -603,7 +604,7 @@ install_crypto_key(Key) -> %% Miscellanous tests, mainly to get better coverage. debug_info(erlang_v1, Module, ok, _Opts) -> {ok, [Module]}; -debug_info(erlang_v1, Module, error, _Opts) -> +debug_info(erlang_v1, _Module, error, _Opts) -> {error, unknown_format}. custom_debug_info(Config) when is_list(Config) -> @@ -727,6 +728,23 @@ utf8_atoms(Config) when is_list(Config) -> NoUtf8AtomForms = [{attribute,Anno,module,no_utf8_atom}|Forms], error = compile:forms(NoUtf8AtomForms, [binary, r19]). +utf8_functions(Config) when is_list(Config) -> + Anno = erl_anno:new(1), + Atom = binary_to_atom(<<"こんにちは"/utf8>>, utf8), + Forms = [{attribute,Anno,compile,[export_all]}, + {function,Anno,Atom,0,[{clause,Anno,[],[],[{atom,Anno,world}]}]}], + + Utf8FunctionForms = [{attribute,Anno,module,utf8_function}|Forms], + {ok,utf8_function,Utf8FunctionBin} = + compile:forms(Utf8FunctionForms, [binary]), + {ok,{utf8_function,[{atoms,_}]}} = + beam_lib:chunks(Utf8FunctionBin, [atoms]), + code:load_binary(utf8_function, "compile_SUITE", Utf8FunctionBin), + world = utf8_function:Atom(), + + NoUtf8FunctionForms = [{attribute,Anno,module,no_utf8_function}|Forms], + error = compile:forms(NoUtf8FunctionForms, [binary, r19]). + extra_chunks(Config) when is_list(Config) -> Anno = erl_anno:new(1), Forms = [{attribute,Anno,module,extra_chunks}], @@ -777,9 +795,9 @@ env_1(Simple, Target) -> %% Test pretty-printing in Core Erlang format and then try to %% compile the generated Core Erlang files. -core(Config) when is_list(Config) -> +core_pp(Config) when is_list(Config) -> PrivDir = proplists:get_value(priv_dir, Config), - Outdir = filename:join(PrivDir, "core"), + Outdir = filename:join(PrivDir, atom_to_list(?FUNCTION_NAME)), ok = file:make_dir(Outdir), TestBeams = get_unique_beam_files(), @@ -787,11 +805,11 @@ core(Config) when is_list(Config) -> {raw_abstract_v1,Abstr}}]}} = beam_lib:chunks(Beam, [abstract_code]), {Mod,Abstr} end || Beam <- TestBeams], - test_lib:p_run(fun(F) -> do_core(F, Outdir) end, Abstr). + test_lib:p_run(fun(F) -> do_core_pp(F, Outdir) end, Abstr). -do_core({M,A}, Outdir) -> +do_core_pp({M,A}, Outdir) -> try - do_core_1(M, A, Outdir) + do_core_pp_1(M, A, Outdir) catch throw:{error,Error} -> io:format("*** compilation failure '~p' for module ~s\n", @@ -803,7 +821,7 @@ do_core({M,A}, Outdir) -> error end. -do_core_1(M, A, Outdir) -> +do_core_pp_1(M, A, Outdir) -> {ok,M,Core0} = compile:forms(A, [to_core]), CoreFile = filename:join(Outdir, atom_to_list(M)++".core"), CorePP = core_pp:format(Core0), @@ -1347,6 +1365,47 @@ env_compiler_options(_Config) -> end, lists:foreach(F, Cases). +%% Test options for compatibility with previous major versions of OTP. + +bc_options(Config) -> + DataDir = proplists:get_value(data_dir, Config), + + 101 = highest_opcode(DataDir, small_float, [no_line_info]), + + 103 = highest_opcode(DataDir, big, + [no_record_opt,no_line_info,no_stack_trimming]), + + 125 = highest_opcode(DataDir, small_float, [no_line_info,no_float_opt]), + + 132 = highest_opcode(DataDir, small, + [no_record_opt,no_float_opt,no_line_info]), + + 136 = highest_opcode(DataDir, big, [no_record_opt,no_line_info]), + + 153 = highest_opcode(DataDir, big, [no_record_opt]), + 153 = highest_opcode(DataDir, big, [r16]), + 153 = highest_opcode(DataDir, big, [r17]), + 153 = highest_opcode(DataDir, big, [r18]), + 153 = highest_opcode(DataDir, big, [r19]), + 153 = highest_opcode(DataDir, small_float, [r16]), + 153 = highest_opcode(DataDir, small_float, []), + + 158 = highest_opcode(DataDir, small_maps, [r17]), + 158 = highest_opcode(DataDir, small_maps, [r18]), + 158 = highest_opcode(DataDir, small_maps, [r19]), + 158 = highest_opcode(DataDir, small_maps, []), + + 159 = highest_opcode(DataDir, big, []), + + ok. + +highest_opcode(DataDir, Mod, Opt) -> + Src = filename:join(DataDir, atom_to_list(Mod)++".erl"), + {ok,Mod,Beam} = compile:file(Src, [binary|Opt]), + {ok,{Mod,[{"Code",Code}]}} = beam_lib:chunks(Beam, ["Code"]), + <<16:32,0:32,HighestOpcode:32,_/binary>> = Code, + HighestOpcode. + %%% %%% Utilities. %%% diff --git a/lib/compiler/test/compile_SUITE_data/small_float.erl b/lib/compiler/test/compile_SUITE_data/small_float.erl new file mode 100644 index 0000000000..5cbb5aef83 --- /dev/null +++ b/lib/compiler/test/compile_SUITE_data/small_float.erl @@ -0,0 +1,5 @@ +-module(small_float). +-export([f/1]). + +f(F) when is_float(F) -> + F / 2. diff --git a/lib/compiler/test/compiler.cover b/lib/compiler/test/compiler.cover index 2be079944f..3fd7fc1937 100644 --- a/lib/compiler/test/compiler.cover +++ b/lib/compiler/test/compiler.cover @@ -1,5 +1,5 @@ {incl_app,compiler,details}. %% -*- erlang -*- -{excl_mods,compiler,[core_scan,core_parse,sys_pre_expand]}. +{excl_mods,compiler,[core_scan,core_parse]}. diff --git a/lib/compiler/vsn.mk b/lib/compiler/vsn.mk index 463c264a5f..5c87304a01 100644 --- a/lib/compiler/vsn.mk +++ b/lib/compiler/vsn.mk @@ -1 +1 @@ -COMPILER_VSN = 7.1 +COMPILER_VSN = 7.0.4 diff --git a/lib/crypto/c_src/crypto.c b/lib/crypto/c_src/crypto.c index 3f0439ed80..793cff166c 100644 --- a/lib/crypto/c_src/crypto.c +++ b/lib/crypto/c_src/crypto.c @@ -1089,9 +1089,6 @@ static void init_algorithms_types(ErlNifEnv* env) #ifndef OPENSSL_NO_RC4 algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"rc4"); #endif -#if defined(HAVE_GCM) - algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"aes_gcm"); -#endif #if defined(HAVE_CHACHA20_POLY1305) algo_cipher[algo_cipher_cnt++] = enif_make_atom(env,"chacha20_poly1305"); #endif diff --git a/lib/crypto/doc/src/notes.xml b/lib/crypto/doc/src/notes.xml index 58b9db3539..62b013e463 100644 --- a/lib/crypto/doc/src/notes.xml +++ b/lib/crypto/doc/src/notes.xml @@ -31,118 +31,6 @@ </header> <p>This document describes the changes made to the Crypto application.</p> -<section><title>Crypto 4.0</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - LibreSSL can now be used by the modernized crypto app.</p> - <p> - Own Id: OTP-14247</p> - </item> - <item> - <p> - Add compile option <c>-compile(no_native)</c> in modules - with <c>on_load</c> directive which is not yet supported - by HiPE.</p> - <p> - Own Id: OTP-14316 Aux Id: PR-1390 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Add basic support for CMAC</p> - <p> - Own Id: OTP-13779 Aux Id: ERL-82 PR-1138 </p> - </item> - <item> - <p> - Removed functions deprecated in crypto-3.0 first released - in OTP-R16B01</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-13873</p> - </item> - <item> - <p> - The <c>crypto</c> application now supports OpenSSL 1.1.</p> - <p> - Own Id: OTP-13900</p> - </item> - <item> - <p> - Allow Erlang/OTP to use OpenSSL in FIPS-140 mode, in - order to satisfy specific security requirements (mostly - by different parts of the US federal government). </p> - <p> - See the new crypto users guide "FIPS mode" chapter about - building and using the FIPS support which is disabled by - default.</p> - <p> - (Thanks to dszoboszlay and legoscia)</p> - <p> - Own Id: OTP-13921 Aux Id: PR-1180 </p> - </item> - <item> - <p> - Crypto chacha20-poly1305 as in RFC 7539 enabled for - OpenSSL >= 1.1.</p> - <p> - Thanks to mururu.</p> - <p> - Own Id: OTP-14092 Aux Id: PR-1291 </p> - </item> - <item> - <p> - RSA key generation added to <c>crypto:generate_key/2</c>. - Thanks to wiml.</p> - <p> - An interface is also added to - <c>public_key:generate_key/1</c>.</p> - <p> - Own Id: OTP-14140 Aux Id: ERL-165, PR-1299 </p> - </item> - <item> - <p> - Raised minimum requeriment for OpenSSL version to - OpenSSL-9.0.8.c although we recommend a much higher - version, that is a version that is still maintained - officially by the OpenSSL project. Note that using such - an old version may restrict the crypto algorithms - supported.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14171</p> - </item> - <item> - <p> - Deprecate crypto:rand_uniform/2 as it is not - cryptographically strong</p> - <p> - Own Id: OTP-14274</p> - </item> - <item> - <p> - The Crypto application now supports generation of - cryptographically strong random numbers (floats < 1.0 - and integer arbitrary ranges) as a plugin to the 'rand' - module.</p> - <p> - Own Id: OTP-14317 Aux Id: PR-1372 </p> - </item> - </list> - </section> - -</section> - <section><title>Crypto 3.7.4</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/crypto/vsn.mk b/lib/crypto/vsn.mk index 796e3b6d84..f3e0623ac9 100644 --- a/lib/crypto/vsn.mk +++ b/lib/crypto/vsn.mk @@ -1 +1 @@ -CRYPTO_VSN = 4.0 +CRYPTO_VSN = 3.7.4 diff --git a/lib/debugger/doc/src/notes.xml b/lib/debugger/doc/src/notes.xml index fa85567a84..532a17bd81 100644 --- a/lib/debugger/doc/src/notes.xml +++ b/lib/debugger/doc/src/notes.xml @@ -33,42 +33,6 @@ <p>This document describes the changes made to the Debugger application.</p> -<section><title>Debugger 4.2.2</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> The Erlang shell, <c>qlc:string_to_handle()</c>, and - the Debugger (the Evaluator area and Edit variable window - of the Bindings area) can parse pids, ports, references, - and external funs, as long as they can be created in the - running system. </p> - <p> - Own Id: OTP-14296</p> - </item> - <item> - <p> Fix editing of simple binary values in the Bindings - area of the Debugger's Attach Process Window. </p> - <p> - Own Id: OTP-14318</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> Miscellaneous updates due to atoms containing - arbitrary Unicode characters. </p> - <p> - Own Id: OTP-14285</p> - </item> - </list> - </section> - -</section> - <section><title>Debugger 4.2.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/debugger/vsn.mk b/lib/debugger/vsn.mk index 3534570ef5..f5440865ef 100644 --- a/lib/debugger/vsn.mk +++ b/lib/debugger/vsn.mk @@ -1 +1 @@ -DEBUGGER_VSN = 4.2.2 +DEBUGGER_VSN = 4.2.1 diff --git a/lib/dialyzer/doc/src/notes.xml b/lib/dialyzer/doc/src/notes.xml index 41f02ad752..f7613b3145 100644 --- a/lib/dialyzer/doc/src/notes.xml +++ b/lib/dialyzer/doc/src/notes.xml @@ -32,56 +32,6 @@ <p>This document describes the changes made to the Dialyzer application.</p> -<section><title>Dialyzer 3.2</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Analyzing modules with binary construction with huge - strings is now much faster. The compiler also compiles - such modules slightly faster.</p> - <p> - Own Id: OTP-14125 Aux Id: ERL-308 </p> - </item> - <item> - <p> The peak memory consumption is reduced. </p> - <p> - Own Id: OTP-14127</p> - </item> - <item> - <p>TypEr has been removed as separate application and is - now a part of the Dialyzer application. Documentation for - TypEr has been added in the Dialyzer application.</p> - <p> - Own Id: OTP-14336</p> - </item> - <item> - <p>The format of debug information that is stored in BEAM - files (when <c>debug_info</c> is used) has been changed. - The purpose of the change is to better support other - BEAM-based languages such as Elixir or LFE.</p> - <p>All tools included in OTP (dialyzer, debugger, cover, - and so on) will handle both the new format and the - previous format. Tools that retrieve the debug - information using <c>beam_lib:chunk(Beam, - [abstract_code])</c> will continue to work with both the - new and old format. Tools that call - <c>beam_lib:chunk(Beam, ["Abst"])</c> will not work with - the new format.</p> - <p>For more information, see the description of - <c>debug_info</c> in the documentation for - <c>beam_lib</c> and the description of the - <c>{debug_info,{Backend,Data}}</c> option in the - documentation for <c>compile</c>.</p> - <p> - Own Id: OTP-14369 Aux Id: PR-1367 </p> - </item> - </list> - </section> - -</section> - <section><title>Dialyzer 3.1.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/dialyzer/vsn.mk b/lib/dialyzer/vsn.mk index 4a1a7c25a0..0919fba834 100644 --- a/lib/dialyzer/vsn.mk +++ b/lib/dialyzer/vsn.mk @@ -1 +1 @@ -DIALYZER_VSN = 3.2 +DIALYZER_VSN = 3.1 diff --git a/lib/diameter/doc/src/notes.xml b/lib/diameter/doc/src/notes.xml index 374d38ec09..50f568abaa 100644 --- a/lib/diameter/doc/src/notes.xml +++ b/lib/diameter/doc/src/notes.xml @@ -43,32 +43,6 @@ first.</p> <!-- ===================================================================== --> -<section><title>diameter 1.12.3</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p>Typos have been fixed.</p> - <p> - Own Id: OTP-14387</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> The usage of module <c>erl_anno</c> is corrected. - </p> - <p> - Own Id: OTP-14389</p> - </item> - </list> - </section> - -</section> - <section><title>diameter 1.12.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/diameter/vsn.mk b/lib/diameter/vsn.mk index 7c2b9bc895..94d9d72a48 100644 --- a/lib/diameter/vsn.mk +++ b/lib/diameter/vsn.mk @@ -17,5 +17,5 @@ # %CopyrightEnd% APPLICATION = diameter -DIAMETER_VSN = 1.12.3 +DIAMETER_VSN = 1.12.2 APP_VSN = $(APPLICATION)-$(DIAMETER_VSN)$(PRE_VSN) diff --git a/lib/edoc/doc/src/notes.xml b/lib/edoc/doc/src/notes.xml index 27b5a3ac3d..4982488335 100644 --- a/lib/edoc/doc/src/notes.xml +++ b/lib/edoc/doc/src/notes.xml @@ -32,28 +32,6 @@ <p>This document describes the changes made to the EDoc application.</p> -<section><title>Edoc 0.9</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p>To support stable builds, <c>edoc</c> no longer - includes time stamps in the footer for generated - files.</p> - <p> - Own Id: OTP-14277</p> - </item> - <item> - <p> Miscellaneous updates due to atoms containing - arbitrary Unicode characters. </p> - <p> - Own Id: OTP-14285</p> - </item> - </list> - </section> - -</section> - <section><title>Edoc 0.8.1</title> <section><title>Improvements and New Features</title> diff --git a/lib/edoc/vsn.mk b/lib/edoc/vsn.mk index 1a933b2ad8..d66802fdac 100644 --- a/lib/edoc/vsn.mk +++ b/lib/edoc/vsn.mk @@ -1 +1 @@ -EDOC_VSN = 0.9 +EDOC_VSN = 0.8.1 diff --git a/lib/erl_docgen/doc/src/notes.xml b/lib/erl_docgen/doc/src/notes.xml index 1ce7eed6a8..4824a755d9 100644 --- a/lib/erl_docgen/doc/src/notes.xml +++ b/lib/erl_docgen/doc/src/notes.xml @@ -31,36 +31,7 @@ </header> <p>This document describes the changes made to the <em>erl_docgen</em> application.</p> - <section><title>Erl_Docgen 0.7</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Sort index of C functions alphabetically in the sidebar.</p> - <p> - Own Id: OTP-14333 Aux Id: ERL-393 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Improvements in the OTP documentation style.</p> - <p> - (Thanks to Mariano Guerra)</p> - <p> - Own Id: OTP-14371 Aux Id: PR-1215 </p> - </item> - </list> - </section> - -</section> - -<section><title>Erl_Docgen 0.6.1</title> + <section><title>Erl_Docgen 0.6.1</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/erl_interface/doc/src/notes.xml b/lib/erl_interface/doc/src/notes.xml index e586960c06..77d503b4d2 100644 --- a/lib/erl_interface/doc/src/notes.xml +++ b/lib/erl_interface/doc/src/notes.xml @@ -31,39 +31,6 @@ </header> <p>This document describes the changes made to the Erl_interface application.</p> -<section><title>Erl_Interface 3.10</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Fix bug where gethostname would incorrectly fail with - enametoolong on Linux.</p> - <p> - Own Id: OTP-14310</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Remove generation of atoms in old latin1 external format. - The old latin1 format can still be decoded but is never - generated by <c>term_to_binary</c> or sent on a - distribution channel. The new utf8 format for atoms was - introduced in OTP R16. An OTP 20 node can therefore not - connect to nodes older than R16.</p> - <p> - Own Id: OTP-14337</p> - </item> - </list> - </section> - -</section> - <section><title>Erl_Interface 3.9.3</title> <section><title>Improvements and New Features</title> diff --git a/lib/erl_interface/vsn.mk b/lib/erl_interface/vsn.mk index 01fcee86dd..563694a0c1 100644 --- a/lib/erl_interface/vsn.mk +++ b/lib/erl_interface/vsn.mk @@ -1,2 +1,2 @@ -EI_VSN = 3.10 +EI_VSN = 3.9.3 ERL_INTERFACE_VSN = $(EI_VSN) diff --git a/lib/eunit/doc/src/notes.xml b/lib/eunit/doc/src/notes.xml index 2a4ca6d12c..cd4e230254 100644 --- a/lib/eunit/doc/src/notes.xml +++ b/lib/eunit/doc/src/notes.xml @@ -33,21 +33,6 @@ </header> <p>This document describes the changes made to the EUnit application.</p> -<section><title>Eunit 2.3.3</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p>The surefire reports from <c>eunit</c> will no longer - have names with embedded double quotes.</p> - <p> - Own Id: OTP-14287</p> - </item> - </list> - </section> - -</section> - <section><title>Eunit 2.3.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/eunit/vsn.mk b/lib/eunit/vsn.mk index 107ad5c101..7eee75ee10 100644 --- a/lib/eunit/vsn.mk +++ b/lib/eunit/vsn.mk @@ -1 +1 @@ -EUNIT_VSN = 2.3.3 +EUNIT_VSN = 2.3.2 diff --git a/lib/hipe/doc/src/notes.xml b/lib/hipe/doc/src/notes.xml index 9167d0aaec..38f8dbfea3 100644 --- a/lib/hipe/doc/src/notes.xml +++ b/lib/hipe/doc/src/notes.xml @@ -31,101 +31,6 @@ </header> <p>This document describes the changes made to HiPE.</p> -<section><title>Hipe 3.16</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Fix hipe compiler flags <c>o0</c> and <c>o1</c> that have - previously been ignored by mistake.</p> - <p> - Own Id: OTP-13862 Aux Id: PR-1154 </p> - </item> - <item> - <p> - Fix LLVM backend to not convert all remote calls to own - module, like <c>?MODULE:foo()</c>, into local calls.</p> - <p> - Own Id: OTP-13983</p> - </item> - <item> - <p> - Hipe optional LLVM backend does require LLVM version 3.9 - or later as older versions forced strong dependencies on - erts internals structures.</p> - <p> - Own Id: OTP-14238</p> - </item> - <item> - <p> - Fix a bug that has been seen causing failed loading of - hipe compiled modules on NetBSD due to unaligned data - pointers.</p> - <p> - Own Id: OTP-14302 Aux Id: ERL-376, PR-1386 </p> - </item> - <item> - <p> - Fix miscompilation bug in hipe that could cause wrong - function clause to be called from non-tail calls, where - the return value is unused, if the right function clause - is only reachable from those non-tail calls.</p> - <p> - Own Id: OTP-14306 Aux Id: ERL-278, PR-1392 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Improve hipe compilation time for large functions.</p> - <p> - Own Id: OTP-13810 Aux Id: PR-1124 </p> - </item> - <item> - <p>Replaced usage of deprecated symbolic <seealso - marker="erts:erlang#type-time_unit"><c>time - unit</c></seealso> representations.</p> - <p> - Own Id: OTP-13831 Aux Id: OTP-13735 </p> - </item> - <item> - <p> - Speed up hipe compile time register allocation for larger - function.</p> - <p> - Own Id: OTP-13879</p> - </item> - <item> - <p> - Various code generation improvements.</p> - <p> - Own Id: OTP-14261 Aux Id: PR-1360 </p> - </item> - <item> - <p> - Improve hipe compiler to generate code with better CPU - register utilization at runtime by the use of 'Live Range - Splitting' techniques.</p> - <p> - Own Id: OTP-14293 Aux Id: PR-1380 </p> - </item> - <item> - <p> - Allow HiPE to run on VM built with - <c>--enable-m32-build</c>.</p> - <p> - Own Id: OTP-14330 Aux Id: PR-1397 </p> - </item> - </list> - </section> - -</section> - <section><title>Hipe 3.15.4</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/hipe/vsn.mk b/lib/hipe/vsn.mk index 0ef4aa7f09..172d976931 100644 --- a/lib/hipe/vsn.mk +++ b/lib/hipe/vsn.mk @@ -1 +1 @@ -HIPE_VSN = 3.16 +HIPE_VSN = 3.15.4 diff --git a/lib/inets/doc/src/http_uri.xml b/lib/inets/doc/src/http_uri.xml index 696b7dfa31..acf0d2163a 100644 --- a/lib/inets/doc/src/http_uri.xml +++ b/lib/inets/doc/src/http_uri.xml @@ -45,6 +45,7 @@ this module:</p> <p><c>boolean() = true | false</c></p> <p><c>string()</c> = list of ASCII characters</p> + <p><c>unicode_binary()</c> = binary() with characters encoded in the UTF-8 coding standard</p> </section> @@ -53,22 +54,22 @@ <p>Type definitions that are related to URI:</p> <taglist> - <tag><c>uri() = string()</c></tag> + <tag><c>uri() = string() | unicode:unicode_binary()</c></tag> <item><p>Syntax according to the URI definition in RFC 3986, for example, "http://www.erlang.org/"</p></item> - <tag><c>user_info() = string()</c></tag> + <tag><c>user_info() = string() | unicode:unicode_binary()</c></tag> <item><p></p></item> <tag><c>scheme() = atom()</c></tag> <item><p>Example: http, https</p></item> - <tag><c>host() = string()</c></tag> + <tag><c>host() = string() | unicode:unicode_binary()</c></tag> <item><p></p></item> <tag><c>port() = pos_integer()</c></tag> <item><p></p></item> - <tag><c>path() = string()</c></tag> + <tag><c>path() = string() | unicode:unicode_binary()</c></tag> <item><p>Represents a file path or directory path</p></item> - <tag><c>query() = string()</c></tag> + <tag><c>query() = string() | unicode:unicode_binary()</c></tag> <item><p></p></item> - <tag><c>fragment() = string()</c></tag> + <tag><c>fragment() = string() | unicode:unicode_binary()</c></tag> <item><p></p></item> </taglist> @@ -83,7 +84,7 @@ <fsummary>Decodes a hexadecimal encoded URI.</fsummary> <type> - <v>HexEncodedURI = string() - A possibly hexadecimal encoded URI</v> + <v>HexEncodedURI = string() | unicode:unicode_binary() - A possibly hexadecimal encoded URI</v> <v>URI = uri()</v> </type> @@ -98,7 +99,7 @@ <fsummary>Encodes a hexadecimal encoded URI.</fsummary> <type> <v>URI = uri()</v> - <v>HexEncodedURI = string() - Hexadecimal encoded URI</v> + <v>HexEncodedURI = string() | unicode:unicode_binary() - Hexadecimal encoded URI</v> </type> <desc> @@ -145,7 +146,7 @@ <p>Scheme validation fun is to be defined as follows:</p> <code> -fun(SchemeStr :: string()) -> +fun(SchemeStr :: string() | unicode:unicode_binary()) -> valid | {error, Reason :: term()}. </code> diff --git a/lib/inets/doc/src/httpc.xml b/lib/inets/doc/src/httpc.xml index 4217b3c4fb..46484977c9 100644 --- a/lib/inets/doc/src/httpc.xml +++ b/lib/inets/doc/src/httpc.xml @@ -344,7 +344,7 @@ <tag><c><![CDATA[ssl]]></c></tag> <item> <p>This is the <c>SSL/TLS</c> connectin configuration option.</p> - <p>Defaults to <c>[]</c>. See <seealso marker="ssl:ssl">ssl:connect/[2, 3,4]</seealso> for availble options.</p> + <p>Defaults to <c>[]</c>. See <seealso marker="ssl:ssl">ssl:connect/[2,3,4]</seealso> for available options.</p> </item> <tag><c><![CDATA[autoredirect]]></c></tag> diff --git a/lib/inets/doc/src/notes.xml b/lib/inets/doc/src/notes.xml index 7c36d2831a..1fad94fac1 100644 --- a/lib/inets/doc/src/notes.xml +++ b/lib/inets/doc/src/notes.xml @@ -38,9 +38,11 @@ <section><title>Fixed Bugs and Malfunctions</title> <list> <item> - <p>Typos have been fixed.</p> <p> - Own Id: OTP-14387</p> + The close of a chunked file reception crashed in a + certain timing sequence.</p> + <p> + Own Id: OTP-14391 Aux Id: seq13306 </p> </item> </list> </section> @@ -397,6 +399,21 @@ </section> + <section><title>Inets 6.1.1.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Mend ipv6_host_with_brackets option in httpc</p> + <p> + Own Id: OTP-13417</p> + </item> + </list> + </section> + + </section> + <section><title>Inets 6.1.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/inets/src/ftp/ftp.erl b/lib/inets/src/ftp/ftp.erl index 2a225e930e..e0430654eb 100644 --- a/lib/inets/src/ftp/ftp.erl +++ b/lib/inets/src/ftp/ftp.erl @@ -100,6 +100,12 @@ ftp_extension = ?FTP_EXT_DEFAULT }). +-record(recv_chunk_closing, { + dconn_closed = false, + pos_compl_received = false, + client_called_us = false + }). + -type shortage_reason() :: 'etnospc' | 'epnospc'. -type restriction_reason() :: 'epath' | 'efnamena' | 'elogin' | 'enotbinary'. @@ -1343,6 +1349,25 @@ handle_call({_,{recv_chunk_start, RemoteFile}}, From, #state{chunk = false} handle_call({_, recv_chunk}, _, #state{chunk = false} = State) -> {reply, {error, "ftp:recv_chunk_start/2 not called"}, State}; +handle_call({_, recv_chunk}, _From, #state{chunk = true, + caller = #recv_chunk_closing{dconn_closed = true, + pos_compl_received = true + } + } = State0) -> + %% The ftp:recv_chunk call was the last event we waited for, finnish and clean up + ?DBG("recv_chunk_closing ftp:recv_chunk, last event",[]), + activate_ctrl_connection(State0), + {reply, ok, State0#state{caller = undefined, + chunk = false, + client = undefined}}; + +handle_call({_, recv_chunk}, From, #state{chunk = true, + caller = #recv_chunk_closing{} = R + } = State) -> + %% Waiting for more, don't care what + ?DBG("recv_chunk_closing ftp:recv_chunk, get more",[]), + {noreply, State#state{client = From, caller = R#recv_chunk_closing{client_called_us=true}}}; + handle_call({_, recv_chunk}, From, #state{chunk = true} = State0) -> State = activate_data_connection(State0), {noreply, State#state{client = From, caller = recv_chunk}}; @@ -1480,19 +1505,24 @@ handle_info({Cls, Socket}, #state{dsock = {Trpt,Socket}, file_close(Fd), progress_report({transfer_size, 0}, State), activate_ctrl_connection(State), + ?DBG("Data channel close",[]), {noreply, State#state{dsock = undefined, data = <<>>}}; handle_info({Cls, Socket}, #state{dsock = {Trpt,Socket}, + client = Client, caller = recv_chunk} = State) when {Cls,Trpt}=={tcp_closed,tcp} ; {Cls,Trpt}=={ssl_closed,ssl} -> + ?DBG("Data channel close recv_chunk",[]), activate_ctrl_connection(State), - {noreply, State#state{dsock = undefined, data = <<>>, - caller = recv_chunk_closed + {noreply, State#state{dsock = undefined, + caller = #recv_chunk_closing{dconn_closed = true, + client_called_us = Client =/= undefined} }}; handle_info({Cls, Socket}, #state{dsock = {Trpt,Socket}, caller = recv_bin, data = Data} = State) when {Cls,Trpt}=={tcp_closed,tcp} ; {Cls,Trpt}=={ssl_closed,ssl} -> + ?DBG("Data channel close",[]), activate_ctrl_connection(State), {noreply, State#state{dsock = undefined, data = <<>>, caller = {recv_bin, Data}}}; @@ -1500,6 +1530,7 @@ handle_info({Cls, Socket}, #state{dsock = {Trpt,Socket}, caller = recv_bin, handle_info({Cls, Socket}, #state{dsock = {Trpt,Socket}, data = Data, caller = {handle_dir_result, Dir}} = State) when {Cls,Trpt}=={tcp_closed,tcp} ; {Cls,Trpt}=={ssl_closed,ssl} -> + ?DBG("Data channel close",[]), activate_ctrl_connection(State), {noreply, State#state{dsock = undefined, caller = {handle_dir_result, Dir, Data}, @@ -2047,14 +2078,28 @@ handle_ctrl_result({pos_prel, _}, #state{client = From, %%-------------------------------------------------------------------------- %% File handling - chunk_transfer complete + handle_ctrl_result({pos_compl, _}, #state{client = From, - caller = recv_chunk_closed} - = State0) -> + caller = #recv_chunk_closing{dconn_closed = true, + client_called_us = true, + pos_compl_received = false + }} + = State0) when From =/= undefined -> + %% The pos_compl was the last event we waited for, finnish and clean up + ?DBG("recv_chunk_closing pos_compl, last event",[]), gen_server:reply(From, ok), + activate_ctrl_connection(State0), {noreply, State0#state{caller = undefined, chunk = false, client = undefined}}; +handle_ctrl_result({pos_compl, _}, #state{caller = #recv_chunk_closing{}=R} + = State0) -> + %% Waiting for more, don't care what + ?DBG("recv_chunk_closing pos_compl, wait more",[]), + {noreply, State0#state{caller = R#recv_chunk_closing{pos_compl_received=true}}}; + + %%-------------------------------------------------------------------------- %% File handling - recv_file handle_ctrl_result({pos_prel, _}, #state{caller = {recv_file, _}} = State0) -> diff --git a/lib/inets/src/http_lib/http_uri.erl b/lib/inets/src/http_lib/http_uri.erl index cb3e107ccf..4568d165e7 100644 --- a/lib/inets/src/http_lib/http_uri.erl +++ b/lib/inets/src/http_lib/http_uri.erl @@ -102,16 +102,23 @@ parse(AbsURI, Opts) -> reserved() -> sets:from_list([$;, $:, $@, $&, $=, $+, $,, $/, $?, - $#, $[, $], $<, $>, $\", ${, $}, $|, + $#, $[, $], $<, $>, $\", ${, $}, $|, %" $\\, $', $^, $%, $ ]). -encode(URI) -> +encode(URI) when is_list(URI) -> Reserved = reserved(), - lists:append([uri_encode(Char, Reserved) || Char <- URI]). + lists:append([uri_encode(Char, Reserved) || Char <- URI]); +encode(URI) when is_binary(URI) -> + Reserved = reserved(), + << <<(uri_encode_binary(Char, Reserved))/binary>> || <<Char>> <= URI >>. -decode(String) -> - do_decode(String). +decode(String) when is_list(String) -> + do_decode(String); +decode(String) when is_binary(String) -> + do_decode_binary(String). +do_decode([$+|Rest]) -> + [$ |do_decode(Rest)]; do_decode([$%,Hex1,Hex2|Rest]) -> [hex2dec(Hex1)*16+hex2dec(Hex2)|do_decode(Rest)]; do_decode([First|Rest]) -> @@ -119,6 +126,14 @@ do_decode([First|Rest]) -> do_decode([]) -> []. +do_decode_binary(<<$+, Rest/bits>>) -> + <<$ , (do_decode_binary(Rest))/binary>>; +do_decode_binary(<<$%, Hex:2/binary, Rest/bits>>) -> + <<(binary_to_integer(Hex, 16)), (do_decode_binary(Rest))/binary>>; +do_decode_binary(<<First:1/binary, Rest/bits>>) -> + <<First/binary, (do_decode_binary(Rest))/binary>>; +do_decode_binary(<<>>) -> + <<>>. %%%======================================================================== %%% Internal functions @@ -162,9 +177,30 @@ extract_scheme(Str, Opts) -> {error, Error} end; _ -> - {ok, list_to_atom(http_util:to_lower(Str))} + {ok, to_atom(http_util:to_lower(Str))} end. +to_atom(S) when is_list(S) -> + list_to_atom(S); +to_atom(S) when is_binary(S) -> + binary_to_atom(S, unicode). + +parse_uri_rest(Scheme, DefaultPort, <<"//", URIPart/binary>>, Opts) -> + {Authority, PathQueryFragment} = + split_uri(URIPart, "[/?#]", {URIPart, <<"">>}, 1, 0), + {RawPath, QueryFragment} = + split_uri(PathQueryFragment, "[?#]", {PathQueryFragment, <<"">>}, 1, 0), + {Query, Fragment} = + split_uri(QueryFragment, "#", {QueryFragment, <<"">>}, 1, 0), + {UserInfo, HostPort} = split_uri(Authority, "@", {<<"">>, Authority}, 1, 1), + {Host, Port} = parse_host_port(Scheme, DefaultPort, HostPort, Opts), + Path = path(RawPath), + case lists:keyfind(fragment, 1, Opts) of + {fragment, true} -> + {ok, {Scheme, UserInfo, Host, Port, Path, Query, Fragment}}; + _ -> + {ok, {Scheme, UserInfo, Host, Port, Path, Query}} + end; parse_uri_rest(Scheme, DefaultPort, "//" ++ URIPart, Opts) -> {Authority, PathQueryFragment} = split_uri(URIPart, "[/?#]", {URIPart, ""}, 1, 0), @@ -185,6 +221,11 @@ parse_uri_rest(Scheme, DefaultPort, "//" ++ URIPart, Opts) -> %% In this version of the function, we no longer need %% the Scheme argument, but just in case... +parse_host_port(_Scheme, DefaultPort, <<"[", HostPort/binary>>, Opts) -> %ipv6 + {Host, ColonPort} = split_uri(HostPort, "\\]", {HostPort, <<"">>}, 1, 1), + Host2 = maybe_ipv6_host_with_brackets(Host, Opts), + {_, Port} = split_uri(ColonPort, ":", {<<"">>, DefaultPort}, 0, 1), + {Host2, int_port(Port)}; parse_host_port(_Scheme, DefaultPort, "[" ++ HostPort, Opts) -> %ipv6 {Host, ColonPort} = split_uri(HostPort, "\\]", {HostPort, ""}, 1, 1), Host2 = maybe_ipv6_host_with_brackets(Host, Opts), @@ -198,12 +239,19 @@ parse_host_port(_Scheme, DefaultPort, HostPort, _Opts) -> split_uri(UriPart, SplitChar, NoMatchResult, SkipLeft, SkipRight) -> case re:run(UriPart, SplitChar, [{capture, first}]) of {match, [{Match, _}]} -> - {string:substr(UriPart, 1, Match + 1 - SkipLeft), - string:substr(UriPart, Match + 1 + SkipRight, length(UriPart))}; + {string:slice(UriPart, 0, Match + 1 - SkipLeft), + string:slice(UriPart, Match + SkipRight, string:length(UriPart))}; nomatch -> NoMatchResult end. +maybe_ipv6_host_with_brackets(Host, Opts) when is_binary(Host) -> + case lists:keysearch(ipv6_host_with_brackets, 1, Opts) of + {value, {ipv6_host_with_brackets, true}} -> + <<"[", Host/binary, "]">>; + _ -> + Host + end; maybe_ipv6_host_with_brackets(Host, Opts) -> case lists:keysearch(ipv6_host_with_brackets, 1, Opts) of {value, {ipv6_host_with_brackets, true}} -> @@ -212,15 +260,18 @@ maybe_ipv6_host_with_brackets(Host, Opts) -> Host end. - int_port(Port) when is_integer(Port) -> Port; +int_port(Port) when is_binary(Port) -> + binary_to_integer(Port); int_port(Port) when is_list(Port) -> list_to_integer(Port); %% This is the case where no port was found and there was no default port int_port(no_default_port) -> throw({error, no_default_port}). +path(<<"">>) -> + <<"/">>; path("") -> "/"; path(Path) -> @@ -234,6 +285,14 @@ uri_encode(Char, Reserved) -> [Char] end. +uri_encode_binary(Char, Reserved) -> + case sets:is_element(Char, Reserved) of + true -> + << $%, (integer_to_binary(Char, 16))/binary >>; + false -> + <<Char>> + end. + hex2dec(X) when (X>=$0) andalso (X=<$9) -> X-$0; hex2dec(X) when (X>=$A) andalso (X=<$F) -> X-$A+10; hex2dec(X) when (X>=$a) andalso (X=<$f) -> X-$a+10. diff --git a/lib/inets/src/http_lib/http_util.erl b/lib/inets/src/http_lib/http_util.erl index 20c342dd66..78b0aa2885 100644 --- a/lib/inets/src/http_lib/http_util.erl +++ b/lib/inets/src/http_lib/http_util.erl @@ -35,10 +35,10 @@ %%% Internal application API %%%========================================================================= to_upper(Str) -> - string:to_upper(Str). + string:uppercase(Str). to_lower(Str) -> - string:to_lower(Str). + string:lowercase(Str). %% Example: Mon, 09-Dec-2002 13:46:00 GMT convert_netscapecookie_date([_D,_A,_Y, $,, $ , diff --git a/lib/inets/src/http_server/httpd_util.erl b/lib/inets/src/http_server/httpd_util.erl index a647f04ddc..4f771015a6 100644 --- a/lib/inets/src/http_server/httpd_util.erl +++ b/lib/inets/src/http_server/httpd_util.erl @@ -333,7 +333,9 @@ rfc1123_date(LocalTime) -> {{YYYY,MM,DD},{Hour,Min,Sec}} = case calendar:local_time_to_universal_time_dst(LocalTime) of [Gmt] -> Gmt; - [_,Gmt] -> Gmt + [_,Gmt] -> Gmt; + % Should not happen, but handle the empty list to prevent an error. + [] -> LocalTime end, DayNumber = calendar:day_of_the_week({YYYY,MM,DD}), lists:flatten( diff --git a/lib/inets/test/ftp_SUITE.erl b/lib/inets/test/ftp_SUITE.erl index de4456551d..3dfec01ba2 100644 --- a/lib/inets/test/ftp_SUITE.erl +++ b/lib/inets/test/ftp_SUITE.erl @@ -97,6 +97,7 @@ ftp_tests()-> recv_bin_twice, recv_chunk, recv_chunk_twice, + recv_chunk_three_times, type, quote, error_elogin, @@ -639,13 +640,57 @@ recv_chunk_twice(Config0) -> find_diff(ReceivedContents1, Contents1), find_diff(ReceivedContents2, Contents2). +recv_chunk_three_times() -> + [{doc, "Receive two files using chunk-wise."}, + {timetrap,{seconds,120}}]. +recv_chunk_three_times(Config0) -> + File1 = "big_file1.txt", + File2 = "big_file2.txt", + File3 = "big_file3.txt", + Contents1 = list_to_binary( lists:duplicate(1000, lists:seq(0,255)) ), + Contents2 = crypto:strong_rand_bytes(1200), + Contents3 = list_to_binary( lists:duplicate(1000, lists:seq(255,0,-1)) ), + + Config = set_state([reset, {mkfile,File1,Contents1}, {mkfile,File2,Contents2}, {mkfile,File3,Contents3}], Config0), + Pid = proplists:get_value(ftp, Config), + {{error, "ftp:recv_chunk_start/2 not called"},_} = recv_chunk(Pid, <<>>), + + ok = ftp:recv_chunk_start(Pid, id2ftp(File1,Config)), + {ok, ReceivedContents1, Nchunks1} = recv_chunk(Pid, <<>>), + + ok = ftp:recv_chunk_start(Pid, id2ftp(File2,Config)), + {ok, ReceivedContents2, _Nchunks2} = recv_chunk(Pid, <<>>), + + ok = ftp:recv_chunk_start(Pid, id2ftp(File3,Config)), + {ok, ReceivedContents3, _Nchunks3} = recv_chunk(Pid, <<>>, 10000, 0, Nchunks1), + + find_diff(ReceivedContents1, Contents1), + find_diff(ReceivedContents2, Contents2), + find_diff(ReceivedContents3, Contents3). + + + recv_chunk(Pid, Acc) -> - recv_chunk(Pid, Acc, 0). + recv_chunk(Pid, Acc, 0, 0, undefined). + + + +%% ExpectNchunks :: integer() | undefined +recv_chunk(Pid, Acc, DelayMilliSec, N, ExpectNchunks) when N+1 < ExpectNchunks -> + %% for all I in integer(), I < undefined + recv_chunk1(Pid, Acc, DelayMilliSec, N, ExpectNchunks); + +recv_chunk(Pid, Acc, DelayMilliSec, N, ExpectNchunks) -> + %% N >= ExpectNchunks-1 + timer:sleep(DelayMilliSec), + recv_chunk1(Pid, Acc, DelayMilliSec, N, ExpectNchunks). + -recv_chunk(Pid, Acc, N) -> +recv_chunk1(Pid, Acc, DelayMilliSec, N, ExpectNchunks) -> + ct:log("Call ftp:recv_chunk",[]), case ftp:recv_chunk(Pid) of ok -> {ok, Acc, N}; - {ok, Bin} -> recv_chunk(Pid, <<Acc/binary, Bin/binary>>, N+1); + {ok, Bin} -> recv_chunk(Pid, <<Acc/binary, Bin/binary>>, DelayMilliSec, N+1, ExpectNchunks); Error -> {Error, N} end. diff --git a/lib/inets/test/httpd_basic_SUITE.erl b/lib/inets/test/httpd_basic_SUITE.erl index f413248092..a0a38ca103 100644 --- a/lib/inets/test/httpd_basic_SUITE.erl +++ b/lib/inets/test/httpd_basic_SUITE.erl @@ -42,7 +42,8 @@ all() -> escaped_url_in_error_body, script_timeout, slowdose, - keep_alive_timeout + keep_alive_timeout, + invalid_rfc1123_date ]. groups() -> @@ -383,6 +384,16 @@ slowdose(Config) when is_list(Config) -> end. %%------------------------------------------------------------------------- + +invalid_rfc1123_date() -> + [{doc, "Test that a non-DST date is handled correcly"}]. +invalid_rfc1123_date(Config) when is_list(Config) -> + Rfc1123FormattedDate = "Sun, 26 Mar 2017 01:00:00 GMT", + NonDSTDateTime = {{2017, 03, 26},{1, 0, 0}}, + Rfc1123FormattedDate =:= httpd_util:rfc1123_date(NonDSTDateTime). + + +%%------------------------------------------------------------------------- %% Internal functions %%------------------------------------------------------------------------- diff --git a/lib/inets/test/uri_SUITE.erl b/lib/inets/test/uri_SUITE.erl index b26c645821..d055af59e3 100644 --- a/lib/inets/test/uri_SUITE.erl +++ b/lib/inets/test/uri_SUITE.erl @@ -25,6 +25,7 @@ -module(uri_SUITE). +-include_lib("eunit/include/eunit.hrl"). -include_lib("common_test/include/ct.hrl"). -include("inets_test_lib.hrl"). @@ -50,7 +51,8 @@ all() -> fragments, escaped, hexed_query, - scheme_validation + scheme_validation, + encode_decode ]. %%-------------------------------------------------------------------- @@ -73,7 +75,10 @@ end_per_testcase(_Case, _Config) -> ipv4(Config) when is_list(Config) -> {ok, {http,[],"127.0.0.1",80,"/foobar.html",[]}} = - http_uri:parse("http://127.0.0.1/foobar.html"). + http_uri:parse("http://127.0.0.1/foobar.html"), + + {ok, {http,<<>>,<<"127.0.0.1">>,80,<<"/foobar.html">>,<<>>}} = + http_uri:parse(<<"http://127.0.0.1/foobar.html">>). ipv6(Config) when is_list(Config) -> {ok, {http,[],"2010:836B:4179::836B:4179",80,"/foobar.html",[]}} = @@ -89,24 +94,52 @@ ipv6(Config) when is_list(Config) -> [{foo, false}]), {error, {malformed_url, _, "http://2010:836B:4179::836B:4179/foobar.html"}} = - http_uri:parse("http://2010:836B:4179::836B:4179/foobar.html"). + http_uri:parse("http://2010:836B:4179::836B:4179/foobar.html"), + + {ok, {http,<<>>,<<"2010:836B:4179::836B:4179">>,80,<<"/foobar.html">>,<<>>}} = + http_uri:parse(<<"http://[2010:836B:4179::836B:4179]/foobar.html">>), + {ok, {http,<<>>,<<"[2010:836B:4179::836B:4179]">>,80,<<"/foobar.html">>,<<>>}} = + http_uri:parse(<<"http://[2010:836B:4179::836B:4179]/foobar.html">>, + [{ipv6_host_with_brackets, true}]), + {ok, {http,<<>>,<<"2010:836B:4179::836B:4179">>,80,<<"/foobar.html">>,<<>>}} = + http_uri:parse(<<"http://[2010:836B:4179::836B:4179]/foobar.html">>, + [{ipv6_host_with_brackets, false}]), + {ok, {http,<<>>,<<"2010:836B:4179::836B:4179">>,80,<<"/foobar.html">>,<<>>}} = + http_uri:parse(<<"http://[2010:836B:4179::836B:4179]/foobar.html">>, + [{foo, false}]), + {error, + {malformed_url, _, <<"http://2010:836B:4179::836B:4179/foobar.html">>}} = + http_uri:parse(<<"http://2010:836B:4179::836B:4179/foobar.html">>). host(Config) when is_list(Config) -> {ok, {http,[],"localhost",8888,"/foobar.html",[]}} = - http_uri:parse("http://localhost:8888/foobar.html"). + http_uri:parse("http://localhost:8888/foobar.html"), + + {ok, {http,<<>>,<<"localhost">>,8888,<<"/foobar.html">>,<<>>}} = + http_uri:parse(<<"http://localhost:8888/foobar.html">>). userinfo(Config) when is_list(Config) -> {ok, {http,"nisse:foobar","localhost",8888,"/foobar.html",[]}} = - http_uri:parse("http://nisse:foobar@localhost:8888/foobar.html"). + http_uri:parse("http://nisse:foobar@localhost:8888/foobar.html"), + + {ok, {http,<<"nisse:foobar">>,<<"localhost">>,8888,<<"/foobar.html">>,<<>>}} = + http_uri:parse(<<"http://nisse:foobar@localhost:8888/foobar.html">>). scheme(Config) when is_list(Config) -> {error, no_scheme} = http_uri:parse("localhost/foobar.html"), {error, {malformed_url, _, _}} = - http_uri:parse("localhost:8888/foobar.html"). + http_uri:parse("localhost:8888/foobar.html"), + + {error, no_scheme} = http_uri:parse(<<"localhost/foobar.html">>), + {error, {malformed_url, _, _}} = + http_uri:parse(<<"localhost:8888/foobar.html">>). queries(Config) when is_list(Config) -> {ok, {http,[],"localhost",8888,"/foobar.html","?foo=bar&foobar=42"}} = - http_uri:parse("http://localhost:8888/foobar.html?foo=bar&foobar=42"). + http_uri:parse("http://localhost:8888/foobar.html?foo=bar&foobar=42"), + + {ok, {http,<<>>,<<"localhost">>,8888,<<"/foobar.html">>,<<"?foo=bar&foobar=42">>}} = + http_uri:parse(<<"http://localhost:8888/foobar.html?foo=bar&foobar=42">>). fragments(Config) when is_list(Config) -> {ok, {http,[],"localhost",80,"/",""}} = @@ -142,6 +175,41 @@ fragments(Config) when is_list(Config) -> http_uri:parse("http://localhost?query#", [{fragment,true}]), {ok, {http,[],"localhost",80,"/path","?query","#"}} = http_uri:parse("http://localhost/path?query#", [{fragment,true}]), + + + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"">>}} = + http_uri:parse(<<"http://localhost#fragment">>), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"">>}} = + http_uri:parse(<<"http://localhost/path#fragment">>), + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"?query">>}} = + http_uri:parse(<<"http://localhost?query#fragment">>), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"?query">>}} = + http_uri:parse(<<"http://localhost/path?query#fragment">>), + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"">>,<<"#fragment">>}} = + http_uri:parse(<<"http://localhost#fragment">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"">>,<<"#fragment">>}} = + http_uri:parse(<<"http://localhost/path#fragment">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"?query">>,<<"#fragment">>}} = + http_uri:parse(<<"http://localhost?query#fragment">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"?query">>,<<"#fragment">>}} = + http_uri:parse(<<"http://localhost/path?query#fragment">>, + [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"">>,<<"">>}} = + http_uri:parse(<<"http://localhost">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"">>,<<"">>}} = + http_uri:parse(<<"http://localhost/path">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"?query">>,<<"">>}} = + http_uri:parse(<<"http://localhost?query">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"?query">>,<<"">>}} = + http_uri:parse(<<"http://localhost/path?query">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"">>,<<"#">>}} = + http_uri:parse(<<"http://localhost#">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"">>,<<"#">>}} = + http_uri:parse(<<"http://localhost/path#">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/">>,<<"?query">>,<<"#">>}} = + http_uri:parse(<<"http://localhost?query#">>, [{fragment,true}]), + {ok, {http,<<>>,<<"localhost">>,80,<<"/path">>,<<"?query">>,<<"#">>}} = + http_uri:parse(<<"http://localhost/path?query#">>, [{fragment,true}]), ok. escaped(Config) when is_list(Config) -> @@ -152,7 +220,16 @@ escaped(Config) when is_list(Config) -> {ok, {http,[],"www.somedomain.com",80,"/%25abc",[]}} = http_uri:parse("http://www.somedomain.com/%25abc"), {ok, {http,[],"www.somedomain.com",80,"/%25abc", "?foo=bar"}} = - http_uri:parse("http://www.somedomain.com/%25abc?foo=bar"). + http_uri:parse("http://www.somedomain.com/%25abc?foo=bar"), + + {ok, {http,<<>>,<<"www.somedomain.com">>,80,<<"/%2Eabc">>,<<>>}} = + http_uri:parse(<<"http://www.somedomain.com/%2Eabc">>), + {ok, {http,<<>>,<<"www.somedomain.com">>,80,<<"/%252Eabc">>,<<>>}} = + http_uri:parse(<<"http://www.somedomain.com/%252Eabc">>), + {ok, {http,<<>>,<<"www.somedomain.com">>,80,<<"/%25abc">>,<<>>}} = + http_uri:parse(<<"http://www.somedomain.com/%25abc">>), + {ok, {http,<<>>,<<"www.somedomain.com">>,80,<<"/%25abc">>, <<"?foo=bar">>}} = + http_uri:parse(<<"http://www.somedomain.com/%25abc?foo=bar">>). hexed_query(doc) -> [{doc, "Solves OTP-6191"}]; @@ -196,6 +273,17 @@ scheme_validation(Config) when is_list(Config) -> http_uri:parse("https://localhost#fragment", [{scheme_validation_fun, none}]). +encode_decode(Config) when is_list(Config) -> + ?assertEqual("foo%20bar", http_uri:encode("foo bar")), + ?assertEqual(<<"foo%20bar">>, http_uri:encode(<<"foo bar">>)), + + ?assertEqual("foo bar", http_uri:decode("foo+bar")), + ?assertEqual(<<"foo bar">>, http_uri:decode(<<"foo+bar">>)), + ?assertEqual("foo bar", http_uri:decode("foo%20bar")), + ?assertEqual(<<"foo bar">>, http_uri:decode(<<"foo%20bar">>)), + ?assertEqual("foo\r\n", http_uri:decode("foo%0D%0A")), + ?assertEqual(<<"foo\r\n">>, http_uri:decode(<<"foo%0D%0A">>)). + %%-------------------------------------------------------------------- %% Internal Functions ------------------------------------------------ diff --git a/lib/jinterface/doc/src/notes.xml b/lib/jinterface/doc/src/notes.xml index 9ae188be6c..30f607c357 100644 --- a/lib/jinterface/doc/src/notes.xml +++ b/lib/jinterface/doc/src/notes.xml @@ -31,26 +31,6 @@ </header> <p>This document describes the changes made to the Jinterface application.</p> -<section><title>Jinterface 1.8</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Remove generation of atoms in old latin1 external format. - The old latin1 format can still be decoded but is never - generated by <c>term_to_binary</c> or sent on a - distribution channel. The new utf8 format for atoms was - introduced in OTP R16. An OTP 20 node can therefore not - connect to nodes older than R16.</p> - <p> - Own Id: OTP-14337</p> - </item> - </list> - </section> - -</section> - <section><title>Jinterface 1.7.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/jinterface/vsn.mk b/lib/jinterface/vsn.mk index 373e2dab22..c29d9df3d7 100644 --- a/lib/jinterface/vsn.mk +++ b/lib/jinterface/vsn.mk @@ -1 +1 @@ -JINTERFACE_VSN = 1.8 +JINTERFACE_VSN = 1.7.1 diff --git a/lib/kernel/doc/src/disk_log.xml b/lib/kernel/doc/src/disk_log.xml index aebeacee28..570d3ef9bd 100644 --- a/lib/kernel/doc/src/disk_log.xml +++ b/lib/kernel/doc/src/disk_log.xml @@ -968,6 +968,12 @@ <c>read_write</c>. </p> </item> + <tag><c>{quiet, Boolean}</c></tag> + <item> + <p>Specifies if messages will be sent to + <c>error_logger</c> on recoverable errors with + the log files. Defaults to <c>true</c>.</p> + </item> </taglist> <p><c>open/1</c> returns <c>{ok, <anno>Log</anno>}</c> if the log file is successfully opened. If the file is diff --git a/lib/kernel/doc/src/error_logger.xml b/lib/kernel/doc/src/error_logger.xml index 814e8eac46..27db00819f 100644 --- a/lib/kernel/doc/src/error_logger.xml +++ b/lib/kernel/doc/src/error_logger.xml @@ -4,7 +4,7 @@ <erlref> <header> <copyright> - <year>1996</year><year>2016</year> + <year>1996</year><year>2017</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -163,6 +163,19 @@ ok</pre> </desc> </func> <func> + <name name="get_format_depth" arity="0"/> + <fsummary>Get the value of the Kernel application variable + <c>error_logger_format_depth</c>.</fsummary> + <desc> + <p>Returns <c>max(10, Depth)</c>, where <c>Depth</c> is the + value of + <seealso marker="kernel:kernel_app#error_logger_format_depth"> + error_logger_format_depth</seealso> + in the Kernel application, if Depth is an integer. Otherwise, + <c>unlimited</c> is returned.</p> + </desc> + </func> + <func> <name name="info_msg" arity="1"/> <name name="info_msg" arity="2"/> <fsummary>Send a standard information event to the error logger.</fsummary> diff --git a/lib/kernel/doc/src/kernel_app.xml b/lib/kernel/doc/src/kernel_app.xml index 28af155493..9fccb4c7ac 100644 --- a/lib/kernel/doc/src/kernel_app.xml +++ b/lib/kernel/doc/src/kernel_app.xml @@ -419,6 +419,29 @@ MaxT = TickTime + TickTime / 4</code> using this service.</p> <p>Defaults to <c>false</c>.</p> </item> + <tag><c>shell_history = enabled | disabled </c></tag> + <item> + <p>Specifies whether shell history should be logged to disk + between usages of <c>erl</c>.</p> + </item> + <tag><c>shell_history_drop = [string()]</c></tag> + <item> + <p>Specific log lines that should not be persisted. For + example <c>["q().", "init:stop()."]</c> will allow to + ignore commands that shut the node down. Defaults to + <c>[]</c>.</p> + </item> + <tag><c>shell_history_file_bytes = integer()</c></tag> + <item> + <p>how many bytes the shell should remember. By default, the + value is set to 512kb, and the minimal value is 50kb.</p> + </item> + <tag><c>shell_history_path = string()</c></tag> + <item> + <p>Specifies where the shell history files will be stored. + defaults to the user's cache directory as returned by + <c>filename:basedir(user_cache, "erlang-history")</c>.</p> + </item> <tag><c>shutdown_func = {Mod, Func}</c></tag> <item> <p>Where:</p> diff --git a/lib/kernel/doc/src/notes.xml b/lib/kernel/doc/src/notes.xml index efb822f1b3..7127a59a0c 100644 --- a/lib/kernel/doc/src/notes.xml +++ b/lib/kernel/doc/src/notes.xml @@ -31,90 +31,6 @@ </header> <p>This document describes the changes made to the Kernel application.</p> -<section><title>Kernel 5.3</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Fix bug where gethostname would incorrectly fail with - enametoolong on Linux.</p> - <p> - Own Id: OTP-14310</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Since Unicode is now allowed in atoms an extra check is - needed for node names, which are restricted to Latin-1.</p> - <p> - Own Id: OTP-13805</p> - </item> - <item> - <p>Replaced usage of deprecated symbolic <seealso - marker="erts:erlang#type-time_unit"><c>time - unit</c></seealso> representations.</p> - <p> - Own Id: OTP-13831 Aux Id: OTP-13735 </p> - </item> - <item> - <p><c>file:write_file(Name, Data, [raw])</c> would turn - <c>Data</c> into a single binary before writing. This - meant it could not take advantage of the <c>writev()</c> - system call if it was given a list of binaries and told - to write with <c>raw</c> mode.</p> - <p> - Own Id: OTP-13909</p> - </item> - <item> - <p>The performance of the <c>disk_log</c> has been - somewhat improved in some corner cases (big items), and - the documentation has been clarified. </p> - <p> - Own Id: OTP-14057 Aux Id: PR-1245 </p> - </item> - <item> - <p>Functions for detecting changed code has been added. - <c>code:modified_modules/0</c> returns all currently - loaded modules that have changed on disk. - <c>code:module_status/1</c> returns the status for a - module. In the shell and in <c>c</c> module, <c>mm/0</c> - is short for <c>code:modified_modules/0</c>, and - <c>lm/0</c> reloads all currently loaded modules that - have changed on disk.</p> - <p> - Own Id: OTP-14059</p> - </item> - <item> - <p> - Introduce an event manager in Erlang to handle OS - signals. A subset of OS signals may be subscribed to and - those are described in the Kernel application.</p> - <p> - Own Id: OTP-14186</p> - </item> - <item> - <p> Sockets can now be bound to device (SO_BINDTODEVICE) - on platforms where it is supported. </p> <p> This has - been implemented e.g to support VRF-Lite under Linux; see - <url - href="https://www.kernel.org/doc/Documentation/networking/vrf.txt"> - VRF </url>, and GitHub pull request <url - href="https://github.com/erlang/otp/pull/1326">#1326</url>. - </p> - <p> - Own Id: OTP-14357 Aux Id: PR-1326 </p> - </item> - </list> - </section> - -</section> - <section><title>Kernel 5.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/kernel/src/Makefile b/lib/kernel/src/Makefile index 2a89faaf13..78aa6192a9 100644 --- a/lib/kernel/src/Makefile +++ b/lib/kernel/src/Makefile @@ -85,6 +85,7 @@ MODULES = \ global_group \ global_search \ group \ + group_history \ heart \ hipe_unified_loader \ inet \ diff --git a/lib/kernel/src/disk_log.erl b/lib/kernel/src/disk_log.erl index 50a20c918c..70cbf1c87c 100644 --- a/lib/kernel/src/disk_log.erl +++ b/lib/kernel/src/disk_log.erl @@ -638,6 +638,8 @@ check_arg([{mode, read_only}|Tail], Res) -> check_arg(Tail, Res#arg{mode = read_only}); check_arg([{mode, read_write}|Tail], Res) -> check_arg(Tail, Res#arg{mode = read_write}); +check_arg([{quiet, Boolean}|Tail], Res) when is_boolean(Boolean) -> + check_arg(Tail, Res#arg{quiet = Boolean}); check_arg(Arg, _) -> {error, {badarg, Arg}}. @@ -1276,7 +1278,8 @@ compare_arg(_Attr, _Val, _A) -> do_open(A) -> #arg{type = Type, format = Format, name = Name, head = Head0, file = FName, repair = Repair, size = Size, mode = Mode, - version = V} = A, + quiet = Quiet, version = V} = A, + disk_log_1:set_quiet(Quiet), Head = mk_head(Head0, Format), case do_open2(Type, Format, Name, FName, Repair, Size, Mode, Head, V) of {ok, Ret, Extra, FormatType, NoItems} -> diff --git a/lib/kernel/src/disk_log.hrl b/lib/kernel/src/disk_log.hrl index c9fda8bef5..a362881f40 100644 --- a/lib/kernel/src/disk_log.hrl +++ b/lib/kernel/src/disk_log.hrl @@ -69,13 +69,14 @@ | {file, FileName :: file:filename()} | {linkto, LinkTo :: none | pid()} | {repair, Repair :: true | false | truncate} - | {type, Type :: dlog_type} + | {type, Type :: dlog_type()} | {format, Format :: dlog_format()} | {size, Size :: dlog_size()} | {distributed, Nodes :: [node()]} | {notify, boolean()} | {head, Head :: dlog_head_opt()} | {head_func, MFA :: {atom(), atom(), list()}} + | {quiet, boolean()} | {mode, Mode :: dlog_mode()}. -type dlog_options() :: [dlog_option()]. -type dlog_repair() :: 'truncate' | boolean(). @@ -102,6 +103,7 @@ head = none, mode = read_write :: dlog_mode(), notify = false :: boolean(), + quiet = false :: boolean(), options = [] :: dlog_options()}). -record(cache, %% Cache for logged terms (per file descriptor). diff --git a/lib/kernel/src/disk_log_1.erl b/lib/kernel/src/disk_log_1.erl index d83c30f35f..10c22e1ad6 100644 --- a/lib/kernel/src/disk_log_1.erl +++ b/lib/kernel/src/disk_log_1.erl @@ -37,6 +37,7 @@ -export([get_wrap_size/1]). -export([is_head/1]). -export([position/3, truncate_at/3, fwrite/4, fclose/2]). +-export([set_quiet/1, is_quiet/0]). -compile({inline,[{scan_f2,7}]}). @@ -500,7 +501,10 @@ lh(H, _F) -> % cannot happen repair(In, File) -> FSz = file_size(File), - error_logger:info_msg("disk_log: repairing ~tp ...\n", [File]), + case is_quiet() of + true -> ok; + _ -> error_logger:info_msg("disk_log: repairing ~tp ...\n", [File]) + end, Tmp = add_ext(File, "TMP"), {ok, {_Alloc, Out, {0, _}, _FileSize}} = new_int_file(Tmp, none), scan_f_read(<<>>, In, Out, File, FSz, Tmp, ?MAX_CHUNK_SIZE, 0, 0). @@ -769,8 +773,11 @@ mf_int_chunk(Handle, {FileNo, Pos}, Bin, N) -> NFileNo = inc(FileNo, Handle#handle.maxF), case catch int_open(FName, true, read_only, any) of {error, _Reason} -> - error_logger:info_msg("disk_log: chunk error. File ~tp missing.\n\n", - [FName]), + case is_quiet() of + true -> ok; + _ -> error_logger:info_msg("disk_log: chunk error. File ~tp missing.\n\n", + [FName]) + end, mf_int_chunk(Handle, {NFileNo, 0}, [], N); {ok, {_Alloc, FdC, _HeadSize, _FileSize}} -> case chunk(FdC, FName, Pos, Bin, N) of @@ -797,9 +804,12 @@ mf_int_chunk_read_only(Handle, {FileNo, Pos}, Bin, N) -> NFileNo = inc(FileNo, Handle#handle.maxF), case catch int_open(FName, true, read_only, any) of {error, _Reason} -> - error_logger:info_msg("disk_log: chunk error. File ~tp missing.\n\n", - [FName]), - mf_int_chunk_read_only(Handle, {NFileNo, 0}, [], N); + case is_quiet() of + true -> ok; + _ -> error_logger:info_msg("disk_log: chunk error. File ~tp missing.\n\n", + [FName]) + end, + mf_int_chunk_read_only(Handle, {NFileNo, 0}, [], N); {ok, {_Alloc, FdC, _HeadSize, _FileSize}} -> case do_chunk_read_only(FdC, FName, Pos, Bin, N) of {NewFdC, eof} -> @@ -1549,6 +1559,12 @@ fclose(#cache{fd = Fd, c = C}, FileName) -> _ = write_cache_close(Fd, FileName, C), file:close(Fd). +set_quiet(Bool) -> + put(quiet, Bool). + +is_quiet() -> + get(quiet) =:= true. + %% -> {Reply, #cache{}}; Reply = ok | Error write_cache(Fd, _FileName, []) -> {ok, #cache{fd = Fd}}; diff --git a/lib/kernel/src/error_logger.erl b/lib/kernel/src/error_logger.erl index 81f1bf8d97..9bf8547745 100644 --- a/lib/kernel/src/error_logger.erl +++ b/lib/kernel/src/error_logger.erl @@ -31,6 +31,8 @@ handle_event/2, handle_call/2, handle_info/2, terminate/2]). +-export([get_format_depth/0, limit_term/1]). + -define(buffer_size, 10). %%----------------------------------------------------------------- @@ -518,3 +520,21 @@ string_p1([H|T]) when is_list(H) -> end; string_p1([]) -> true; string_p1(_) -> false. + +-spec limit_term(term()) -> term(). + +limit_term(Term) -> + case get_format_depth() of + unlimited -> Term; + D -> io_lib:limit_term(Term, D) + end. + +-spec get_format_depth() -> 'unlimited' | pos_integer(). + +get_format_depth() -> + case application:get_env(kernel, error_logger_format_depth) of + {ok, Depth} when is_integer(Depth) -> + max(10, Depth); + undefined -> + unlimited + end. diff --git a/lib/kernel/src/group.erl b/lib/kernel/src/group.erl index b5e73117dd..0eeaaad8d2 100644 --- a/lib/kernel/src/group.erl +++ b/lib/kernel/src/group.erl @@ -33,7 +33,7 @@ start(Drv, Shell, Options) -> server(Drv, Shell, Options) -> process_flag(trap_exit, true), edlin:init(), - put(line_buffer, proplists:get_value(line_buffer, Options, [])), + put(line_buffer, proplists:get_value(line_buffer, Options, group_history:load())), put(read_mode, list), put(user_drv, Drv), put(expand_fun, @@ -783,6 +783,7 @@ save_line_buffer("\n", Lines) -> save_line_buffer(Line, [Line|_Lines]=Lines) -> save_line_buffer(Lines); save_line_buffer(Line, Lines) -> + group_history:add(Line), save_line_buffer([Line|Lines]). save_line_buffer(Lines) -> diff --git a/lib/kernel/src/group_history.erl b/lib/kernel/src/group_history.erl new file mode 100644 index 0000000000..5ca5e2d1dd --- /dev/null +++ b/lib/kernel/src/group_history.erl @@ -0,0 +1,341 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2017. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% +-module(group_history). +-export([load/0, add/1]). + +%% Make a minimal size that should encompass set of lines and then make +%% a file rotation for N files of this size. +-define(DEFAULT_HISTORY_FILE, "erlang-shell-log"). +-define(MAX_HISTORY_FILES, 10). +-define(DEFAULT_SIZE, 1024*512). % 512 kb total default +-define(DEFAULT_STATUS, disabled). +-define(MIN_HISTORY_SIZE, (50*1024)). % 50 kb, in bytes +-define(DEFAULT_DROP, []). +-define(DISK_LOG_FORMAT, internal). % since we want repairs +-define(LOG_NAME, '$#group_history'). +-define(VSN, {0,1,0}). + +%%%%%%%%%%%%%% +%%% PUBLIC %%% +%%%%%%%%%%%%%% + +%% @doc Loads the shell history from memory. This function should only be +%% called from group:server/3 to inject itself in the previous commands +%% stack. +-spec load() -> [string()]. +load() -> + wait_for_kernel_safe_sup(), + case history_status() of + enabled -> + case open_log() of + {ok, ?LOG_NAME} -> + read_full_log(?LOG_NAME); + {repaired, ?LOG_NAME, {recovered, Good}, {badbytes, Bad}} -> + report_repairs(?LOG_NAME, Good, Bad), + read_full_log(?LOG_NAME); + {error, {need_repair, _FileName}} -> + repair_log(?LOG_NAME); + {error, {arg_mismatch, repair, true, false}} -> + repair_log(?LOG_NAME); + {error, {name_already_open, _}} -> + show_rename_warning(), + read_full_log(?LOG_NAME); + {error, {size_mismatch, Current, New}} -> + show_size_warning(Current, New), + resize_log(?LOG_NAME, Current, New), + load(); + {error, {invalid_header, {vsn, Version}}} -> + upgrade_version(?LOG_NAME, Version), + load(); + {error, Reason} -> + handle_open_error(Reason), + disable_history(), + [] + end; + _ -> + [] + end. + +%% @doc adds a log line to the erlang history log, if configured to do so. +-spec add(iodata()) -> ok. +add(Line) -> add(Line, history_status()). + +add(Line, enabled) -> + case lists:member(Line, to_drop()) of + false -> + case disk_log:log(?LOG_NAME, Line) of + ok -> + ok; + {error, no_such_log} -> + _ = open_log(), % a wild attempt we hope works! + disk_log:log(?LOG_NAME, Line); + {error, _Other} -> + % just ignore, we're too late + ok + end; + true -> + ok + end; +add(_Line, disabled) -> + ok. + +%%%%%%%%%%%%%%% +%%% PRIVATE %%% +%%%%%%%%%%%%%%% + +%% Because loading the shell happens really damn early, processes we depend on +%% might not be there yet. Luckily, the load function is called from the shell +%% after a new process has been spawned, so we can block in here +wait_for_kernel_safe_sup() -> + case whereis(kernel_safe_sup) of + undefined -> + timer:sleep(50), + wait_for_kernel_safe_sup(); + _ -> + ok + end. + +%% Repair the log out of band +repair_log(Name) -> + Opts = lists:keydelete(size, 1, log_options()), + case disk_log:open(Opts) of + {repaired, ?LOG_NAME, {recovered, Good}, {badbytes, Bad}} -> + report_repairs(?LOG_NAME, Good, Bad); + _ -> + ok + end, + _ = disk_log:close(Name), + load(). + +%% Return whether the shell history is enabled or not +-spec history_status() -> enabled | disabled. +history_status() -> + case is_user() orelse application:get_env(kernel, shell_history) of + true -> disabled; % don't run for user proc + {ok, enabled} -> enabled; + undefined -> ?DEFAULT_STATUS; + _ -> disabled + end. + +%% Return whether the user process is running this +-spec is_user() -> boolean(). +is_user() -> + case process_info(self(), registered_name) of + {registered_name, user} -> true; + _ -> false + end. + +%% Open a disk_log file while ensuring the required path is there. +open_log() -> + Opts = log_options(), + _ = ensure_path(Opts), + disk_log:open(Opts). + +%% Return logger options +log_options() -> + Path = find_path(), + File = filename:join([Path, ?DEFAULT_HISTORY_FILE]), + Size = find_wrap_values(), + [{name, ?LOG_NAME}, + {file, File}, + {repair, true}, + {format, internal}, + {type, wrap}, + {size, Size}, + {distributed, []}, + {notify, false}, + {head, {vsn, ?VSN}}, + {quiet, true}, + {mode, read_write}]. + +-spec ensure_path([{file, string()} | {atom(), _}, ...]) -> ok | {error, term()}. +ensure_path(Opts) -> + {file, Path} = lists:keyfind(file, 1, Opts), + filelib:ensure_dir(Path). + +%% @private read the logs from an already open file. Treat closed files +%% as wrong and returns an empty list to avoid crash loops in the shell. +-spec read_full_log(term()) -> [string()]. +read_full_log(Name) -> + case disk_log:chunk(Name, start) of + {error, no_such_log} -> + show_unexpected_close_warning(), + []; + eof -> + []; + {Cont, Logs} -> + lists:reverse(maybe_drop_header(Logs) ++ read_full_log(Name, Cont)) + end. + +read_full_log(Name, Cont) -> + case disk_log:chunk(Name, Cont) of + {error, no_such_log} -> + show_unexpected_close_warning(), + []; + eof -> + []; + {NextCont, Logs} -> + maybe_drop_header(Logs) ++ read_full_log(Name, NextCont) + end. + +maybe_drop_header([{vsn, _} | Rest]) -> Rest; +maybe_drop_header(Logs) -> Logs. + +-spec handle_open_error(_) -> ok. +handle_open_error({arg_mismatch, OptName, CurrentVal, NewVal}) -> + show('$#erlang-history-arg-mismatch', + "Log file argument ~p changed value from ~p to ~p " + "and cannot be automatically updated. Please clear the " + "history files and try again.~n", + [OptName, CurrentVal, NewVal]); +handle_open_error({not_a_log_file, FileName}) -> + show_invalid_file_warning(FileName); +handle_open_error({invalid_index_file, FileName}) -> + show_invalid_file_warning(FileName); +handle_open_error({invalid_header, Term}) -> + show('$#erlang-history-invalid-header', + "Shell history expects to be able to use the log files " + "which currently have unknown headers (~p) and may belong to " + "another mechanism. History logging will be " + "disabled.~n", + [Term]); +handle_open_error({file_error, FileName, Reason}) -> + show('$#erlang-history-file-error', + "Error handling File ~s. Reason: ~p~n" + "History logging will be disabled.~n", + [FileName, Reason]); +handle_open_error(Err) -> + show_unexpected_warning({disk_log, open, 1}, Err). + +find_wrap_values() -> + ConfSize = case application:get_env(kernel, shell_history_file_bytes) of + undefined -> ?DEFAULT_SIZE; + {ok, S} -> S + end, + SizePerFile = max(?MIN_HISTORY_SIZE, ConfSize div ?MAX_HISTORY_FILES), + FileCount = if SizePerFile > ?MIN_HISTORY_SIZE -> + ?MAX_HISTORY_FILES + ; SizePerFile =< ?MIN_HISTORY_SIZE -> + max(1, ConfSize div SizePerFile) + end, + {SizePerFile, FileCount}. + +report_repairs(_, _, 0) -> + %% just a regular close repair + ok; +report_repairs(_, Good, Bad) -> + show('$#erlang-history-report-repairs', + "The shell history log file was corrupted and was repaired. " + "~p bytes were recovered and ~p were lost.~n", [Good, Bad]). + +resize_log(Name, _OldSize, NewSize) -> + show('$#erlang-history-resize-attempt', + "Attempting to resize the log history file to ~p...", [NewSize]), + Opts = lists:keydelete(size, 1, log_options()), + _ = case disk_log:open(Opts) of + {error, {need_repair, _}} -> + _ = repair_log(Name), + disk_log:open(Opts); + _ -> + ok + end, + case disk_log:change_size(Name, NewSize) of + ok -> + show('$#erlang-history-resize-result', + "ok~n", []); + {error, {new_size_too_small, _}} -> + show('$#erlang-history-resize-result', + "failed (new size is too small)~n", []), + disable_history(); + {error, Reason} -> + show('$#erlang-history-resize-result', + "failed (~p)~n", [Reason]), + disable_history() + end. + +upgrade_version(_Name, Unsupported) -> + %% We only know of one version and can't support a newer one + show('$#erlang-history-upgrade', + "The version for the shell logs found on disk (~p) is " + "not supported by the current version (~p)~n", + [Unsupported, ?VSN]), + disable_history(). + +disable_history() -> + show('$#erlang-history-disable', "Disabling shell history logging.~n", []), + application:set_env(kernel, shell_history, force_disabled). + +find_path() -> + case application:get_env(kernel, shell_history_path) of + undefined -> filename:basedir(user_cache, "erlang-history"); + {ok, Path} -> Path + end. + +to_drop() -> + case application:get_env(kernel, shell_history_drop) of + undefined -> + application:set_env(kernel, shell_history_drop, ?DEFAULT_DROP), + ?DEFAULT_DROP; + {ok, V} when is_list(V) -> [Ln++"\n" || Ln <- V]; + {ok, _} -> ?DEFAULT_DROP + end. + +%%%%%%%%%%%%%%%%%%%%%%%% +%%% Output functions %%% +%%%%%%%%%%%%%%%%%%%%%%%% +show_rename_warning() -> + show('$#erlang-history-rename-warn', + "A history file with a different path has already " + "been started for the shell of this node. The old " + "name will keep being used for this session.~n", + []). + +show_invalid_file_warning(FileName) -> + show('$#erlang-history-invalid-file', + "Shell history expects to be able to use the file ~s " + "which currently exists and is not a file usable for " + "history logging purposes. History logging will be " + "disabled.~n", [FileName]). + +show_unexpected_warning({M,F,A}, Term) -> + show('$#erlang-history-unexpected-return', + "unexpected return value from ~p:~p/~p: ~p~n" + "shell history will be disabled for this session.~n", + [M,F,A,Term]). + +show_unexpected_close_warning() -> + show('$#erlang-history-unexpected-close', + "The shell log file has mysteriousy closed. Ignoring " + "currently unread history.~n", []). + +show_size_warning(_Current, _New) -> + show('$#erlang-history-size', + "The configured log history file size is different from " + "the size of the log file on disk.~n", []). + +show(Key, Format, Args) -> + case get(Key) of + undefined -> + io:format(standard_error, Format, Args), + put(Key, true), + ok; + true -> + ok + end. diff --git a/lib/kernel/src/kernel.app.src b/lib/kernel/src/kernel.app.src index 25e4ddd95c..1128ee3ec5 100644 --- a/lib/kernel/src/kernel.app.src +++ b/lib/kernel/src/kernel.app.src @@ -44,6 +44,7 @@ global_group, global_search, group, + group_history, heart, hipe_unified_loader, inet6_tcp, diff --git a/lib/kernel/src/kernel.erl b/lib/kernel/src/kernel.erl index 6e5022a405..cba57088ec 100644 --- a/lib/kernel/src/kernel.erl +++ b/lib/kernel/src/kernel.erl @@ -123,7 +123,7 @@ init([]) -> restart => permanent, shutdown => 2000, type => worker, - modeules => [file, file_server, file_io_server, prim_file]}, + modules => [file, file_server, file_io_server, prim_file]}, StdError = #{id => standard_error, start => {standard_error, start_link, []}, diff --git a/lib/kernel/test/code_SUITE.erl b/lib/kernel/test/code_SUITE.erl index 7831777726..afc32283ba 100644 --- a/lib/kernel/test/code_SUITE.erl +++ b/lib/kernel/test/code_SUITE.erl @@ -107,14 +107,19 @@ init_per_testcase(big_boot_embedded, Config) -> _Else -> {skip, "Needs crypto!"} end; -init_per_testcase(on_load_embedded, Config) -> +init_per_testcase(on_load_embedded, Config0) -> LibRoot = code:lib_dir(), LinkName = filename:join(LibRoot, "on_load_app-1.0"), - [{link_name,LinkName}|Config]; + Config = [{link_name,LinkName}|Config0], + init_per_testcase(Config); init_per_testcase(_Func, Config) -> + init_per_testcase(Config). + +init_per_testcase(Config) -> P = code:get_path(), [{code_path, P}|Config]. + end_per_testcase(module_status, Config) -> code:purge(?TESTMOD), code:delete(?TESTMOD), diff --git a/lib/kernel/test/disk_log_SUITE.erl b/lib/kernel/test/disk_log_SUITE.erl index 069df5a11d..2b11a0381f 100644 --- a/lib/kernel/test/disk_log_SUITE.erl +++ b/lib/kernel/test/disk_log_SUITE.erl @@ -2493,6 +2493,7 @@ error_repair(Conf) when is_list(Conf) -> del(File, No), ok = file:del_dir(Dir), + error_logger:add_report_handler(?MODULE, self()), %% repair a file P1 = pps(), {ok, n} = disk_log:open([{name, n}, {file, File}, {type, wrap}, @@ -2509,6 +2510,8 @@ error_repair(Conf) when is_list(Conf) -> ok = disk_log:close(n), true = (P1 == pps()), del(File, No), + receive {info_msg, _, "disk_log: repairing" ++ _, _} -> ok + after 1000 -> ct:fail(failed) end, %% yet another repair P2 = pps(), @@ -2525,6 +2528,8 @@ error_repair(Conf) when is_list(Conf) -> ok = disk_log:close(n), true = (P2 == pps()), del(File, No), + receive {info_msg, _, "disk_log: repairing" ++ _, _} -> ok + after 1000 -> ct:fail(failed) end, %% Repair, large term Big = term_to_binary(lists:duplicate(66000,$a)), @@ -2540,6 +2545,8 @@ error_repair(Conf) when is_list(Conf) -> ok = disk_log:close(n), Got = Big, del(File, No), + receive {info_msg, _, "disk_log: repairing" ++ _, _} -> ok + after 1000 -> ct:fail(failed) end, %% A term a little smaller than a chunk, then big terms. BigSmall = mk_bytes(1024*64-8-12), @@ -2560,6 +2567,8 @@ error_repair(Conf) when is_list(Conf) -> {type, halt}, {format, internal}]), ok = disk_log:close(n), file:delete(File), + receive {info_msg, _, "disk_log: repairing" ++ _, _} -> ok + after 1000 -> ct:fail(failed) end, %% The header is recovered. {ok,n} = @@ -2573,12 +2582,13 @@ error_repair(Conf) when is_list(Conf) -> crash(File, 30), {repaired,n,{recovered,3},{badbytes,15}} = disk_log:open([{name, n}, {file, File}, {type, halt}, - {format, internal},{repair,true}, + {format, internal},{repair,true}, {quiet, true}, {head_func, {?MODULE, head_fun, [{ok,"head"}]}}]), ["head",'of',terms] = get_all_terms(n), ok = disk_log:close(n), - + error_logger:delete_report_handler(?MODULE), file:delete(File), + {messages, []} = process_info(self(), messages), ok. @@ -5007,6 +5017,9 @@ init(Tester) -> handle_event({error_report, _GL, {Pid, crash_report, Report}}, Tester) -> Tester ! {crash_report, Pid, Report}, {ok, Tester}; +handle_event({info_msg, _GL, {Pid, F,A}}, Tester) -> + Tester ! {info_msg, Pid, F, A}, + {ok, Tester}; handle_event(_Event, State) -> {ok, State}. diff --git a/lib/kernel/vsn.mk b/lib/kernel/vsn.mk index 4edecd8969..76b020e8ed 100644 --- a/lib/kernel/vsn.mk +++ b/lib/kernel/vsn.mk @@ -1 +1 @@ -KERNEL_VSN = 5.3 +KERNEL_VSN = 5.2 diff --git a/lib/megaco/doc/src/notes.xml b/lib/megaco/doc/src/notes.xml index aa3d434dd7..a05a339003 100644 --- a/lib/megaco/doc/src/notes.xml +++ b/lib/megaco/doc/src/notes.xml @@ -37,21 +37,7 @@ section is the version number of Megaco.</p> - <section><title>Megaco 3.18.2</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p>Typos have been fixed.</p> - <p> - Own Id: OTP-14387</p> - </item> - </list> - </section> - -</section> - -<section><title>Megaco 3.18.1</title> + <section><title>Megaco 3.18.1</title> <section><title>Improvements and New Features</title> <list> diff --git a/lib/megaco/vsn.mk b/lib/megaco/vsn.mk index ca8049daee..0d40f863b2 100644 --- a/lib/megaco/vsn.mk +++ b/lib/megaco/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = megaco -MEGACO_VSN = 3.18.2 +MEGACO_VSN = 3.18.1 PRE_VSN = APP_VSN = "$(APPLICATION)-$(MEGACO_VSN)$(PRE_VSN)" diff --git a/lib/mnesia/doc/src/notes.xml b/lib/mnesia/doc/src/notes.xml index 3ca4026190..062a62298b 100644 --- a/lib/mnesia/doc/src/notes.xml +++ b/lib/mnesia/doc/src/notes.xml @@ -39,26 +39,7 @@ thus constitutes one section in this document. The title of each section is the version number of Mnesia.</p> - <section><title>Mnesia 4.15</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Removed the wrapping of select continuations in extension - plugin handling. This might require the user to rewrite - user backend plugin if used.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14039</p> - </item> - </list> - </section> - -</section> - -<section><title>Mnesia 4.14.3</title> + <section><title>Mnesia 4.14.3</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/mnesia/vsn.mk b/lib/mnesia/vsn.mk index 81b15d65db..e272a469bb 100644 --- a/lib/mnesia/vsn.mk +++ b/lib/mnesia/vsn.mk @@ -1 +1 @@ -MNESIA_VSN = 4.15 +MNESIA_VSN = 4.14.3 diff --git a/lib/observer/doc/src/notes.xml b/lib/observer/doc/src/notes.xml index dd043d1014..bd9aa265f8 100644 --- a/lib/observer/doc/src/notes.xml +++ b/lib/observer/doc/src/notes.xml @@ -32,39 +32,6 @@ <p>This document describes the changes made to the Observer application.</p> -<section><title>Observer 2.4</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Show dirty-scheduler threads in performance monitor graph - and add a column with maximum allocated memory in the - Memory Allocators table.</p> - <p> - Own Id: OTP-14137</p> - </item> - <item> - <p> - Keep table and port selection after refresh of tables. - Store settings before shutdown and restore when starting - application.</p> - <p> - Own Id: OTP-14270</p> - </item> - <item> - <p> - When observing a node older than OTP-19.0, a pop-up will - be displayed when trying to access port information. - Earlier, observer would crash in this situation.</p> - <p> - Own Id: OTP-14345 Aux Id: ERL-399 </p> - </item> - </list> - </section> - -</section> - <section><title>Observer 2.3.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/observer/src/etop.erl b/lib/observer/src/etop.erl index 925f4456bb..2093e6a0d7 100644 --- a/lib/observer/src/etop.erl +++ b/lib/observer/src/etop.erl @@ -180,10 +180,16 @@ stop(Opts) -> end, unregister(etop_server). -update(#opts{store=Store,node=Node,tracing=Tracing}=Opts) -> +update(#opts{store=Store,node=Node,tracing=Tracing,intv=Interval}=Opts) -> Pid = spawn_link(Node,observer_backend,etop_collect,[self()]), Info = receive {Pid,I} -> I - after 1000 -> exit(connection_lost) + after Interval -> + %% Took more than the update interval to fetch + %% data. Either the connection is lost or the + %% fetching took too long... + io:format("Timeout when waiting for process info from " + "node ~p; exiting~n", [Node]), + exit(timeout) end, #etop_info{procinfo=ProcInfo} = Info, ProcInfo1 = diff --git a/lib/observer/src/observer_alloc_wx.erl b/lib/observer/src/observer_alloc_wx.erl index 9506a2b380..ef425f0874 100644 --- a/lib/observer/src/observer_alloc_wx.erl +++ b/lib/observer/src/observer_alloc_wx.erl @@ -194,14 +194,17 @@ code_change(_, _, State) -> %%%%%%%%%% restart_fetcher(Node, #state{panel=Panel, wins=Wins0, time=Ti} = State) -> - SysInfo = observer_wx:try_rpc(Node, observer_backend, sys_info, []), - Info = alloc_info(SysInfo), - Max = lists:foldl(fun calc_max/2, #{}, Info), - {Wins, Samples} = add_data(Info, {0, queue:new()}, Wins0, Ti, true), - erlang:send_after(1000 div ?DISP_FREQ, self(), {refresh, 0}), - wxWindow:refresh(Panel), - precalc(State#state{active=true, appmon=Node, time=Ti#ti{tick=0}, - wins=Wins, samples=Samples, max=Max}). + case rpc:call(Node, observer_backend, sys_info, []) of + {badrpc, _} -> State; + SysInfo -> + Info = alloc_info(SysInfo), + Max = lists:foldl(fun calc_max/2, #{}, Info), + {Wins, Samples} = add_data(Info, {0, queue:new()}, Wins0, Ti, true), + erlang:send_after(1000 div ?DISP_FREQ, self(), {refresh, 0}), + wxWindow:refresh(Panel), + precalc(State#state{active=true, appmon=Node, time=Ti#ti{tick=0}, + wins=Wins, samples=Samples, max=Max}) + end. precalc(#state{samples=Data0, paint=Paint, time=Ti, wins=Wins0}=State) -> Wins = [precalc(Ti, Data0, Paint, Win) || Win <- Wins0], diff --git a/lib/observer/src/observer_pro_wx.erl b/lib/observer/src/observer_pro_wx.erl index ffa6f6d3b4..3083297f31 100644 --- a/lib/observer/src/observer_pro_wx.erl +++ b/lib/observer/src/observer_pro_wx.erl @@ -67,12 +67,14 @@ -record(holder, {parent, info, - etop, + next=[], sort=#sort{}, accum=[], + next_accum=[], attrs, node, - backend_pid + backend_pid, + old_backend=false }). -record(state, {parent, @@ -226,7 +228,7 @@ handle_info({holder_updated, Count}, State0=#state{grid=Grid}) -> wxListCtrl:setItemCount(Grid, Count), Count > 0 andalso wxListCtrl:refreshItems(Grid, 0, Count-1), - + observer_wx:set_status(io_lib:format("Number of Processes: ~w", [Count])), {noreply, State}; handle_info(refresh_interval, #state{holder=Holder}=State) -> @@ -459,13 +461,13 @@ rm_selected(_, [], [], AccIds, AccPids) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%TABLE HOLDER%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% init_table_holder(Parent, Accum0, Attrs) -> - Backend = spawn_link(node(), observer_backend,etop_collect,[self()]), + process_flag(trap_exit, true), + Backend = spawn_link(node(), observer_backend, procs_info, [self()]), Accum = case Accum0 of true -> true; false -> [] end, table_holder(#holder{parent=Parent, - etop=#etop_info{}, info=array:new(), node=node(), backend_pid=Backend, @@ -474,7 +476,7 @@ init_table_holder(Parent, Accum0, Attrs) -> }). table_holder(#holder{info=Info, attrs=Attrs, - node=Node, backend_pid=Backend}=S0) -> + node=Node, backend_pid=Backend, old_backend=Old}=S0) -> receive {get_row, From, Row, Col} -> get_row(From, Row, Col, Info), @@ -482,14 +484,25 @@ table_holder(#holder{info=Info, attrs=Attrs, {get_attr, From, Row} -> get_attr(From, Row, Attrs), table_holder(S0); + {procs_info, Backend, Procs} -> + State = handle_update(Procs, S0), + table_holder(State); + {'EXIT', Backend, normal} when Old =:= false -> + S1 = update_complete(S0), + table_holder(S1#holder{backend_pid=undefined}); {Backend, EtopInfo=#etop_info{}} -> - State = handle_update(EtopInfo, S0), + State = handle_update_old(EtopInfo, S0), table_holder(State#holder{backend_pid=undefined}); refresh when is_pid(Backend)-> table_holder(S0); %% Already updating refresh -> - Pid = spawn_link(Node,observer_backend,etop_collect,[self()]), - table_holder(S0#holder{backend_pid=Pid}); + Pid = case Old of + true -> + spawn_link(Node, observer_backend, etop_collect, [self()]); + false -> + spawn_link(Node, observer_backend, procs_info, [self()]) + end, + table_holder(S0#holder{backend_pid=Pid}); {change_sort, Col} -> State = change_sort(Col, S0), table_holder(State); @@ -502,7 +515,6 @@ table_holder(#holder{info=Info, attrs=Attrs, {get_name_or_pid, From, Indices} -> get_name_or_pid(From, Indices, Info), table_holder(S0); - {get_node, From} -> From ! {self(), Node}, table_holder(S0); @@ -511,36 +523,50 @@ table_holder(#holder{info=Info, attrs=Attrs, true -> table_holder(S0); false -> - self() ! refresh, - table_holder(S0#holder{node=NewNode}) - end; + _ = rpc:call(NewNode, code, ensure_loaded, [observer_backend]), + case rpc:call(NewNode, erlang, function_exported, + [observer_backend,procs_info, 1]) of + true -> + self() ! refresh, + table_holder(S0#holder{node=NewNode, old_backend=false}); + false -> + self() ! refresh, + table_holder(S0#holder{node=NewNode, old_backend=true}); + _ -> + table_holder(S0) + end + end; {accum, Bool} -> table_holder(change_accum(Bool,S0)); {get_accum, From} -> From ! {self(), S0#holder.accum == true}, table_holder(S0); {dump, Fd} -> - EtopInfo = (S0#holder.etop)#etop_info{procinfo=array:to_list(Info)}, - %% The empty #etop_info{} below is a dummy previous info - %% value. It is used by etop to calculate the scheduler - %% utilization since last update. When dumping to file, - %% there is no previous measurement to use, so we just add - %% a dummy here, and the value shown will be since the - %% tool was started. - etop_txt:do_update(Fd, EtopInfo, #etop_info{}, #opts{node=Node}), - file:close(Fd), - table_holder(S0); + Collector = spawn_link(Node, observer_backend, etop_collect,[self()]), + receive + {Collector, EtopInfo=#etop_info{}} -> + etop_txt:do_update(Fd, EtopInfo, #etop_info{}, #opts{node=Node}), + file:close(Fd), + table_holder(S0); + {'EXIT', Collector, _} -> + table_holder(S0) + end; stop -> ok; - What -> - io:format("Table holder got ~p~n",[What]), + {'EXIT', Backend, normal} -> + table_holder(S0); + {'EXIT', Backend, _Reason} -> + %% Node crashed will be noticed soon.. + table_holder(S0#holder{backend_pid=undefined}); + _What -> + %% io:format("~p: Table holder got ~p~n",[?MODULE, _What]), table_holder(S0) end. change_sort(Col, S0=#holder{parent=Parent, info=Data, sort=Sort0}) -> {Sort, ProcInfo}=sort(Col, Sort0, Data), Parent ! {holder_updated, array:size(Data)}, - S0#holder{info=ProcInfo, sort=Sort}. + S0#holder{info=array:from_list(ProcInfo), sort=Sort}. change_accum(true, S0) -> S0#holder{accum=true}; @@ -548,23 +574,45 @@ change_accum(false, S0=#holder{info=Info}) -> self() ! refresh, S0#holder{accum=lists:sort(array:to_list(Info))}. -handle_update(EI=#etop_info{procinfo=ProcInfo0}, - S0=#holder{parent=Parent, sort=Sort=#sort{sort_key=KeyField}}) -> - {ProcInfo1, S1} = accum(ProcInfo0, S0), +handle_update_old(#etop_info{procinfo=ProcInfo0}, + S0=#holder{parent=Parent, sort=Sort=#sort{sort_key=KeyField}}) -> + {ProcInfo1, Accum} = accum(ProcInfo0, S0), {_SO, ProcInfo} = sort(KeyField, Sort#sort{sort_key=undefined}, ProcInfo1), - Parent ! {holder_updated, array:size(ProcInfo)}, - S1#holder{info=ProcInfo, etop=EI#etop_info{procinfo=[]}}. + Info = array:from_list(ProcInfo), + Parent ! {holder_updated, array:size(Info)}, + S0#holder{info=Info, accum=Accum}. + +handle_update(ProcInfo0, S0=#holder{next=Next, sort=#sort{sort_key=KeyField}}) -> + {ProcInfo1, Accum} = accum(ProcInfo0, S0), + Sort = sort_fun(KeyField, true), + Merge = merge_fun(KeyField), + Merged = Merge(Sort(ProcInfo1), Next), + case Accum of + true -> S0#holder{next=Merged}; + _List -> S0#holder{next=Merged, next_accum=Accum} + end. -accum(ProcInfo, State=#holder{accum=true}) -> - {ProcInfo, State}; -accum(ProcInfo0, State=#holder{accum=Previous}) -> +update_complete(#holder{parent=Parent, sort=#sort{sort_incr=Incr}, + next=ProcInfo, accum=Accum, next_accum=NextAccum}=S0) -> + Info = case Incr of + true -> array:from_list(ProcInfo); + false -> array:from_list(lists:reverse(ProcInfo)) + end, + Parent ! {holder_updated, array:size(Info)}, + S0#holder{info=Info, accum= Accum =:= true orelse NextAccum, + next=[], next_accum=[]}. + +accum(ProcInfo, #holder{accum=true}) -> + {ProcInfo, true}; +accum(ProcInfo0, #holder{accum=Previous, next_accum=Next}) -> + Accum = [{Pid, Reds} || #etop_proc_info{pid=Pid, reds=Reds} <- ProcInfo0], ProcInfo = lists:sort(ProcInfo0), - {accum2(ProcInfo,Previous,[]), State#holder{accum=ProcInfo}}. + {accum2(ProcInfo,Previous,[]), lists:merge(lists:sort(Accum), Next)}. -accum2([PI=#etop_proc_info{pid=Pid, reds=Reds, runtime=RT}|PIs], - [#etop_proc_info{pid=Pid, reds=OldReds, runtime=OldRT}|Old], Acc) -> - accum2(PIs, Old, [PI#etop_proc_info{reds=Reds-OldReds, runtime=RT-OldRT}|Acc]); -accum2(PIs=[#etop_proc_info{pid=Pid}|_], [#etop_proc_info{pid=OldPid}|Old], Acc) +accum2([PI=#etop_proc_info{pid=Pid, reds=Reds}|PIs], + [{Pid, OldReds}|Old], Acc) -> + accum2(PIs, Old, [PI#etop_proc_info{reds=Reds-OldReds}|Acc]); +accum2(PIs=[#etop_proc_info{pid=Pid}|_], [{OldPid,_}|Old], Acc) when Pid > OldPid -> accum2(PIs, Old, Acc); accum2([PI|PIs], Old, Acc) -> @@ -575,14 +623,52 @@ sort(Col, Opt, Table) when not is_list(Table) -> sort(Col,Opt,array:to_list(Table)); sort(Col, Opt=#sort{sort_key=Col, sort_incr=Bool}, Table) -> - {Opt#sort{sort_incr=not Bool}, - array:from_list(lists:reverse(Table))}; -sort(Col, S=#sort{sort_incr=true}, Table) -> - {S#sort{sort_key=Col}, - array:from_list(lists:keysort(col_to_element(Col), Table))}; -sort(Col, S=#sort{sort_incr=false}, Table) -> - {S#sort{sort_key=Col}, - array:from_list(lists:reverse(lists:keysort(col_to_element(Col), Table)))}. + {Opt#sort{sort_incr=not Bool},lists:reverse(Table)}; +sort(Col, S=#sort{sort_incr=Incr}, Table) -> + Sort = sort_fun(Col, Incr), + {S#sort{sort_key=Col}, Sort(Table)}. + +sort_fun(?COL_NAME, true) -> + fun(Table) -> lists:sort(fun sort_name/2, Table) end; +sort_fun(?COL_NAME, false) -> + fun(Table) -> lists:sort(fun sort_name_rev/2, Table) end; +sort_fun(Col, true) -> + N = col_to_element(Col), + fun(Table) -> lists:keysort(N, Table) end; +sort_fun(Col, false) -> + N = col_to_element(Col), + fun(Table) -> lists:reverse(lists:keysort(N, Table)) end. + +merge_fun(?COL_NAME) -> + fun(A,B) -> lists:merge(fun sort_name/2, A, B) end; +merge_fun(Col) -> + KeyField = col_to_element(Col), + fun(A,B) -> lists:keymerge(KeyField, A, B) end. + + +sort_name(#etop_proc_info{name={_,_,_}=A}, #etop_proc_info{name={_,_,_}=B}) -> + A =< B; +sort_name(#etop_proc_info{name=A}, #etop_proc_info{name=B}) + when is_atom(A), is_atom(B) -> + A =< B; +sort_name(#etop_proc_info{name=Reg}, #etop_proc_info{name={M,_F,_A}}) + when is_atom(Reg) -> + Reg < M; +sort_name(#etop_proc_info{name={M,_,_}}, #etop_proc_info{name=Reg}) + when is_atom(Reg) -> + M < Reg. + +sort_name_rev(#etop_proc_info{name={_,_,_}=A}, #etop_proc_info{name={_,_,_}=B}) -> + A >= B; +sort_name_rev(#etop_proc_info{name=A}, #etop_proc_info{name=B}) + when is_atom(A), is_atom(B) -> + A >= B; +sort_name_rev(#etop_proc_info{name=Reg}, #etop_proc_info{name={M,_F,_A}}) + when is_atom(Reg) -> + Reg >= M; +sort_name_rev(#etop_proc_info{name={M,_,_}}, #etop_proc_info{name=Reg}) + when is_atom(Reg) -> + M >= Reg. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/lib/observer/src/observer_wx.erl b/lib/observer/src/observer_wx.erl index 739761e7fd..9b9e80f479 100644 --- a/lib/observer/src/observer_wx.erl +++ b/lib/observer/src/observer_wx.erl @@ -143,7 +143,8 @@ setup(#state{frame = Frame} = State) -> wxFrame:setTitle(Frame, atom_to_list(node())), wxStatusBar:setStatusText(StatusBar, atom_to_list(node())), - wxNotebook:connect(Notebook, command_notebook_page_changed, [{skip, true}]), + wxNotebook:connect(Notebook, command_notebook_page_changed, + [{skip, true}, {id, ?ID_NOTEBOOK}]), wxFrame:connect(Frame, close_window, []), wxMenu:connect(Frame, command_menu_selected), wxFrame:show(Frame), @@ -230,12 +231,13 @@ setup(#state{frame = Frame} = State) -> %%Callbacks handle_event(#wx{event=#wxNotebook{type=command_notebook_page_changed, nSel=Next}}, - #state{active_tab=Previous, node=Node, panels=Panels} = State) -> + #state{active_tab=Previous, node=Node, panels=Panels, status_bar=SB} = State) -> {_, Obj, _} = lists:nth(Next+1, Panels), case wx_object:get_pid(Obj) of Previous -> {noreply, State}; Pid -> + wxStatusBar:setStatusText(SB, ""), Previous ! not_active, Pid ! {active, Node}, {noreply, State#state{active_tab=Pid}} diff --git a/lib/observer/vsn.mk b/lib/observer/vsn.mk index 21edfcd184..ca9ad72473 100644 --- a/lib/observer/vsn.mk +++ b/lib/observer/vsn.mk @@ -1 +1 @@ -OBSERVER_VSN = 2.4 +OBSERVER_VSN = 2.3.1 diff --git a/lib/parsetools/doc/src/notes.xml b/lib/parsetools/doc/src/notes.xml index e9f81fb86e..5a16445577 100644 --- a/lib/parsetools/doc/src/notes.xml +++ b/lib/parsetools/doc/src/notes.xml @@ -31,33 +31,6 @@ </header> <p>This document describes the changes made to the Parsetools application.</p> -<section><title>Parsetools 2.1.5</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Minor documentation fixes</p> - <p> - Own Id: OTP-14276 Aux Id: PR-1357 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> Miscellaneous updates due to atoms containing - arbitrary Unicode characters. </p> - <p> - Own Id: OTP-14285</p> - </item> - </list> - </section> - -</section> - <section><title>Parsetools 2.1.4</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/parsetools/vsn.mk b/lib/parsetools/vsn.mk index 502ca00a47..d102c63730 100644 --- a/lib/parsetools/vsn.mk +++ b/lib/parsetools/vsn.mk @@ -1 +1 @@ -PARSETOOLS_VSN = 2.1.5 +PARSETOOLS_VSN = 2.1.4 diff --git a/lib/public_key/doc/src/notes.xml b/lib/public_key/doc/src/notes.xml index 64592a6d87..dd83e1961d 100644 --- a/lib/public_key/doc/src/notes.xml +++ b/lib/public_key/doc/src/notes.xml @@ -35,41 +35,6 @@ <file>notes.xml</file> </header> -<section><title>Public_Key 1.4.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Bug for <c>public_key:generate_key({namedCurve,OID})</c> - fixed.</p> - <p> - Own Id: OTP-14258</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Modernized internal representation used for crl - validation by use of maps.</p> - <p> - Own Id: OTP-14111</p> - </item> - <item> - <p> - Support EC key in pkix_sign/2</p> - <p> - Own Id: OTP-14294</p> - </item> - </list> - </section> - -</section> - <section><title>Public_Key 1.4</title> <section><title>Improvements and New Features</title> diff --git a/lib/public_key/vsn.mk b/lib/public_key/vsn.mk index 83a77d2a28..b94768ae77 100644 --- a/lib/public_key/vsn.mk +++ b/lib/public_key/vsn.mk @@ -1 +1 @@ -PUBLIC_KEY_VSN = 1.4.1 +PUBLIC_KEY_VSN = 1.4 diff --git a/lib/runtime_tools/doc/src/notes.xml b/lib/runtime_tools/doc/src/notes.xml index 388fa54fa8..49615cf077 100644 --- a/lib/runtime_tools/doc/src/notes.xml +++ b/lib/runtime_tools/doc/src/notes.xml @@ -32,41 +32,6 @@ <p>This document describes the changes made to the Runtime_Tools application.</p> -<section><title>Runtime_Tools 1.12</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Add compile option <c>-compile(no_native)</c> in modules - with <c>on_load</c> directive which is not yet supported - by HiPE.</p> - <p> - Own Id: OTP-14316 Aux Id: PR-1390 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> Sockets can now be bound to device (SO_BINDTODEVICE) - on platforms where it is supported. </p> <p> This has - been implemented e.g to support VRF-Lite under Linux; see - <url - href="https://www.kernel.org/doc/Documentation/networking/vrf.txt"> - VRF </url>, and GitHub pull request <url - href="https://github.com/erlang/otp/pull/1326">#1326</url>. - </p> - <p> - Own Id: OTP-14357 Aux Id: PR-1326 </p> - </item> - </list> - </section> - -</section> - <section><title>Runtime_Tools 1.11.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/runtime_tools/src/observer_backend.erl b/lib/runtime_tools/src/observer_backend.erl index 1e0d2d642e..d36af257ce 100644 --- a/lib/runtime_tools/src/observer_backend.erl +++ b/lib/runtime_tools/src/observer_backend.erl @@ -23,7 +23,7 @@ -export([vsn/0]). %% observer stuff --export([sys_info/0, get_port_list/0, +-export([sys_info/0, get_port_list/0, procs_info/1, get_table/3, get_table_list/2, fetch_stats/2]). %% etop stuff @@ -293,6 +293,23 @@ fetch_stats_loop(Parent, Time) -> try erlang:memory() catch _:_ -> [] end}, fetch_stats_loop(Parent, Time) end. + +%% +%% Chunk sending process info to etop/observer +%% +procs_info(Collector) -> + All = processes(), + Send = fun Send (Pids) -> + try lists:split(10000, Pids) of + {First, Rest} -> + Collector ! {procs_info, self(), etop_collect(First, [])}, + Send(Rest) + catch _:_ -> + Collector ! {procs_info, self(), etop_collect(Pids, [])} + end + end, + Send(All). + %% %% etop backend %% diff --git a/lib/runtime_tools/vsn.mk b/lib/runtime_tools/vsn.mk index 5ee39a25fe..8ec532de76 100644 --- a/lib/runtime_tools/vsn.mk +++ b/lib/runtime_tools/vsn.mk @@ -1 +1 @@ -RUNTIME_TOOLS_VSN = 1.12 +RUNTIME_TOOLS_VSN = 1.11.1 diff --git a/lib/sasl/doc/src/notes.xml b/lib/sasl/doc/src/notes.xml index 03b8a333a9..8684151b3a 100644 --- a/lib/sasl/doc/src/notes.xml +++ b/lib/sasl/doc/src/notes.xml @@ -31,23 +31,6 @@ </header> <p>This document describes the changes made to the SASL application.</p> -<section><title>SASL 3.0.4</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Documented default values for the 'mod' and - 'start_phases' fields in .app files were not allowed as - actual values in a .app file. This is now corrected.</p> - <p> - Own Id: OTP-14029</p> - </item> - </list> - </section> - -</section> - <section><title>SASL 3.0.3</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/sasl/src/sasl_report.erl b/lib/sasl/src/sasl_report.erl index 14702b0ad2..eb454155d5 100644 --- a/lib/sasl/src/sasl_report.erl +++ b/lib/sasl/src/sasl_report.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2016. All Rights Reserved. +%% Copyright Ericsson AB 1996-2017. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -68,12 +68,12 @@ write_report2(IO, Fd, Head, progress, Report) -> Format = format_key_val(Report), write_report_action(IO, Fd, Head, "~s", [Format]); write_report2(IO, Fd, Head, crash_report, Report) -> - Depth = get_depth(), + Depth = error_logger:get_format_depth(), Format = proc_lib:format(Report, latin1, Depth), write_report_action(IO, Fd, Head, "~s", [Format]). supervisor_format(Args0) -> - case get_depth() of + case error_logger:get_format_depth() of unlimited -> {" Supervisor: ~p~n" " Context: ~p~n" @@ -102,14 +102,6 @@ format_key_val([{Tag,Data}|Rep]) -> format_key_val(_) -> []. -get_depth() -> - case application:get_env(kernel, error_logger_format_depth) of - {ok, Depth} when is_integer(Depth) -> - max(10, Depth); - undefined -> - unlimited - end. - sup_get(Tag, Report) -> case lists:keysearch(Tag, 1, Report) of {value, {_, Value}} -> diff --git a/lib/sasl/vsn.mk b/lib/sasl/vsn.mk index 2608ca5e00..6aa662a743 100644 --- a/lib/sasl/vsn.mk +++ b/lib/sasl/vsn.mk @@ -1 +1 @@ -SASL_VSN = 3.0.4 +SASL_VSN = 3.0.3 diff --git a/lib/snmp/src/agent/snmp_generic.erl b/lib/snmp/src/agent/snmp_generic.erl index fc23e16ef1..4738525341 100644 --- a/lib/snmp/src/agent/snmp_generic.erl +++ b/lib/snmp/src/agent/snmp_generic.erl @@ -413,20 +413,21 @@ table_check_status(NameDb, Col, ?'RowStatus_createAndGo', RowIndex, Cols) -> false -> % it's ok to use snmpa_local_db:table_construct_row since it's % side effect free and we only use the result temporary. - case catch snmpa_local_db:table_construct_row( + try snmpa_local_db:table_construct_row( NameDb, RowIndex, ?'RowStatus_createAndGo', Cols) of - {'EXIT', _Reason} -> - ?vtrace( - "failed construct row (createAndGo): " - " n Reason: ~p" - " n Stack: ~p", - [_Reason, erlang:get_stacktrace()]), - {noCreation, Col}; % Bad RowIndex Row -> case lists:member(noinit, tuple_to_list(Row)) of false -> {noError, 0}; _Found -> {inconsistentValue, Col} end + catch + _:_Reason -> + ?vtrace( + "failed construct row (createAndGo): " + " n Reason: ~p" + " n Stack: ~p", + [_Reason, erlang:get_stacktrace()]), + {noCreation, Col} % Bad RowIndex end; true -> {inconsistentValue, Col} end; @@ -435,17 +436,18 @@ table_check_status(NameDb, Col, ?'RowStatus_createAndGo', RowIndex, Cols) -> table_check_status(NameDb, Col, ?'RowStatus_createAndWait', RowIndex, Cols) -> case table_row_exists(NameDb, RowIndex) of false -> - case catch snmpa_local_db:table_construct_row( + try snmpa_local_db:table_construct_row( NameDb, RowIndex, ?'RowStatus_createAndGo', Cols) of - {'EXIT', _Reason} -> + _Row -> + {noError, 0} + catch + _:_Reason -> ?vtrace( "failed construct row (createAndWait): " " n Reason: ~p" " n Stack: ~p", [_Reason, erlang:get_stacktrace()]), - {noCreation, Col}; % Bad RowIndex - _Row -> - {noError, 0} + {noCreation, Col} % Bad RowIndex end; true -> {inconsistentValue, Col} end; diff --git a/lib/ssh/doc/src/notes.xml b/lib/ssh/doc/src/notes.xml index ac31ab14a6..bddae00dd2 100644 --- a/lib/ssh/doc/src/notes.xml +++ b/lib/ssh/doc/src/notes.xml @@ -30,123 +30,6 @@ <file>notes.xml</file> </header> -<section><title>Ssh 4.5</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - The internal handling of SSH options is re-written.</p> - <p> - Previously there were no checks if a client option was - given to a daemon or vice versa. This is corrected now. - If your code has e.g. a client-only option in a call to - start a daemon, the call will fail.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-12872</p> - </item> - <item> - <p> - Modernization of diffie-hellman algorithms. Removed an - outdated weak algorithm and added stronger replacements - to keep interoperability with other ssh clients and - servers. The default ordering is also adjusted.</p> - <p> - Retired: key-exchange <c>diffie-hellman-group1-sha1</c>. - It is not enabled by default, but can be enabled with the - option <c>preferred-algorithms</c>.</p> - <p> - Added: key-exchange <c>diffie-hellman-group16-sha512</c>, - <c>diffie-hellman-group18-sha512</c> and - <c>diffie-hellman-group14-sha256</c>.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14110</p> - </item> - <item> - <p> - Modernized internal representation of sftp by use of - maps.</p> - <p> - Own Id: OTP-14117</p> - </item> - <item> - <p> - The draft-ietf-curdle-ssh-ext-info-05 and - draft-ietf-curdle-rsa-sha2-05 are (partially) - implemented. </p> - <p> - The extension <c>server-sig-algs</c> and the signature - algorithms <c>rsa-sha2-256</c> and <c>rsa-sha2-512</c> - are implemented. The <c>rsa-sha2-*</c> are not enabled by - default yet since they has a bug. Use option - preferred_algorithms with key public key to try them.</p> - <p> - Own Id: OTP-14193</p> - </item> - <item> - <p> - Ssh:connect, ssh:shell and ssh:start_channel now accept - an IP-tuple as Host destination argument.</p> - <p> - Own Id: OTP-14243</p> - </item> - <item> - <p> - The function <c>ssh:daemon_info/1</c> now returns Host - and Profile as well as the Port info.</p> - <p> - Own Id: OTP-14259</p> - </item> - <item> - <p> - Removed the option <c>public_key_alg</c> which was - deprecated in 18.2. Use <c>pref_public_key_algs</c> - instead.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14263</p> - </item> - <item> - <p> - The ssh application is refactored regarding daemon - starting. The resolution of contradicting <c>Host</c> - argument and <c>ip</c> option were not described. There - were also strange corner cases when the <c>'any'</c> - value was used in <c>Host</c> argument or <c>ip</c> - option. This is (hopefully) resolved now, but it may - cause incompatibilities for code using both <c>Host</c> - and the <c>ip</c> option. The value 'loopback' has been - added for a correct way of naming those addresses.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14264</p> - </item> - <item> - <p> - The supervisor code is refactored. The naming of - listening IP-Port-Profile tripples are slightly changed - to improve consistency in strange corner cases as - resolved by OTP-14264</p> - <p> - Own Id: OTP-14267 Aux Id: OTP-14266 </p> - </item> - <item> - <p> - The <c>idle_time</c> option can now be used in daemons.</p> - <p> - Own Id: OTP-14312</p> - </item> - </list> - </section> - -</section> - <section><title>Ssh 4.4.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/ssh/doc/src/ssh.xml b/lib/ssh/doc/src/ssh.xml index 84b7cdd7a1..5c9ce3d5fb 100644 --- a/lib/ssh/doc/src/ssh.xml +++ b/lib/ssh/doc/src/ssh.xml @@ -246,10 +246,12 @@ <tag><c><![CDATA[{pref_public_key_algs, list()}]]></c></tag> <item> <p>List of user (client) public key algorithms to try to use.</p> - <p>The default value is - <c><![CDATA[['ssh-rsa','ssh-dss','ecdsa-sha2-nistp256','ecdsa-sha2-nistp384','ecdsa-sha2-nistp521'] ]]></c> + <p>The default value is the <c>public_key</c> entry in + <seealso marker="#default_algorithms/0">ssh:default_algorithms/0</seealso>. + </p> + <p>If there is no public key of a specified type available, the corresponding entry is ignored. + Note that the available set is dependent on the underlying cryptolib and current user's public keys. </p> - <p>If there is no public key of a specified type available, the corresponding entry is ignored.</p> </item> <tag><c><![CDATA[{preferred_algorithms, algs_list()}]]></c></tag> @@ -293,6 +295,15 @@ connection. For <c>gen_tcp</c> the time is in milli-seconds and the default value is <c>infinity</c>.</p> </item> + + <tag><c><![CDATA[{auth_methods, string()}]]></c></tag> + <item> + <p>Comma-separated string that determines which + authentication methods that the client shall support and + in which order they are tried. Defaults to + <c><![CDATA["publickey,keyboard-interactive,password"]]></c></p> + </item> + <tag><c><![CDATA[{user, string()}]]></c></tag> <item> <p>Provides a username. If this option is not given, <c>ssh</c> @@ -300,6 +311,7 @@ <c><![CDATA[USER]]></c> on UNIX, <c><![CDATA[USERNAME]]></c> on Windows).</p> </item> + <tag><c><![CDATA[{password, string()}]]></c></tag> <item> <p>Provides a password for password authentication. @@ -307,6 +319,7 @@ password, if the password authentication method is attempted.</p> </item> + <tag><c><![CDATA[{key_cb, key_cb()}]]></c></tag> <item> <p>Module implementing the behaviour <seealso @@ -316,6 +329,7 @@ module via the options passed to it under the key 'key_cb_private'. </p> </item> + <tag><c><![CDATA[{quiet_mode, atom() = boolean()}]]></c></tag> <item> <p>If <c>true</c>, the client does not print anything on authorization.</p> @@ -466,6 +480,7 @@ authentication methods that the server is to support and in what order they are tried. Defaults to <c><![CDATA["publickey,keyboard-interactive,password"]]></c></p> + <p>Note that the client is free to use any order and to exclude methods.</p> </item> <tag><c><![CDATA[{auth_method_kb_interactive_data, PromptTexts}]]></c> diff --git a/lib/ssh/src/ssh_connection_handler.erl b/lib/ssh/src/ssh_connection_handler.erl index 342583306b..6a6b9896cb 100644 --- a/lib/ssh/src/ssh_connection_handler.erl +++ b/lib/ssh/src/ssh_connection_handler.erl @@ -434,11 +434,7 @@ init_ssh_record(Role, Socket, Opts) -> init_ssh_record(Role, _Socket, PeerAddr, Opts) -> KeyCb = ?GET_OPT(key_cb, Opts), - AuthMethods = - case Role of - server -> ?GET_OPT(auth_methods, Opts); - client -> undefined - end, + AuthMethods = ?GET_OPT(auth_methods, Opts), S0 = #ssh{role = Role, key_cb = KeyCb, opts = Opts, @@ -1705,15 +1701,18 @@ handle_ssh_msg_ext_info(#ssh_msg_ext_info{data=Data}, D0) -> lists:foldl(fun ext_info/2, D0, Data). -ext_info({"server-sig-algs",SigAlgs}, D0 = #data{ssh_params=#ssh{role=client}=Ssh0}) -> +ext_info({"server-sig-algs",SigAlgs}, D0 = #data{ssh_params=#ssh{role=client, + userauth_pubkeys=ClientSigAlgs}=Ssh0}) -> %% Make strings to eliminate risk of beeing bombed with odd strings that fills the atom table: SupportedAlgs = lists:map(fun erlang:atom_to_list/1, ssh_transport:supported_algorithms(public_key)), - Ssh = Ssh0#ssh{userauth_pubkeys = - [list_to_atom(SigAlg) || SigAlg <- string:tokens(SigAlgs,","), - %% length of SigAlg is implicitly checked by member: - lists:member(SigAlg, SupportedAlgs) - ]}, - D0#data{ssh_params = Ssh}; + ServerSigAlgs = [list_to_atom(SigAlg) || SigAlg <- string:tokens(SigAlgs,","), + %% length of SigAlg is implicitly checked by the comparison + %% in member/2: + lists:member(SigAlg, SupportedAlgs) + ], + CommonAlgs = [Alg || Alg <- ServerSigAlgs, + lists:member(Alg, ClientSigAlgs)], + D0#data{ssh_params = Ssh0#ssh{userauth_pubkeys = CommonAlgs} }; ext_info(_, D0) -> %% Not implemented diff --git a/lib/ssh/src/ssh_file.erl b/lib/ssh/src/ssh_file.erl index 6692432fcf..33792da38f 100644 --- a/lib/ssh/src/ssh_file.erl +++ b/lib/ssh/src/ssh_file.erl @@ -75,10 +75,7 @@ host_key(Algorithm, Opts) -> Password = proplists:get_value(identity_pass_phrase(Algorithm), Opts, ignore), case decode(File, Password) of {ok,Key} -> - case ssh_transport:valid_key_sha_alg(Key,Algorithm) of - true -> {ok,Key}; - false -> {error,bad_keytype_in_file} - end; + check_key_type(Key, Algorithm); {error,DecodeError} -> {error,DecodeError} end. @@ -104,10 +101,20 @@ is_host_key(Key, PeerName, Algorithm, Opts) -> user_key(Algorithm, Opts) -> File = file_name(user, identity_key_filename(Algorithm), Opts), Password = proplists:get_value(identity_pass_phrase(Algorithm), Opts, ignore), - decode(File, Password). + case decode(File, Password) of + {ok, Key} -> + check_key_type(Key, Algorithm); + Error -> + Error + end. %% Internal functions %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +check_key_type(Key, Algorithm) -> + case ssh_transport:valid_key_sha_alg(Key,Algorithm) of + true -> {ok,Key}; + false -> {error,bad_keytype_in_file} + end. file_base_name('ssh-rsa' ) -> "ssh_host_rsa_key"; file_base_name('rsa-sha2-256' ) -> "ssh_host_rsa_key"; diff --git a/lib/ssh/src/ssh_options.erl b/lib/ssh/src/ssh_options.erl index 0886d5b34d..12c0190082 100644 --- a/lib/ssh/src/ssh_options.erl +++ b/lib/ssh/src/ssh_options.erl @@ -293,12 +293,6 @@ default(server) -> class => user_options }, - {auth_methods, def} => - #{default => ?SUPPORTED_AUTH_METHODS, - chk => fun check_string/1, - class => user_options - }, - {auth_method_kb_interactive_data, def} => #{default => undefined, % Default value can be constructed when User is known chk => fun({S1,S2,S3,B}) -> @@ -398,6 +392,12 @@ default(server) -> class => user_options }, + {preferred_algorithms, def} => + #{default => ssh:default_algorithms(), + chk => fun check_preferred_algorithms/1, + class => user_options + }, + %%%%% Undocumented {infofun, def} => #{default => fun(_,_,_) -> void end, @@ -436,12 +436,24 @@ default(client) -> }, {pref_public_key_algs, def} => - #{default => - ssh_transport:supported_algorithms(public_key), - chk => - fun check_pref_public_key_algs/1, - class => - ssh + #{default => ssh_transport:default_algorithms(public_key) -- ['rsa-sha2-256', + 'rsa-sha2-512'], + chk => fun check_pref_public_key_algs/1, + class => user_options + }, + + {preferred_algorithms, def} => + #{default => [{K,Vs} || {K,Vs0} <- ssh:default_algorithms(), + Vs <- [case K of + public_key -> + Vs0 -- ['rsa-sha2-256', + 'rsa-sha2-512']; + _ -> + Vs0 + end] + ], + chk => fun check_preferred_algorithms/1, + class => user_options }, {dh_gex_limits, def} => @@ -509,12 +521,6 @@ default(common) -> class => user_options }, - {preferred_algorithms, def} => - #{default => ssh:default_algorithms(), - chk => fun check_preferred_algorithms/1, - class => user_options - }, - {id_string, def} => #{default => undefined, % FIXME: see ssh_transport:ssh_vsn/0 chk => fun(random) -> @@ -582,6 +588,21 @@ default(common) -> class => user_options }, + {auth_methods, def} => + #{default => ?SUPPORTED_AUTH_METHODS, + chk => fun(As) -> + try + Sup = string:tokens(?SUPPORTED_AUTH_METHODS, ","), + New = string:tokens(As, ","), + [] == [X || X <- New, + not lists:member(X,Sup)] + catch + _:_ -> false + end + end, + class => user_options + }, + %%%%% Undocumented {transport, def} => #{default => ?DEFAULT_TRANSPORT, @@ -808,16 +829,23 @@ valid_hash(X, _) -> error_in_check(X, "Expect atom or list in fingerprint spec" %%%---------------------------------------------------------------- check_preferred_algorithms(Algs) -> + [error_in_check(K,"Bad preferred_algorithms key") + || {K,_} <- Algs, + not lists:keymember(K,1,ssh:default_algorithms())], + try alg_duplicates(Algs, [], []) of [] -> {true, - [try ssh_transport:supported_algorithms(Key) - of - DefAlgs -> handle_pref_alg(Key,Vals,DefAlgs) - catch - _:_ -> error_in_check(Key,"Bad preferred_algorithms key") - end || {Key,Vals} <- Algs] + [case proplists:get_value(Key, Algs) of + undefined -> + {Key,DefAlgs}; + Vals -> + handle_pref_alg(Key,Vals,SupAlgs) + end + || {{Key,DefAlgs}, {Key,SupAlgs}} <- lists:zip(ssh:default_algorithms(), + ssh_transport:supported_algorithms()) + ] }; Dups -> diff --git a/lib/ssh/src/ssh_transport.erl b/lib/ssh/src/ssh_transport.erl index 25c64a4f25..1a15798080 100644 --- a/lib/ssh/src/ssh_transport.erl +++ b/lib/ssh/src/ssh_transport.erl @@ -92,10 +92,7 @@ default_algorithms(cipher) -> default_algorithms(mac) -> supported_algorithms(mac, same(['AEAD_AES_128_GCM', 'AEAD_AES_256_GCM'])); -default_algorithms(public_key) -> - supported_algorithms(public_key, ['rsa-sha2-256', - 'rsa-sha2-384', - 'rsa-sha2-512']); + default_algorithms(Alg) -> supported_algorithms(Alg, []). @@ -122,10 +119,9 @@ supported_algorithms(public_key) -> {'ecdsa-sha2-nistp384', [{public_keys,ecdsa}, {hashs,sha384}, {ec_curve,secp384r1}]}, {'ecdsa-sha2-nistp521', [{public_keys,ecdsa}, {hashs,sha512}, {ec_curve,secp521r1}]}, {'ecdsa-sha2-nistp256', [{public_keys,ecdsa}, {hashs,sha256}, {ec_curve,secp256r1}]}, + {'ssh-rsa', [{public_keys,rsa}, {hashs,sha} ]}, {'rsa-sha2-256', [{public_keys,rsa}, {hashs,sha256} ]}, - {'rsa-sha2-384', [{public_keys,rsa}, {hashs,sha384} ]}, {'rsa-sha2-512', [{public_keys,rsa}, {hashs,sha512} ]}, - {'ssh-rsa', [{public_keys,rsa}, {hashs,sha} ]}, {'ssh-dss', [{public_keys,dss}, {hashs,sha} ]} % Gone in OpenSSH 7.3.p1 ]); @@ -724,14 +720,28 @@ kex_ext_info(Role, Opts) -> end. ext_info_message(#ssh{role=client, - send_ext_info=true} = Ssh0) -> - %% FIXME: no extensions implemented - {ok, "", Ssh0}; + send_ext_info=true, + opts=Opts} = Ssh0) -> + %% Since no extension sent by the client is implemented, we add a fake one + %% to be able to test the framework. + %% Remove this when there is one and update ssh_protocol_SUITE whare it is used. + case proplists:get_value(ext_info_client, ?GET_OPT(tstflg,Opts)) of + true -> + Msg = #ssh_msg_ext_info{nr_extensions = 1, + data = [{"[email protected]", "Testing,PleaseIgnore"}] + }, + {SshPacket, Ssh} = ssh_packet(Msg, Ssh0), + {ok, SshPacket, Ssh}; + _ -> + {ok, "", Ssh0} + end; ext_info_message(#ssh{role=server, - send_ext_info=true} = Ssh0) -> + send_ext_info=true, + opts = Opts} = Ssh0) -> AlgsList = lists:map(fun erlang:atom_to_list/1, - ssh_transport:default_algorithms(public_key)), + proplists:get_value(public_key, + ?GET_OPT(preferred_algorithms, Opts))), Msg = #ssh_msg_ext_info{nr_extensions = 1, data = [{"server-sig-algs", string:join(AlgsList,",")}] }, diff --git a/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl b/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl index 410a9ea983..0995182623 100644 --- a/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl +++ b/lib/ssh/test/property_test/ssh_eqc_encode_decode.erl @@ -284,8 +284,18 @@ fix_asym(#ssh_msg_global_request{name=N} = M) -> M#ssh_msg_global_request{name = fix_asym(#ssh_msg_debug{message=D,language=L} = M) -> M#ssh_msg_debug{message = binary_to_list(D), language = binary_to_list(L)}; fix_asym(#ssh_msg_kexinit{cookie=C} = M) -> M#ssh_msg_kexinit{cookie = <<C:128>>}; + +fix_asym(#ssh_msg_kexdh_reply{public_host_key = Key} = M) -> M#ssh_msg_kexdh_reply{public_host_key = key_sigalg(Key)}; +fix_asym(#ssh_msg_kex_dh_gex_reply{public_host_key = Key} = M) -> M#ssh_msg_kex_dh_gex_reply{public_host_key = key_sigalg(Key)}; +fix_asym(#ssh_msg_kex_ecdh_reply{public_host_key = Key} = M) -> M#ssh_msg_kex_ecdh_reply{public_host_key = key_sigalg(Key)}; + fix_asym(M) -> M. +%%% Keys now contains an sig-algorithm name +key_sigalg(#'RSAPublicKey'{} = Key) -> {Key,'ssh-rsa'}; +key_sigalg({_, #'Dss-Parms'{}} = Key) -> {Key,'ssh-dss'}; +key_sigalg({#'ECPoint'{}, {namedCurve,OID}} = Key) -> {Key,"ecdsa-sha2-256"}. + %%% Message codes 30 and 31 are overloaded depending on kex family so arrange the decoder %%% input as the test object does decode_state(<<30,_/binary>>=Msg, KexFam) -> <<KexFam/binary, Msg/binary>>; diff --git a/lib/ssh/test/ssh_algorithms_SUITE.erl b/lib/ssh/test/ssh_algorithms_SUITE.erl index 6e6269d3e0..736461624d 100644 --- a/lib/ssh/test/ssh_algorithms_SUITE.erl +++ b/lib/ssh/test/ssh_algorithms_SUITE.erl @@ -68,7 +68,7 @@ groups() -> TagGroupSet ++ AlgoTcSet. -tags() -> [kex,cipher,mac,compression]. +tags() -> [kex,cipher,mac,compression,public_key]. two_way_tags() -> [cipher,mac,compression]. %%-------------------------------------------------------------------- @@ -123,20 +123,35 @@ init_per_group(Group, Config) -> Tag = proplists:get_value(name, hd(proplists:get_value(tc_group_path, Config))), Alg = Group, - PA = - case split(Alg) of - [_] -> - [Alg]; - [A1,A2] -> - [{client2server,[A1]}, - {server2client,[A2]}] - end, - ct:log("Init tests for tag=~p alg=~p",[Tag,PA]), - PrefAlgs = {preferred_algorithms,[{Tag,PA}]}, - start_std_daemon([PrefAlgs], - [{pref_algs,PrefAlgs} | Config]) + init_per_group(Tag, Alg, Config) end. + +init_per_group(public_key=Tag, Alg, Config) -> + ct:log("Init tests for public_key ~p",[Alg]), + PrefAlgs = {preferred_algorithms,[{Tag,[Alg]}]}, + %% Daemon started later in init_per_testcase + [{pref_algs,PrefAlgs}, + {tag_alg,{Tag,Alg}} + | Config]; + +init_per_group(Tag, Alg, Config) -> + PA = + case split(Alg) of + [_] -> + [Alg]; + [A1,A2] -> + [{client2server,[A1]}, + {server2client,[A2]}] + end, + ct:log("Init tests for tag=~p alg=~p",[Tag,PA]), + PrefAlgs = {preferred_algorithms,[{Tag,PA}]}, + start_std_daemon([PrefAlgs], + [{pref_algs,PrefAlgs}, + {tag_alg,{Tag,Alg}} + | Config]). + + end_per_group(_Alg, Config) -> case proplists:get_value(srvr_pid,Config) of Pid when is_pid(Pid) -> @@ -148,23 +163,49 @@ end_per_group(_Alg, Config) -> -init_per_testcase(sshc_simple_exec_os_cmd, Config) -> - start_pubkey_daemon([proplists:get_value(pref_algs,Config)], Config); -init_per_testcase(_TC, Config) -> - Config. +init_per_testcase(TC, Config) -> + init_per_testcase(TC, proplists:get_value(tag_alg,Config), Config). -end_per_testcase(sshc_simple_exec_os_cmd, Config) -> - case proplists:get_value(srvr_pid,Config) of - Pid when is_pid(Pid) -> - ssh:stop_daemon(Pid), - ct:log("stopped ~p",[proplists:get_value(srvr_addr,Config)]); - _ -> - ok +init_per_testcase(_, {public_key,Alg}, Config) -> + Opts = pubkey_opts(Config), + case {ssh_file:user_key(Alg,Opts), ssh_file:host_key(Alg,Opts)} of + {{ok,_}, {ok,_}} -> + start_pubkey_daemon([proplists:get_value(pref_algs,Config)], + [{extra_daemon,true}|Config]); + {{ok,_}, _} -> + {skip, "No host key"}; + + {_, {ok,_}} -> + {skip, "No user key"}; + + _ -> + {skip, "Neither host nor user key"} end; -end_per_testcase(_TC, Config) -> + +init_per_testcase(sshc_simple_exec_os_cmd, _, Config) -> + start_pubkey_daemon([proplists:get_value(pref_algs,Config)], + [{extra_daemon,true}|Config]); + +init_per_testcase(_, _, Config) -> Config. + +end_per_testcase(_TC, Config) -> + case proplists:get_value(extra_daemon, Config, false) of + true -> + case proplists:get_value(srvr_pid,Config) of + Pid when is_pid(Pid) -> + ssh:stop_daemon(Pid), + ct:log("stopped ~p",[proplists:get_value(srvr_addr,Config)]), + Config; + _ -> + Config + end; + _ -> + Config + end. + %%-------------------------------------------------------------------- %% Test Cases -------------------------------------------------------- %%-------------------------------------------------------------------- @@ -260,8 +301,9 @@ sshc_simple_exec_os_cmd(Config) -> %%-------------------------------------------------------------------- %% Connect to the ssh server of the OS -sshd_simple_exec(_Config) -> +sshd_simple_exec(Config) -> ConnectionRef = ssh_test_lib:connect(22, [{silently_accept_hosts, true}, + proplists:get_value(pref_algs,Config), {user_interaction, false}]), {ok, ChannelId0} = ssh_connection:session_channel(ConnectionRef, infinity), success = ssh_connection:exec(ConnectionRef, ChannelId0, @@ -318,29 +360,32 @@ concat(A1, A2) -> list_to_atom(lists:concat([A1," + ",A2])). split(Alg) -> ssh_test_lib:to_atoms(string:tokens(atom_to_list(Alg), " + ")). specific_test_cases(Tag, Alg, SshcAlgos, SshdAlgos, TypeSSH) -> - [simple_exec, simple_sftp] ++ - case supports(Tag, Alg, SshcAlgos) of - true when TypeSSH == openSSH -> - [sshc_simple_exec_os_cmd]; - _ -> - [] - end ++ - case supports(Tag, Alg, SshdAlgos) of - true -> - [sshd_simple_exec]; - _ -> - [] - end ++ - case {Tag,Alg} of - {kex,_} when Alg == 'diffie-hellman-group-exchange-sha1' ; - Alg == 'diffie-hellman-group-exchange-sha256' -> - [simple_exec_groups, - simple_exec_groups_no_match_too_large, - simple_exec_groups_no_match_too_small - ]; - _ -> - [] - end. + case Tag of + public_key -> []; + _ -> [simple_exec, simple_sftp] + end + ++ case supports(Tag, Alg, SshcAlgos) of + true when TypeSSH == openSSH -> + [sshc_simple_exec_os_cmd]; + _ -> + [] + end ++ + case supports(Tag, Alg, SshdAlgos) of + true -> + [sshd_simple_exec]; + _ -> + [] + end ++ + case {Tag,Alg} of + {kex,_} when Alg == 'diffie-hellman-group-exchange-sha1' ; + Alg == 'diffie-hellman-group-exchange-sha256' -> + [simple_exec_groups, + simple_exec_groups_no_match_too_large, + simple_exec_groups_no_match_too_small + ]; + _ -> + [] + end. supports(Tag, Alg, Algos) -> lists:all(fun(A) -> @@ -370,19 +415,30 @@ start_std_daemon(Opts, Config) -> ct:log("started ~p:~p ~p",[Host,Port,Opts]), [{srvr_pid,Pid},{srvr_addr,{Host,Port}} | Config]. + start_pubkey_daemon(Opts0, Config) -> - Opts = [{auth_methods,"publickey"}|Opts0], - {Pid, Host, Port} = ssh_test_lib:std_daemon1(Config, Opts), - ct:log("started pubkey_daemon ~p:~p ~p",[Host,Port,Opts]), + ct:log("starting pubkey_daemon",[]), + Opts = pubkey_opts(Config) ++ Opts0, + {Pid, Host, Port} = ssh_test_lib:daemon([{failfun, fun ssh_test_lib:failfun/2} + | Opts]), + ct:log("started ~p:~p ~p",[Host,Port,Opts]), [{srvr_pid,Pid},{srvr_addr,{Host,Port}} | Config]. +pubkey_opts(Config) -> + SystemDir = filename:join(proplists:get_value(priv_dir,Config), "system"), + [{auth_methods,"publickey"}, + {system_dir, SystemDir}]. + + setup_pubkey(Config) -> DataDir = proplists:get_value(data_dir, Config), UserDir = proplists:get_value(priv_dir, Config), - ssh_test_lib:setup_dsa(DataDir, UserDir), - ssh_test_lib:setup_rsa(DataDir, UserDir), - ssh_test_lib:setup_ecdsa("256", DataDir, UserDir), + Keys = + [ssh_test_lib:setup_dsa(DataDir, UserDir), + ssh_test_lib:setup_rsa(DataDir, UserDir), + ssh_test_lib:setup_ecdsa("256", DataDir, UserDir)], + ssh_test_lib:write_auth_keys(Keys, UserDir), % 'authorized_keys' shall contain ALL pub keys Config. diff --git a/lib/ssh/test/ssh_basic_SUITE.erl b/lib/ssh/test/ssh_basic_SUITE.erl index b80c3ed5e2..62e2a585e4 100644 --- a/lib/ssh/test/ssh_basic_SUITE.erl +++ b/lib/ssh/test/ssh_basic_SUITE.erl @@ -612,7 +612,7 @@ exec_key_differs(Config, UserPKAlgs) -> {_Pid, _Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir}, {user_dir, SystemUserDir}, {preferred_algorithms, - [{public_key,['ssh-rsa']}]}]), + [{public_key,['ssh-rsa'|UserPKAlgs]}]}]), ct:sleep(500), IO = ssh_test_lib:start_io_server(), @@ -1173,13 +1173,10 @@ login_bad_pwd_no_retry3(Config) -> login_bad_pwd_no_retry(Config, "password,publickey,keyboard-interactive"). login_bad_pwd_no_retry4(Config) -> - login_bad_pwd_no_retry(Config, "password,other,keyboard-interactive"). + login_bad_pwd_no_retry(Config, "password,keyboard-interactive"). login_bad_pwd_no_retry5(Config) -> - login_bad_pwd_no_retry(Config, "password,other,keyboard-interactive,password,password"). - - - + login_bad_pwd_no_retry(Config, "password,keyboard-interactive,password,password"). login_bad_pwd_no_retry(Config, AuthMethods) -> diff --git a/lib/ssh/test/ssh_property_test_SUITE.erl b/lib/ssh/test/ssh_property_test_SUITE.erl index 9b2a84d8e4..5ea60d8a8f 100644 --- a/lib/ssh/test/ssh_property_test_SUITE.erl +++ b/lib/ssh/test/ssh_property_test_SUITE.erl @@ -55,6 +55,9 @@ groups() -> init_per_suite(Config) -> ct_property_test:init_per_suite(Config). +end_per_suite(Config) -> + Config. + %%% One group in this suite happens to support only QuickCheck, so skip it %%% if we run proper. init_per_group(client_server, Config) -> diff --git a/lib/ssh/test/ssh_protocol_SUITE.erl b/lib/ssh/test/ssh_protocol_SUITE.erl index 5a6e0638a7..0385e30ad1 100644 --- a/lib/ssh/test/ssh_protocol_SUITE.erl +++ b/lib/ssh/test/ssh_protocol_SUITE.erl @@ -59,7 +59,8 @@ all() -> {group,service_requests}, {group,authentication}, {group,packet_size_error}, - {group,field_size_error} + {group,field_size_error}, + {group,ext_info} ]. groups() -> @@ -90,7 +91,12 @@ groups() -> bad_service_name_then_correct ]}, {authentication, [], [client_handles_keyboard_interactive_0_pwds - ]} + ]}, + {ext_info, [], [no_ext_info_s1, + no_ext_info_s2, + ext_info_s, + ext_info_c + ]} ]. @@ -644,7 +650,113 @@ client_info_line(_Config) -> ok end. +%%%-------------------------------------------------------------------- +%%% The server does not send the extension because +%%% the client does not tell the server to send it +no_ext_info_s1(Config) -> + %% Start the dameon + Server = {Pid,_,_} = ssh_test_lib:daemon([{send_ext_info,true}, + {system_dir, system_dir(Config)}]), + {ok,AfterKexState} = connect_and_kex([{server,Server}|Config]), + {ok,_} = + ssh_trpt_test_lib:exec( + [{send, #ssh_msg_service_request{name = "ssh-userauth"}}, + {match, #ssh_msg_service_accept{name = "ssh-userauth"}, receive_msg} + ], AfterKexState), + ssh:stop_daemon(Pid). + +%%%-------------------------------------------------------------------- +%%% The server does not send the extension because +%%% the server is not configured to send it +no_ext_info_s2(Config) -> + %% Start the dameon + Server = {Pid,_,_} = ssh_test_lib:daemon([{send_ext_info,false}, + {system_dir, system_dir(Config)}]), + {ok,AfterKexState} = connect_and_kex([{extra_options,[{recv_ext_info,true}]}, + {server,Server} + | Config]), + {ok,_} = + ssh_trpt_test_lib:exec( + [{send, #ssh_msg_service_request{name = "ssh-userauth"}}, + {match, #ssh_msg_service_accept{name = "ssh-userauth"}, receive_msg} + ], AfterKexState), + ssh:stop_daemon(Pid). + +%%%-------------------------------------------------------------------- +%%% The server sends the extension +ext_info_s(Config) -> + %% Start the dameon + Server = {Pid,_,_} = ssh_test_lib:daemon([{send_ext_info,true}, + {system_dir, system_dir(Config)}]), + {ok,AfterKexState} = connect_and_kex([{extra_options,[{recv_ext_info,true}]}, + {server,Server} + | Config]), + {ok,_} = + ssh_trpt_test_lib:exec( + [{match, #ssh_msg_ext_info{_='_'}, receive_msg} + ], + AfterKexState), + ssh:stop_daemon(Pid). + +%%%-------------------------------------------------------------------- +%%% The client sends the extension +ext_info_c(Config) -> + {User,_Pwd} = server_user_password(Config), + + %% Create a listening socket as server socket: + {ok,InitialState} = ssh_trpt_test_lib:exec(listen), + HostPort = ssh_trpt_test_lib:server_host_port(InitialState), + + Parent = self(), + %% Start a process handling one connection on the server side: + Pid = + spawn_link( + fun() -> + Result = + ssh_trpt_test_lib:exec( + [{set_options, [print_ops, print_messages]}, + {accept, [{system_dir, system_dir(Config)}, + {user_dir, user_dir(Config)}, + {recv_ext_info, true} + ]}, + receive_hello, + {send, hello}, + + {send, ssh_msg_kexinit}, + {match, #ssh_msg_kexinit{_='_'}, receive_msg}, + + {match, #ssh_msg_kexdh_init{_='_'}, receive_msg}, + {send, ssh_msg_kexdh_reply}, + + {send, #ssh_msg_newkeys{}}, + {match, #ssh_msg_newkeys{_='_'}, receive_msg}, + + {match, #ssh_msg_ext_info{_='_'}, receive_msg}, + + close_socket, + print_state + ], + InitialState), + Parent ! {result,self(),Result} + end), + + %% connect to it with a regular Erlang SSH client + %% (expect error due to the close_socket in daemon): + {error,_} = std_connect(HostPort, Config, + [{preferred_algorithms,[{kex,[?DEFAULT_KEX]}, + {cipher,?DEFAULT_CIPHERS} + ]}, + {tstflg, [{ext_info_client,true}]}, + {send_ext_info, true} + ] + ), + %% Check that the daemon got expected result: + receive + {result, Pid, {ok,_}} -> ok; + {result, Pid, Error} -> ct:fail("Error: ~p",[Error]) + end. + %%%================================================================ %%%==== Internal functions ======================================== %%%================================================================ @@ -751,10 +863,12 @@ connect_and_kex(Config, InitialState) -> [{preferred_algorithms,[{kex,[?DEFAULT_KEX]}, {cipher,?DEFAULT_CIPHERS} ]}, - {silently_accept_hosts, true}, + {silently_accept_hosts, true}, {recv_ext_info, false}, {user_dir, user_dir(Config)}, - {user_interaction, false}]}, + {user_interaction, false} + | proplists:get_value(extra_options,Config,[]) + ]}, receive_hello, {send, hello}, {send, ssh_msg_kexinit}, diff --git a/lib/ssh/test/ssh_test_lib.erl b/lib/ssh/test/ssh_test_lib.erl index 36ae2525da..7b273fecef 100644 --- a/lib/ssh/test/ssh_test_lib.erl +++ b/lib/ssh/test/ssh_test_lib.erl @@ -500,8 +500,12 @@ setup_ecdsa_auth_keys(_Size, Dir, UserDir) -> setup_auth_keys(Keys, Dir) -> AuthKeys = public_key:ssh_encode(Keys, auth_keys), AuthKeysFile = filename:join(Dir, "authorized_keys"), - file:write_file(AuthKeysFile, AuthKeys). + ok = file:write_file(AuthKeysFile, AuthKeys), + AuthKeys. +write_auth_keys(Keys, Dir) -> + AuthKeysFile = filename:join(Dir, "authorized_keys"), + file:write_file(AuthKeysFile, Keys). del_dirs(Dir) -> case file:list_dir(Dir) of diff --git a/lib/ssh/test/ssh_to_openssh_SUITE.erl b/lib/ssh/test/ssh_to_openssh_SUITE.erl index a3d596a1c9..4d6aa93d4e 100644 --- a/lib/ssh/test/ssh_to_openssh_SUITE.erl +++ b/lib/ssh/test/ssh_to_openssh_SUITE.erl @@ -107,6 +107,9 @@ init_per_testcase(erlang_server_openssh_client_public_key_rsa, Config) -> chk_key(sshc, 'ssh-rsa', ".ssh/id_rsa", Config); init_per_testcase(erlang_client_openssh_server_publickey_dsa, Config) -> chk_key(sshd, 'ssh-dss', ".ssh/id_dsa", Config); +init_per_testcase(erlang_client_openssh_server_publickey_rsa, Config) -> + chk_key(sshd, 'ssh-rsa', ".ssh/id_rsa", Config); + init_per_testcase(erlang_server_openssh_client_renegotiate, Config) -> case os:type() of {unix,_} -> ssh:start(), Config; @@ -322,65 +325,44 @@ erlang_client_openssh_server_setenv(Config) when is_list(Config) -> %% setenv not meaningfull on erlang ssh daemon! %%-------------------------------------------------------------------- -erlang_client_openssh_server_publickey_rsa() -> - [{doc, "Validate using rsa publickey."}]. -erlang_client_openssh_server_publickey_rsa(Config) when is_list(Config) -> - {ok,[[Home]]} = init:get_argument(home), - KeyFile = filename:join(Home, ".ssh/id_rsa"), - case file:read_file(KeyFile) of - {ok, Pem} -> - case public_key:pem_decode(Pem) of - [{_,_, not_encrypted}] -> - ConnectionRef = - ssh_test_lib:connect(?SSH_DEFAULT_PORT, - [{pref_public_key_algs, ['ssh-rsa','ssh-dss']}, - {user_interaction, false}, - silently_accept_hosts]), - {ok, Channel} = - ssh_connection:session_channel(ConnectionRef, infinity), - ok = ssh_connection:close(ConnectionRef, Channel), - ok = ssh:close(ConnectionRef); - _ -> - {skip, {error, "Has pass phrase can not be used by automated test case"}} - end; - _ -> - {skip, "no ~/.ssh/id_rsa"} - end. - +erlang_client_openssh_server_publickey_rsa(Config) -> + erlang_client_openssh_server_publickey_X(Config, 'ssh-rsa'). + +erlang_client_openssh_server_publickey_dsa(Config) -> + erlang_client_openssh_server_publickey_X(Config, 'ssh-dss'). -%%-------------------------------------------------------------------- -erlang_client_openssh_server_publickey_dsa() -> - [{doc, "Validate using dsa publickey."}]. -erlang_client_openssh_server_publickey_dsa(Config) when is_list(Config) -> + +erlang_client_openssh_server_publickey_X(Config, Alg) -> ConnectionRef = - ssh_test_lib:connect(?SSH_DEFAULT_PORT, - [{pref_public_key_algs, ['ssh-dss','ssh-rsa']}, - {user_interaction, false}, - silently_accept_hosts]), + ssh_test_lib:connect(?SSH_DEFAULT_PORT, + [{pref_public_key_algs, [Alg]}, + {user_interaction, false}, + {auth_methods, "publickey"}, + silently_accept_hosts]), {ok, Channel} = - ssh_connection:session_channel(ConnectionRef, infinity), + ssh_connection:session_channel(ConnectionRef, infinity), ok = ssh_connection:close(ConnectionRef, Channel), ok = ssh:close(ConnectionRef). %%-------------------------------------------------------------------- erlang_server_openssh_client_public_key_dsa() -> - [{timetrap, {seconds,(?TIMEOUT div 1000)+10}}, - {doc, "Validate using dsa publickey."}]. + [{timetrap, {seconds,(?TIMEOUT div 1000)+10}}]. erlang_server_openssh_client_public_key_dsa(Config) when is_list(Config) -> - erlang_server_openssh_client_public_key_X(Config, ssh_dsa). + erlang_server_openssh_client_public_key_X(Config, 'ssh-dss'). -erlang_server_openssh_client_public_key_rsa() -> - [{timetrap, {seconds,(?TIMEOUT div 1000)+10}}, - {doc, "Validate using rsa publickey."}]. +erlang_server_openssh_client_public_key_rsa() -> + [{timetrap, {seconds,(?TIMEOUT div 1000)+10}}]. erlang_server_openssh_client_public_key_rsa(Config) when is_list(Config) -> - erlang_server_openssh_client_public_key_X(Config, ssh_rsa). + erlang_server_openssh_client_public_key_X(Config, 'ssh-rsa'). -erlang_server_openssh_client_public_key_X(Config, _PubKeyAlg) -> +erlang_server_openssh_client_public_key_X(Config, Alg) -> SystemDir = proplists:get_value(data_dir, Config), PrivDir = proplists:get_value(priv_dir, Config), KnownHosts = filename:join(PrivDir, "known_hosts"), {Pid, Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir}, + {preferred_algorithms,[{public_key, [Alg]}]}, + {auth_methods, "publickey"}, {failfun, fun ssh_test_lib:failfun/2}]), ct:sleep(500), @@ -401,7 +383,7 @@ erlang_server_openssh_client_renegotiate(Config) -> KnownHosts = filename:join(PrivDir, "known_hosts"), {Pid, Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir}, - {failfun, fun ssh_test_lib:failfun/2}]), + {failfun, fun ssh_test_lib:failfun/2}]), ct:sleep(500), RenegLimitK = 3, diff --git a/lib/ssh/vsn.mk b/lib/ssh/vsn.mk index 7208baca6e..48332d2e5a 100644 --- a/lib/ssh/vsn.mk +++ b/lib/ssh/vsn.mk @@ -1,5 +1,5 @@ #-*-makefile-*- ; force emacs to enter makefile-mode -SSH_VSN = 4.5 +SSH_VSN = 4.4.2 APP_VSN = "ssh-$(SSH_VSN)" diff --git a/lib/ssl/doc/src/notes.xml b/lib/ssl/doc/src/notes.xml index 88ba12b83b..29ec3f9d57 100644 --- a/lib/ssl/doc/src/notes.xml +++ b/lib/ssl/doc/src/notes.xml @@ -28,75 +28,15 @@ <p>This document describes the changes made to the SSL application.</p> -<section><title>SSL 8.2</title> +<section><title>SSL 8.1.3</title> - <section><title>Improvements and New Features</title> + <section><title>Fixed Bugs and Malfunctions</title> <list> <item> <p> - TLS-1.2 clients will now always send hello messages on - its own format, as opposed to earlier versions that will - send the hello on the lowest supported version, this is a - change supported by the latest RFC.</p> - <p> - This will make interoperability with some newer servers - smoother. Potentially, but unlikely, this could cause a - problem with older servers if they do not adhere to the - RFC and ignore unknown extensions.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-13820</p> - </item> - <item> - <p> - Allow Erlang/OTP to use OpenSSL in FIPS-140 mode, in - order to satisfy specific security requirements (mostly - by different parts of the US federal government). </p> - <p> - See the new crypto users guide "FIPS mode" chapter about - building and using the FIPS support which is disabled by - default.</p> - <p> - (Thanks to dszoboszlay and legoscia)</p> - <p> - Own Id: OTP-13921 Aux Id: PR-1180 </p> - </item> - <item> - <p> - Implemented DTLS cookie generation, requiered by spec, - instead of using hardcode value.</p> - <p> - Own Id: OTP-14076</p> - </item> - <item> - <p> - Extend connection_information/[1,2] . The values - session_id, master_secret, client_random and - server_random can no be accessed by - connection_information/2. Note only session_id will be - added to connection_information/1. The rational is that - values concerning the connection security should have to - be explicitly requested.</p> - <p> - Own Id: OTP-14291</p> - </item> - <item> - <p> - Chacha cipher suites are currently not tested enough to - be most prefered ones</p> - <p> - Own Id: OTP-14382</p> - </item> - <item> - <p> - Basic support for DTLS that been tested together with - OpenSSL.</p> - <p> - Test by providing the option {protocol, dtls} to the ssl - API functions connect and listen.</p> + Remove debug printout</p> <p> - Own Id: OTP-14388</p> + Own Id: OTP-14396</p> </item> </list> </section> diff --git a/lib/ssl/doc/src/ssl.xml b/lib/ssl/doc/src/ssl.xml index 2c09122fe6..2940ccb1e7 100644 --- a/lib/ssl/doc/src/ssl.xml +++ b/lib/ssl/doc/src/ssl.xml @@ -127,7 +127,7 @@ <item><p><c>hostname() | ipaddress()</c></p></item> <tag><c>hostname() =</c></tag> - <item><p><c>string()</c></p></item> + <item><p><c>string() - DNS hostname</c></p></item> <tag><c>ip_address() =</c></tag> <item><p><c>{N1,N2,N3,N4} % IPv4 | {K1,K2,K3,K4,K5,K6,K7,K8} % IPv6 @@ -249,7 +249,7 @@ be PEER, CA, ROOT-CA; if 2 the path can be PEER, CA, CA, ROOT-CA, and so on. The default value is 1.</p></item> - <tag><c>{verify_fun, {Verifyfun :: fun(), InitialUserState :: + <tag><marker id="verify_fun"/><c>{verify_fun, {Verifyfun :: fun(), InitialUserState :: term()}}</c></tag> <item><p>The verification fun is to be defined as follows:</p> @@ -582,15 +582,23 @@ fun(srp, Username :: string(), UserState :: term()) -> <item><p>Specifies the username and password to use to authenticate to the server.</p></item> - <tag><c>{server_name_indication, hostname()}</c></tag> - <item><p>Can be specified when upgrading a TCP socket to a TLS - socket to use the TLS Server Name Indication extension.</p></item> + <tag><c>{server_name_indication, HostName :: hostname()}</c></tag> + <item><p>Specify the hostname to be used in TLS Server Name Indication extension. + Is usefull when upgrading a TCP socket to a TLS socket or if the hostname can not be + derived from the Host argument to <seealso marker="ssl#connect-3">ssl:connect/3</seealso>. + Will also cause the client to preform host name verification of the peer certificate + <seealso marker="public_key:public_key#pkix_verify_hostname-2">public_key:pkix_verify_hostname(PeerCert, [{dns_id, HostName}])</seealso> + </p> during the x509-path validation. If the check fails the error {bad_cert, hostname_check_failiure} will be + propagated to the path validation fun <seealso marker="#verify_fun">verify_fun</seealso> + </item> <tag><c>{server_name_indication, disable}</c></tag> <item> <p>When starting a TLS connection without upgrade, the Server Name - Indication extension is sent if possible. This option can be - used to disable that behavior.</p> + Indication extension is sent if possible that is can be derived from the Host argument + to <seealso marker="ssl#connect-3">ssl:connect/3</seealso>. + This option can be used to disable that behavior.</p> + <note><p> Note that this also disables the default host name verification check of the peer certificate.</p></note> </item> <tag><c>{fallback, boolean()}</c></tag> <item> diff --git a/lib/ssl/src/dtls_connection.erl b/lib/ssl/src/dtls_connection.erl index 9937373e6e..f078b87bce 100644 --- a/lib/ssl/src/dtls_connection.erl +++ b/lib/ssl/src/dtls_connection.erl @@ -53,7 +53,7 @@ %% Data handling -export([encode_data/3, passive_receive/2, next_record_if_active/1, handle_common_event/4, - send/3, socket/5]). + send/3, socket/5, setopts/3, getopts/3]). %% gen_statem state functions -export([init/3, error/3, downgrade/3, %% Initiation and take down states @@ -203,6 +203,11 @@ select_sni_extension(_) -> socket(Pid, Transport, Socket, Connection, _) -> dtls_socket:socket(Pid, Transport, Socket, Connection). +setopts(Transport, Socket, Other) -> + dtls_socket:setopts(Transport, Socket, Other). +getopts(Transport, Socket, Tag) -> + dtls_socket:getopts(Transport, Socket, Tag). + %%==================================================================== %% tls_connection_sup API %%==================================================================== @@ -688,16 +693,18 @@ next_record(#state{unprocessed_handshake_events = N} = State) when N > 0 -> {no_record, State#state{unprocessed_handshake_events = N-1}}; next_record(#state{protocol_buffers = - #protocol_buffers{dtls_cipher_texts = [CT | Rest]} + #protocol_buffers{dtls_cipher_texts = [#ssl_tls{epoch = Epoch} = CT | Rest]} = Buffers, - connection_states = ConnStates0} = State) -> - case dtls_record:decode_cipher_text(CT, ConnStates0) of - {Plain, ConnStates} -> - {Plain, State#state{protocol_buffers = - Buffers#protocol_buffers{dtls_cipher_texts = Rest}, - connection_states = ConnStates}}; - #alert{} = Alert -> - {Alert, State} + connection_states = ConnectionStates} = State) -> + CurrentRead = dtls_record:get_connection_state_by_epoch(Epoch, ConnectionStates, read), + case dtls_record:replay_detect(CT, CurrentRead) of + false -> + decode_cipher_text(State#state{connection_states = ConnectionStates}) ; + true -> + %% Ignore replayed record + next_record(State#state{protocol_buffers = + Buffers#protocol_buffers{dtls_cipher_texts = Rest}, + connection_states = ConnectionStates}) end; next_record(#state{role = server, socket = {Listener, {Client, _}}, @@ -770,6 +777,17 @@ next_event(StateName, Record, {next_state, StateName, State, [{next_event, internal, Alert} | Actions]} end. +decode_cipher_text(#state{protocol_buffers = #protocol_buffers{dtls_cipher_texts = [ CT | Rest]} = Buffers, + connection_states = ConnStates0} = State) -> + case dtls_record:decode_cipher_text(CT, ConnStates0) of + {Plain, ConnStates} -> + {Plain, State#state{protocol_buffers = + Buffers#protocol_buffers{dtls_cipher_texts = Rest}, + connection_states = ConnStates}}; + #alert{} = Alert -> + {Alert, State} + end. + dtls_version(hello, Version, #state{role = server} = State) -> State#state{negotiated_version = Version}; %%Inital version dtls_version(_,_, State) -> diff --git a/lib/ssl/src/dtls_handshake.erl b/lib/ssl/src/dtls_handshake.erl index 1ed63f8a83..37a46b862e 100644 --- a/lib/ssl/src/dtls_handshake.erl +++ b/lib/ssl/src/dtls_handshake.erl @@ -65,9 +65,8 @@ client_hello(Host, Port, Cookie, ConnectionStates, TLSVersion = dtls_v1:corresponding_tls_version(Version), CipherSuites = ssl_handshake:available_suites(UserSuites, TLSVersion), - Extensions = ssl_handshake:client_hello_extensions(Host, TLSVersion, CipherSuites, + Extensions = ssl_handshake:client_hello_extensions(TLSVersion, CipherSuites, SslOpts, ConnectionStates, Renegotiation), - Id = ssl_session:client_id({Host, Port, SslOpts}, Cache, CacheCb, OwnCert), #client_hello{session_id = Id, diff --git a/lib/ssl/src/dtls_record.erl b/lib/ssl/src/dtls_record.erl index 6a418c6fb1..8a7f8c1d0a 100644 --- a/lib/ssl/src/dtls_record.erl +++ b/lib/ssl/src/dtls_record.erl @@ -46,7 +46,7 @@ is_higher/2, supported_protocol_versions/0, is_acceptable_version/2, hello_version/2]). --export([save_current_connection_state/2, next_epoch/2]). +-export([save_current_connection_state/2, next_epoch/2, get_connection_state_by_epoch/3, replay_detect/2]). -export([init_connection_state_seq/2, current_connection_state_epoch/2]). @@ -55,6 +55,8 @@ -type dtls_version() :: ssl_record:ssl_version(). -type dtls_atom_version() :: dtlsv1 | 'dtlsv1.2'. +-define(REPLAY_WINDOW_SIZE, 64). + -compile(inline). %%==================================================================== @@ -73,7 +75,7 @@ init_connection_states(Role, BeastMitigation) -> Initial = initial_connection_state(ConnectionEnd, BeastMitigation), Current = Initial#{epoch := 0}, InitialPending = ssl_record:empty_connection_state(ConnectionEnd, BeastMitigation), - Pending = InitialPending#{epoch => undefined}, + Pending = InitialPending#{epoch => undefined, replay_window => init_replay_window(?REPLAY_WINDOW_SIZE)}, #{saved_read => Current, current_read => Current, pending_read => Pending, @@ -96,11 +98,13 @@ save_current_connection_state(#{current_write := Current} = States, write) -> next_epoch(#{pending_read := Pending, current_read := #{epoch := Epoch}} = States, read) -> - States#{pending_read := Pending#{epoch := Epoch + 1}}; + States#{pending_read := Pending#{epoch := Epoch + 1, + replay_window := init_replay_window(?REPLAY_WINDOW_SIZE)}}; next_epoch(#{pending_write := Pending, current_write := #{epoch := Epoch}} = States, write) -> - States#{pending_write := Pending#{epoch := Epoch + 1}}. + States#{pending_write := Pending#{epoch := Epoch + 1, + replay_window := init_replay_window(?REPLAY_WINDOW_SIZE)}}. get_connection_state_by_epoch(Epoch, #{current_write := #{epoch := Epoch} = Current}, write) -> @@ -411,6 +415,7 @@ hello_version(Version, Versions) -> lowest_protocol_version(Versions) end. + %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- @@ -419,6 +424,7 @@ initial_connection_state(ConnectionEnd, BeastMitigation) -> ssl_record:initial_security_params(ConnectionEnd), epoch => undefined, sequence_number => 0, + replay_window => init_replay_window(?REPLAY_WINDOW_SIZE), beast_mitigation => BeastMitigation, compression_state => undefined, cipher_state => undefined, @@ -499,8 +505,9 @@ decode_cipher_text(#ssl_tls{type = Type, version = Version, {PlainFragment, CipherState} -> {Plain, CompressionS1} = ssl_record:uncompress(CompAlg, PlainFragment, CompressionS0), - ReadState = ReadState0#{compression_state => CompressionS1, + ReadState0 = ReadState0#{compression_state => CompressionS1, cipher_state => CipherState}, + ReadState = update_replay_window(Seq, ReadState0), ConnnectionStates = set_connection_state_by_epoch(ReadState, Epoch, ConnnectionStates0, read), {CipherText#ssl_tls{fragment = Plain}, ConnnectionStates}; #alert{} = Alert -> @@ -523,7 +530,8 @@ decode_cipher_text(#ssl_tls{type = Type, version = Version, {Plain, CompressionS1} = ssl_record:uncompress(CompAlg, PlainFragment, CompressionS0), - ReadState = ReadState1#{compression_state => CompressionS1}, + ReadState2 = ReadState1#{compression_state => CompressionS1}, + ReadState = update_replay_window(Seq, ReadState2), ConnnectionStates = set_connection_state_by_epoch(ReadState, Epoch, ConnnectionStates0, read), {CipherText#ssl_tls{fragment = Plain}, ConnnectionStates}; false -> @@ -555,3 +563,38 @@ mac_hash({Major, Minor}, MacAlg, MacSecret, Epoch, SeqNo, Type, Length, Fragment calc_aad(Type, {MajVer, MinVer}, Epoch, SeqNo) -> <<?UINT16(Epoch), ?UINT48(SeqNo), ?BYTE(Type), ?BYTE(MajVer), ?BYTE(MinVer)>>. + +init_replay_window(Size) -> + #{size => Size, + top => Size, + bottom => 0, + mask => 0 bsl 64 + }. + +replay_detect(#ssl_tls{sequence_number = SequenceNumber}, #{replay_window := Window}) -> + is_replay(SequenceNumber, Window). + + +is_replay(SequenceNumber, #{bottom := Bottom}) when SequenceNumber < Bottom -> + true; +is_replay(SequenceNumber, #{size := Size, + top := Top, + bottom := Bottom, + mask := Mask}) when (SequenceNumber >= Bottom) andalso (SequenceNumber =< Top) -> + Index = (SequenceNumber rem Size), + (Index band Mask) == 1; + +is_replay(_, _) -> + false. + +update_replay_window(SequenceNumber, #{replay_window := #{size := Size, + top := Top, + bottom := Bottom, + mask := Mask0} = Window0} = ConnectionStates) -> + NoNewBits = SequenceNumber - Top, + Index = SequenceNumber rem Size, + Mask = (Mask0 bsl NoNewBits) bor Index, + Window = Window0#{top => SequenceNumber, + bottom => Bottom + NoNewBits, + mask => Mask}, + ConnectionStates#{replay_window := Window}. diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index dfee4b2243..5421bdef99 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -112,7 +112,7 @@ connect(Host, Port, Options) -> connect(Host, Port, Options, Timeout) when (is_integer(Timeout) andalso Timeout >= 0) or (Timeout == infinity) -> try - {ok, Config} = handle_options(Options, client), + {ok, Config} = handle_options(Options, client, Host), case Config#config.connection_cb of tls_connection -> tls_socket:connect(Host,Port,Config,Timeout); @@ -455,7 +455,16 @@ setopts(#sslsocket{pid = Pid}, Options0) when is_pid(Pid), is_list(Options0) -> _:_ -> {error, {options, {not_a_proplist, Options0}}} end; - +setopts(#sslsocket{pid = {{udp, _}, #config{transport_info = {Transport,_,_,_}}}} = ListenSocket, Options) when is_list(Options) -> + try dtls_socket:setopts(Transport, ListenSocket, Options) of + ok -> + ok; + {error, InetError} -> + {error, {options, {socket_options, Options, InetError}}} + catch + _:Error -> + {error, {options, {socket_options, Options, Error}}} + end; setopts(#sslsocket{pid = {_, #config{transport_info = {Transport,_,_,_}}}} = ListenSocket, Options) when is_list(Options) -> try tls_socket:setopts(Transport, ListenSocket, Options) of ok -> @@ -632,8 +641,12 @@ do_listen(Port, #config{transport_info = {Transport, _, _, _}} = Config, dtls_c %% Handle extra ssl options given to ssl_accept -spec handle_options([any()], #ssl_options{}) -> #ssl_options{} ; ([any()], client | server) -> {ok, #config{}}. +handle_options(Opts, Role) -> + handle_options(Opts, Role, undefined). + + handle_options(Opts0, #ssl_options{protocol = Protocol, cacerts = CaCerts0, - cacertfile = CaCertFile0} = InheritedSslOpts) -> + cacertfile = CaCertFile0} = InheritedSslOpts, _) -> RecordCB = record_cb(Protocol), CaCerts = handle_option(cacerts, Opts0, CaCerts0), {Verify, FailIfNoPeerCert, CaCertDefault, VerifyFun, PartialChainHanlder, @@ -666,7 +679,7 @@ handle_options(Opts0, #ssl_options{protocol = Protocol, cacerts = CaCerts0, end; %% Handle all options in listen and connect -handle_options(Opts0, Role) -> +handle_options(Opts0, Role, Host) -> Opts = proplists:expand([{binary, [{mode, binary}]}, {list, [{mode, list}]}], Opts0), assert_proplist(Opts), @@ -738,7 +751,9 @@ handle_options(Opts0, Role) -> make_next_protocol_selector( handle_option(client_preferred_next_protocols, Opts, undefined)), log_alert = handle_option(log_alert, Opts, true), - server_name_indication = handle_option(server_name_indication, Opts, undefined), + server_name_indication = handle_option(server_name_indication, Opts, + default_option_role(client, + server_name_indication_default(Host), Role)), sni_hosts = handle_option(sni_hosts, Opts, []), sni_fun = handle_option(sni_fun, Opts, undefined), honor_cipher_order = handle_option(honor_cipher_order, Opts, @@ -982,12 +997,20 @@ validate_option(next_protocols_advertised = Opt, Value) when is_list(Value) -> validate_option(next_protocols_advertised, undefined) -> undefined; -validate_option(server_name_indication, Value) when is_list(Value) -> +validate_option(server_name_indication = Opt, Value) when is_list(Value) -> + %% RFC 6066, Section 3: Currently, the only server names supported are + %% DNS hostnames + case inet_parse:domain(Value) of + false -> + throw({error, {options, {{Opt, Value}}}}); + true -> + Value + end; +validate_option(server_name_indication, undefined = Value) -> Value; validate_option(server_name_indication, disable) -> - disable; -validate_option(server_name_indication, undefined) -> undefined; + validate_option(sni_hosts, []) -> []; validate_option(sni_hosts, [{Hostname, SSLOptions} | Tail]) when is_list(Hostname) -> @@ -1445,3 +1468,8 @@ include_security_info([Item | Items]) -> false -> include_security_info(Items) end. + +server_name_indication_default(Host) when is_list(Host) -> + Host; +server_name_indication_default(_) -> + undefined. diff --git a/lib/ssl/src/ssl_certificate.erl b/lib/ssl/src/ssl_certificate.erl index 2046ec75b3..0dd5e5c5cf 100644 --- a/lib/ssl/src/ssl_certificate.erl +++ b/lib/ssl/src/ssl_certificate.erl @@ -125,7 +125,7 @@ file_to_crls(File, DbHandle) -> %% Description: Validates ssl/tls specific extensions %%-------------------------------------------------------------------- validate(_,{extension, #'Extension'{extnID = ?'id-ce-extKeyUsage', - extnValue = KeyUse}}, UserState = {Role, _,_, _, _}) -> + extnValue = KeyUse}}, UserState = {Role, _,_, _, _, _}) -> case is_valid_extkey_usage(KeyUse, Role) of true -> {valid, UserState}; @@ -138,8 +138,15 @@ validate(_, {bad_cert, _} = Reason, _) -> {fail, Reason}; validate(_, valid, UserState) -> {valid, UserState}; -validate(_, valid_peer, UserState) -> - {valid, UserState}. +validate(Cert, valid_peer, UserState = {client, _,_, Hostname, _, _}) when Hostname =/= undefined -> + case public_key:pkix_verify_hostname(Cert, [{dns_id, Hostname}]) of + true -> + {valid, UserState}; + false -> + {fail, {bad_cert, hostname_check_failed}} + end; +validate(_, valid_peer, UserState) -> + {valid, UserState}. %%-------------------------------------------------------------------- -spec is_valid_key_usage(list(), term()) -> boolean(). diff --git a/lib/ssl/src/ssl_connection.erl b/lib/ssl/src/ssl_connection.erl index d9707115d5..fb87662c7b 100644 --- a/lib/ssl/src/ssl_connection.erl +++ b/lib/ssl/src/ssl_connection.erl @@ -501,13 +501,7 @@ certify(internal, #certificate{} = Cert, crl_db = CRLDbInfo, ssl_options = Opts} = State, Connection) -> case ssl_handshake:certify(Cert, CertDbHandle, CertDbRef, - Opts#ssl_options.depth, - Opts#ssl_options.verify, - Opts#ssl_options.verify_fun, - Opts#ssl_options.partial_chain, - Opts#ssl_options.crl_check, - CRLDbInfo, - Role) of + Opts, CRLDbInfo, Role) of {PeerCert, PublicKeyInfo} -> handle_peer_cert(Role, PeerCert, PublicKeyInfo, State#state{client_certificate_requested = false}, Connection); @@ -848,7 +842,6 @@ handle_common_event(internal, #change_cipher_spec{type = <<1>>}, StateName, StateName, State); handle_common_event(_Type, Msg, StateName, #state{negotiated_version = Version} = State, _) -> - ct:pal("Unexpected msg ~p", [Msg]), Alert = ?ALERT_REC(?FATAL,?UNEXPECTED_MESSAGE), handle_own_alert(Alert, Version, {StateName, Msg}, State). @@ -912,14 +905,14 @@ handle_call({new_user, User}, From, StateName, handle_call({get_opts, OptTags}, From, _, #state{socket = Socket, transport_cb = Transport, - socket_options = SockOpts}, _) -> - OptsReply = get_socket_opts(Transport, Socket, OptTags, SockOpts, []), + socket_options = SockOpts}, Connection) -> + OptsReply = get_socket_opts(Connection, Transport, Socket, OptTags, SockOpts, []), {keep_state_and_data, [{reply, From, OptsReply}]}; handle_call({set_opts, Opts0}, From, StateName, #state{socket_options = Opts1, socket = Socket, - transport_cb = Transport} = State0, _) -> - {Reply, Opts} = set_socket_opts(Transport, Socket, Opts0, Opts1, []), + transport_cb = Transport} = State0, Connection) -> + {Reply, Opts} = set_socket_opts(Connection, Transport, Socket, Opts0, Opts1, []), State = State0#state{socket_options = Opts}, handle_active_option(Opts#socket_options.active, StateName, From, Reply, State); @@ -1917,42 +1910,39 @@ call(FsmPid, Event) -> {error, closed} end. -get_socket_opts(_,_,[], _, Acc) -> +get_socket_opts(_, _,_,[], _, Acc) -> {ok, Acc}; -get_socket_opts(Transport, Socket, [mode | Tags], SockOpts, Acc) -> - get_socket_opts(Transport, Socket, Tags, SockOpts, +get_socket_opts(Connection, Transport, Socket, [mode | Tags], SockOpts, Acc) -> + get_socket_opts(Connection, Transport, Socket, Tags, SockOpts, [{mode, SockOpts#socket_options.mode} | Acc]); -get_socket_opts(Transport, Socket, [packet | Tags], SockOpts, Acc) -> +get_socket_opts(Connection, Transport, Socket, [packet | Tags], SockOpts, Acc) -> case SockOpts#socket_options.packet of {Type, headers} -> - get_socket_opts(Transport, Socket, Tags, SockOpts, [{packet, Type} | Acc]); + get_socket_opts(Connection, Transport, Socket, Tags, SockOpts, [{packet, Type} | Acc]); Type -> - get_socket_opts(Transport, Socket, Tags, SockOpts, [{packet, Type} | Acc]) + get_socket_opts(Connection, Transport, Socket, Tags, SockOpts, [{packet, Type} | Acc]) end; -get_socket_opts(Transport, Socket, [header | Tags], SockOpts, Acc) -> - get_socket_opts(Transport, Socket, Tags, SockOpts, +get_socket_opts(Connection, Transport, Socket, [header | Tags], SockOpts, Acc) -> + get_socket_opts(Connection, Transport, Socket, Tags, SockOpts, [{header, SockOpts#socket_options.header} | Acc]); -get_socket_opts(Transport, Socket, [active | Tags], SockOpts, Acc) -> - get_socket_opts(Transport, Socket, Tags, SockOpts, +get_socket_opts(Connection, Transport, Socket, [active | Tags], SockOpts, Acc) -> + get_socket_opts(Connection, Transport, Socket, Tags, SockOpts, [{active, SockOpts#socket_options.active} | Acc]); -get_socket_opts(Transport, Socket, [Tag | Tags], SockOpts, Acc) -> - try tls_socket:getopts(Transport, Socket, [Tag]) of - {ok, [Opt]} -> - get_socket_opts(Transport, Socket, Tags, SockOpts, [Opt | Acc]); - {error, Error} -> - {error, {options, {socket_options, Tag, Error}}} - catch - %% So that inet behavior does not crash our process - _:Error -> {error, {options, {socket_options, Tag, Error}}} +get_socket_opts(Connection, Transport, Socket, [Tag | Tags], SockOpts, Acc) -> + case Connection:getopts(Transport, Socket, [Tag]) of + {ok, [Opt]} -> + get_socket_opts(Connection, Transport, Socket, Tags, SockOpts, [Opt | Acc]); + {error, Reason} -> + {error, {options, {socket_options, Tag, Reason}}} end; -get_socket_opts(_, _,Opts, _,_) -> +get_socket_opts(_,_, _,Opts, _,_) -> {error, {options, {socket_options, Opts, function_clause}}}. -set_socket_opts(_,_, [], SockOpts, []) -> +set_socket_opts(_,_,_, [], SockOpts, []) -> {ok, SockOpts}; -set_socket_opts(Transport, Socket, [], SockOpts, Other) -> +set_socket_opts(ConnectionCb, Transport, Socket, [], SockOpts, Other) -> %% Set non emulated options - try tls_socket:setopts(Transport, Socket, Other) of + try ConnectionCb:setopts(Transport, Socket, Other) of ok -> {ok, SockOpts}; {error, InetError} -> @@ -1963,13 +1953,13 @@ set_socket_opts(Transport, Socket, [], SockOpts, Other) -> {{error, {options, {socket_options, Other, Error}}}, SockOpts} end; -set_socket_opts(Transport,Socket, [{mode, Mode}| Opts], SockOpts, Other) +set_socket_opts(ConnectionCb, Transport,Socket, [{mode, Mode}| Opts], SockOpts, Other) when Mode == list; Mode == binary -> - set_socket_opts(Transport, Socket, Opts, + set_socket_opts(ConnectionCb, Transport, Socket, Opts, SockOpts#socket_options{mode = Mode}, Other); -set_socket_opts(_, _, [{mode, _} = Opt| _], SockOpts, _) -> +set_socket_opts(_, _, _, [{mode, _} = Opt| _], SockOpts, _) -> {{error, {options, {socket_options, Opt}}}, SockOpts}; -set_socket_opts(Transport,Socket, [{packet, Packet}| Opts], SockOpts, Other) +set_socket_opts(ConnectionCb, Transport,Socket, [{packet, Packet}| Opts], SockOpts, Other) when Packet == raw; Packet == 0; Packet == 1; @@ -1985,26 +1975,26 @@ set_socket_opts(Transport,Socket, [{packet, Packet}| Opts], SockOpts, Other) Packet == httph; Packet == http_bin; Packet == httph_bin -> - set_socket_opts(Transport, Socket, Opts, + set_socket_opts(ConnectionCb, Transport, Socket, Opts, SockOpts#socket_options{packet = Packet}, Other); -set_socket_opts(_, _, [{packet, _} = Opt| _], SockOpts, _) -> +set_socket_opts(_, _, _, [{packet, _} = Opt| _], SockOpts, _) -> {{error, {options, {socket_options, Opt}}}, SockOpts}; -set_socket_opts(Transport, Socket, [{header, Header}| Opts], SockOpts, Other) +set_socket_opts(ConnectionCb, Transport, Socket, [{header, Header}| Opts], SockOpts, Other) when is_integer(Header) -> - set_socket_opts(Transport, Socket, Opts, + set_socket_opts(ConnectionCb, Transport, Socket, Opts, SockOpts#socket_options{header = Header}, Other); -set_socket_opts(_, _, [{header, _} = Opt| _], SockOpts, _) -> +set_socket_opts(_, _, _, [{header, _} = Opt| _], SockOpts, _) -> {{error,{options, {socket_options, Opt}}}, SockOpts}; -set_socket_opts(Transport, Socket, [{active, Active}| Opts], SockOpts, Other) +set_socket_opts(ConnectionCb, Transport, Socket, [{active, Active}| Opts], SockOpts, Other) when Active == once; Active == true; Active == false -> - set_socket_opts(Transport, Socket, Opts, + set_socket_opts(ConnectionCb, Transport, Socket, Opts, SockOpts#socket_options{active = Active}, Other); -set_socket_opts(_, _, [{active, _} = Opt| _], SockOpts, _) -> +set_socket_opts(_,_, _, [{active, _} = Opt| _], SockOpts, _) -> {{error, {options, {socket_options, Opt}} }, SockOpts}; -set_socket_opts(Transport, Socket, [Opt | Opts], SockOpts, Other) -> - set_socket_opts(Transport, Socket, Opts, SockOpts, [Opt | Other]). +set_socket_opts(ConnectionCb, Transport, Socket, [Opt | Opts], SockOpts, Other) -> + set_socket_opts(ConnectionCb, Transport, Socket, Opts, SockOpts, [Opt | Other]). start_or_recv_cancel_timer(infinity, _RecvFrom) -> undefined; diff --git a/lib/ssl/src/ssl_handshake.erl b/lib/ssl/src/ssl_handshake.erl index 2520fee238..58db8449d6 100644 --- a/lib/ssl/src/ssl_handshake.erl +++ b/lib/ssl/src/ssl_handshake.erl @@ -50,7 +50,7 @@ finished/5, next_protocol/1]). %% Handle handshake messages --export([certify/10, client_certificate_verify/6, certificate_verify/6, verify_signature/5, +-export([certify/6, client_certificate_verify/6, certificate_verify/6, verify_signature/5, master_secret/4, server_key_exchange_hash/2, verify_connection/6, init_handshake_history/0, update_handshake_history/3, verify_server_key/5 ]). @@ -68,7 +68,7 @@ select_session/11, supported_ecc/1, available_signature_algs/4]). %% Extensions handling --export([client_hello_extensions/6, +-export([client_hello_extensions/5, handle_client_hello_extensions/9, %% Returns server hello extensions handle_server_hello_extensions/9, select_curve/2, select_curve/3 ]). @@ -119,7 +119,7 @@ server_hello(SessionId, Version, ConnectionStates, Extensions) -> server_hello_done() -> #server_hello_done{}. -client_hello_extensions(Host, Version, CipherSuites, +client_hello_extensions(Version, CipherSuites, #ssl_options{signature_algs = SupportedHashSigns, eccs = SupportedECCs} = SslOpts, ConnectionStates, Renegotiation) -> {EcPointFormats, EllipticCurves} = @@ -142,7 +142,7 @@ client_hello_extensions(Host, Version, CipherSuites, next_protocol_negotiation = encode_client_protocol_negotiation(SslOpts#ssl_options.next_protocol_selector, Renegotiation), - sni = sni(Host, SslOpts#ssl_options.server_name_indication)}. + sni = sni(SslOpts#ssl_options.server_name_indication)}. %%-------------------------------------------------------------------- -spec certificate(der_cert(), db_handle(), certdb_ref(), client | server) -> #certificate{} | #alert{}. @@ -388,24 +388,26 @@ verify_signature(_, Hash, {HashAlgo, _SignAlg}, Signature, %%-------------------------------------------------------------------- --spec certify(#certificate{}, db_handle(), certdb_ref(), integer() | nolimit, - verify_peer | verify_none, {fun(), term}, fun(), term(), term(), +-spec certify(#certificate{}, db_handle(), certdb_ref(), #ssl_options{}, term(), client | server) -> {der_cert(), public_key_info()} | #alert{}. %% %% Description: Handles a certificate handshake message %%-------------------------------------------------------------------- certify(#certificate{asn1_certificates = ASN1Certs}, CertDbHandle, CertDbRef, - MaxPathLen, _Verify, ValidationFunAndState0, PartialChain, CRLCheck, CRLDbHandle, Role) -> + Opts, CRLDbHandle, Role) -> + [PeerCert | _] = ASN1Certs, try {TrustedCert, CertPath} = - ssl_certificate:trusted_cert_and_path(ASN1Certs, CertDbHandle, CertDbRef, PartialChain), - ValidationFunAndState = validation_fun_and_state(ValidationFunAndState0, Role, + ssl_certificate:trusted_cert_and_path(ASN1Certs, CertDbHandle, CertDbRef, + Opts#ssl_options.partial_chain), + ValidationFunAndState = validation_fun_and_state(Opts#ssl_options.verify_fun, Role, CertDbHandle, CertDbRef, - CRLCheck, CRLDbHandle, CertPath), + Opts#ssl_options.server_name_indication, + Opts#ssl_options.crl_check, CRLDbHandle, CertPath), case public_key:pkix_path_validation(TrustedCert, CertPath, - [{max_path_length, MaxPathLen}, + [{max_path_length, Opts#ssl_options.depth}, {verify_fun, ValidationFunAndState}]) of {ok, {PublicKeyInfo,_}} -> {PeerCert, PublicKeyInfo}; @@ -1522,25 +1524,16 @@ select_shared_curve([Curve | Rest], Curves) -> select_shared_curve(Rest, Curves) end. -%% RFC 6066, Section 3: Currently, the only server names supported are -%% DNS hostnames -sni(_, disable) -> +sni(undefined) -> undefined; -sni(Host, undefined) -> - sni1(Host); -sni(_Host, SNIOption) -> - sni1(SNIOption). - -sni1(Hostname) -> - case inet_parse:domain(Hostname) of - false -> undefined; - true -> #sni{hostname = Hostname} - end. +sni(Hostname) -> + #sni{hostname = Hostname}. + %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- validation_fun_and_state({Fun, UserState0}, Role, CertDbHandle, CertDbRef, - CRLCheck, CRLDbHandle, CertPath) -> + ServerNameIndication, CRLCheck, CRLDbHandle, CertPath) -> {fun(OtpCert, {extension, _} = Extension, {SslState, UserState}) -> case ssl_certificate:validate(OtpCert, Extension, @@ -1557,9 +1550,9 @@ validation_fun_and_state({Fun, UserState0}, Role, CertDbHandle, CertDbRef, (OtpCert, VerifyResult, {SslState, UserState}) -> apply_user_fun(Fun, OtpCert, VerifyResult, UserState, SslState, CertPath) - end, {{Role, CertDbHandle, CertDbRef, CRLCheck, CRLDbHandle}, UserState0}}; + end, {{Role, CertDbHandle, CertDbRef, ServerNameIndication, CRLCheck, CRLDbHandle}, UserState0}}; validation_fun_and_state(undefined, Role, CertDbHandle, CertDbRef, - CRLCheck, CRLDbHandle, CertPath) -> + ServerNameIndication, CRLCheck, CRLDbHandle, CertPath) -> {fun(OtpCert, {extension, _} = Extension, SslState) -> ssl_certificate:validate(OtpCert, Extension, @@ -1568,8 +1561,10 @@ validation_fun_and_state(undefined, Role, CertDbHandle, CertDbRef, (VerifyResult == valid_peer) -> case crl_check(OtpCert, CRLCheck, CertDbHandle, CertDbRef, CRLDbHandle, VerifyResult, CertPath) of - valid -> - {VerifyResult, SslState}; + valid -> + ssl_certificate:validate(OtpCert, + VerifyResult, + SslState); Reason -> {fail, Reason} end; @@ -1577,10 +1572,10 @@ validation_fun_and_state(undefined, Role, CertDbHandle, CertDbRef, ssl_certificate:validate(OtpCert, VerifyResult, SslState) - end, {Role, CertDbHandle, CertDbRef, CRLCheck, CRLDbHandle}}. + end, {Role, CertDbHandle, CertDbRef, ServerNameIndication, CRLCheck, CRLDbHandle}}. apply_user_fun(Fun, OtpCert, VerifyResult, UserState0, - {_, CertDbHandle, CertDbRef, CRLCheck, CRLDbHandle} = SslState, CertPath) when + {_, CertDbHandle, CertDbRef, _, CRLCheck, CRLDbHandle} = SslState, CertPath) when (VerifyResult == valid) or (VerifyResult == valid_peer) -> case Fun(OtpCert, VerifyResult, UserState0) of {Valid, UserState} when (Valid == valid) or (Valid == valid_peer) -> diff --git a/lib/ssl/src/tls_connection.erl b/lib/ssl/src/tls_connection.erl index a289ee0a65..96c3ab86e9 100644 --- a/lib/ssl/src/tls_connection.erl +++ b/lib/ssl/src/tls_connection.erl @@ -60,7 +60,7 @@ %% Data handling -export([passive_receive/2, next_record_if_active/1, handle_common_event/4, send/3, - socket/5]). + socket/5, setopts/3, getopts/3]). %% gen_statem state functions -export([init/3, error/3, downgrade/3, %% Initiation and take down states @@ -195,6 +195,10 @@ callback_mode() -> socket(Pid, Transport, Socket, Connection, Tracker) -> tls_socket:socket(Pid, Transport, Socket, Connection, Tracker). +setopts(Transport, Socket, Other) -> + tls_socket:setopts(Transport, Socket, Other). +getopts(Transport, Socket, Tag) -> + tls_socket:getopts(Transport, Socket, Tag). %%-------------------------------------------------------------------- %% State functions diff --git a/lib/ssl/src/tls_handshake.erl b/lib/ssl/src/tls_handshake.erl index 9da7b43be3..b54540393a 100644 --- a/lib/ssl/src/tls_handshake.erl +++ b/lib/ssl/src/tls_handshake.erl @@ -56,7 +56,7 @@ client_hello(Host, Port, ConnectionStates, Version = tls_record:highest_protocol_version(Versions), #{security_parameters := SecParams} = ssl_record:pending_connection_state(ConnectionStates, read), AvailableCipherSuites = ssl_handshake:available_suites(UserSuites, Version), - Extensions = ssl_handshake:client_hello_extensions(Host, Version, + Extensions = ssl_handshake:client_hello_extensions(Version, AvailableCipherSuites, SslOpts, ConnectionStates, Renegotiation), CipherSuites = diff --git a/lib/ssl/test/make_certs.erl b/lib/ssl/test/make_certs.erl index e14f7f60c4..74505169a0 100644 --- a/lib/ssl/test/make_certs.erl +++ b/lib/ssl/test/make_certs.erl @@ -385,6 +385,7 @@ req_cnf(Root, C) -> "subjectAltName = email:copy\n"]. ca_cnf(Root, C = #config{issuing_distribution_point = true}) -> + Hostname = net_adm:localhost(), ["# Purpose: Configuration for CAs.\n" "\n" "ROOTDIR = " ++ Root ++ "\n" @@ -434,7 +435,7 @@ ca_cnf(Root, C = #config{issuing_distribution_point = true}) -> "keyUsage = nonRepudiation, digitalSignature, keyEncipherment\n" "subjectKeyIdentifier = hash\n" "authorityKeyIdentifier = keyid,issuer:always\n" - "subjectAltName = email:copy\n" + "subjectAltName = DNS.1:" ++ Hostname ++ "\n" "issuerAltName = issuer:copy\n" "crlDistributionPoints=@crl_section\n" @@ -449,7 +450,7 @@ ca_cnf(Root, C = #config{issuing_distribution_point = true}) -> "keyUsage = digitalSignature\n" "subjectKeyIdentifier = hash\n" "authorityKeyIdentifier = keyid,issuer:always\n" - "subjectAltName = email:copy\n" + "subjectAltName = DNS.1:" ++ Hostname ++ "\n" "issuerAltName = issuer:copy\n" "\n" @@ -458,12 +459,13 @@ ca_cnf(Root, C = #config{issuing_distribution_point = true}) -> "keyUsage = cRLSign, keyCertSign\n" "subjectKeyIdentifier = hash\n" "authorityKeyIdentifier = keyid:always,issuer:always\n" - "subjectAltName = email:copy\n" + "subjectAltName = DNS.1:" ++ Hostname ++ "\n" "issuerAltName = issuer:copy\n" "crlDistributionPoints=@crl_section\n" ]; ca_cnf(Root, C = #config{issuing_distribution_point = false}) -> + Hostname = net_adm:localhost(), ["# Purpose: Configuration for CAs.\n" "\n" "ROOTDIR = " ++ Root ++ "\n" @@ -513,7 +515,7 @@ ca_cnf(Root, C = #config{issuing_distribution_point = false}) -> "keyUsage = nonRepudiation, digitalSignature, keyEncipherment\n" "subjectKeyIdentifier = hash\n" "authorityKeyIdentifier = keyid,issuer:always\n" - "subjectAltName = email:copy\n" + "subjectAltName = DNS.1:" ++ Hostname ++ "\n" "issuerAltName = issuer:copy\n" %"crlDistributionPoints=@crl_section\n" @@ -528,7 +530,7 @@ ca_cnf(Root, C = #config{issuing_distribution_point = false}) -> "keyUsage = digitalSignature\n" "subjectKeyIdentifier = hash\n" "authorityKeyIdentifier = keyid,issuer:always\n" - "subjectAltName = email:copy\n" + "subjectAltName = DNS.1:" ++ Hostname ++ "\n" "issuerAltName = issuer:copy\n" "\n" diff --git a/lib/ssl/test/ssl_basic_SUITE.erl b/lib/ssl/test/ssl_basic_SUITE.erl index 018110514c..58870a3419 100644 --- a/lib/ssl/test/ssl_basic_SUITE.erl +++ b/lib/ssl/test/ssl_basic_SUITE.erl @@ -2880,10 +2880,10 @@ der_input(Config) when is_list(Config) -> Size = ets:info(CADb, size), - SeverVerifyOpts = ssl_test_lib:ssl_options(server_verification_opts, Config), + SeverVerifyOpts = ssl_test_lib:ssl_options(server_opts, Config), {ServerCert, ServerKey, ServerCaCerts, DHParams} = der_input_opts([{dhfile, DHParamFile} | SeverVerifyOpts]), - ClientVerifyOpts = ssl_test_lib:ssl_options(client_verification_opts, Config), + ClientVerifyOpts = ssl_test_lib:ssl_options(client_opts, Config), {ClientCert, ClientKey, ClientCaCerts, DHParams} = der_input_opts([{dhfile, DHParamFile} | ClientVerifyOpts]), ServerOpts = [{verify, verify_peer}, {fail_if_no_peer_cert, true}, diff --git a/lib/ssl/test/ssl_bench_SUITE.erl b/lib/ssl/test/ssl_bench_SUITE.erl index 960ddf7808..ae2928b1c3 100644 --- a/lib/ssl/test/ssl_bench_SUITE.erl +++ b/lib/ssl/test/ssl_bench_SUITE.erl @@ -410,13 +410,19 @@ ssl_opts(connect_der) -> [{verify, verify_peer} | ssl_opts("client_der")]; ssl_opts(Role) -> CertData = cert_data(Role), - [{active, false}, - {depth, 2}, - {reuseaddr, true}, - {mode,binary}, - {nodelay, true}, - {ciphers, [{dhe_rsa,aes_256_cbc,sha}]} - |CertData]. + Opts = [{active, false}, + {depth, 2}, + {reuseaddr, true}, + {mode,binary}, + {nodelay, true}, + {ciphers, [{dhe_rsa,aes_256_cbc,sha}]} + |CertData], + case Role of + "client" ++ _ -> + [{server_name_indication, disable} | Opts]; + "server" ++ _ -> + Opts + end. cert_data(Der) when Der =:= "server_der"; Der =:= "client_der" -> [Role,_] = string:tokens(Der, "_"), diff --git a/lib/ssl/test/ssl_test_lib.erl b/lib/ssl/test/ssl_test_lib.erl index f44d1d5523..68a99c28dd 100644 --- a/lib/ssl/test/ssl_test_lib.erl +++ b/lib/ssl/test/ssl_test_lib.erl @@ -34,13 +34,13 @@ run_where(_) -> ClientNode = node(), ServerNode = node(), - {ok, Host} = rpc:call(ServerNode, inet, gethostname, []), + Host = rpc:call(ServerNode, net_adm, localhost, []), {ClientNode, ServerNode, Host}. run_where(_, ipv6) -> ClientNode = node(), ServerNode = node(), - {ok, Host} = rpc:call(ServerNode, inet, gethostname, []), + Host = rpc:call(ServerNode, net_adm, localhost, []), {ClientNode, ServerNode, Host}. node_to_hostip(Node) -> @@ -1394,6 +1394,18 @@ portable_open_port(Exe, Args) -> open_port({spawn_executable, AbsPath}, [{args, Args}, stderr_to_stdout]). +supports_ssl_tls_version(sslv2 = Version) -> + case os:cmd("openssl version") of + "OpenSSL 1" ++ _ -> + false; + _ -> + VersionFlag = version_flag(Version), + Exe = "openssl", + Args = ["s_client", VersionFlag], + Port = ssl_test_lib:portable_open_port(Exe, Args), + do_supports_ssl_tls_version(Port) + end; + supports_ssl_tls_version(Version) -> VersionFlag = version_flag(Version), Exe = "openssl", @@ -1403,6 +1415,8 @@ supports_ssl_tls_version(Version) -> do_supports_ssl_tls_version(Port) -> receive + {Port, {data, "u"}} -> + false; {Port, {data, "unknown option" ++ _}} -> false; {Port, {data, Data}} -> diff --git a/lib/ssl/test/x509_test.erl b/lib/ssl/test/x509_test.erl index c36e96013b..4da1537ef6 100644 --- a/lib/ssl/test/x509_test.erl +++ b/lib/ssl/test/x509_test.erl @@ -105,7 +105,7 @@ root_cert(Role, PrivKey, Opts) -> validity = validity(Opts), subject = Issuer, subjectPublicKeyInfo = public_key(PrivKey), - extensions = extensions(ca, Opts) + extensions = extensions(Role, ca, Opts) }, public_key:pkix_sign(OTPTBS, PrivKey). @@ -175,22 +175,27 @@ validity(Opts) -> #'Validity'{notBefore={generalTime, Format(DefFrom)}, notAfter ={generalTime, Format(DefTo)}}. -extensions(Type, Opts) -> +extensions(Role, Type, Opts) -> Exts = proplists:get_value(extensions, Opts, []), - lists:flatten([extension(Ext) || Ext <- default_extensions(Type, Exts)]). + lists:flatten([extension(Ext) || Ext <- default_extensions(Role, Type, Exts)]). %% Common extension: name_constraints, policy_constraints, ext_key_usage, inhibit_any, %% auth_key_id, subject_key_id, policy_mapping, -default_extensions(ca, Exts) -> +default_extensions(_, ca, Exts) -> Def = [{key_usage, [keyCertSign, cRLSign]}, {basic_constraints, default}], add_default_extensions(Def, Exts); -default_extensions(peer, Exts) -> - Def = [{key_usage, [digitalSignature, keyAgreement]}], - add_default_extensions(Def, Exts). +default_extensions(server, peer, Exts) -> + Hostname = net_adm:localhost(), + Def = [{key_usage, [digitalSignature, keyAgreement]}, + {subject_alt, Hostname}], + add_default_extensions(Def, Exts); +default_extensions(_, peer, Exts) -> + Exts. + add_default_extensions(Def, Exts) -> Filter = fun({Key, _}, D) -> lists:keydelete(Key, 1, D); @@ -228,6 +233,10 @@ extension({key_usage, Value}) -> #'Extension'{extnID = ?'id-ce-keyUsage', extnValue = Value, critical = false}; +extension({subject_alt, Hostname}) -> + #'Extension'{extnID = ?'id-ce-subjectAltName', + extnValue = [{dNSName, Hostname}], + critical = false}; extension({Id, Data, Critical}) -> #'Extension'{extnID = Id, extnValue = Data, critical = Critical}. @@ -309,7 +318,7 @@ cert(Role, #'OTPCertificate'{tbsCertificate = #'OTPTBSCertificate'{subject = Iss validity = validity(CertOpts), subject = subject(Contact, atom_to_list(Role) ++ Name), subjectPublicKeyInfo = public_key(Key), - extensions = extensions(Type, + extensions = extensions(Role, Type, add_default_extensions([{auth_key_id, {auth_key_oid(Role), Issuer, SNr}}], CertOpts)) }, diff --git a/lib/stdlib/doc/src/ets.xml b/lib/stdlib/doc/src/ets.xml index 0d1e5c6e3a..342be80f98 100644 --- a/lib/stdlib/doc/src/ets.xml +++ b/lib/stdlib/doc/src/ets.xml @@ -1495,17 +1495,37 @@ is_integer(X), is_integer(Y), X + Y < 4711]]></code> <fsummary>Match and replace objects atomically in an ETS table</fsummary> <desc> <p>Matches the objects in the table <c><anno>Tab</anno></c> using a - <seealso marker="#match_spec">match specification</seealso>. If - an object is matched, the existing object is replaced with - the match specification result, which <em>must</em> retain - the original key or the operation will fail with <c>badarg</c>.</p> + <seealso marker="#match_spec">match specification</seealso>. For each + matched object, the existing object is replaced with + the match specification result.</p> + <p>The match-and-replace operation for each individual object is guaranteed to be + <seealso marker="#concurrency">atomic and isolated</seealso>. The + <c>select_replace</c> table iteration as a whole, like all other select functions, + does not give such guarantees.</p> + <p>The match specifiction must be guaranteed to <em>retain the key</em> + of any matched object. If not, <c>select_replace</c> will fail with <c>badarg</c> + without updating any objects.</p> <p>For the moment, due to performance and semantic constraints, tables of type <c>bag</c> are not yet supported.</p> <p>The function returns the total number of replaced objects.</p> - <note> - <p>The match/replacement operation atomicity scope is limited - to each individual object.</p> - </note> + <p><em>Example</em></p> + <p>For all 2-tuples with a list in second position, add atom <c>'marker'</c> first in the list:</p> + <pre> +1> <input>T = ets:new(x,[]), ets:insert(T, {key, [1, 2, 3]}).</input> +true +2> <input>MS = ets:fun2ms(fun({K, L}) when is_list(L) -> {K, [marker | L]} end).</input> +[{{'$1','$2'},[{is_list,'$2'}],[{{'$1',[marker|'$2']}}]}] +3> <input>ets:select_replace(T, MS).</input> +1 +4> <input>ets:tab2list(T).</input> +[{key,[marker,1,2,3]}] + </pre> + <p>A generic single object compare-and-swap operation:</p> + <pre> +[Old] = ets:lookup(T, Key), +New = update_object(Old), +Success = (1 =:= ets:select_replace(T, [{Old, [], [{const, New}]}])), + </pre> </desc> </func> diff --git a/lib/stdlib/doc/src/gen_event.xml b/lib/stdlib/doc/src/gen_event.xml index 09bde3e397..012737c390 100644 --- a/lib/stdlib/doc/src/gen_event.xml +++ b/lib/stdlib/doc/src/gen_event.xml @@ -358,7 +358,7 @@ gen_event:stop -----> Module:terminate/2 <v> Name = atom()</v> <v> GlobalName = ViaName = term()</v> <v>Options = [Option]</v> - <v> Option = {debug,Dbgs} | {timeout,Time} | {spawn_opt,SOpts}</v> + <v> Option = {debug,Dbgs} | {timeout,Time} | {hibernate_after,HibernateAfterTimeout} | {spawn_opt,SOpts}</v> <v> Dbgs = [Dbg]</v> <v> Dbg = trace | log | statistics | {log_to_file,FileName} | {install,{Func,FuncState}}</v> <v> SOpts = [term()]</v> @@ -385,7 +385,7 @@ gen_event:stop -----> Module:terminate/2 <v> Name = atom()</v> <v> GlobalName = ViaName = term()</v> <v>Options = [Option]</v> - <v> Option = {debug,Dbgs} | {timeout,Time} | {spawn_opt,SOpts}</v> + <v> Option = {debug,Dbgs} | {timeout,Time} | {hibernate_after,HibernateAfterTimeout} | {spawn_opt,SOpts}</v> <v> Dbgs = [Dbg]</v> <v> Dbg = trace | log | statistics | {log_to_file,FileName} | {install,{Func,FuncState}}</v> <v> SOpts = [term()]</v> @@ -419,6 +419,12 @@ gen_event:stop -----> Module:terminate/2 <seealso marker="kernel:global"><c>global</c></seealso>. Thus, <c>{via,global,GlobalName}</c> is a valid reference.</p> </item> + <item> + <p>If option <c>{hibernate_after,HibernateAfterTimeout}</c> is present, the <c>gen_event</c> + process awaits any message for <c>HibernateAfterTimeout</c> milliseconds and + if no message is received, the process goes into hibernation automatically + (by calling <seealso marker="proc_lib#hibernate/3"><c>proc_lib:hibernate/3</c></seealso>).</p> + </item> </list> <p>If the event manager is successfully created, the function returns <c>{ok,Pid}</c>, where <c>Pid</c> is the pid of diff --git a/lib/stdlib/doc/src/gen_server.xml b/lib/stdlib/doc/src/gen_server.xml index a8006bb870..7d137fc772 100644 --- a/lib/stdlib/doc/src/gen_server.xml +++ b/lib/stdlib/doc/src/gen_server.xml @@ -199,7 +199,7 @@ gen_server:abcast -----> Module:handle_cast/2 <type> <v>Module = atom()</v> <v>Options = [Option]</v> - <v> Option = {debug,Dbgs}</v> + <v> Option = {debug,Dbgs} | {hibernate_after,HibernateAfterTimeout}</v> <v> Dbgs = [Dbg]</v> <v> Dbg = trace | log | statistics</v> <v> | {log_to_file,FileName} | {install,{Func,FuncState}}</v> @@ -334,7 +334,7 @@ gen_server:abcast -----> Module:handle_cast/2 <v>Module = atom()</v> <v>Args = term()</v> <v>Options = [Option]</v> - <v> Option = {debug,Dbgs} | {timeout,Time} | {spawn_opt,SOpts}</v> + <v> Option = {debug,Dbgs} | {timeout,Time} | {hibernate_after,HibernateAfterTimeout} | {spawn_opt,SOpts}</v> <v> Dbgs = [Dbg]</v> <v> Dbg = trace | log | statistics | {log_to_file,FileName} | {install,{Func,FuncState}}</v> <v> SOpts = [term()]</v> @@ -364,7 +364,7 @@ gen_server:abcast -----> Module:handle_cast/2 <v>Module = atom()</v> <v>Args = term()</v> <v>Options = [Option]</v> - <v> Option = {debug,Dbgs} | {timeout,Time} | {spawn_opt,SOpts}</v> + <v> Option = {debug,Dbgs} | {timeout,Time} | {hibernate_after,HibernateAfterTimeout} | {spawn_opt,SOpts}</v> <v> Dbgs = [Dbg]</v> <v> Dbg = trace | log | statistics | {log_to_file,FileName} | {install,{Func,FuncState}}</v> <v> SOpts = [term()]</v> @@ -417,6 +417,12 @@ gen_server:abcast -----> Module:handle_cast/2 returns <c>{error,timeout}</c>.</p> </item> <item> + <p>If option <c>{hibernate_after,HibernateAfterTimeout}</c> is present, the <c>gen_server</c> + process awaits any message for <c>HibernateAfterTimeout</c> milliseconds and + if no message is received, the process goes into hibernation automatically + (by calling <seealso marker="proc_lib#hibernate/3"><c>proc_lib:hibernate/3</c></seealso>).</p> + </item> + <item> <p>If option <c>{debug,Dbgs}</c> is present, the corresponding <c>sys</c> function is called for each item in <c>Dbgs</c>; see diff --git a/lib/stdlib/doc/src/gen_statem.xml b/lib/stdlib/doc/src/gen_statem.xml index 17a3a3c83c..1aac88c308 100644 --- a/lib/stdlib/doc/src/gen_statem.xml +++ b/lib/stdlib/doc/src/gen_statem.xml @@ -453,6 +453,21 @@ handle_event(_, _, State, Data) -> </desc> </datatype> <datatype> + <name name="hibernate_after_opt"/> + <desc> + <p> + hibernate_after option that can be used when starting + a <c>gen_statem</c> server through, + <seealso marker="#enter_loop/4"><c>enter_loop/4-6</c></seealso>. + </p> + <p>If option<seealso marker="#type-hibernate_after_opt"><c>{hibernate_after,HibernateAfterTimeout}</c></seealso> is present, the <c>gen_statem</c> + process awaits any message for <c>HibernateAfterTimeout</c> milliseconds and + if no message is received, the process goes into hibernation automatically + (by calling <seealso marker="proc_lib#hibernate/3"><c>proc_lib:hibernate/3</c></seealso>). + </p> + </desc> + </datatype> + <datatype> <name name="start_opt"/> <desc> <p> @@ -1551,6 +1566,13 @@ handle_event(_, _, State, Data) -> </p> </item> <item> + <p>If option<seealso marker="#type-hibernate_after_opt"><c>{hibernate_after,HibernateAfterTimeout}</c></seealso> is present, the <c>gen_statem</c> + process awaits any message for <c>HibernateAfterTimeout</c> milliseconds and + if no message is received, the process goes into hibernation automatically + (by calling <seealso marker="proc_lib#hibernate/3"><c>proc_lib:hibernate/3</c></seealso>). + </p> + </item> + <item> <p> If option <seealso marker="#type-debug_opt"><c>{debug,Dbgs}</c></seealso> diff --git a/lib/stdlib/doc/src/notes.xml b/lib/stdlib/doc/src/notes.xml index b4b6eac9d3..a8a252cb35 100644 --- a/lib/stdlib/doc/src/notes.xml +++ b/lib/stdlib/doc/src/notes.xml @@ -31,384 +31,6 @@ </header> <p>This document describes the changes made to the STDLIB application.</p> -<section><title>STDLIB 3.4</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p>For many releases, it has been legal to override a BIF - with a local function having the same name. However, - calling a local function with the same name as guard BIF - as filter in a list comprehension was not allowed.</p> - <p> - Own Id: OTP-13690</p> - </item> - <item> - <p> A new (default) pseudo-random number generator - algorithm Xoroshiro116+ has been implemented in the - <c>rand</c> module. </p><p> The old algorithm - implementations had a number of flaws so they are all - deprecated, but corrected versions of two of them have - been added. See the documentation. </p> - <p> - Own Id: OTP-14295 Aux Id: PR-1372 </p> - </item> - <item> - <p> The Erlang shell, <c>qlc:string_to_handle()</c>, and - the Debugger (the Evaluator area and Edit variable window - of the Bindings area) can parse pids, ports, references, - and external funs, as long as they can be created in the - running system. </p> - <p> - Own Id: OTP-14296</p> - </item> - <item> - <p> The state machine engine <c>gen_statem</c> can now - handle generic time-outs (multiple named) as well as - absolute time-out time. See the documentation. </p><p> - The <c>gen_statem</c> callback <c>Module:init/1</c> has - become mandatory to harmonize with other <c>gen_*</c> - modules. This may be an incompatibility for - <c>gen_statem</c> callback modules that use - <c>gen_statem:enter_loop/4-6</c>. </p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14531</p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> - Improved unicode support for strings. Added normalization - functions in the <c>unicode</c> module. Extended the - <c>string</c> module API with new functions with improved - unicode handling and that works on grapheme clusters. The - new functions operates on the <c>unicode:chardata()</c> - type, thus they also accept <c>UTF-8 binaries</c> as - input. </p> - <p> - The old string API have been marked as obsolete. The - return values have been changed for some error cases.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-10289 Aux Id: OTP-10309 </p> - </item> - <item> - <p>There are two new guard BIFs '<c>floor/1</c>' and - '<c>ceil/1</c>'. They both return integers. In the - '<c>math</c>' module, there are two new BIFs with the - same names that return floating point values.</p> - <p> - Own Id: OTP-13692</p> - </item> - <item> - <p> - Making code_change, terminate and handle_info callbacks - optional in the OTP behaviours.</p> - <p> - Own Id: OTP-13801</p> - </item> - <item> - <p> The support for Dets files created with Erlang/OTP R7 - and earlier is removed. </p> - <p> - Own Id: OTP-13830</p> - </item> - <item> - <p>Replaced usage of deprecated symbolic <seealso - marker="erts:erlang#type-time_unit"><c>time - unit</c></seealso> representations.</p> - <p> - Own Id: OTP-13831 Aux Id: OTP-13735 </p> - </item> - <item> - <p>The function <c>fmod/2</c> has been added to the - <c>math</c> module.</p> - <p> - Own Id: OTP-14000</p> - </item> - <item> - <p>The EXIT signals received from processes using - <c>proc_lib</c> now looks like EXIT signals from - processes that were spawned using <c>spawn_link</c>. In - particular, that means that the stack trace is now - included in the EXIT signal so that it can see where the - process crashed.</p> - <p> - Own Id: OTP-14001</p> - </item> - <item> - <p><c>sets:add_element/2</c> is faster when adding an - element that is already present, and - <c>sets:del_element/2</c> is faster when the element to - be deleted is not present. This optimization can make - certain operations, such as sets:union/2 with many - overlapping elements, up to two orders of magnitude - faster.</p> - <p> - Own Id: OTP-14035</p> - </item> - <item> - <p> - Add information in doc about supervisor shutdown reason - when maximum restart frequency is reached.</p> - <p> - Own Id: OTP-14037 Aux Id: PR-1233 </p> - </item> - <item> - <p> - Added <c>rand:jump/[0|1]</c> functions.</p> - <p> - Own Id: OTP-14038 Aux Id: PR-1235 </p> - </item> - <item> - <p>Functions for detecting changed code has been added. - <c>code:modified_modules/0</c> returns all currently - loaded modules that have changed on disk. - <c>code:module_status/1</c> returns the status for a - module. In the shell and in <c>c</c> module, <c>mm/0</c> - is short for <c>code:modified_modules/0</c>, and - <c>lm/0</c> reloads all currently loaded modules that - have changed on disk.</p> - <p> - Own Id: OTP-14059</p> - </item> - <item> - <p>Each assert macro in <c>assert.hrl</c> now has a - corresponding version with an extra argument, for adding - comments to assertions. These can for example be printed - as part of error reports, to clarify the meaning of the - check that failed.</p> - <p> - Own Id: OTP-14066</p> - </item> - <item> - <p><c>error_logger_tty_h</c> and - <c>error_logger_file_h</c> now inserts the node - information for nonlocal messages before the message - itself instead of after, both for readability and so as - not to change the line termination property at the end of - the message.</p> - <p> - Own Id: OTP-14068</p> - </item> - <item> - <p>The Erlang code linter checks for badly formed type - constraints. </p> - <p> - Own Id: OTP-14070 Aux Id: PR-1214 </p> - </item> - <item> - <p>By default, there will now be a warning when - <c>export_all</c> is used. The warning can be disabled - using <c>nowarn_export_all</c>.</p> - <p> - Own Id: OTP-14071</p> - </item> - <item> - <p>When a <c>gen_server</c> process crashes, the - stacktrace for the client will be printed to facilitate - debugging.</p> - <p> - Own Id: OTP-14089</p> - </item> - <item> - <p> - Optimized ETS operations by changing table identifier - type from integer to reference. The reference enables a - more direct mapping to the table with less potential lock - contention and makes especially creation and deletion of - tables scale much better.</p> - <p> - The change of the opaque type for the ETS table - identifiers may cause failure in code that make faulty - assumptions about this opaque type.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14094</p> - </item> - <item> - <p><c>take/2</c> has been added to <c>dict</c>, - <c>orddict</c>, and <c>gb_trees</c>. <c>take_any/2</c> - has been added to <c>gb_trees</c>.</p> - <p> - Own Id: OTP-14102</p> - </item> - <item> - <p> - Extend gen_event API to handle options as well.</p> - <p> - Own Id: OTP-14123</p> - </item> - <item> - <p> - Advice on how to tune the supervisor restart frequency - (intensity and period) is added to System Documentation - - Design Principles - Supervisor Behaviour.</p> - <p> - Own Id: OTP-14168 Aux Id: PR-1289 </p> - </item> - <item> - <p> - Replace gen_fsm with gen_statem</p> - <p> - Own Id: OTP-14183</p> - </item> - <item> - <p>The shell functions <c>c/1</c> and <c>c/2</c> have - been extended so that if the argument is a module name - instead of a file name, it automatically locates the - .beam file and the corresponding source file, and then - recompiles the module using the same compiler options - (plus any options passed to c/2). If compilation fails, - the old beam file is preserved. Also adds <c>c(Mod, Opts, - Filter)</c>, where the Filter argument allows you to - remove old compiler options before the new options are - added.</p> <p>New utility functions <c>file_find/2/3</c> - and <c>find_source/1/2/3</c> have been added to - <c>filelib</c>.</p> - <p> - Own Id: OTP-14190</p> - </item> - <item> - <p><c>erl_tar</c> in previous versions of OTP only - supports the USTAR format. That limited path names to at - most 255 bytes, and did not support Unicode characters in - names in a portable way.</p> - <p><c>erl_tar</c> now has support for reading tar - archives in the formats currently in common use, such as - v7, STAR, USTAR, PAX, and GNU tar's extensions to the - STAR/USTAR format. When writing tar archives, - <c>erl_tar</c> can now write them in the <c>PAX</c> - format if necessary (for example, to support very long - filenames or filenames with Unicode characters). If - possible, <c>erl_tar</c> will still write tar archives in - the USTAR for maximum portability.</p> - <p> - Own Id: OTP-14226</p> - </item> - <item> - <p><c>base64:mime_decode/1</c> has been optimized so that - it is now almost as fast as<c>base64:decode/1</c>; it - used be noticably slower.</p> - <p> - Own Id: OTP-14245</p> - </item> - <item> - <p><c>erl_tar</c> will now strip any leading '<c>/</c>' - from pathnames when extracting files from a tar archive - and write a message to the error logger. There is also - new check for directory traversal attacks; if a relative - path points above the current working directory the - extraction will be aborted.</p> - <p> - Own Id: OTP-14278</p> - </item> - <item> - <p> Miscellaneous updates due to atoms containing - arbitrary Unicode characters. </p> - <p> - Own Id: OTP-14285</p> - </item> - <item> - <p> - The Crypto application now supports generation of - cryptographically strong random numbers (floats < 1.0 - and integer arbitrary ranges) as a plugin to the 'rand' - module.</p> - <p> - Own Id: OTP-14317 Aux Id: PR-1372 </p> - </item> - <item> - <p> - Add new function <c>ets:select_replace/2</c> which - performs atomic "compare-and-swap" operations for ETS - objects using match specifications.</p> - <p> - Own Id: OTP-14319 Aux Id: PR-1076 </p> - </item> - <item> - <p> The Erlang code linter checks for bad <c>dialyzer</c> - attributes. </p> - <p> - Own Id: OTP-14323</p> - </item> - <item> - <p> Two new functions has been implemented in the - <c>rand</c> module; <c>normal/2</c> and - <c>normal_s/3</c>, that both produce normal distribution - (pseudo) random numbers with mean value and variance - according to arguments. </p> - <p> - Own Id: OTP-14328 Aux Id: PR-1382 </p> - </item> - <item> - <p> - Upgraded the OTP internal PCRE library from version 8.33 - to version 8.40. This library is used for implementation - of the <seealso marker="stdlib:re"><c>re</c></seealso> - regular expressions module.</p> - <p> - Besides various bug fixes, the new version allows for - better stack protection. In order to utilize this - feature, the stack size of normal scheduler threads is - now by default set to 128 kilo words on all platforms. - The stack size of normal scheduler threads can be set - upon system start by passing the <seealso - marker="erts:erl#sched_thread_stack_size"><c>+sss</c></seealso> - command line argument to the <seealso - marker="erts:erl"><c>erl</c></seealso> command.</p> - <p> - See <url - href="http://pcre.org/original/changelog.txt"><c>http://pcre.org/original/changelog.txt</c></url> - for information about changes made to PCRE between the - versions 8.33 and 8.40.</p> - <p> - *** POTENTIAL INCOMPATIBILITY ***</p> - <p> - Own Id: OTP-14331 Aux Id: ERL-208 </p> - </item> - <item> - <p> - Added function <c>re:version/0</c> which returns - information about the OTP internal PCRE version used for - implementation of the <c>re</c> module.</p> - <p> - Own Id: OTP-14347 Aux Id: PR-1412 </p> - </item> - <item> - <p>The format of debug information that is stored in BEAM - files (when <c>debug_info</c> is used) has been changed. - The purpose of the change is to better support other - BEAM-based languages such as Elixir or LFE.</p> - <p>All tools included in OTP (dialyzer, debugger, cover, - and so on) will handle both the new format and the - previous format. Tools that retrieve the debug - information using <c>beam_lib:chunk(Beam, - [abstract_code])</c> will continue to work with both the - new and old format. Tools that call - <c>beam_lib:chunk(Beam, ["Abst"])</c> will not work with - the new format.</p> - <p>For more information, see the description of - <c>debug_info</c> in the documentation for - <c>beam_lib</c> and the description of the - <c>{debug_info,{Backend,Data}}</c> option in the - documentation for <c>compile</c>.</p> - <p> - Own Id: OTP-14369 Aux Id: PR-1367 </p> - </item> - </list> - </section> - -</section> - <section><title>STDLIB 3.3</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/stdlib/src/erl_lint.erl b/lib/stdlib/src/erl_lint.erl index 7c40058dd8..d53a31db0d 100644 --- a/lib/stdlib/src/erl_lint.erl +++ b/lib/stdlib/src/erl_lint.erl @@ -92,6 +92,14 @@ value_option(Flag, Default, On, OnVal, Off, OffVal, Opts) -> :: dict:dict(ta(), line()) }). + +%% Are we outside or inside a catch or try/catch? +-type catch_scope() :: 'none' + | 'after_old_catch' + | 'after_try' + | 'wrong_part_of_try' + | 'try_catch'. + %% Define the lint state record. %% 'called' and 'exports' contain {Line, {Function, Arity}}, %% the other function collections contain {Function, Arity}. @@ -135,7 +143,9 @@ value_option(Flag, Default, On, OnVal, Off, OffVal, Opts) -> types = dict:new() %Type definitions :: dict:dict(ta(), #typeinfo{}), exp_types=gb_sets:empty() %Exported types - :: gb_sets:set(ta()) + :: gb_sets:set(ta()), + catch_scope = none %Inside/outside try or catch + :: catch_scope() }). -type lint_state() :: #lint{}. @@ -223,7 +233,15 @@ format_error({redefine_old_bif_import,{F,A}}) -> format_error({redefine_bif_import,{F,A}}) -> io_lib:format("import directive overrides auto-imported BIF ~w/~w~n" " - use \"-compile({no_auto_import,[~w/~w]}).\" to resolve name clash", [F,A,F,A]); - +format_error({get_stacktrace,wrong_part_of_try}) -> + "erlang:get_stacktrace/0 used in the wrong part of 'try' expression. " + "(Use it in the block between 'catch' and 'end'.)"; +format_error({get_stacktrace,after_old_catch}) -> + "erlang:get_stacktrace/0 used following an old-style 'catch' " + "may stop working in a future release. (Use it inside 'try'.)"; +format_error({get_stacktrace,after_try}) -> + "erlang:get_stacktrace/0 used following a 'try' expression " + "may stop working in a future release. (Use it inside 'try'.)"; format_error({deprecated, MFA, ReplacementMFA, Rel}) -> io_lib:format("~s is deprecated and will be removed in ~s; use ~s", [format_mfa(MFA), Rel, format_mfa(ReplacementMFA)]); @@ -568,7 +586,10 @@ start(File, Opts) -> false, Opts)}, {missing_spec_all, bool_option(warn_missing_spec_all, nowarn_missing_spec_all, - false, Opts)} + false, Opts)}, + {get_stacktrace, + bool_option(warn_get_stacktrace, nowarn_get_stacktrace, + true, Opts)} ], Enabled1 = [Category || {Category,true} <- Enabled0], Enabled = ordsets:from_list(Enabled1), @@ -1405,8 +1426,9 @@ call_function(Line, F, A, #lint{usage=Usage0,called=Cd,func=Func,file=File}=St) %% function(Line, Name, Arity, Clauses, State) -> State. function(Line, Name, Arity, Cs, St0) -> - St1 = define_function(Line, Name, Arity, St0#lint{func={Name,Arity}}), - clauses(Cs, St1). + St1 = St0#lint{func={Name,Arity},catch_scope=none}, + St2 = define_function(Line, Name, Arity, St1), + clauses(Cs, St2). -spec define_function(line(), atom(), arity(), lint_state()) -> lint_state(). @@ -2338,22 +2360,24 @@ expr({call,Line,F,As}, Vt, St0) -> expr({'try',Line,Es,Scs,Ccs,As}, Vt, St0) -> %% Currently, we don't allow any exports because later %% passes cannot handle exports in combination with 'after'. - {Evt0,St1} = exprs(Es, Vt, St0), + {Evt0,St1} = exprs(Es, Vt, St0#lint{catch_scope=wrong_part_of_try}), TryLine = {'try',Line}, Uvt = vtunsafe(TryLine, Evt0, Vt), Evt1 = vtupdate(Uvt, Evt0), - {Sccs,St2} = icrt_clauses(Scs++Ccs, TryLine, vtupdate(Evt1, Vt), St1), + {Sccs,St2} = try_clauses(Scs, Ccs, TryLine, + vtupdate(Evt1, Vt), St1), Rvt0 = Sccs, Rvt1 = vtupdate(vtunsafe(TryLine, Rvt0, Vt), Rvt0), Evt2 = vtmerge(Evt1, Rvt1), {Avt0,St} = exprs(As, vtupdate(Evt2, Vt), St2), Avt1 = vtupdate(vtunsafe(TryLine, Avt0, Vt), Avt0), Avt = vtmerge(Evt2, Avt1), - {Avt,St}; + {Avt,St#lint{catch_scope=after_try}}; expr({'catch',Line,E}, Vt, St0) -> %% No new variables added, flag new variables as unsafe. {Evt,St} = expr(E, Vt, St0), - {vtupdate(vtunsafe({'catch',Line}, Evt, Vt), Evt),St}; + {vtupdate(vtunsafe({'catch',Line}, Evt, Vt), Evt), + St#lint{catch_scope=after_old_catch}}; expr({match,_Line,P,E}, Vt, St0) -> {Evt,St1} = expr(E, Vt, St0), {Pvt,Bvt,St2} = pattern(P, vtupdate(Evt, Vt), St1), @@ -3173,6 +3197,17 @@ is_module_dialyzer_option(Option) -> error_handling,race_conditions,no_missing_calls, specdiffs,overspecs,underspecs,unknown]). +%% try_catch_clauses(Scs, Ccs, In, ImportVarTable, State) -> +%% {UpdVt,State}. + +try_clauses(Scs, Ccs, In, Vt, St0) -> + {Csvt0,St1} = icrt_clauses(Scs, Vt, St0), + St2 = St1#lint{catch_scope=try_catch}, + {Csvt1,St3} = icrt_clauses(Ccs, Vt, St2), + Csvt = Csvt0 ++ Csvt1, + UpdVt = icrt_export(Csvt, Vt, In, St3), + {UpdVt,St3}. + %% icrt_clauses(Clauses, In, ImportVarTable, State) -> %% {UpdVt,State}. @@ -3657,7 +3692,8 @@ has_wildcard_field([]) -> false. check_remote_function(Line, M, F, As, St0) -> St1 = deprecated_function(Line, M, F, As, St0), St2 = check_qlc_hrl(Line, M, F, As, St1), - format_function(Line, M, F, As, St2). + St3 = check_get_stacktrace(Line, M, F, As, St2), + format_function(Line, M, F, As, St3). %% check_qlc_hrl(Line, ModName, FuncName, [Arg], State) -> State %% Add warning if qlc:q/1,2 has been called but qlc.hrl has not @@ -3706,6 +3742,23 @@ deprecated_function(Line, M, F, As, St) -> St end. +check_get_stacktrace(Line, erlang, get_stacktrace, [], St) -> + case St of + #lint{catch_scope=none} -> + St; + #lint{catch_scope=try_catch} -> + St; + #lint{catch_scope=Scope} -> + case is_warn_enabled(get_stacktrace, St) of + false -> + St; + true -> + add_warning(Line, {get_stacktrace,Scope}, St) + end + end; +check_get_stacktrace(_, _, _, _, St) -> + St. + -dialyzer({no_match, deprecated_type/5}). deprecated_type(L, M, N, As, St) -> diff --git a/lib/stdlib/src/error_logger_file_h.erl b/lib/stdlib/src/error_logger_file_h.erl index 0b262de3ab..76f89841b9 100644 --- a/lib/stdlib/src/error_logger_file_h.erl +++ b/lib/stdlib/src/error_logger_file_h.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2016. All Rights Reserved. +%% Copyright Ericsson AB 1996-2017. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -57,7 +57,7 @@ init(File, PrevHandler) -> process_flag(trap_exit, true), case file:open(File, [write]) of {ok,Fd} -> - Depth = get_depth(), + Depth = error_logger:get_format_depth(), State = #st{fd=Fd,filename=File,prev_handler=PrevHandler, depth=Depth}, {ok, State}; @@ -65,14 +65,6 @@ init(File, PrevHandler) -> Error end. -get_depth() -> - case application:get_env(kernel, error_logger_format_depth) of - {ok, Depth} when is_integer(Depth) -> - max(10, Depth); - undefined -> - unlimited - end. - handle_event({_Type, GL, _Msg}, State) when node(GL) =/= node() -> {ok, State}; handle_event(Event, State) -> diff --git a/lib/stdlib/src/error_logger_tty_h.erl b/lib/stdlib/src/error_logger_tty_h.erl index 2f2fd65252..8f0d7b0362 100644 --- a/lib/stdlib/src/error_logger_tty_h.erl +++ b/lib/stdlib/src/error_logger_tty_h.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2016. All Rights Reserved. +%% Copyright Ericsson AB 1996-2017. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -44,7 +44,7 @@ %% This one is used when we takeover from the simple error_logger. init({[], {error_logger, Buf}}) -> User = set_group_leader(), - Depth = get_depth(), + Depth = error_logger:get_format_depth(), State = #st{user=User,prev_handler=error_logger,depth=Depth}, write_events(State, Buf), {ok, State}; @@ -56,17 +56,9 @@ init({[], {error_logger_tty_h, PrevHandler}}) -> %% This one is used when we are started directly. init([]) -> User = set_group_leader(), - Depth = get_depth(), + Depth = error_logger:get_format_depth(), {ok, #st{user=User,prev_handler=[],depth=Depth}}. -get_depth() -> - case application:get_env(kernel, error_logger_format_depth) of - {ok, Depth} when is_integer(Depth) -> - max(10, Depth); - undefined -> - unlimited - end. - handle_event({_Type, GL, _Msg}, State) when node(GL) =/= node() -> {ok, State}; handle_event(Event, State) -> diff --git a/lib/stdlib/src/escript.erl b/lib/stdlib/src/escript.erl index 6e8f780f7c..f2629a47c2 100644 --- a/lib/stdlib/src/escript.erl +++ b/lib/stdlib/src/escript.erl @@ -284,8 +284,9 @@ start(EscriptOptions) -> io:format("escript: ~s\n", [Str]), my_halt(127); _:Reason -> + Stk = erlang:get_stacktrace(), io:format("escript: Internal error: ~p\n", [Reason]), - io:format("~p\n", [erlang:get_stacktrace()]), + io:format("~p\n", [Stk]), my_halt(127) end. diff --git a/lib/stdlib/src/gen.erl b/lib/stdlib/src/gen.erl index 597830cf9a..257c829801 100644 --- a/lib/stdlib/src/gen.erl +++ b/lib/stdlib/src/gen.erl @@ -26,7 +26,7 @@ %%% %%% The standard behaviour should export init_it/6. %%%----------------------------------------------------------------- --export([start/5, start/6, debug_options/2, +-export([start/5, start/6, debug_options/2, hibernate_after/1, name/1, unregister_name/1, get_proc_name/1, get_parent/0, call/3, call/4, reply/2, stop/1, stop/3]). @@ -408,6 +408,14 @@ spawn_opts(Options) -> [] end. +hibernate_after(Options) -> + case lists:keyfind(hibernate_after, 1, Options) of + {_,HibernateAfterTimeout} -> + HibernateAfterTimeout; + false -> + infinity + end. + debug_options(Name, Opts) -> case lists:keyfind(debug, 1, Opts) of {_,Options} -> diff --git a/lib/stdlib/src/gen_event.erl b/lib/stdlib/src/gen_event.erl index 4c80464680..da2b0da3ca 100644 --- a/lib/stdlib/src/gen_event.erl +++ b/lib/stdlib/src/gen_event.erl @@ -37,7 +37,7 @@ stop/1, stop/3, notify/2, sync_notify/2, add_handler/3, add_sup_handler/3, delete_handler/3, swap_handler/3, - swap_sup_handler/3, which_handlers/1, call/3, call/4, wake_hib/4]). + swap_sup_handler/3, which_handlers/1, call/3, call/4, wake_hib/5]). -export([init_it/6, system_continue/3, @@ -186,8 +186,9 @@ init_it(Starter, Parent, Name0, _, _, Options) -> process_flag(trap_exit, true), Name = gen:name(Name0), Debug = gen:debug_options(Name, Options), + HibernateAfterTimeout = gen:hibernate_after(Options), proc_lib:init_ack(Starter, {ok, self()}), - loop(Parent, Name, [], Debug, false). + loop(Parent, Name, [], HibernateAfterTimeout, Debug, false). -spec add_handler(emgr_ref(), handler(), term()) -> term(). add_handler(M, Handler, Args) -> rpc(M, {add_handler, Handler, Args}). @@ -264,81 +265,83 @@ send(M, Cmd) -> M ! Cmd, ok. -loop(Parent, ServerName, MSL, Debug, true) -> - proc_lib:hibernate(?MODULE, wake_hib, [Parent, ServerName, MSL, Debug]); -loop(Parent, ServerName, MSL, Debug, _) -> - fetch_msg(Parent, ServerName, MSL, Debug, false). +loop(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, true) -> + proc_lib:hibernate(?MODULE, wake_hib, [Parent, ServerName, MSL, HibernateAfterTimeout, Debug]); +loop(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, _) -> + fetch_msg(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, false). -wake_hib(Parent, ServerName, MSL, Debug) -> - fetch_msg(Parent, ServerName, MSL, Debug, true). +wake_hib(Parent, ServerName, MSL, HibernateAfterTimeout, Debug) -> + fetch_msg(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, true). -fetch_msg(Parent, ServerName, MSL, Debug, Hib) -> +fetch_msg(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, Hib) -> receive {system, From, Req} -> sys:handle_system_msg(Req, From, Parent, ?MODULE, Debug, - [ServerName, MSL, Hib],Hib); + [ServerName, MSL, HibernateAfterTimeout, Hib],Hib); {'EXIT', Parent, Reason} -> terminate_server(Reason, Parent, MSL, ServerName); Msg when Debug =:= [] -> - handle_msg(Msg, Parent, ServerName, MSL, []); + handle_msg(Msg, Parent, ServerName, MSL, HibernateAfterTimeout, []); Msg -> Debug1 = sys:handle_debug(Debug, fun print_event/3, ServerName, {in, Msg}), - handle_msg(Msg, Parent, ServerName, MSL, Debug1) + handle_msg(Msg, Parent, ServerName, MSL, HibernateAfterTimeout, Debug1) + after HibernateAfterTimeout -> + loop(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, true) end. -handle_msg(Msg, Parent, ServerName, MSL, Debug) -> +handle_msg(Msg, Parent, ServerName, MSL, HibernateAfterTimeout, Debug) -> case Msg of {notify, Event} -> {Hib,MSL1} = server_notify(Event, handle_event, MSL, ServerName), - loop(Parent, ServerName, MSL1, Debug, Hib); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib); {_From, Tag, {sync_notify, Event}} -> {Hib, MSL1} = server_notify(Event, handle_event, MSL, ServerName), reply(Tag, ok), - loop(Parent, ServerName, MSL1, Debug, Hib); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib); {'EXIT', From, Reason} -> MSL1 = handle_exit(From, Reason, MSL, ServerName), - loop(Parent, ServerName, MSL1, Debug, false); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, false); {_From, Tag, {call, Handler, Query}} -> {Hib, Reply, MSL1} = server_call(Handler, Query, MSL, ServerName), reply(Tag, Reply), - loop(Parent, ServerName, MSL1, Debug, Hib); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib); {_From, Tag, {add_handler, Handler, Args}} -> {Hib, Reply, MSL1} = server_add_handler(Handler, Args, MSL), reply(Tag, Reply), - loop(Parent, ServerName, MSL1, Debug, Hib); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib); {_From, Tag, {add_sup_handler, Handler, Args, SupP}} -> {Hib, Reply, MSL1} = server_add_sup_handler(Handler, Args, MSL, SupP), reply(Tag, Reply), - loop(Parent, ServerName, MSL1, Debug, Hib); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib); {_From, Tag, {delete_handler, Handler, Args}} -> {Reply, MSL1} = server_delete_handler(Handler, Args, MSL, ServerName), reply(Tag, Reply), - loop(Parent, ServerName, MSL1, Debug, false); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, false); {_From, Tag, {swap_handler, Handler1, Args1, Handler2, Args2}} -> {Hib, Reply, MSL1} = server_swap_handler(Handler1, Args1, Handler2, Args2, MSL, ServerName), reply(Tag, Reply), - loop(Parent, ServerName, MSL1, Debug, Hib); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib); {_From, Tag, {swap_sup_handler, Handler1, Args1, Handler2, Args2, Sup}} -> {Hib, Reply, MSL1} = server_swap_handler(Handler1, Args1, Handler2, Args2, MSL, Sup, ServerName), reply(Tag, Reply), - loop(Parent, ServerName, MSL1, Debug, Hib); + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib); {_From, Tag, stop} -> catch terminate_server(normal, Parent, MSL, ServerName), reply(Tag, ok); {_From, Tag, which_handlers} -> reply(Tag, the_handlers(MSL)), - loop(Parent, ServerName, MSL, Debug, false); + loop(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, false); {_From, Tag, get_modules} -> reply(Tag, get_modules(MSL)), - loop(Parent, ServerName, MSL, Debug, false); + loop(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, false); Other -> {Hib, MSL1} = server_notify(Other, handle_info, MSL, ServerName), - loop(Parent, ServerName, MSL1, Debug, Hib) + loop(Parent, ServerName, MSL1, HibernateAfterTimeout, Debug, Hib) end. terminate_server(Reason, Parent, MSL, ServerName) -> @@ -392,18 +395,18 @@ terminate_supervised(Pid, Reason, MSL, SName) -> %%----------------------------------------------------------------- %% Callback functions for system messages handling. %%----------------------------------------------------------------- -system_continue(Parent, Debug, [ServerName, MSL, Hib]) -> - loop(Parent, ServerName, MSL, Debug, Hib). +system_continue(Parent, Debug, [ServerName, MSL, HibernateAfterTimeout, Hib]) -> + loop(Parent, ServerName, MSL, HibernateAfterTimeout, Debug, Hib). -spec system_terminate(_, _, _, [_]) -> no_return(). -system_terminate(Reason, Parent, _Debug, [ServerName, MSL, _Hib]) -> +system_terminate(Reason, Parent, _Debug, [ServerName, MSL, _HibernateAfterTimeout, _Hib]) -> terminate_server(Reason, Parent, MSL, ServerName). %%----------------------------------------------------------------- %% Module here is sent in the system msg change_code. It specifies %% which module should be changed. %%----------------------------------------------------------------- -system_code_change([ServerName, MSL, Hib], Module, OldVsn, Extra) -> +system_code_change([ServerName, MSL, HibernateAfterTimeout, Hib], Module, OldVsn, Extra) -> MSL1 = lists:zf(fun(H) when H#handler.module =:= Module -> {ok, NewState} = Module:code_change(OldVsn, @@ -412,12 +415,12 @@ system_code_change([ServerName, MSL, Hib], Module, OldVsn, Extra) -> (_) -> true end, MSL), - {ok, [ServerName, MSL1, Hib]}. + {ok, [ServerName, MSL1, HibernateAfterTimeout, Hib]}. -system_get_state([_ServerName, MSL, _Hib]) -> +system_get_state([_ServerName, MSL, _HibernateAfterTimeout, _Hib]) -> {ok, [{Mod,Id,State} || #handler{module=Mod, id=Id, state=State} <- MSL]}. -system_replace_state(StateFun, [ServerName, MSL, Hib]) -> +system_replace_state(StateFun, [ServerName, MSL, HibernateAfterTimeout, Hib]) -> {NMSL, NStates} = lists:unzip([begin Cur = {Mod,Id,State}, @@ -429,7 +432,7 @@ system_replace_state(StateFun, [ServerName, MSL, Hib]) -> {HS, Cur} end end || #handler{module=Mod, id=Id, state=State}=HS <- MSL]), - {ok, NStates, [ServerName, NMSL, Hib]}. + {ok, NStates, [ServerName, NMSL, HibernateAfterTimeout, Hib]}. %%----------------------------------------------------------------- %% Format debug messages. Print them as the call-back module sees @@ -798,7 +801,7 @@ get_modules(MSL) -> %% Status information %%----------------------------------------------------------------- format_status(Opt, StatusData) -> - [PDict, SysState, Parent, _Debug, [ServerName, MSL, _Hib]] = StatusData, + [PDict, SysState, Parent, _Debug, [ServerName, MSL, _HibernateAfterTimeout, _Hib]] = StatusData, Header = gen:format_status_header("Status for event handler", ServerName), FmtMSL = [case erlang:function_exported(Mod, format_status, 2) of diff --git a/lib/stdlib/src/gen_fsm.erl b/lib/stdlib/src/gen_fsm.erl index f9d4286a7c..9ef0ca818c 100644 --- a/lib/stdlib/src/gen_fsm.erl +++ b/lib/stdlib/src/gen_fsm.erl @@ -113,7 +113,7 @@ sync_send_all_state_event/2, sync_send_all_state_event/3, reply/2, start_timer/2,send_event_after/2,cancel_timer/1, - enter_loop/4, enter_loop/5, enter_loop/6, wake_hib/6]). + enter_loop/4, enter_loop/5, enter_loop/6, wake_hib/7]). %% Internal exports -export([init_it/6, @@ -329,7 +329,8 @@ enter_loop(Mod, Options, StateName, StateData, ServerName, Timeout) -> Name = gen:get_proc_name(ServerName), Parent = gen:get_parent(), Debug = gen:debug_options(Name, Options), - loop(Parent, Name, StateName, StateData, Mod, Timeout, Debug). + HibernateAfterTimeout = gen:hibernate_after(Options), + loop(Parent, Name, StateName, StateData, Mod, Timeout, HibernateAfterTimeout, Debug). %%% --------------------------------------------------- %%% Initiate the new process. @@ -343,13 +344,14 @@ init_it(Starter, self, Name, Mod, Args, Options) -> init_it(Starter, Parent, Name0, Mod, Args, Options) -> Name = gen:name(Name0), Debug = gen:debug_options(Name, Options), - case catch Mod:init(Args) of + HibernateAfterTimeout = gen:hibernate_after(Options), + case catch Mod:init(Args) of {ok, StateName, StateData} -> proc_lib:init_ack(Starter, {ok, self()}), - loop(Parent, Name, StateName, StateData, Mod, infinity, Debug); + loop(Parent, Name, StateName, StateData, Mod, infinity, HibernateAfterTimeout, Debug); {ok, StateName, StateData, Timeout} -> proc_lib:init_ack(Starter, {ok, self()}), - loop(Parent, Name, StateName, StateData, Mod, Timeout, Debug); + loop(Parent, Name, StateName, StateData, Mod, Timeout, HibernateAfterTimeout, Debug); {stop, Reason} -> gen:unregister_name(Name0), proc_lib:init_ack(Starter, {error, Reason}), @@ -371,68 +373,77 @@ init_it(Starter, Parent, Name0, Mod, Args, Options) -> %%----------------------------------------------------------------- %% The MAIN loop %%----------------------------------------------------------------- -loop(Parent, Name, StateName, StateData, Mod, hibernate, Debug) -> +loop(Parent, Name, StateName, StateData, Mod, hibernate, HibernateAfterTimeout, Debug) -> proc_lib:hibernate(?MODULE,wake_hib, - [Parent, Name, StateName, StateData, Mod, + [Parent, Name, StateName, StateData, Mod, HibernateAfterTimeout, Debug]); -loop(Parent, Name, StateName, StateData, Mod, Time, Debug) -> + +loop(Parent, Name, StateName, StateData, Mod, infinity, HibernateAfterTimeout, Debug) -> + receive + Msg -> + decode_msg(Msg,Parent, Name, StateName, StateData, Mod, infinity, HibernateAfterTimeout, Debug, false) + after HibernateAfterTimeout -> + loop(Parent, Name, StateName, StateData, Mod, hibernate, HibernateAfterTimeout, Debug) + end; + +loop(Parent, Name, StateName, StateData, Mod, Time, HibernateAfterTimeout, Debug) -> Msg = receive Input -> Input after Time -> {'$gen_event', timeout} end, - decode_msg(Msg,Parent, Name, StateName, StateData, Mod, Time, Debug, false). + decode_msg(Msg,Parent, Name, StateName, StateData, Mod, Time, HibernateAfterTimeout, Debug, false). -wake_hib(Parent, Name, StateName, StateData, Mod, Debug) -> +wake_hib(Parent, Name, StateName, StateData, Mod, HibernateAfterTimeout, Debug) -> Msg = receive Input -> Input end, - decode_msg(Msg, Parent, Name, StateName, StateData, Mod, hibernate, Debug, true). + decode_msg(Msg, Parent, Name, StateName, StateData, Mod, hibernate, HibernateAfterTimeout, Debug, true). -decode_msg(Msg,Parent, Name, StateName, StateData, Mod, Time, Debug, Hib) -> +decode_msg(Msg,Parent, Name, StateName, StateData, Mod, Time, HibernateAfterTimeout, Debug, Hib) -> case Msg of {system, From, Req} -> sys:handle_system_msg(Req, From, Parent, ?MODULE, Debug, - [Name, StateName, StateData, Mod, Time], Hib); + [Name, StateName, StateData, Mod, Time, HibernateAfterTimeout], Hib); {'EXIT', Parent, Reason} -> terminate(Reason, Name, Msg, Mod, StateName, StateData, Debug); _Msg when Debug =:= [] -> - handle_msg(Msg, Parent, Name, StateName, StateData, Mod, Time); + handle_msg(Msg, Parent, Name, StateName, StateData, Mod, Time, HibernateAfterTimeout); _Msg -> Debug1 = sys:handle_debug(Debug, fun print_event/3, {Name, StateName}, {in, Msg}), handle_msg(Msg, Parent, Name, StateName, StateData, - Mod, Time, Debug1) + Mod, Time, HibernateAfterTimeout, Debug1) end. %%----------------------------------------------------------------- %% Callback functions for system messages handling. %%----------------------------------------------------------------- -system_continue(Parent, Debug, [Name, StateName, StateData, Mod, Time]) -> - loop(Parent, Name, StateName, StateData, Mod, Time, Debug). +system_continue(Parent, Debug, [Name, StateName, StateData, Mod, Time, HibernateAfterTimeout]) -> + loop(Parent, Name, StateName, StateData, Mod, Time, HibernateAfterTimeout, Debug). -spec system_terminate(term(), _, _, [term(),...]) -> no_return(). system_terminate(Reason, _Parent, Debug, - [Name, StateName, StateData, Mod, _Time]) -> + [Name, StateName, StateData, Mod, _Time, _HibernateAfterTimeout]) -> terminate(Reason, Name, [], Mod, StateName, StateData, Debug). -system_code_change([Name, StateName, StateData, Mod, Time], +system_code_change([Name, StateName, StateData, Mod, Time, HibernateAfterTimeout], _Module, OldVsn, Extra) -> case catch Mod:code_change(OldVsn, StateName, StateData, Extra) of {ok, NewStateName, NewStateData} -> - {ok, [Name, NewStateName, NewStateData, Mod, Time]}; + {ok, [Name, NewStateName, NewStateData, Mod, Time, HibernateAfterTimeout]}; Else -> Else end. -system_get_state([_Name, StateName, StateData, _Mod, _Time]) -> +system_get_state([_Name, StateName, StateData, _Mod, _Time, _HibernateAfterTimeout]) -> {ok, {StateName, StateData}}. -system_replace_state(StateFun, [Name, StateName, StateData, Mod, Time]) -> +system_replace_state(StateFun, [Name, StateName, StateData, Mod, Time, HibernateAfterTimeout]) -> Result = {NStateName, NStateData} = StateFun({StateName, StateData}), - {ok, Result, [Name, NStateName, NStateData, Mod, Time]}. + {ok, Result, [Name, NStateName, NStateData, Mod, Time, HibernateAfterTimeout]}. %%----------------------------------------------------------------- %% Format debug messages. Print them as the call-back module sees @@ -467,19 +478,19 @@ print_event(Dev, return, {Name, StateName}) -> io:format(Dev, "*DBG* ~p switched to state ~w~n", [Name, StateName]). -handle_msg(Msg, Parent, Name, StateName, StateData, Mod, _Time) -> %No debug here +handle_msg(Msg, Parent, Name, StateName, StateData, Mod, _Time, HibernateAfterTimeout) -> %No debug here From = from(Msg), case catch dispatch(Msg, Mod, StateName, StateData) of {next_state, NStateName, NStateData} -> - loop(Parent, Name, NStateName, NStateData, Mod, infinity, []); + loop(Parent, Name, NStateName, NStateData, Mod, infinity, HibernateAfterTimeout, []); {next_state, NStateName, NStateData, Time1} -> - loop(Parent, Name, NStateName, NStateData, Mod, Time1, []); + loop(Parent, Name, NStateName, NStateData, Mod, Time1, HibernateAfterTimeout, []); {reply, Reply, NStateName, NStateData} when From =/= undefined -> reply(From, Reply), - loop(Parent, Name, NStateName, NStateData, Mod, infinity, []); + loop(Parent, Name, NStateName, NStateData, Mod, infinity, HibernateAfterTimeout, []); {reply, Reply, NStateName, NStateData, Time1} when From =/= undefined -> reply(From, Reply), - loop(Parent, Name, NStateName, NStateData, Mod, Time1, []); + loop(Parent, Name, NStateName, NStateData, Mod, Time1, HibernateAfterTimeout, []); {stop, Reason, NStateData} -> terminate(Reason, Name, Msg, Mod, StateName, NStateData, []); {stop, Reason, Reply, NStateData} when From =/= undefined -> @@ -490,7 +501,7 @@ handle_msg(Msg, Parent, Name, StateName, StateData, Mod, _Time) -> %No debug her {'EXIT', {undef, [{Mod, handle_info, [_,_,_], _}|_]}} -> error_logger:warning_msg("** Undefined handle_info in ~p~n" "** Unhandled message: ~p~n", [Mod, Msg]), - loop(Parent, Name, StateName, StateData, Mod, infinity, []); + loop(Parent, Name, StateName, StateData, Mod, infinity, HibernateAfterTimeout, []); {'EXIT', What} -> terminate(What, Name, Msg, Mod, StateName, StateData, []); Reply -> @@ -498,23 +509,23 @@ handle_msg(Msg, Parent, Name, StateName, StateData, Mod, _Time) -> %No debug her Name, Msg, Mod, StateName, StateData, []) end. -handle_msg(Msg, Parent, Name, StateName, StateData, Mod, _Time, Debug) -> +handle_msg(Msg, Parent, Name, StateName, StateData, Mod, _Time, HibernateAfterTimeout, Debug) -> From = from(Msg), case catch dispatch(Msg, Mod, StateName, StateData) of {next_state, NStateName, NStateData} -> Debug1 = sys:handle_debug(Debug, fun print_event/3, {Name, NStateName}, return), - loop(Parent, Name, NStateName, NStateData, Mod, infinity, Debug1); + loop(Parent, Name, NStateName, NStateData, Mod, infinity, HibernateAfterTimeout, Debug1); {next_state, NStateName, NStateData, Time1} -> Debug1 = sys:handle_debug(Debug, fun print_event/3, {Name, NStateName}, return), - loop(Parent, Name, NStateName, NStateData, Mod, Time1, Debug1); + loop(Parent, Name, NStateName, NStateData, Mod, Time1, HibernateAfterTimeout, Debug1); {reply, Reply, NStateName, NStateData} when From =/= undefined -> Debug1 = reply(Name, From, Reply, Debug, NStateName), - loop(Parent, Name, NStateName, NStateData, Mod, infinity, Debug1); + loop(Parent, Name, NStateName, NStateData, Mod, infinity, HibernateAfterTimeout, Debug1); {reply, Reply, NStateName, NStateData, Time1} when From =/= undefined -> Debug1 = reply(Name, From, Reply, Debug, NStateName), - loop(Parent, Name, NStateName, NStateData, Mod, Time1, Debug1); + loop(Parent, Name, NStateName, NStateData, Mod, Time1, HibernateAfterTimeout, Debug1); {stop, Reason, NStateData} -> terminate(Reason, Name, Msg, Mod, StateName, NStateData, Debug); {stop, Reason, Reply, NStateData} when From =/= undefined -> @@ -645,7 +656,7 @@ get_msg(Msg) -> Msg. %% Status information %%----------------------------------------------------------------- format_status(Opt, StatusData) -> - [PDict, SysState, Parent, Debug, [Name, StateName, StateData, Mod, _Time]] = + [PDict, SysState, Parent, Debug, [Name, StateName, StateData, Mod, _Time, _HibernateAfterTimeout]] = StatusData, Header = gen:format_status_header("Status for state machine", Name), diff --git a/lib/stdlib/src/gen_server.erl b/lib/stdlib/src/gen_server.erl index e628fec00f..9f6568473d 100644 --- a/lib/stdlib/src/gen_server.erl +++ b/lib/stdlib/src/gen_server.erl @@ -94,7 +94,7 @@ cast/2, reply/2, abcast/2, abcast/3, multi_call/2, multi_call/3, multi_call/4, - enter_loop/3, enter_loop/4, enter_loop/5, wake_hib/5]). + enter_loop/3, enter_loop/4, enter_loop/5, wake_hib/6]). %% System exports -export([system_continue/3, @@ -107,8 +107,6 @@ %% Internal exports -export([init_it/6]). --import(error_logger, [format/2]). - %%%========================================================================= %%% API %%%========================================================================= @@ -307,7 +305,8 @@ enter_loop(Mod, Options, State, ServerName, Timeout) -> Name = gen:get_proc_name(ServerName), Parent = gen:get_parent(), Debug = gen:debug_options(Name, Options), - loop(Parent, Name, State, Mod, Timeout, Debug). + HibernateAfterTimeout = gen:hibernate_after(Options), + loop(Parent, Name, State, Mod, Timeout, HibernateAfterTimeout, Debug). %%%======================================================================== %%% Gen-callback functions @@ -325,13 +324,14 @@ init_it(Starter, self, Name, Mod, Args, Options) -> init_it(Starter, Parent, Name0, Mod, Args, Options) -> Name = gen:name(Name0), Debug = gen:debug_options(Name, Options), + HibernateAfterTimeout = gen:hibernate_after(Options), case catch Mod:init(Args) of {ok, State} -> proc_lib:init_ack(Starter, {ok, self()}), - loop(Parent, Name, State, Mod, infinity, Debug); + loop(Parent, Name, State, Mod, infinity, HibernateAfterTimeout, Debug); {ok, State, Timeout} -> proc_lib:init_ack(Starter, {ok, self()}), - loop(Parent, Name, State, Mod, Timeout, Debug); + loop(Parent, Name, State, Mod, Timeout, HibernateAfterTimeout, Debug); {stop, Reason} -> %% For consistency, we must make sure that the %% registered name (if any) is unregistered before @@ -362,37 +362,46 @@ init_it(Starter, Parent, Name0, Mod, Args, Options) -> %%% --------------------------------------------------- %%% The MAIN loop. %%% --------------------------------------------------- -loop(Parent, Name, State, Mod, hibernate, Debug) -> - proc_lib:hibernate(?MODULE,wake_hib,[Parent, Name, State, Mod, Debug]); -loop(Parent, Name, State, Mod, Time, Debug) -> +loop(Parent, Name, State, Mod, hibernate, HibernateAfterTimeout, Debug) -> + proc_lib:hibernate(?MODULE,wake_hib,[Parent, Name, State, Mod, HibernateAfterTimeout, Debug]); + +loop(Parent, Name, State, Mod, infinity, HibernateAfterTimeout, Debug) -> + receive + Msg -> + decode_msg(Msg, Parent, Name, State, Mod, infinity, HibernateAfterTimeout, Debug, false) + after HibernateAfterTimeout -> + loop(Parent, Name, State, Mod, hibernate, HibernateAfterTimeout, Debug) + end; + +loop(Parent, Name, State, Mod, Time, HibernateAfterTimeout, Debug) -> Msg = receive Input -> Input after Time -> timeout end, - decode_msg(Msg, Parent, Name, State, Mod, Time, Debug, false). + decode_msg(Msg, Parent, Name, State, Mod, Time, HibernateAfterTimeout, Debug, false). -wake_hib(Parent, Name, State, Mod, Debug) -> +wake_hib(Parent, Name, State, Mod, HibernateAfterTimeout, Debug) -> Msg = receive Input -> Input end, - decode_msg(Msg, Parent, Name, State, Mod, hibernate, Debug, true). + decode_msg(Msg, Parent, Name, State, Mod, hibernate, HibernateAfterTimeout, Debug, true). -decode_msg(Msg, Parent, Name, State, Mod, Time, Debug, Hib) -> +decode_msg(Msg, Parent, Name, State, Mod, Time, HibernateAfterTimeout, Debug, Hib) -> case Msg of {system, From, Req} -> sys:handle_system_msg(Req, From, Parent, ?MODULE, Debug, - [Name, State, Mod, Time], Hib); + [Name, State, Mod, Time, HibernateAfterTimeout], Hib); {'EXIT', Parent, Reason} -> terminate(Reason, Name, undefined, Msg, Mod, State, Debug); _Msg when Debug =:= [] -> - handle_msg(Msg, Parent, Name, State, Mod); + handle_msg(Msg, Parent, Name, State, Mod, HibernateAfterTimeout); _Msg -> Debug1 = sys:handle_debug(Debug, fun print_event/3, Name, {in, Msg}), - handle_msg(Msg, Parent, Name, State, Mod, Debug1) + handle_msg(Msg, Parent, Name, State, Mod, HibernateAfterTimeout, Debug1) end. %%% --------------------------------------------------- @@ -659,65 +668,65 @@ try_terminate(Mod, Reason, State) -> %%% Message handling functions %%% --------------------------------------------------- -handle_msg({'$gen_call', From, Msg}, Parent, Name, State, Mod) -> +handle_msg({'$gen_call', From, Msg}, Parent, Name, State, Mod, HibernateAfterTimeout) -> Result = try_handle_call(Mod, Msg, From, State), case Result of {ok, {reply, Reply, NState}} -> reply(From, Reply), - loop(Parent, Name, NState, Mod, infinity, []); + loop(Parent, Name, NState, Mod, infinity, HibernateAfterTimeout, []); {ok, {reply, Reply, NState, Time1}} -> reply(From, Reply), - loop(Parent, Name, NState, Mod, Time1, []); + loop(Parent, Name, NState, Mod, Time1, HibernateAfterTimeout, []); {ok, {noreply, NState}} -> - loop(Parent, Name, NState, Mod, infinity, []); + loop(Parent, Name, NState, Mod, infinity, HibernateAfterTimeout, []); {ok, {noreply, NState, Time1}} -> - loop(Parent, Name, NState, Mod, Time1, []); + loop(Parent, Name, NState, Mod, Time1, HibernateAfterTimeout, []); {ok, {stop, Reason, Reply, NState}} -> {'EXIT', R} = (catch terminate(Reason, Name, From, Msg, Mod, NState, [])), reply(From, Reply), exit(R); - Other -> handle_common_reply(Other, Parent, Name, From, Msg, Mod, State) + Other -> handle_common_reply(Other, Parent, Name, From, Msg, Mod, HibernateAfterTimeout, State) end; -handle_msg(Msg, Parent, Name, State, Mod) -> +handle_msg(Msg, Parent, Name, State, Mod, HibernateAfterTimeout) -> Reply = try_dispatch(Msg, Mod, State), - handle_common_reply(Reply, Parent, Name, undefined, Msg, Mod, State). + handle_common_reply(Reply, Parent, Name, undefined, Msg, Mod, HibernateAfterTimeout, State). -handle_msg({'$gen_call', From, Msg}, Parent, Name, State, Mod, Debug) -> +handle_msg({'$gen_call', From, Msg}, Parent, Name, State, Mod, HibernateAfterTimeout, Debug) -> Result = try_handle_call(Mod, Msg, From, State), case Result of {ok, {reply, Reply, NState}} -> Debug1 = reply(Name, From, Reply, NState, Debug), - loop(Parent, Name, NState, Mod, infinity, Debug1); + loop(Parent, Name, NState, Mod, infinity, HibernateAfterTimeout, Debug1); {ok, {reply, Reply, NState, Time1}} -> Debug1 = reply(Name, From, Reply, NState, Debug), - loop(Parent, Name, NState, Mod, Time1, Debug1); + loop(Parent, Name, NState, Mod, Time1, HibernateAfterTimeout, Debug1); {ok, {noreply, NState}} -> Debug1 = sys:handle_debug(Debug, fun print_event/3, Name, {noreply, NState}), - loop(Parent, Name, NState, Mod, infinity, Debug1); + loop(Parent, Name, NState, Mod, infinity, HibernateAfterTimeout, Debug1); {ok, {noreply, NState, Time1}} -> Debug1 = sys:handle_debug(Debug, fun print_event/3, Name, {noreply, NState}), - loop(Parent, Name, NState, Mod, Time1, Debug1); + loop(Parent, Name, NState, Mod, Time1, HibernateAfterTimeout, Debug1); {ok, {stop, Reason, Reply, NState}} -> {'EXIT', R} = (catch terminate(Reason, Name, From, Msg, Mod, NState, Debug)), _ = reply(Name, From, Reply, NState, Debug), exit(R); Other -> - handle_common_reply(Other, Parent, Name, From, Msg, Mod, State, Debug) + handle_common_reply(Other, Parent, Name, From, Msg, Mod, HibernateAfterTimeout, State, Debug) end; -handle_msg(Msg, Parent, Name, State, Mod, Debug) -> +handle_msg(Msg, Parent, Name, State, Mod, HibernateAfterTimeout, Debug) -> Reply = try_dispatch(Msg, Mod, State), - handle_common_reply(Reply, Parent, Name, undefined, Msg, Mod, State, Debug). + handle_common_reply(Reply, Parent, Name, undefined, Msg, Mod, HibernateAfterTimeout, State, Debug). -handle_common_reply(Reply, Parent, Name, From, Msg, Mod, State) -> +handle_common_reply(Reply, Parent, Name, From, Msg, Mod, HibernateAfterTimeout, State) -> case Reply of {ok, {noreply, NState}} -> - loop(Parent, Name, NState, Mod, infinity, []); + loop(Parent, Name, NState, Mod, infinity, HibernateAfterTimeout, []); {ok, {noreply, NState, Time1}} -> - loop(Parent, Name, NState, Mod, Time1, []); + loop(Parent, Name, NState, Mod, Time1, HibernateAfterTimeout, []); {ok, {stop, Reason, NState}} -> terminate(Reason, Name, From, Msg, Mod, NState, []); {'EXIT', ExitReason, ReportReason} -> @@ -726,16 +735,16 @@ handle_common_reply(Reply, Parent, Name, From, Msg, Mod, State) -> terminate({bad_return_value, BadReply}, Name, From, Msg, Mod, State, []) end. -handle_common_reply(Reply, Parent, Name, From, Msg, Mod, State, Debug) -> +handle_common_reply(Reply, Parent, Name, From, Msg, Mod, HibernateAfterTimeout, State, Debug) -> case Reply of {ok, {noreply, NState}} -> Debug1 = sys:handle_debug(Debug, fun print_event/3, Name, {noreply, NState}), - loop(Parent, Name, NState, Mod, infinity, Debug1); + loop(Parent, Name, NState, Mod, infinity, HibernateAfterTimeout, Debug1); {ok, {noreply, NState, Time1}} -> Debug1 = sys:handle_debug(Debug, fun print_event/3, Name, {noreply, NState}), - loop(Parent, Name, NState, Mod, Time1, Debug1); + loop(Parent, Name, NState, Mod, Time1, HibernateAfterTimeout, Debug1); {ok, {stop, Reason, NState}} -> terminate(Reason, Name, From, Msg, Mod, NState, Debug); {'EXIT', ExitReason, ReportReason} -> @@ -753,26 +762,26 @@ reply(Name, {To, Tag}, Reply, State, Debug) -> %%----------------------------------------------------------------- %% Callback functions for system messages handling. %%----------------------------------------------------------------- -system_continue(Parent, Debug, [Name, State, Mod, Time]) -> - loop(Parent, Name, State, Mod, Time, Debug). +system_continue(Parent, Debug, [Name, State, Mod, Time, HibernateAfterTimeout]) -> + loop(Parent, Name, State, Mod, Time, HibernateAfterTimeout, Debug). -spec system_terminate(_, _, _, [_]) -> no_return(). -system_terminate(Reason, _Parent, Debug, [Name, State, Mod, _Time]) -> +system_terminate(Reason, _Parent, Debug, [Name, State, Mod, _Time, _HibernateAfterTimeout]) -> terminate(Reason, Name, undefined, [], Mod, State, Debug). -system_code_change([Name, State, Mod, Time], _Module, OldVsn, Extra) -> +system_code_change([Name, State, Mod, Time, HibernateAfterTimeout], _Module, OldVsn, Extra) -> case catch Mod:code_change(OldVsn, State, Extra) of - {ok, NewState} -> {ok, [Name, NewState, Mod, Time]}; + {ok, NewState} -> {ok, [Name, NewState, Mod, Time, HibernateAfterTimeout]}; Else -> Else end. -system_get_state([_Name, State, _Mod, _Time]) -> +system_get_state([_Name, State, _Mod, _Time, _HibernateAfterTimeout]) -> {ok, State}. -system_replace_state(StateFun, [Name, State, Mod, Time]) -> +system_replace_state(StateFun, [Name, State, Mod, Time, HibernateAfterTimeout]) -> NState = StateFun(State), - {ok, NState, [Name, NState, Mod, Time]}. + {ok, NState, [Name, NState, Mod, Time, HibernateAfterTimeout]}. %%----------------------------------------------------------------- %% Format debug messages. Print them as the call-back module sees @@ -802,10 +811,10 @@ print_event(Dev, Event, Name) -> %%% Terminate the server. %%% --------------------------------------------------- + -spec terminate(_, _, _, _, _, _, _) -> no_return(). terminate(Reason, Name, From, Msg, Mod, State, Debug) -> terminate(Reason, Reason, Name, From, Msg, Mod, State, Debug). - -spec terminate(_, _, _, _, _, _, _, _) -> no_return(). terminate(ExitReason, ReportReason, Name, From, Msg, Mod, State, Debug) -> Reply = try_terminate(Mod, ExitReason, State), @@ -850,17 +859,17 @@ error_info(Reason, Name, From, Msg, State, Debug) -> end end; _ -> - Reason - end, + error_logger:limit_term(Reason) + end, {ClientFmt, ClientArgs} = client_stacktrace(From), - format("** Generic server ~p terminating \n" - "** Last message in was ~p~n" - "** When Server state == ~p~n" - "** Reason for termination == ~n** ~p~n" ++ ClientFmt, - [Name, Msg, State, Reason1] ++ ClientArgs), + LimitedState = error_logger:limit_term(State), + error_logger:format("** Generic server ~p terminating \n" + "** Last message in was ~p~n" + "** When Server state == ~p~n" + "** Reason for termination == ~n** ~p~n" ++ ClientFmt, + [Name, Msg, LimitedState, Reason1] ++ ClientArgs), sys:print_log(Debug), ok. - client_stacktrace(undefined) -> {"", []}; client_stacktrace({From, _Tag}) -> @@ -885,7 +894,7 @@ client_stacktrace(From) when is_pid(From) -> %% Status information %%----------------------------------------------------------------- format_status(Opt, StatusData) -> - [PDict, SysState, Parent, Debug, [Name, State, Mod, _Time]] = StatusData, + [PDict, SysState, Parent, Debug, [Name, State, Mod, _Time, _HibernateAfterTimeout]] = StatusData, Header = gen:format_status_header("Status for generic server", Name), Log = sys:get_debug(log, Debug, []), Specfic = case format_status(Opt, Mod, PDict, State) of diff --git a/lib/stdlib/src/gen_statem.erl b/lib/stdlib/src/gen_statem.erl index 6f566b8beb..b5e9da1e66 100644 --- a/lib/stdlib/src/gen_statem.erl +++ b/lib/stdlib/src/gen_statem.erl @@ -369,9 +369,12 @@ event_type(Type) -> Dbgs :: ['trace' | 'log' | 'statistics' | 'debug' | {'logfile', string()}]}. +-type hibernate_after_opt() :: + {'hibernate_after', HibernateAfterTimeout :: timeout()}. -type start_opt() :: debug_opt() | {'timeout', Time :: timeout()} + | hibernate_after_opt() | {'spawn_opt', [proc_lib:spawn_option()]}. -type start_ret() :: {'ok', pid()} | 'ignore' | {'error', term()}. @@ -544,14 +547,14 @@ reply({To,Tag}, Reply) when is_pid(To) -> %% started by proc_lib into a state machine using %% the same arguments as you would have returned from init/1 -spec enter_loop( - Module :: module(), Opts :: [debug_opt()], + Module :: module(), Opts :: [debug_opt() | hibernate_after_opt()], State :: state(), Data :: data()) -> no_return(). enter_loop(Module, Opts, State, Data) -> enter_loop(Module, Opts, State, Data, self()). %% -spec enter_loop( - Module :: module(), Opts :: [debug_opt()], + Module :: module(), Opts :: [debug_opt() | hibernate_after_opt()], State :: state(), Data :: data(), Server_or_Actions :: server_name() | pid() | [action()]) -> @@ -565,7 +568,7 @@ enter_loop(Module, Opts, State, Data, Server_or_Actions) -> end. %% -spec enter_loop( - Module :: module(), Opts :: [debug_opt()], + Module :: module(), Opts :: [debug_opt() | hibernate_after_opt()], State :: state(), Data :: data(), Server :: server_name() | pid(), Actions :: [action()] | action()) -> @@ -605,7 +608,8 @@ enter(Module, Opts, State, Data, Server, Actions, Parent) -> %% The values should already have been type checked Name = gen:get_proc_name(Server), Debug = gen:debug_options(Name, Opts), - Events = [], + HibernateAfterTimeout = gen:hibernate_after(Opts), + Events = [], P = [], Event = {internal,init_state}, %% We enforce {postpone,false} to ensure that @@ -648,6 +652,7 @@ enter(Module, Opts, State, Data, Server, Actions, Parent) -> timer_refs => TimerRefs, timer_types => TimerTypes, hibernate => Hibernate, + hibernate_after => HibernateAfterTimeout, cancel_timers => CancelTimers }, NewDebug = sys_debug(Debug, S, State, {enter,Event,State}), @@ -854,7 +859,7 @@ loop_hibernate(Parent, Debug, S) -> {wakeup_from_hibernate,3}}). %% Entry point for wakeup_from_hibernate/3 -loop_receive(Parent, Debug, S) -> +loop_receive(Parent, Debug, #{hibernate_after := HibernateAfterTimeout} = S) -> receive Msg -> case Msg of @@ -956,6 +961,9 @@ loop_receive(Parent, Debug, S) -> loop_receive_result( Parent, Debug, S, Hibernate, Event) end + after + HibernateAfterTimeout -> + loop_hibernate(Parent, Debug, S) end. loop_receive_result( @@ -1714,6 +1722,8 @@ error_info( end; _ -> {Reason,Stacktrace} end, + [LimitedP, LimitedFmtData, LimitedFixedReason] = + [error_logger:limit_term(D) || D <- [P, FmtData, FixedReason]], CBMode = case StateEnter of true -> @@ -1747,8 +1757,8 @@ error_info( [] -> []; [Event|_] -> [Event] end] ++ - [FmtData, - Class,FixedReason, + [LimitedFmtData, + Class,LimitedFixedReason, CBMode] ++ case Q of [_|[_|_] = Events] -> [Events]; @@ -1756,7 +1766,7 @@ error_info( end ++ case P of [] -> []; - _ -> [P] + _ -> [LimitedP] end ++ case FixedStacktrace of [] -> []; diff --git a/lib/stdlib/src/io_lib.erl b/lib/stdlib/src/io_lib.erl index 5ed2f4d888..9d447418f8 100644 --- a/lib/stdlib/src/io_lib.erl +++ b/lib/stdlib/src/io_lib.erl @@ -84,6 +84,8 @@ -export([write_unicode_string/1, write_unicode_char/1, deep_unicode_char_list/1]). +-export([limit_term/2]). + -export_type([chars/0, latin1_string/0, continuation/0, fread_error/0, fread_item/0, format_spec/0]). @@ -911,3 +913,116 @@ binrev(L) -> binrev(L, T) -> list_to_binary(lists:reverse(L, T)). + +-spec limit_term(term(), non_neg_integer()) -> term(). + +%% The intention is to mimic the depth limitation of io_lib:write() +%% and io_lib_pretty:print(). The leaves ('...') should never be +%% seen when printed with the same depth. Bitstrings are never +%% truncated, which is OK as long as they are not sent to other nodes. +limit_term(Term, Depth) -> + try test_limit(Term, Depth) of + ok -> Term + catch + throw:limit -> + limit(Term, Depth) + end. + +limit(_, 0) -> '...'; +limit([H|T]=L, D) -> + if + D =:= 1 -> '...'; + true -> + case printable_list(L) of + true -> L; + false -> + [limit(H, D-1)|limit_tail(T, D-1)] + end + end; +limit(Term, D) when is_map(Term) -> + limit_map(Term, D); +limit({}=T, _D) -> T; +limit(T, D) when is_tuple(T) -> + if + D =:= 1 -> '...'; + true -> + list_to_tuple([limit(element(1, T), D-1)| + limit_tail(tl(tuple_to_list(T)), D-1)]) + end; +limit(<<_/bitstring>>=Term, D) -> limit_bitstring(Term, D); +limit(Term, _D) -> Term. + +limit_tail([], _D) -> []; +limit_tail(_, 1) -> ['...']; +limit_tail([H|T], D) -> + [limit(H, D-1)|limit_tail(T, D-1)]; +limit_tail(Other, D) -> + limit(Other, D-1). + +%% Cannot limit maps properly since there is no guarantee that +%% maps:from_list() creates a map with the same internal ordering of +%% the selected associations as in Map. +limit_map(Map, D) -> + maps:from_list(erts_internal:maps_to_list(Map, D)). +%% maps:from_list(limit_map_body(erts_internal:maps_to_list(Map, D), D)). + +%% limit_map_body(_, 0) -> [{'...', '...'}]; +%% limit_map_body([], _) -> []; +%% limit_map_body([{K,V}], D) -> [limit_map_assoc(K, V, D)]; +%% limit_map_body([{K,V}|KVs], D) -> +%% [limit_map_assoc(K, V, D) | limit_map_body(KVs, D-1)]. + +%% limit_map_assoc(K, V, D) -> +%% {limit(K, D-1), limit(V, D-1)}. + +limit_bitstring(B, _D) -> B. %% Keeps all printable binaries. + +test_limit(_, 0) -> throw(limit); +test_limit([H|T]=L, D) when is_integer(D) -> + if + D =:= 1 -> throw(limit); + true -> + case printable_list(L) of + true -> ok; + false -> + test_limit(H, D-1), + test_limit_tail(T, D-1) + end + end; +test_limit(Term, D) when is_map(Term) -> + test_limit_map(Term, D); +test_limit({}, _D) -> ok; +test_limit(T, D) when is_tuple(T) -> + test_limit_tuple(T, 1, tuple_size(T), D); +test_limit(<<_/bitstring>>=Term, D) -> test_limit_bitstring(Term, D); +test_limit(_Term, _D) -> ok. + +test_limit_tail([], _D) -> ok; +test_limit_tail(_, 1) -> throw(limit); +test_limit_tail([H|T], D) -> + test_limit(H, D-1), + test_limit_tail(T, D-1); +test_limit_tail(Other, D) -> + test_limit(Other, D-1). + +test_limit_tuple(_T, I, Sz, _D) when I > Sz -> ok; +test_limit_tuple(_, _, _, 1) -> throw(limit); +test_limit_tuple(T, I, Sz, D) -> + test_limit(element(I, T), D-1), + test_limit_tuple(T, I+1, Sz, D-1). + +test_limit_map(_Map, _D) -> ok. +%% test_limit_map_body(erts_internal:maps_to_list(Map, D), D). + +%% test_limit_map_body(_, 0) -> throw(limit); +%% test_limit_map_body([], _) -> ok; +%% test_limit_map_body([{K,V}], D) -> test_limit_map_assoc(K, V, D); +%% test_limit_map_body([{K,V}|KVs], D) -> +%% test_limit_map_assoc(K, V, D), +%% test_limit_map_body(KVs, D-1). + +%% test_limit_map_assoc(K, V, D) -> +%% test_limit(K, D-1), +%% test_limit(V, D-1). + +test_limit_bitstring(_, _) -> ok. diff --git a/lib/stdlib/src/proc_lib.erl b/lib/stdlib/src/proc_lib.erl index 363705b0f4..3fa54cd0d5 100644 --- a/lib/stdlib/src/proc_lib.erl +++ b/lib/stdlib/src/proc_lib.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2016. All Rights Reserved. +%% Copyright Ericsson AB 1996-2017. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -264,12 +264,12 @@ exit_p(Class, Reason, Stacktrace) -> case get('$initial_call') of {M,F,A} when is_atom(M), is_atom(F), is_integer(A) -> MFA = {M,F,make_dummy_args(A, [])}, - crash_report(Class, Reason, MFA), + crash_report(Class, Reason, MFA, Stacktrace), erlang:raise(exit, exit_reason(Class, Reason, Stacktrace), Stacktrace); _ -> %% The process dictionary has been cleared or %% possibly modified. - crash_report(Class, Reason, []), + crash_report(Class, Reason, [], Stacktrace), erlang:raise(exit, exit_reason(Class, Reason, Stacktrace), Stacktrace) end. @@ -499,26 +499,28 @@ trans_init(M, F, A) when is_atom(M), is_atom(F) -> %% Generate a crash report. %% ----------------------------------------------------- -crash_report(exit, normal, _) -> ok; -crash_report(exit, shutdown, _) -> ok; -crash_report(exit, {shutdown,_}, _) -> ok; -crash_report(Class, Reason, StartF) -> - OwnReport = my_info(Class, Reason, StartF), +crash_report(exit, normal, _, _) -> ok; +crash_report(exit, shutdown, _, _) -> ok; +crash_report(exit, {shutdown,_}, _, _) -> ok; +crash_report(Class, Reason, StartF, Stacktrace) -> + OwnReport = my_info(Class, Reason, StartF, Stacktrace), LinkReport = linked_info(self()), Rep = [OwnReport,LinkReport], error_logger:error_report(crash_report, Rep). -my_info(Class, Reason, []) -> - my_info_1(Class, Reason); -my_info(Class, Reason, StartF) -> - [{initial_call, StartF}|my_info_1(Class, Reason)]. +my_info(Class, Reason, [], Stacktrace) -> + my_info_1(Class, Reason, Stacktrace); +my_info(Class, Reason, StartF, Stacktrace) -> + [{initial_call, StartF}| + my_info_1(Class, Reason, Stacktrace)]. -my_info_1(Class, Reason) -> +my_info_1(Class, Reason, Stacktrace) -> [{pid, self()}, get_process_info(self(), registered_name), - {error_info, {Class,Reason,erlang:get_stacktrace()}}, + {error_info, {Class,Reason,Stacktrace}}, get_ancestors(self()), - get_process_info(self(), messages), + get_process_info(self(), message_queue_len), + get_messages(self()), get_process_info(self(), links), get_cleaned_dictionary(self()), get_process_info(self(), trap_exit), @@ -538,12 +540,49 @@ get_ancestors(Pid) -> {ancestors,[]} end. +%% The messages and the dictionary are possibly limited too much if +%% some error handles output the messages or the dictionary using ~P +%% or ~W with depth greater than the depth used here (the depth of +%% control characters P and W takes precedence over the depth set by +%% application variable error_logger_format_depth). However, it is +%% assumed that all report handlers call proc_lib:format(). +get_messages(Pid) -> + Messages = get_process_messages(Pid), + {messages, error_logger:limit_term(Messages)}. + +get_process_messages(Pid) -> + Depth = error_logger:get_format_depth(), + case Pid =/= self() orelse Depth =:= unlimited of + true -> + {messages, Messages} = get_process_info(Pid, messages), + Messages; + false -> + %% If there are more messages than Depth, garbage + %% collection can sometimes be avoided by collecting just + %% enough messages for the crash report. It is assumed the + %% process is about to die anyway. + receive_messages(Depth) + end. + +receive_messages(0) -> []; +receive_messages(N) -> + receive + M -> + [M|receive_messages(N - 1)] + after 0 -> + [] + end. + get_cleaned_dictionary(Pid) -> case get_process_info(Pid,dictionary) of - {dictionary,Dict} -> {dictionary,clean_dict(Dict)}; + {dictionary,Dict} -> {dictionary,cleaned_dict(Dict)}; _ -> {dictionary,[]} end. +cleaned_dict(Dict) -> + CleanDict = clean_dict(Dict), + error_logger:limit_term(CleanDict). + clean_dict([{'$ancestors',_}|Dict]) -> clean_dict(Dict); clean_dict([{'$initial_call',_}|Dict]) -> @@ -581,20 +620,24 @@ make_neighbour_reports1([P|Ps]) -> make_neighbour_reports1([]) -> []. +%% Do not include messages or process dictionary, even if +%% error_logger_format_depth is unlimited. make_neighbour_report(Pid) -> [{pid, Pid}, get_process_info(Pid, registered_name), get_initial_call(Pid), get_process_info(Pid, current_function), get_ancestors(Pid), - get_process_info(Pid, messages), + get_process_info(Pid, message_queue_len), + %% get_messages(Pid), get_process_info(Pid, links), - get_cleaned_dictionary(Pid), + %% get_cleaned_dictionary(Pid), get_process_info(Pid, trap_exit), get_process_info(Pid, status), get_process_info(Pid, heap_size), get_process_info(Pid, stack_size), - get_process_info(Pid, reductions) + get_process_info(Pid, reductions), + get_process_info(Pid, current_stacktrace) ]. get_initial_call(Pid) -> @@ -721,24 +764,37 @@ format(CrashReport, Encoding) -> format([OwnReport,LinkReport], Encoding, Depth) -> Extra = {Encoding,Depth}, - OwnFormat = format_report(OwnReport, Extra), - LinkFormat = format_report(LinkReport, Extra), + MyIndent = " ", + OwnFormat = format_report(OwnReport, MyIndent, Extra), + LinkFormat = format_link_report(LinkReport, MyIndent, Extra), Str = io_lib:format(" crasher:~n~ts neighbours:~n~ts", [OwnFormat, LinkFormat]), lists:flatten(Str). -format_report(Rep, Extra) when is_list(Rep) -> - format_rep(Rep, Extra); -format_report(Rep, {Enc,_}) -> - io_lib:format("~"++modifier(Enc)++"p~n", [Rep]). - -format_rep([{initial_call,InitialCall}|Rep], {_Enc,Depth}=Extra) -> - [format_mfa(InitialCall, Depth)|format_rep(Rep, Extra)]; -format_rep([{error_info,{Class,Reason,StackTrace}}|Rep], Extra) -> - [format_exception(Class, Reason, StackTrace, Extra)|format_rep(Rep, Extra)]; -format_rep([{Tag,Data}|Rep], Extra) -> - [format_tag(Tag, Data, Extra)|format_rep(Rep, Extra)]; -format_rep(_, _Extra) -> +format_link_report([Link|Reps], Indent, Extra) -> + Rep = case Link of + {neighbour,Rep0} -> Rep0; + _ -> Link + end, + LinkIndent = [" ",Indent], + [Indent,"neighbour:\n",format_report(Rep, LinkIndent, Extra)| + format_link_report(Reps, Indent, Extra)]; +format_link_report([], _, _) -> + []. + +format_report(Rep, Indent, Extra) when is_list(Rep) -> + format_rep(Rep, Indent, Extra); +format_report(Rep, Indent, {Enc,Depth}) -> + io_lib:format("~s~"++modifier(Enc)++"P~n", [Indent, Rep, Depth]). + +format_rep([{initial_call,InitialCall}|Rep], Indent, Extra) -> + [format_mfa(Indent, InitialCall, Extra)|format_rep(Rep, Indent, Extra)]; +format_rep([{error_info,{Class,Reason,StackTrace}}|Rep], Indent, Extra) -> + [format_exception(Class, Reason, StackTrace, Extra)| + format_rep(Rep, Indent, Extra)]; +format_rep([{Tag,Data}|Rep], Indent, Extra) -> + [format_tag(Indent, Tag, Data, Extra)|format_rep(Rep, Indent, Extra)]; +format_rep(_, _, _Extra) -> []. format_exception(Class, Reason, StackTrace, {Enc,_}=Extra) -> @@ -749,14 +805,14 @@ format_exception(Class, Reason, StackTrace, {Enc,_}=Extra) -> [EI, lib:format_exception(1+length(EI), Class, Reason, StackTrace, StackFun, PF, Enc), "\n"]. -format_mfa({M,F,Args}=StartF, Depth) -> +format_mfa(Indent, {M,F,Args}=StartF, Extra) -> try A = length(Args), - [" initial call: ",atom_to_list(M),$:,atom_to_list(F),$/, + [Indent,"initial call: ",atom_to_list(M),$:,atom_to_list(F),$/, integer_to_list(A),"\n"] catch error:_ -> - format_tag(initial_call, StartF, Depth) + format_tag(Indent, initial_call, StartF, Extra) end. pp_fun({Enc,Depth}) -> @@ -769,12 +825,12 @@ pp_fun({Enc,Depth}) -> io_lib:format("~." ++ integer_to_list(I) ++ P, [Term|Tl]) end. -format_tag(Tag, Data, {_Enc,Depth}) -> +format_tag(Indent, Tag, Data, {_Enc,Depth}) -> case Depth of unlimited -> - io_lib:format(" ~p: ~80.18p~n", [Tag, Data]); + io_lib:format("~s~p: ~80.18p~n", [Indent, Tag, Data]); _ -> - io_lib:format(" ~p: ~80.18P~n", [Tag, Data, Depth]) + io_lib:format("~s~p: ~80.18P~n", [Indent, Tag, Data, Depth]) end. modifier(latin1) -> ""; diff --git a/lib/stdlib/src/qlc.erl b/lib/stdlib/src/qlc.erl index 20aaa2638c..535ca57a6b 100644 --- a/lib/stdlib/src/qlc.erl +++ b/lib/stdlib/src/qlc.erl @@ -1392,8 +1392,10 @@ next_loop(Pid, L, N) when N =/= 0 -> {caught, throw, Error, [?THROWN_ERROR | _]} -> Error; {caught, Class, Reason, Stacktrace} -> - _ = (catch erlang:error(foo)), - erlang:raise(Class, Reason, Stacktrace ++ erlang:get_stacktrace()); + CurrentStacktrace = try erlang:error(foo) + catch error:_ -> erlang:get_stacktrace() + end, + erlang:raise(Class, Reason, Stacktrace ++ CurrentStacktrace); error -> erlang:error({qlc_cursor_pid_no_longer_exists, Pid}) end; diff --git a/lib/stdlib/test/dummy_h.erl b/lib/stdlib/test/dummy_h.erl index 45ccfee240..7a9eb11f98 100644 --- a/lib/stdlib/test/dummy_h.erl +++ b/lib/stdlib/test/dummy_h.erl @@ -43,6 +43,9 @@ handle_event(hibernate, _State) -> {ok,[],hibernate}; handle_event(wakeup, _State) -> {ok,[]}; +handle_event({From, handle_event}, _State) -> + From ! handled_event, + {ok,[]}; handle_event(Event, Parent) -> Parent ! {dummy_h, Event}, {ok, Parent}. @@ -75,6 +78,9 @@ handle_info(wake, _State) -> {ok, []}; handle_info(gnurf, _State) -> {ok, []}; +handle_info({From, handle_info}, _State) -> + From ! handled_info, + {ok, []}; handle_info(Info, Parent) -> Parent ! {dummy_h, Info}, {ok, Parent}. diff --git a/lib/stdlib/test/erl_lint_SUITE.erl b/lib/stdlib/test/erl_lint_SUITE.erl index 03cad2c093..02524679fa 100644 --- a/lib/stdlib/test/erl_lint_SUITE.erl +++ b/lib/stdlib/test/erl_lint_SUITE.erl @@ -65,7 +65,8 @@ maps/1,maps_type/1,maps_parallel_match/1, otp_11851/1,otp_11879/1,otp_13230/1, record_errors/1, otp_11879_cont/1, - non_latin1_module/1, otp_14323/1]). + non_latin1_module/1, otp_14323/1, + get_stacktrace/1]). suite() -> [{ct_hooks,[ts_install_cth]}, @@ -85,7 +86,8 @@ all() -> too_many_arguments, basic_errors, bin_syntax_errors, predef, maps, maps_type, maps_parallel_match, otp_11851, otp_11879, otp_13230, - record_errors, otp_11879_cont, non_latin1_module, otp_14323]. + record_errors, otp_11879_cont, non_latin1_module, otp_14323, + get_stacktrace]. groups() -> [{unused_vars_warn, [], @@ -3980,6 +3982,63 @@ otp_14323(Config) -> [] = run(Config, Ts), ok. +get_stacktrace(Config) -> + Ts = [{old_catch, + <<"t1() -> + catch error(foo), + erlang:get_stacktrace(). + ">>, + [], + {warnings,[{3,erl_lint,{get_stacktrace,after_old_catch}}]}}, + {nowarn_get_stacktrace, + <<"t1() -> + catch error(foo), + erlang:get_stacktrace(). + ">>, + [nowarn_get_stacktrace], + []}, + {try_catch, + <<"t1(X) -> + try abs(X) of + _ -> + erlang:get_stacktrace() + catch + _:_ -> ok + end. + + t2() -> + try error(foo) + catch _:_ -> ok + end, + erlang:get_stacktrace(). + + t3() -> + try error(foo) + catch _:_ -> + try error(bar) + catch _:_ -> + ok + end, + erlang:get_stacktrace() + end. + + no_warning(X) -> + try + abs(X) + catch + _:_ -> + erlang:get_stacktrace() + end. + ">>, + [], + {warnings,[{4,erl_lint,{get_stacktrace,wrong_part_of_try}}, + {13,erl_lint,{get_stacktrace,after_try}}, + {22,erl_lint,{get_stacktrace,after_try}}]}}], + + run(Config, Ts), + ok. + + run(Config, Tests) -> F = fun({N,P,Ws,E}, BadL) -> case catch run_test(Config, P, Ws) of diff --git a/lib/stdlib/test/ets_SUITE.erl b/lib/stdlib/test/ets_SUITE.erl index 4c2f6cfe9e..05451a83fb 100644 --- a/lib/stdlib/test/ets_SUITE.erl +++ b/lib/stdlib/test/ets_SUITE.erl @@ -1349,6 +1349,26 @@ t_select_replace(Config) when is_list(Config) -> NeqPairs(X,Y)) || X <- Terms, Y <- Terms], + %% Wrap entire tuple with 'const' + [[begin + Old = {Key, 1, 2}, + ets:insert(T2, Old), + 1 = ets:select_replace(T2, [{Old, [], [{const, New}]}]), + [New] = ets:lookup(T2, Key), + ets:delete(T2, Key) + end || New <- [{Key, 1, 2}, {Key, 3, 4}, {Key, 1}, {Key, 1, 2, 3}, {Key}] + ] + || Key <- [{1, tuple}, {nested, {tuple, {a,b}}} | Terms]], + + %% 'const' wrap does not work with maps or variables in keys + [[begin + Old = {Key, 1, 2}, + {'EXIT',{badarg,_}} = (catch ets:select_replace(T2, [{Old, [], [{const, New}]}])) + end || New <- [{Key, 1, 2}, {Key, 3, 4}, {Key, 1}, {Key, 1, 2, 3}, {Key}] + ] + || Key <- [#{a => 1}, {nested, #{a => 1}}, '$1']], + + ets:delete(T2), verify_etsmem(EtsMem). diff --git a/lib/stdlib/test/gen_event_SUITE.erl b/lib/stdlib/test/gen_event_SUITE.erl index 749920f843..880b10117c 100644 --- a/lib/stdlib/test/gen_event_SUITE.erl +++ b/lib/stdlib/test/gen_event_SUITE.erl @@ -26,7 +26,7 @@ end_per_testcase/2]). -export([start/1, add_handler/1, add_sup_handler/1, delete_handler/1, swap_handler/1, swap_sup_handler/1, - notify/1, sync_notify/1, call/1, info/1, hibernate/1, + notify/1, sync_notify/1, call/1, info/1, hibernate/1, auto_hibernate/1, call_format_status/1, call_format_status_anon/1, error_format_status/1, get_state/1, replace_state/1, start_opt/1, @@ -37,7 +37,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> - [start, {group, test_all}, hibernate, + [start, {group, test_all}, hibernate, auto_hibernate, call_format_status, call_format_status_anon, error_format_status, get_state, replace_state, start_opt, {group, undef_callbacks}, undef_in_terminate]. @@ -306,6 +306,48 @@ hibernate(Config) when is_list(Config) -> ok. +auto_hibernate(Config) when is_list(Config) -> + HibernateAfterTimeout = 100, + State = {auto_hibernate_state}, + {ok,Pid} = gen_event:start({local, auto_hibernate_handler}, [{hibernate_after, HibernateAfterTimeout}]), + %% After init test + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + ok = gen_event:add_handler(auto_hibernate_handler, dummy_h, [State]), + %% Get state test + [{dummy_h,false,State}] = sys:get_state(Pid), + is_in_erlang_hibernate(Pid), + %% Call test + {ok, hejhopp} = gen_event:call(auto_hibernate_handler, dummy_h, hejsan), + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Event test + ok = gen_event:notify(auto_hibernate_handler, {self(), handle_event}), + receive + handled_event -> + ok + after 1000 -> + ct:fail(event) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Info test + Pid ! {self(), handle_info}, + receive + handled_info -> + ok + after 1000 -> + ct:fail(info) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + ok = gen_event:stop(auto_hibernate_handler), + ok. + is_in_erlang_hibernate(Pid) -> receive after 1 -> ok end, is_in_erlang_hibernate_1(200, Pid). diff --git a/lib/stdlib/test/gen_fsm_SUITE.erl b/lib/stdlib/test/gen_fsm_SUITE.erl index 7f98526d35..86cf58566b 100644 --- a/lib/stdlib/test/gen_fsm_SUITE.erl +++ b/lib/stdlib/test/gen_fsm_SUITE.erl @@ -44,7 +44,7 @@ -export([undef_in_handle_info/1, undef_in_terminate/1]). --export([hibernate/1,hiber_idle/3,hiber_wakeup/3,hiber_idle/2,hiber_wakeup/2]). +-export([hibernate/1,auto_hibernate/1,hiber_idle/3,hiber_wakeup/3,hiber_idle/2,hiber_wakeup/2]). -export([enter_loop/1]). @@ -68,7 +68,7 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [{group, start}, {group, abnormal}, shutdown, - {group, sys}, hibernate, enter_loop, {group, undef_callbacks}, + {group, sys}, hibernate, auto_hibernate, enter_loop, {group, undef_callbacks}, undef_in_handle_info, undef_in_terminate]. groups() -> @@ -700,6 +700,43 @@ hibernate(Config) when is_list(Config) -> process_flag(trap_exit, OldFl), ok. +%% Auto hibernation +auto_hibernate(Config) when is_list(Config) -> + OldFl = process_flag(trap_exit, true), + HibernateAfterTimeout = 100, + State = {auto_hibernate_state}, + {ok, Pid} = gen_fsm:start_link({local, my_test_name_auto_hibernate}, ?MODULE, {state_data, State}, [{hibernate_after, HibernateAfterTimeout}]), + %% After init test + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Get state test + {_, State} = sys:get_state(my_test_name_auto_hibernate), + is_in_erlang_hibernate(Pid), + %% Sync send event test + 'alive!' = gen_fsm:sync_send_event(Pid,'alive?'), + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Send event test + ok = gen_fsm:send_all_state_event(Pid,{'alive?', self()}), + wfor(yes), + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Info test + Pid ! {self(), handle_info}, + wfor({Pid, handled_info}), + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + stop_it(Pid), + receive + {'EXIT',Pid,normal} -> ok + end, + process_flag(trap_exit, OldFl), + ok. + is_in_erlang_hibernate(Pid) -> receive after 1 -> ok end, is_in_erlang_hibernate_1(200, Pid). @@ -1151,6 +1188,8 @@ idle(badreturn, _From, _Data) -> idle({timeout,Time}, From, _Data) -> gen_fsm:send_event_after(Time, {timeout,Time}), {next_state, timeout, From}; +idle('alive?', _From, Data) -> + {reply, 'alive!', idle, Data}; idle(_, _From, Data) -> {reply, 'eh?', idle, Data}. @@ -1226,6 +1265,9 @@ handle_info(hibernate_later, _SName, _State) -> handle_info({call_undef_fun, {Mod, Fun}}, State, Data) -> Mod:Fun(), {next_state, State, Data}; +handle_info({From, handle_info}, SName, State) -> + From ! {self(), handled_info}, + {next_state, SName, State}; handle_info(Info, _State, Data) -> {stop, {unexpected,Info}, Data}. diff --git a/lib/stdlib/test/gen_server_SUITE.erl b/lib/stdlib/test/gen_server_SUITE.erl index 6f72034b09..7e3c71715e 100644 --- a/lib/stdlib/test/gen_server_SUITE.erl +++ b/lib/stdlib/test/gen_server_SUITE.erl @@ -32,7 +32,7 @@ call_remote_n1/1, call_remote_n2/1, call_remote_n3/1, spec_init/1, spec_init_local_registered_parent/1, spec_init_global_registered_parent/1, - otp_5854/1, hibernate/1, otp_7669/1, call_format_status/1, + otp_5854/1, hibernate/1, auto_hibernate/1, otp_7669/1, call_format_status/1, error_format_status/1, terminate_crash_format/1, get_state/1, replace_state/1, call_with_huge_message_queue/1, undef_handle_call/1, undef_handle_cast/1, undef_handle_info/1, @@ -65,7 +65,7 @@ all() -> call_remote3, call_remote_n1, call_remote_n2, call_remote_n3, spec_init, spec_init_local_registered_parent, - spec_init_global_registered_parent, otp_5854, hibernate, + spec_init_global_registered_parent, otp_5854, hibernate, auto_hibernate, otp_7669, call_format_status, error_format_status, terminate_crash_format, get_state, replace_state, @@ -730,6 +730,58 @@ hibernate(Config) when is_list(Config) -> process_flag(trap_exit, OldFl), ok. +auto_hibernate(Config) when is_list(Config) -> + OldFl = process_flag(trap_exit, true), + HibernateAfterTimeout = 100, + State = {auto_hibernate_state}, + {ok, Pid} = + gen_server:start_link({local, my_test_name_auto_hibernate}, + gen_server_SUITE, {state,State}, [{hibernate_after, HibernateAfterTimeout}]), + %% After init test + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Get state test + State = sys:get_state(my_test_name_auto_hibernate), + is_in_erlang_hibernate(Pid), + %% Call test + ok = gen_server:call(my_test_name_auto_hibernate, started_p), + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Cast test + ok = gen_server:cast(my_test_name_auto_hibernate, {self(),handle_cast}), + receive + {Pid, handled_cast} -> + ok + after 1000 -> + ct:fail(cast) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Info test + Pid ! {self(),handle_info}, + receive + {Pid, handled_info} -> + ok + after 1000 -> + ct:fail(info) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + + ok = gen_server:call(my_test_name_auto_hibernate, stop), + receive + {'EXIT', Pid, stopped} -> + ok + after 5000 -> + ct:fail(gen_server_did_not_die) + end, + process_flag(trap_exit, OldFl), + ok. + is_in_erlang_hibernate(Pid) -> receive after 1 -> ok end, is_in_erlang_hibernate_1(200, Pid). @@ -747,6 +799,23 @@ is_in_erlang_hibernate_1(N, Pid) -> is_in_erlang_hibernate_1(N-1, Pid) end. +is_not_in_erlang_hibernate(Pid) -> + receive after 1 -> ok end, + is_not_in_erlang_hibernate_1(200, Pid). + +is_not_in_erlang_hibernate_1(0, Pid) -> + io:format("~p\n", [erlang:process_info(Pid, current_function)]), + ct:fail(not_in_erlang_hibernate_3); +is_not_in_erlang_hibernate_1(N, Pid) -> + {current_function,MFA} = erlang:process_info(Pid, current_function), + case MFA of + {erlang,hibernate,3} -> + receive after 10 -> ok end, + is_not_in_erlang_hibernate_1(N-1, Pid); + _ -> + ok + end. + %% -------------------------------------- %% Test gen_server:abcast and handle_cast. %% Test all different return values from diff --git a/lib/stdlib/test/gen_statem_SUITE.erl b/lib/stdlib/test/gen_statem_SUITE.erl index 05934b3953..5b9daecfd3 100644 --- a/lib/stdlib/test/gen_statem_SUITE.erl +++ b/lib/stdlib/test/gen_statem_SUITE.erl @@ -40,7 +40,7 @@ all() -> shutdown, stop_and_reply, state_enter, event_order, state_timeout, event_types, generic_timers, code_change, {group, sys}, - hibernate, enter_loop, {group, undef_callbacks}, + hibernate, auto_hibernate, enter_loop, {group, undef_callbacks}, undef_in_terminate]. groups() -> @@ -1284,6 +1284,84 @@ hibernate(Config) -> end, ok = verify_empty_msgq(). +%% Auto-hibernation timeout +auto_hibernate(Config) -> + OldFl = process_flag(trap_exit, true), + HibernateAfterTimeout = 100, + + {ok,Pid} = + gen_statem:start_link( + ?MODULE, start_arg(Config, []), [{hibernate_after, HibernateAfterTimeout}]), + %% After init test + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% After info test + Pid ! {hping, self()}, + receive + {Pid, hpong} -> + ok + after 1000 -> + ct:fail(info) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% After cast test + ok = gen_statem:cast(Pid, {hping, self()}), + receive + {Pid, hpong} -> + ok + after 1000 -> + ct:fail(cast) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% After call test + hpong = gen_statem:call(Pid, hping), + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Timer test 1 + TimerTimeout1 = 50, + ok = gen_statem:call(Pid, {arm_htimer, self(), TimerTimeout1}), + is_not_in_erlang_hibernate(Pid), + timer:sleep(TimerTimeout1), + is_not_in_erlang_hibernate(Pid), + receive + {Pid, htimer_armed} -> + ok + after 1000 -> + ct:fail(timer1) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + %% Timer test 2 + TimerTimeout2 = 150, + ok = gen_statem:call(Pid, {arm_htimer, self(), TimerTimeout2}), + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + receive + {Pid, htimer_armed} -> + ok + after 1000 -> + ct:fail(timer2) + end, + is_not_in_erlang_hibernate(Pid), + timer:sleep(HibernateAfterTimeout), + is_in_erlang_hibernate(Pid), + stop_it(Pid), + process_flag(trap_exit, OldFl), + receive + {'EXIT',Pid,normal} -> ok + after 5000 -> + ct:fail(gen_statem_did_not_die) + end, + ok = verify_empty_msgq(). + is_in_erlang_hibernate(Pid) -> receive after 1 -> ok end, is_in_erlang_hibernate_1(200, Pid). @@ -1704,6 +1782,19 @@ terminate(_Reason, _State, _Data) -> %% State functions +idle(info, {hping,Pid}, _Data) -> + Pid ! {self(), hpong}, + keep_state_and_data; +idle(cast, {hping,Pid}, Data) -> + Pid ! {self(), hpong}, + {keep_state, Data}; +idle({call, From}, hping, _Data) -> + {keep_state_and_data, [{reply, From, hpong}]}; +idle({call, From}, {arm_htimer, Pid, Timeout}, _Data) -> + {keep_state_and_data, [{reply, From, ok}, {timeout, Timeout, {arm_htimer, Pid}}]}; +idle(timeout, {arm_htimer, Pid}, _Data) -> + Pid ! {self(), htimer_armed}, + keep_state_and_data; idle(cast, {connect,Pid}, Data) -> Pid ! accept, {next_state,wfor_conf,Data,infinity}; % NoOp timeout just to test API diff --git a/lib/stdlib/test/io_SUITE.erl b/lib/stdlib/test/io_SUITE.erl index ef3f0be5d7..e2c73371cd 100644 --- a/lib/stdlib/test/io_SUITE.erl +++ b/lib/stdlib/test/io_SUITE.erl @@ -31,7 +31,7 @@ otp_10836/1, io_lib_width_too_small/1, io_with_huge_message_queue/1, format_string/1, maps/1, coverage/1, otp_14178_unicode_atoms/1, otp_14175/1, - otp_14285/1]). + otp_14285/1, limit_term/1]). -export([pretty/2]). @@ -63,7 +63,7 @@ all() -> io_lib_print_binary_depth_one, otp_10302, otp_10755, otp_10836, io_lib_width_too_small, io_with_huge_message_queue, format_string, maps, coverage, otp_14178_unicode_atoms, otp_14175, - otp_14285]. + otp_14285, limit_term]. %% Error cases for output. error_1(Config) when is_list(Config) -> @@ -2373,3 +2373,58 @@ otp_14285(_Config) -> latin1_fmt(Fmt, Args) -> L = fmt(Fmt, Args), true = lists:all(fun is_latin1/1, L). + +limit_term(_Config) -> + {_, 2} = limt([a,b,c], 2), + {_, 2} = limt([a,b,c], 3), + {_, 2} = limt([a,b|c], 2), + {_, 2} = limt([a,b|c], 3), + {_, 2} = limt({a,b,c,[d,e]}, 2), + {_, 2} = limt({a,b,c,[d,e]}, 3), + {_, 2} = limt({a,b,c,[d,e]}, 4), + {_, 1} = limt(<<"foo">>, 18), + ok = blimt(<<"123456789012345678901234567890">>), + {_, 1} = limt(<<7:3>>, 2), + {_, 1} = limt(<<7:21>>, 2), + {_, 1} = limt([], 2), + {_, 1} = limt({}, 2), + {_, 1} = limt(#{}, 2), + {_, 1} = limt(#{[] => {}}, 2), + {_, 1} = limt(#{[] => {}}, 3), + T = #{[] => {},[a] => [b]}, + {_, 1} = limt(T, 2), + {_, 1} = limt(T, 3), + {_, 1} = limt(T, 4), + ok. + +blimt(Binary) -> + blimt(Binary, byte_size(Binary)). + +blimt(_B, 1) -> ok; +blimt(B, D) -> + {_, 1} = limt(B, D), + blimt(B, D - 1). + +limt(Term, Depth) when is_integer(Depth) -> + T1 = io_lib:limit_term(Term, Depth), + S = form(Term, Depth), + S1 = form(T1, Depth), + OK1 = S1 =:= S, + + T2 = io_lib:limit_term(Term, Depth+1), + S2 = form(T2, Depth), + OK2 = S2 =:= S, + + T3 = io_lib:limit_term(Term, Depth-1), + S3 = form(T3, Depth), + OK3 = S3 =/= S, + + R = case {OK1, OK2, OK3} of + {true, true, true} -> 2; + {true, true, false} -> 1; + _ -> 0 + end, + {{S, S1, S2}, R}. + +form(Term, Depth) -> + lists:flatten(io_lib:format("~W", [Term, Depth])). diff --git a/lib/stdlib/vsn.mk b/lib/stdlib/vsn.mk index f062c7fe6e..f7bd21472c 100644 --- a/lib/stdlib/vsn.mk +++ b/lib/stdlib/vsn.mk @@ -1 +1 @@ -STDLIB_VSN = 3.4 +STDLIB_VSN = 3.3 diff --git a/lib/syntax_tools/doc/src/notes.xml b/lib/syntax_tools/doc/src/notes.xml index 11c37459d6..e8de0ffce2 100644 --- a/lib/syntax_tools/doc/src/notes.xml +++ b/lib/syntax_tools/doc/src/notes.xml @@ -32,21 +32,6 @@ <p>This document describes the changes made to the Syntax_Tools application.</p> -<section><title>Syntax_Tools 2.1.2</title> - - <section><title>Improvements and New Features</title> - <list> - <item> - <p> Miscellaneous updates due to atoms containing - arbitrary Unicode characters. </p> - <p> - Own Id: OTP-14285</p> - </item> - </list> - </section> - -</section> - <section><title>Syntax_Tools 2.1.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/syntax_tools/vsn.mk b/lib/syntax_tools/vsn.mk index 9b33f1e1f4..c5e363112b 100644 --- a/lib/syntax_tools/vsn.mk +++ b/lib/syntax_tools/vsn.mk @@ -1 +1 @@ -SYNTAX_TOOLS_VSN = 2.1.2 +SYNTAX_TOOLS_VSN = 2.1.1 diff --git a/lib/tools/doc/src/lcnt.xml b/lib/tools/doc/src/lcnt.xml index 9c8ce148e9..6e66a957ab 100644 --- a/lib/tools/doc/src/lcnt.xml +++ b/lib/tools/doc/src/lcnt.xml @@ -38,7 +38,7 @@ <modulesummary>A runtime system Lock Profiling tool.</modulesummary> <description> <p>The <c>lcnt</c> module is used to profile the internal ethread locks in the - Erlang Runtime System. With <c>lcnt</c> enabled, Internal counters in the + Erlang Runtime System. With <c>lcnt</c> enabled, internal counters in the runtime system are updated each time a lock is taken. The counters stores information about the number of acquisition tries and the number of collisions that has occurred during the acquisition tries. The counters also record the @@ -47,17 +47,17 @@ <p> The data produced by the lock counters will give an estimate on how well the runtime system will behave from a parallelizable view point for the - scenarios tested. This tool was mainly developed to help erlang runtime + scenarios tested. This tool was mainly developed to help Erlang runtime developers iron out potential and generic bottlenecks. </p> <p>Locks in the emulator are named after what type of resource they protect and where in the emulator they are initialized, those are lock 'classes'. Most of those locks are also instantiated several times, and given unique identifiers, to increase locking granularity. Typically an instantiated lock protects a disjunct set of - the resource, i.e ets-tables, processes or ports. In other cases it protects a - specific range of a resource, e.g. <c>pix_lock</c> which protects index to process + the resource, for example ets tables, processes or ports. In other cases it protects a + specific range of a resource, for example <c>pix_lock</c> which protects index to process mappings, and is given a unique number within the class. A unique lock in <c>lcnt</c> - is referenced by a name (class) and an identifier, <c>{Name, Id}</c>. + is referenced by a name (class) and an identifier: <c>{Name, Id}</c>. </p> <p>Some locks in the system are static and protects global resources, for example <c>bif_timers</c> and the <c>run_queue</c> locks. Other locks are dynamic and not diff --git a/lib/tools/doc/src/lcnt_chapter.xml b/lib/tools/doc/src/lcnt_chapter.xml index 6cfdb5cf1b..1981d66117 100644 --- a/lib/tools/doc/src/lcnt_chapter.xml +++ b/lib/tools/doc/src/lcnt_chapter.xml @@ -42,42 +42,32 @@ completed its access to the resource and unlocked it. The <c>lcnt</c> tool measures these lock conflicts. </p> <p> - Locks has an inherent cost in execution time and memory space. It takes time initialize, destroy, aquiring or releasing locks. To decrease lock contention it + Locks have an inherent cost in execution time and memory space. It takes time initialize, destroy, aquiring or releasing locks. To decrease lock contention it some times necessary to use finer grained locking strategies. This will usually also increase the locking overhead and hence there is a tradeoff between lock contention and overhead. In general, lock contention increases with the number of threads running concurrently. The <c>lcnt</c> tool does not measure locking overhead. </p> <section> <title> Enabling lock-counting </title> - <p>For investigation of locks in the emulator we use an internal tool called <c>lcnt</c> (short for lock-count). The VM needs to be compiled with this option enabled. To enable this, use:</p> + <p>For investigation of locks in the emulator we use an internal tool called <c>lcnt</c> (short for lock-count). The VM needs to be compiled with this option enabled. + To compile a lock-counting VM along with a normal VM, use:</p> <pre> cd $ERL_TOP -./configure --enable-lock-counter - </pre> - - <p> - Another way to enable this alongside a normal VM is to compile it at emulator directory level, much like a debug build. To compile it this way do the following, - </p> - <pre> -cd $ERL_TOP/erts/emulator -make lcnt FLAVOR=smp - </pre> - <p> and then starting Erlang with,</p> +./configure --enable-lock-counter</pre> + <p>Start the lock-counting VM like this:</p> <pre> -$ERL_TOP/bin/cerl -lcnt - </pre> - <p>To verify that you lock-counting enabled check that <c>[lock-counting]</c> appears in the status text when the VM is started.</p> +$ERL_TOP/bin/erl -emu_type lcnt</pre> + <p>To verify that lock counting is enabled check that <c>[lock-counting]</c> appears in the status text when the VM is started.</p> <pre> -Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:0] [hipe] - [kernel-poll:false] [lock-counting] - </pre> +Erlang/OTP 20 [erts-9.0] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] + [kernel-poll:false] [lock-counting]</pre> </section> <section> <title>Getting started</title> <p>Once you have a lock counting enabled VM the module <c>lcnt</c> can be used. The module is intended to be used from the current running nodes shell. To access remote nodes use <c>lcnt:clear(Node)</c> and <c>lcnt:collect(Node)</c>. </p> <p>All locks are continuously monitored and its statistics updated. Use <c>lcnt:clear/0</c> to initially clear all counters before running any specific tests. This command will also reset the duration timer internally.</p> - <p>To retrieve lock statistics information use, <c>lcnt:collect/0,1</c>. The collect operation will start a <c>lcnt</c> server if it not already started. All collected data will be built into an erlang term and uploaded to the server and a duration time will also be uploaded. This duration is the time between <c>lcnt:clear/0,1</c> and <c>lcnt:collect/0,1</c>.</p> + <p>To retrieve lock statistics information, use <c>lcnt:collect/0,1</c>. The collect operation will start a <c>lcnt</c> server if it not already started. All collected data will be built into an Erlang term and uploaded to the server and a duration time will also be uploaded. This duration is the time between <c>lcnt:clear/0,1</c> and <c>lcnt:collect/0,1</c>.</p> <p>Once the data is collected to the server it can be filtered, sorted and printed in many different ways.</p> <p>See the <seealso marker="lcnt">reference manual</seealso> for a description of each function.</p> diff --git a/lib/tools/doc/src/notes.xml b/lib/tools/doc/src/notes.xml index ae2b4b00a0..bc17fd5307 100644 --- a/lib/tools/doc/src/notes.xml +++ b/lib/tools/doc/src/notes.xml @@ -31,35 +31,6 @@ </header> <p>This document describes the changes made to the Tools application.</p> -<section><title>Tools 2.10</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - In some situations, <c>make:all()</c> and friends did not - detect changes in include files located in the current - directory. This is now corrected.</p> - <p> - Own Id: OTP-14339 Aux Id: ERL-395 </p> - </item> - </list> - </section> - - - <section><title>Improvements and New Features</title> - <list> - <item> - <p>The <c>make</c> module now accepts the - <c>{emake,Emake}</c> option.</p> - <p> - Own Id: OTP-14253</p> - </item> - </list> - </section> - -</section> - <section><title>Tools 2.9.1</title> <section><title>Improvements and New Features</title> diff --git a/lib/tools/emacs/erlang.el b/lib/tools/emacs/erlang.el index 5a0c814509..438abc2d29 100644 --- a/lib/tools/emacs/erlang.el +++ b/lib/tools/emacs/erlang.el @@ -4706,9 +4706,10 @@ for a tag on the form `module:tag'." (if (fboundp 'advice-add) ;; Emacs 24.4+ - (require 'etags) - (advice-add 'etags-tags-completion-table :around - #'erlang-etags-tags-completion-table-advice) + (progn + (require 'etags) + (advice-add 'etags-tags-completion-table :around + #'erlang-etags-tags-completion-table-advice)) ;; Emacs 23.1-24.3 (defadvice etags-tags-completion-table (around erlang-replace-tags-table diff --git a/lib/tools/test/lcnt_SUITE.erl b/lib/tools/test/lcnt_SUITE.erl index d39a5deeab..2e48b11740 100644 --- a/lib/tools/test/lcnt_SUITE.erl +++ b/lib/tools/test/lcnt_SUITE.erl @@ -29,7 +29,8 @@ -export([t_load/1, t_conflicts/1, t_locations/1, - t_swap_keys/1]). + t_swap_keys/1, + smoke_lcnt/1]). init_per_testcase(_Case, Config) -> Config. @@ -43,7 +44,8 @@ suite() -> {timetrap,{minutes,4}}]. all() -> - [t_load, t_conflicts, t_locations, t_swap_keys]. + [t_load, t_conflicts, t_locations, t_swap_keys, + smoke_lcnt]. %%---------------------------------------------------------------------- %% Tests @@ -146,3 +148,82 @@ t_swap_keys_file([File|Files]) -> ok = lcnt:conflicts(), ok = lcnt:stop(), t_swap_keys_file(Files). + +%% Simple smoke test of actual lock-counting, if running on +%% a run-time with lock-counting enabled. + +smoke_lcnt(Config) -> + case erlang:system_info(build_type) of + lcnt -> + do_smoke_lcnt(Config); + _ -> + {skip,"Lock counting is not enabled"} + end. + +do_smoke_lcnt(Config) -> + PrivDir = proplists:get_value(priv_dir, Config), + SaveFile = filename:join(PrivDir, atom_to_list(?FUNCTION_NAME)), + {Time,ok} = timer:tc(fun() -> lcnt:apply(fun() -> big_bang(200) end) end), + io:format("~p ms\n", [Time]), + ok = lcnt:conflicts(), + ok = lcnt:save(SaveFile), + ok = lcnt:load(SaveFile), + ok = lcnt:conflicts(), + lcnt:stop(). + + +%%% +%%% A slightly modified version of Rickard Green's Big Bang benchmark. +%%% + +big_bang(N) when is_integer(N) -> + Procs = spawn_procs(N), + RMsgs = lists:map(fun (P) -> {done, P} end, Procs), + send_procs(Procs, {procs, Procs, self()}), + receive_msgs(RMsgs), + lists:foreach(fun (P) -> exit(P, normal) end, Procs). + +pinger([], [], true) -> + receive + {procs, Procs, ReportTo} -> + pinger(Procs, [], ReportTo) + end; +pinger([], [], false) -> + receive {ping, From} -> From ! {pong, self()} end, + pinger([],[],false); +pinger([], [], ReportTo) -> + ReportTo ! {done, self()}, + pinger([],[],false); +pinger([],[Po|Pos] = Pongers, ReportTo) -> + receive + {ping, From} -> + From ! {pong, self()}, + pinger([], Pongers, ReportTo); + {pong, Po} -> + pinger([], Pos, ReportTo) + end; +pinger([Pi|Pis], Pongers, ReportTo) -> + receive {ping, From} -> From ! {pong, self()} + after 0 -> ok + end, + Pi ! {ping, self()}, + pinger(Pis, [Pi|Pongers], ReportTo). + +spawn_procs(N) when N =< 0 -> + []; +spawn_procs(N) -> + [spawn_link(fun () -> pinger([], [], true) end) | spawn_procs(N-1)]. + +send_procs([], Msg) -> + Msg; +send_procs([P|Ps], Msg) -> + P ! Msg, + send_procs(Ps, Msg). + +receive_msgs([]) -> + ok; +receive_msgs([M|Ms]) -> + receive + M -> + receive_msgs(Ms) + end. diff --git a/lib/tools/vsn.mk b/lib/tools/vsn.mk index 8aa7814e1d..f60da27c44 100644 --- a/lib/tools/vsn.mk +++ b/lib/tools/vsn.mk @@ -1 +1 @@ -TOOLS_VSN = 2.10 +TOOLS_VSN = 2.9.1 diff --git a/lib/wx/c_src/egl_impl.h b/lib/wx/c_src/egl_impl.h index 719b4926db..7ecd484de5 100644 --- a/lib/wx/c_src/egl_impl.h +++ b/lib/wx/c_src/egl_impl.h @@ -112,7 +112,7 @@ typedef long int int32_t; typedef long long int int64_t; typedef unsigned long long int uint64_t; #elif defined(WIN32) && defined(_MSC_VER) -typedef long int int32_t; +typedef __int32 int32_t; typedef __int64 int64_t; typedef unsigned __int64 uint64_t; #elif defined(WIN32) && defined(__GNUC__) diff --git a/lib/wx/doc/src/notes.xml b/lib/wx/doc/src/notes.xml index 18661d1071..9086117c81 100644 --- a/lib/wx/doc/src/notes.xml +++ b/lib/wx/doc/src/notes.xml @@ -32,21 +32,6 @@ <p>This document describes the changes made to the wxErlang application.</p> -<section><title>Wx 1.8.1</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Fix a livelock that could be caused by <c>wx:batch/1</c>.</p> - <p> - Own Id: OTP-14289</p> - </item> - </list> - </section> - -</section> - <section><title>Wx 1.8</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/wx/vsn.mk b/lib/wx/vsn.mk index b9100e7c87..cfa256fb12 100644 --- a/lib/wx/vsn.mk +++ b/lib/wx/vsn.mk @@ -1 +1 @@ -WX_VSN = 1.8.1 +WX_VSN = 1.8 diff --git a/lib/xmerl/doc/src/notes.xml b/lib/xmerl/doc/src/notes.xml index a7592dc344..ef4831e6ec 100644 --- a/lib/xmerl/doc/src/notes.xml +++ b/lib/xmerl/doc/src/notes.xml @@ -32,25 +32,6 @@ <p>This document describes the changes made to the Xmerl application.</p> -<section><title>Xmerl 1.3.14</title> - - <section><title>Fixed Bugs and Malfunctions</title> - <list> - <item> - <p> - Improves accumulator fun in xmerl_scan so that only one - #xmlText record is returned for strings which have - character references.</p> - <p> - (Thanks to Jimmy Zöger)</p> - <p> - Own Id: OTP-14377 Aux Id: PR-1369 </p> - </item> - </list> - </section> - -</section> - <section><title>Xmerl 1.3.13</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/xmerl/vsn.mk b/lib/xmerl/vsn.mk index 4e741d59a8..1515a4e37d 100644 --- a/lib/xmerl/vsn.mk +++ b/lib/xmerl/vsn.mk @@ -1 +1 @@ -XMERL_VSN = 1.3.14 +XMERL_VSN = 1.3.13 |