httpc: Fix streaming bugs when handling small responses

Own Id: OTP-11992

Correct distirbing mode for httpd:reload_config/2

Own Id: OTP-11914

Improved handling of invalid strings in the HTTP request line.

Impact: May improve memory consumption

Own Id: OTP-11925 Aux Id: Sequence 12601

Fixed a spelling mistake in httpc doc (Thanks to Wasif Riaz Malik)

Own Id: OTP-11538

Application upgrade (appup) files are corrected for the following applications:

asn1, common_test, compiler, crypto, debugger, dialyzer, edoc, eldap, erl_docgen, et, eunit, gs, hipe, inets, observer, odbc, os_mon, otp_mibs, parsetools, percept, public_key, reltool, runtime_tools, ssh, syntax_tools, test_server, tools, typer, webtool, wx, xmerl

A new test utility for testing appup files is added to test_server. This is now used by most applications in OTP.

(Thanks to Tobias Schlager)

Own Id: OTP-11744

ftp now sanitize file name, user name and passwords from <CR> and <LF> tags (Thanks to Sergei Golovan)

Own Id: OTP-11750

Corrected error handling in the HTTP client, making it behave more graceful.

Thanks to Kirilll Zaborsky

Own Id: OTP-11794

Support identity transfer-encoding in httpc.

Thanks to Anthony Ramine

Own Id: OTP-11802

Ignore empty Set-Cookie headers to increase interoperability with servers that violate the RFC.

Thanks to Kirilll Zaborsky

Own Id: OTP-11803

The commit 6189bc07 "inets: httpc improve pipelining" has been reverted, as it turned out to break things rather than improve pipelining utilization. It is instead up to the user to configure httpc and use it wisely to be able to get the most out of pipelining.

Own Id: OTP-11756

Handle all response codes in httpd_util:message/3

Own Id: OTP-11838

Mend max_clients check that was broken and avoid too extensive logging that could cause memory problems.

Own Id: OTP-11557 Aux Id: seq12478

Fix httpd config option 'script_timeout' and fixed httpd config option 'keep_alive_timeout'. Thanks to Johannes Weissl.

Own Id: OTP-11276

Make httpc:request_cancel/[1,2] asynchronous. Previously these functions tried to guarantee request answer would not reach the client, which only worked for some of the use cases. Now these functions are totally asynchronous which makes it the clients responsibility to disregard possible answers to canceled requests.

Also pipelining implementation has been changed to improve the utilization factor. Further investigation of possible enhancements in this area are planned for later.

*** POTENTIAL INCOMPATIBILITY ***

Own Id: OTP-11312

[httpd] Add handling of new response for mod_head (otherwise causing case_clause crash). Also updated logging: Removed logging for keep-alive connections timeout (this is a normal occurrence and not an error) and some access-log body size corrections.

Own Id: OTP-11328

The ftp client now supports ftp over tls (ftps).

Own Id: OTP-11037

httpc: Allow content body in DELETE requests. Thanks to James Wheare.

Own Id: OTP-11190

Add missing brackets to report formatting on ftp_progress process exit. Thanks to Artur Wilniewczyc.

Own Id: OTP-11202

Fix some errors in the inets documentation. Thanks to Johannes Weissl.

Own Id: OTP-11210

Fix various typos in httpd, inets. Thanks to Tomohiko Aono.

Own Id: OTP-11226

Fix httpd config option 'erl_script_nocache'. Thanks to Johannes Weissl.

Own Id: OTP-11260

Reverted incorrect commit that broke cookie handling when using httpc-profiles.

Own Id: OTP-10956

Fix http_request:http_headers/1 to send content-length when length is zero. Thanks to CA Meijer.

Own Id: OTP-10934

Integrate elliptic curve contribution from Andreas Schultz

In order to be able to support elliptic curve cipher suites in SSL/TLS, additions to handle elliptic curve infrastructure has been added to public_key and crypto.

This also has resulted in a rewrite of the crypto API to gain consistency and remove unnecessary overhead. All OTP applications using crypto has been updated to use the new API.

Impact: Elliptic curve cryptography (ECC) offers equivalent security with smaller key sizes than other public key algorithms. Smaller key sizes result in savings for power, memory, bandwidth, and computational cost that make ECC especially attractive for constrained environments.

Own Id: OTP-11009

Fix {stream, {self, once}} in httpc to work as expected. Thanks to Masatake Daimon

Own Id: OTP-11122

httpd: The modules option now defaults to the documented value.

Own Id: OTP-10844

httpc: Fixed persistent connection implementation that was broken by a patch to R13. The patch made persisten connections behaved the same way as pipelining.

Own Id: OTP-10845

httpd: Simplified configuration of ssl in httpd, this also enables all ssl options to be configured. The old and limited way is no longer documented but will be supported for backwards comatibility for some time.

Own Id: OTP-10846

Handle correctly the "No files found or file unavailable" error code. Thanks to Serge Aleynikov

Own Id: OTP-10886

httpc: The HTTP client now supports HTTPS through proxies

Own Id: OTP-10256 Aux Id: kunagi-2 [ce2e800e-c99f-4050-a1c4-f47023d9c7aa-1]

Some examples overflowing the width of PDF pages have been corrected.

Own Id: OTP-10665

Fix autoredirect for POST requests responding 303. Thanks to Hans Svensson.

Own Id: OTP-10765

Make log_alert configurable as option in ssl, SSLLogLevel added as option to inets conf file

Own Id: OTP-11259

Fixed obsolete error report in inets.

Own Id: OTP-11185 Aux Id: seq12357

Minimum bytes per second

New option to http server, {minimum_bytes_per_second, integer()}, for a connection, if it is not reached the socket will close for that specific connection. Can be used to prevent hanging requests from faulty clients.

Own Id: OTP-10392

Better handling of errorI(s) during update of the session database.

Also added and updated some debugging functions which_sessions/10,1 and info/0.

Own Id: OTP-10093

Aux Id: Seq 12062

Removed R14B compatible version of (inets-service and tftp) behaviour definition.

Own Id: OTP-10095

[httpc] Documentation of KeepAlive and Pipeline timeout options have been improved.

Own Id: OTP-10114

[httpc] Cancel request does not work due to incorrect handler table creation (wrong keypos).

Vyacheslav Vorobyov

Own Id: OTP-10092

[httpd] Make the server header configurable with new config option server_tokens. The value of the server header, which was previously hard-coded (at compile time), is now possible to manipulate through the means of the server_tokens config option.

Own Id: OTP-9805

Improve inets support for inets as an included application.

inets_app calls supervisor:start_link/3 directly rather than calling the root supervisor function inets_sup:start_link/0. This precludes using included_applications to start inets without having a wrapper function.

Jay Nelson

Own Id: OTP-9960

[httpc] Add function for retrieving current options, get_options/1,2.

Own Id: OTP-9979

Utility module http_uri now officially supported.

Also, the parse function has been extended with more scheme support and a way to provide your own scheme info.

Own Id: OTP-9983

Aux Id: Seq 12022

[ftp] Fails to open IPv6 connection due to badly formatted IPv6 address in EPRT command. The address part of the command incorrectly contained decimal elements instead of hexadecimal.

Own Id: OTP-9827

Aux Id: Seq 11970

[httpc] Bad Keep Alive Mode. When selecting a session, the "state" of the session (specifically if the server has responded) was not taken into account.

Own Id: OTP-9847

[httpc] The client incorrectly streams 404 responses. The documentation specifies that only 200 and 206 responses shall be streamed.

Shane Evens

Own Id: OTP-9860

[ftpc] Add a config option to specify a data connect timeout. That is how long the ftp client will wait for the server to connect to the data socket. If this timeout occurs, an error will be returned to the caller and the ftp client process will be terminated.

Own Id: OTP-9545

[httpc] Wrong Host header in IPv6 HTTP requests. When a URI with a IPv6 host is parsed, the brackets that encapsulates the address part is removed. This value is then supplied as the host header. This can cause problems with some servers. A workaround for this is to use headers_as_is and provide the host header with the requst call. To solve this a new option has been added, ipv6_host_with_brackets. This option specifies if the host value of the host header shall include the brackets or not. By default, it does not (as before).

Own Id: OTP-9628

[httpd] Fix logging of content length in mod_log.

Garrett Smith

Own Id: OTP-9715

[httpd] Sometimes entries in the transfer log was written with the message size as list of numbers. This list was actually the size as a string, e.g. "123", written with the control sequence ~w. This has now been corrected so that any string is converted to an integer (if possible).

Own Id: OTP-9733

Fixed various problems detected by Dialyzer.

Own Id: OTP-9736

[httpc] Deprecated interface module http has been removed. It has (long) been replaced by http client interface module httpc.

Own Id: OTP-9359

[httpc|httpd] The old ssl implementation (based on OpenSSL), has been deprecated. The config option that specified usage of this version of the ssl app, ossl, has been removed.

Own Id: OTP-9522

[httpd] XSS prevention did not work for hex-encoded URL's.

Own Id: OTP-9655

[httpd] GET request with malformed header date caused server crash (non-fatal) with no reply to client. Will now result in a reply with status code 400.

Own Id: OTP-9674

Aux Id: seq11936

[httpc] Parsing of a cookie expire date should be more forgiving. That is, if the parsing fails, the date should be ignored. Also added support for (yet another) date format: "Tue Jan 01 08:00:01 2036 GMT".

Own Id: OTP-9433

[httpc] Rewrote cookie parsing. Among other things solving cookie processing from www.expedia.com.

Own Id: OTP-9434

[httpd] Fix httpd directory traversal on Windows. Directory traversal was possible on Windows where backward slash is used as directory separator.

András Veres-Szentkirályi.

Own Id: OTP-9561

[httpc|httpd] Added support for IPv6 with ssl.

Own Id: OTP-5566

[httpc] Remove unnecessary usage of iolist_to_binary when processing body (for PUT and POST).

Filipe David Manana

Own Id: OTP-9317

[ftp] FTP client doesn't work with IPv6 host.

Attila Rajmund Nohl

Own Id: OTP-9342 Aux Id: seq11853

[httpd] Peer/sockname resolv doesn't work with IPv6 addrs in HTTP.

Attila Rajmund Nohl.

Own Id: OTP-9343

[httpc] Clients started stand-alone not properly handled. Also it was not documented how to use them, that is that once started, they are represented by a pid() and not by their profile().

Own Id: OTP-9365

[httpc] Add support for upload body streaming (PUT and POST).

For more info, see the definition of the Body argument of the request/4,5 function.

Filipe David Manana

Own Id: OTP-9094

[ftp] Added (type) spec for all exported functions.

Own Id: OTP-9114 Aux Id: seq11799

[httpd] mod_esi:deliver/2 made to accept binary data.

Bernard Duggan

Own Id: OTP-9123

[httpd] Prevent XSS in error pages. Prevent user controlled input from being interpreted as HTML in error pages by encoding the reserved HTML characters.

Michael Santos

Own Id: OTP-9124

[httpd] Improved error messages.

Ricardo Catalinas Jiménez

Own Id: OTP-9157

[httpd] Extended support for file descriptors. In order to be able to bind to a privileged port without running the erlang VM as root, the support for using file descriptors has been improved. It is now possible to add the file descriptor to the config (option fd) when calling the inets:start(httpd, ...) function.

Attila Rajmund Nohl

Own Id: OTP-9202

Aux Id: seq11819

The default ssl kind has now been changed to essl.

ossl will work for as long as the ssl application supports it.

See the httpd socket_type communication property or the httpc request/4,5 function for more info.

Own Id: OTP-9230

*** POTENTIAL INCOMPATIBILITY ***

[httpd] Wrong security property names used in documentation.

security_data_file used instead of data_file.

security_max_retries used instead of max_retries.

security_block_time used instead of block_time.

security_fail_expire_time used instead of fail_expire_time.

security_auth_timeout used instead of auth_timeout.

Garrett Smith

Own Id: OTP-9131

[httpd] Fix timeout message generated by mod_esi. When a mod_esi request times out, the code to send a timeout response was incorrect and generated an internal server error as well as an invalid response line.

Bernard Duggan

Own Id: OTP-9158

[httpc] httpc manager crashes. When a request results in a retry, the request id will be "reused" in the previous implementation a race condition could occur causing the manager to crash.

This is now avoided by using proc_lib:init_ack and gen_server:enter_loop to allow mor advanced initialization of httpc_handlers without blocking the httpc_manger and eliminating extra processes that can cause race conditions.

Own Id: OTP-9246

[httpc] Issuing a request (httpc:request) to an host with the ssl option {ip, {127,0,0,1}} results in an handler crash. The reason was that the connect call resulted in an exit with reason badarg (this was the same for both ssl and gen_tcp).

Exits was not catched. This has now been improved.

Own Id: OTP-9289

Aux Id: seq11845

[httpd] httpd_response:send_chunk handles empty list and empty binary - i.e. no chunk is sent, but it does not handle a list with an empty binary [<<>>]. This will be sent as an empty chunk - which in turn will be encoded by http_chunk to the same as a final chunk, which will make the http client believe that the end of the page is reached.

Own Id: OTP-8906

Miscellaneous inet6 related problems.

Own Id: OTP-8927

Updated http-server to make sure URLs in error-messages are URL-encoded. Added support in http-client to use URL-encoding. Also added the missing include directory for the inets application.

Own Id: OTP-8940

Aux Id: seq11735

Fix format_man_pages so it handles all man sections and remove warnings/errors in various man pages.

Own Id: OTP-8600

[httpc] Pipelined and queued requests not processed when connection closed remotelly.

Own Id: OTP-8906

[httpc] If a request times out (not connect timeout), the handler process exited (normal) but neglected to inform the manager process. For this reason, the manager did not clean up the request table., resulting in a memory leak. Also the manager did not create a monitor for the handler, so in an unforseen handler crash, this could also create a memory leak.

Own Id: OTP-8739

The service tftp was spelled wrong in documentation and in some parts of the code. It should be tftp.

Own Id: OTP-8741 Aux Id: seq11635

[httpc] Replaced the old http client api module (http) with the new, httpc in the users guide.

Own Id: OTP-8742

Eliminated warnings for auto-imported BIF clashes.

Own Id: OTP-8840

[httpc|httpd] - Now allow the use of the "new" ssl, by using the essl tag instead.

See the http_option option in the request/4,5 or the socket-type section of the Communication properties chapter for more info,

Own Id: OTP-7907

Deprecated functions designated to be removed in R14 has been removed. Also, some new functions has been marked as deprecated (the old http client api module).

Own Id: OTP-8564

*** POTENTIAL INCOMPATIBILITY ***

[httpd] - Improved mod_alias. Now able to do better URL rewrites.

See URL aliasing properties and the CGI properties section(s) for more info,

Own Id: OTP-8573

[httpc] - Made cookie handling more case insensitive.

Own Id: OTP-8609

Nicolas Thauvin

[httpc|httpd] - Netscape cookie dates can also be given with a 2-digit year (e.g. 06 = 2006).

Own Id: OTP-8610

Nicolas Thauvin

[httpd] - Added support (again) for the documented debugging features. See the User's Guide Configuration chapter for more info.

Own Id: OTP-8624

[httpc] - Memory leak plugged. The profile manager never cleaned up in its handler database. This meant that with each new request handler, another entry was created that was never deleted. Eventually the request id counter (used as a key) would wrap, but the machine would most likely run out of memory before that happened.

Own Id: OTP-8542

Lev Walkin

[httpc] - https requests with default port (443) not handled properly.

Own Id: OTP-8607

jebu ittiachen

[httpc] - Badly formated error reason for errors occuring during initial connect to a server. Also, the possible error reasons was not properly documented.

Own Id: OTP-8508

Aux Id: seq11407

[httpd] - Issues with ESI erl_script_timeout.

The erl_script_timeout config option is ducumented as a number of seconds. But when parsing the config, in the new format (not a config file), it was handled as if in number of milliseconds.

When the erl-script-timeout time was exceeded, the server incorrectly marked the answer as sent, thereby leaving client hanging (with an incomplete answer). This has been changed, so that now the socket will be closed.

[httpc] - Allow users to pass socket options to the transport module when making requests.

See the socket_opts option in the request/4 or set_options/1,2 for more info,

Own Id: OTP-8352

[httpc] Fix bug crafting Host header when port is not 80.

The host header should include the port number as well as the host name when making a request to a server listening on a port other than the HTTP default of 80. Currently, only the host name is included. This is important to make the http client more compliant with the HTTP specification.

Own Id: OTP-8371

Kelly McLaughlin

[httpc|httpd] http_chunk data handling/passing improvement.

This is a modification to the http_chunk module to forward any full chunk received, regardless of whether the size field for the following chunk has been received yet. This allows http_chunk to be used in situations where a long term HTTP connection is used to send periodic status updates as individual chunks. Previously a given chunk would not be forwarded to the client process until the size for the next chunk had been read which rendered the module difficult to use for the scenario described.

Bernard Duggan

Own Id: OTP-8351

Include the inets test suite in the release of the application.

Own Id: OTP-8349

[httpc] - It is now possible to configure the client to deliver an async reply to more receivers then the calling process.

See the receiver option for more info,

Own Id: OTP-8106

[httpd] - Methods "PUT" and "DELETE" now allowed.

huntermorris@gmail.com

Own Id: OTP-8103

[httpc] Several more or less critical fixes:

Initial call between the httpc manager and request handler was synchronous.

When the manager starts a new request handler, this is no longer a synchronous operation. Previously, the new request handler made the connection to the server and issuing of the first request (the reason for starting it) in the gen_server init function. If the connection for some reason "took some time", the manager hanged, leaving all other activities by that manager also hanging.

[httpd] The server did not fully support the documented module callback api. Specifically, the load function should be able to return the atom ok, but this was not accepted.

Own Id: OTP-8359

Fixing various documentation-related bugs (bad quotes).

Own Id: OTP-8327

Fixing minor Dialyzer and copyright problem(s).

Own Id: OTP-8315

[httpc] - Added basic sanity check of option value combinations.

adam.kocoloski@gmail.com

Own Id: OTP-8056

[ftpc] - Start of the FTP client has been changed in the following way:

Own Id: OTP-8258

The documentation is now built with open source tools (xsltproc and fop) that exists on most platforms. One visible change is that the frames are removed.

Own Id: OTP-8249

[httpc] - Streaming to file did not work.

dizzyd@gmail.com

Own Id: OTP-8204

[ftpc] - The ls/2 function (LIST command) and the nlist/2 function (NLST command) with wildcards did not work properly.

These functions is documented as working on directories, but this is actually not according the standard. The LIST and NLST commands are specified to operate on a directory or other group of files, or a file.

Previously, an attempt was made to check if the listing returned by the server was actually an error message. This was done by changing remote directory (cd) into the (assumed) "directory". This may work if Pathname was actually a directory, but as this is not always the case, this test does not work. Instead, we now return the actual server result and leave the interpretation to the caller.

*** POTENTIAL INCOMPATIBILITY ***

Own Id: OTP-8247

Aux Id: seq11407

[httpc] - Fixes various bugs in timeout and keep-alive queue handling.

Jean-Sébastien Pédron

Own Id: OTP-8248

[httpd] - Unnecessarily strict matching when handling closing sockets.

Own Id: OTP-8280

[httpc] - Raise condition. When http:request is called and httpc_manager selects a session where there's already a pending request, then the connection handler for that session effectively resets its parser, readying it for the response to the second request. But if there are still some inbound packets for the response to the first request, things get confused.

tomas.abrahamsson@gmail.com

Own Id: OTP-8154

[httpc] - Added http option connect_timeout for http client request. The connect_timeout option is used for the initial request, when the client connects to the server. Default value is that of the timeout option.

See the request/4,5 function for more info.

Own Id: OTP-7298

[httpd] - Failed to create listen socket with invalid option combo. The http-server failed to create its listen socket when the bind-address was an IPv4-address (a tuple of size 4) and the ipfamily option was inet6fb4.

Own Id: OTP-8118

Aux Id: seq11321

[httpd] - Removed documentation for non-existing function (httpd_util:header/2,3,4).

Own Id: OTP-8101

[httpd] - When starting inets (the web-server) and supplying a descriptor on the command line (example: erl -httpd_8888 <descriptor>) it is now possible to specify which ip-family to use: inet | inet6 | inet6fb4.

Example: erl -httpd_8888 10|inet6

When starting the web-server either using a file with property list (the proplist_file) or a an property list, using the ipfamily option: {ipfamily, inet | inet6 | inet6fb4}.

Finally, when starting the web-server using the classical apache-style config file, the BindAddress directive has been augmented to allow the specification of the IpFamily: BindAddress blirk.ericsson.se|inet

Default is inet6fb4 which emulates the behaviour of the previous version.

See the Communication properties section for more info.

Own Id: OTP-8069

Aux Id: seq11086

[httpc] - Reception of unexpected data causes handler crash.

Own Id: OTP-8052

[httpc] Added support for web services using only basic auth, with a token as the user part and no password part.

twoggle@gmail.com

Own Id: OTP-7998

[httpc] - Bind HTTP client to IP-addr. It is now possible to specify an alternate ip-address and port to be used when the client connects to the server.

As a side-effect of this, the option ipv6 has been removed and replaced by the ipfamily option.

See http:set_options/1,2 for more info.

*** POTENTIAL INCOMPATIBILITY ***

Own Id: OTP-8004

Updated guard tests (i.e. is_list(L) instead of list(L) and possibly andalso/orelse instead of ","/";").

Own Id: OTP-7994

[httpc] - Remove use of the deprecated regexp module.

Own Id: OTP-8001

[httpc] - The option max_keep_alive_length was not handled properly.

Own Id: OTP-8005