diff options
Diffstat (limited to 'docs/en/gun/2.0/guide')
| -rw-r--r-- | docs/en/gun/2.0/guide/index.html | 4 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/internals_tls_over_tls.asciidoc | 177 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/internals_tls_over_tls/index.html | 221 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/introduction.asciidoc | 2 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/introduction/index.html | 2 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/migrating_from_1.3.asciidoc | 75 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/migrating_from_1.3/index.html | 45 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/protocols.asciidoc | 17 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/protocols/index.html | 8 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/websocket.asciidoc | 8 | ||||
| -rw-r--r-- | docs/en/gun/2.0/guide/websocket/index.html | 12 |
11 files changed, 501 insertions, 70 deletions
diff --git a/docs/en/gun/2.0/guide/index.html b/docs/en/gun/2.0/guide/index.html index b6cb2ed6..cdfa5d76 100644 --- a/docs/en/gun/2.0/guide/index.html +++ b/docs/en/gun/2.0/guide/index.html @@ -76,6 +76,10 @@ <li><a href="websocket/">Using Websocket</a> </li> </ul> +<h2 id="_advanced">Advanced</h2> +<ul><li><a href="internals_tls_over_tls/">Internals: TLS over TLS</a> +</li> +</ul> <h2 id="_additional_information">Additional information</h2> <ul><li><a href="migrating_from_1.3/">Migrating from Gun 1.3 to 2.0</a> </li> diff --git a/docs/en/gun/2.0/guide/internals_tls_over_tls.asciidoc b/docs/en/gun/2.0/guide/internals_tls_over_tls.asciidoc new file mode 100644 index 00000000..07e86698 --- /dev/null +++ b/docs/en/gun/2.0/guide/internals_tls_over_tls.asciidoc @@ -0,0 +1,177 @@ +== Internals: TLS over TLS + +The `ssl` application that comes with Erlang/OTP implements +TLS using an interface equivalent to the `gen_tcp` interface: +you get and manipulate a socket. The TLS encoding and +decoding is applied transparently to the data sent and +received. + +In order to have a TLS layer inside another TLS layer we +need a way to encode the data of the inner layer before +we pass it to the outer layer. We cannot do this with +a socket interface. Thankfully, the `ssl` application comes +with options that allow to transform an `sslsocket()` into +an encoder/decoder. + +The implementation is however a little convoluted as a +result. This chapter aims to give an overview of how it +all works under the hood. + +=== gun_tls_proxy + +The module `gun_tls_proxy` implements an intermediary process +that sits between the Gun process and the TLS process. It is +responsible for routing data from the Gun process to the TLS +process, and from the TLS process to the Gun process. + +In order to obtain the TLS encoded data the `cb_info` option +is given to the `ssl:connect/3` function. This replaces the +default TCP outer socket module with our own custom module. +Gun uses the `gun_tls_proxy_cb` module instead. This module +will forward all messages to the `gun_tls_proxy` process. + +The resulting operations looks like this: + +---- +Gun process <-> gun_tls_proxy <-> sslsocket() <-> gun_tls_proxy <-> "inner socket" +---- + +The "inner socket" is the socket for the Gun connection. +The `gun_tls_proxy` process decides where to send or +receive the data based on where it's coming from. This +is how it knows whether the data has been encoded/decoded +or not. + +Because the `ssl:connect/3` function call is blocking, +a temporary process is used while connecting. This is required +because the `gun_tls_proxy` needs to forward data even while +performing the TLS handshake, otherwise the `ssl:connect/3` +call will not complete. + +The result of the `ssl:connect/3` call is forward to the Gun +process, along with the negotiated protocols when the connection +was successful. + +The `gun_tls_proxy_cb` module does not actually implement +`{active,N}` as requested by the `ssl` application. Instead +it uses `{active,true}`. + +The behavior of the `gun_tls_proxy` process will vary depending +on whether the TLS over TLS is done connection-wide or only +stream-wide. + +=== Connection-wide TLS over TLS + +When used for the entire connection, the `gun_tls_proxy` process +will act like a real socket once connected. The only difference +is how the connection is performed. As mentioned earlier, the +result of the `ssl:connect/3` call is sent back to the Gun process. + +When doing TLS over TLS the processes will end up looking like +this: + +---- +Gun process <-> gun_tls_proxy <-> "inner socket" +---- + +The details of the interactions between `gun_tls_proxy` and +its associated `sslsocket()` have been removed in order to +better illustrate the concept. + +When adding another layer this becomes: + +---- +Gun process <-> gun_tls_proxy <-> gun_tls_proxy <-> sslsocket() +---- + +This is what is done when only HTTP/1.1 and SOCKS proxies are +involved. + +=== Stream-wide TLS over TLS + +The same cannot be done for HTTP/2 proxies. This is because the +HTTP/2 CONNECT method does not apply to the entire connection, +but only to a stream. The proxied data must be wrapped inside +a DATA frame. It cannot be sent directly. This is what must be +done: + +---- +Gun process -> gun_tls_proxy -> Gun process -> "inner socket" +---- + +The "inner socket" is the socket for the HTTP/2 connection. + +In order to tell Gun to continue processing the data, the +`handle_continue` mechanism is introduced. When `gun_tls_proxy` +has TLS encoded the data it sends it back to the Gun process, +wrapped in a `handle_continue` tuple. This tuple contains +enough information to figure out what stream the data belongs +to and what should be done next. Gun will therefore route the +data to the correct layer and continue sending it. + +This solution is also used for receiving data, except in the +reverse order. + +=== Routing to the right stream + +In order to know where to route the data, the `stream_ref` +had to be modified to contain all the references to the +individual streams. So if the tunnel is identified by +`StreamA` and a request on this tunnel is identified +by `StreamB`, then the stream is known as `[StreamA, StreamB]` +to the user. Gun then routes first to `StreamA`, a +tunnel, which continues routing to `StreamB`. + +A problem comes up if an intermediary is a SOCKS server, +for example in the following scenario: + +---- +HTTP/2 proxy <-> SOCKS proxy <-> HTTP/1.1 origin +---- + +The SOCKS protocol doesn't have a concept of stream, +therefore when we refer to request to the origin server +they are `[StreamA, StreamB]`, not `[StreamA, StreamB, StreamC]`. +This is a problem for routing encoded/decoded TLS data +to the SOCKS layer: we don't have a built-in way of referring +to the SOCKS layer. + +The solution is to have a separate `handle_continue_stream_ref` +value that assigns a reference to the SOCKS layers. Gun will +then be able to forward `handle_continue` message, and only +them, to the appropriate layer. + +Gun therefore has two different routing avenues, but the +mechanism remains the same otherwise. + +=== gun_tunnel + +In order to simplify the routing, the `gun_tunnel` module +was introduced. For each intermediary (including the original +CONNECT stream at the HTTP/2 layer) there is a `gun_tunnel` +module. + +Going back to the example above: + +---- +HTTP/2 proxy <-> SOCKS proxy <-> HTTP/1.1 origin +---- + +In this case the modules involved to handle the request +will be as follow: + +---- +gun_http2 <-> gun_tunnel <-> gun_tunnel <-> gun_http +---- + +The `gun_http2` module doesn't do any routing, it just +passes everything to the `gun_tunnel` module. The `gun_tunnel` +module will then do the routing, which involves removing +`StreamA` from `[StreamA, StreamB]` where appropriate. + +The `gun_tunnel` module also receives the TLS encoded/decoded +data and forwards it appropriately. When it comes to sending +data, it will return a `send` command that allows the previous +module to continue sending the data. The `gun_http2` module +will ultimately wrap the data to be sent in a DATA frame and +send it to the "inner socket". diff --git a/docs/en/gun/2.0/guide/internals_tls_over_tls/index.html b/docs/en/gun/2.0/guide/internals_tls_over_tls/index.html new file mode 100644 index 00000000..7ebc2221 --- /dev/null +++ b/docs/en/gun/2.0/guide/internals_tls_over_tls/index.html @@ -0,0 +1,221 @@ +<!DOCTYPE html> +<html lang="en"> + +<head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <meta name="description" content=""> + <meta name="author" content="Loïc Hoguin based on a design from (Soft10) Pol Cámara"> + + <title>Nine Nines: Internals: TLS over TLS</title> + + <link href='https://fonts.googleapis.com/css?family=Open+Sans:400,700,400italic' rel='stylesheet' type='text/css'> + <link href="/css/99s.css?r=7" rel="stylesheet"> + + <link rel="shortcut icon" href="/img/ico/favicon.ico"> + <link rel="apple-touch-icon-precomposed" sizes="114x114" href="/img/ico/apple-touch-icon-114.png"> + <link rel="apple-touch-icon-precomposed" sizes="72x72" href="/img/ico/apple-touch-icon-72.png"> + <link rel="apple-touch-icon-precomposed" href="/img/ico/apple-touch-icon-57.png"> + + +</head> + + +<body class=""> + <header id="page-head"> + <div id="topbar" class="container"> + <div class="row"> + <div class="span2"> + <h1 id="logo"><a href="/" title="99s">99s</a></h1> + </div> + <div class="span10"> + + <div id="side-header"> + <nav> + <ul> + <li><a title="Hear my thoughts" href="/articles">Articles</a></li> + <li><a title="Watch my talks" href="/talks">Talks</a></li> + <li class="active"><a title="Read the docs" href="/docs">Documentation</a></li> + <li><a title="Request my services" href="/services">Consulting & Training</a></li> + </ul> + </nav> + <ul id="social"> + <li> + <a href="https://github.com/ninenines" title="Check my Github repositories"><img src="/img/ico_github.png" data-hover="/img/ico_github_alt.png" alt="Github"></a> + </li> + <li> + <a title="Contact me" href="mailto:[email protected]"><img src="/img/ico_mail.png" data-hover="/img/ico_mail_alt.png"></a> + </li> + </ul> + </div> + </div> + </div> + </div> + + +</header> + +<div id="contents" class="two_col"> +<div class="container"> +<div class="row"> +<div id="docs" class="span9 maincol"> + +<h1 class="lined-header"><span>Internals: TLS over TLS</span></h1> + +<p>The <code>ssl</code> application that comes with Erlang/OTP implements TLS using an interface equivalent to the <code>gen_tcp</code> interface: you get and manipulate a socket. The TLS encoding and decoding is applied transparently to the data sent and received.</p> +<p>In order to have a TLS layer inside another TLS layer we need a way to encode the data of the inner layer before we pass it to the outer layer. We cannot do this with a socket interface. Thankfully, the <code>ssl</code> application comes with options that allow to transform an <code>sslsocket()</code> into an encoder/decoder.</p> +<p>The implementation is however a little convoluted as a result. This chapter aims to give an overview of how it all works under the hood.</p> +<h2 id="_gun_tls_proxy">gun_tls_proxy</h2> +<p>The module <code>gun_tls_proxy</code> implements an intermediary process that sits between the Gun process and the TLS process. It is responsible for routing data from the Gun process to the TLS process, and from the TLS process to the Gun process.</p> +<p>In order to obtain the TLS encoded data the <code>cb_info</code> option is given to the <code>ssl:connect/3</code> function. This replaces the default TCP outer socket module with our own custom module. Gun uses the <code>gun_tls_proxy_cb</code> module instead. This module will forward all messages to the <code>gun_tls_proxy</code> process.</p> +<p>The resulting operations looks like this:</p> +<div class="listingblock"><div class="content"><pre>Gun process <-> gun_tls_proxy <-> sslsocket() <-> gun_tls_proxy <-> "inner socket"</pre></div></div> +<p>The "inner socket" is the socket for the Gun connection. The <code>gun_tls_proxy</code> process decides where to send or receive the data based on where it's coming from. This is how it knows whether the data has been encoded/decoded or not.</p> +<p>Because the <code>ssl:connect/3</code> function call is blocking, a temporary process is used while connecting. This is required because the <code>gun_tls_proxy</code> needs to forward data even while performing the TLS handshake, otherwise the <code>ssl:connect/3</code> call will not complete.</p> +<p>The result of the <code>ssl:connect/3</code> call is forward to the Gun process, along with the negotiated protocols when the connection was successful.</p> +<p>The <code>gun_tls_proxy_cb</code> module does not actually implement <code>{active,N}</code> as requested by the <code>ssl</code> application. Instead it uses <code>{active,true}</code>.</p> +<p>The behavior of the <code>gun_tls_proxy</code> process will vary depending on whether the TLS over TLS is done connection-wide or only stream-wide.</p> +<h2 id="_connection_wide_tls_over_tls">Connection-wide TLS over TLS</h2> +<p>When used for the entire connection, the <code>gun_tls_proxy</code> process will act like a real socket once connected. The only difference is how the connection is performed. As mentioned earlier, the result of the <code>ssl:connect/3</code> call is sent back to the Gun process.</p> +<p>When doing TLS over TLS the processes will end up looking like this:</p> +<div class="listingblock"><div class="content"><pre>Gun process <-> gun_tls_proxy <-> "inner socket"</pre></div></div> +<p>The details of the interactions between <code>gun_tls_proxy</code> and its associated <code>sslsocket()</code> have been removed in order to better illustrate the concept.</p> +<p>When adding another layer this becomes:</p> +<div class="listingblock"><div class="content"><pre>Gun process <-> gun_tls_proxy <-> gun_tls_proxy <-> sslsocket()</pre></div></div> +<p>This is what is done when only HTTP/1.1 and SOCKS proxies are involved.</p> +<h2 id="_stream_wide_tls_over_tls">Stream-wide TLS over TLS</h2> +<p>The same cannot be done for HTTP/2 proxies. This is because the HTTP/2 CONNECT method does not apply to the entire connection, but only to a stream. The proxied data must be wrapped inside a DATA frame. It cannot be sent directly. This is what must be done:</p> +<div class="listingblock"><div class="content"><pre>Gun process -> gun_tls_proxy -> Gun process -> "inner socket"</pre></div></div> +<p>The "inner socket" is the socket for the HTTP/2 connection.</p> +<p>In order to tell Gun to continue processing the data, the <code>handle_continue</code> mechanism is introduced. When <code>gun_tls_proxy</code> has TLS encoded the data it sends it back to the Gun process, wrapped in a <code>handle_continue</code> tuple. This tuple contains enough information to figure out what stream the data belongs to and what should be done next. Gun will therefore route the data to the correct layer and continue sending it.</p> +<p>This solution is also used for receiving data, except in the reverse order.</p> +<h2 id="_routing_to_the_right_stream">Routing to the right stream</h2> +<p>In order to know where to route the data, the <code>stream_ref</code> had to be modified to contain all the references to the individual streams. So if the tunnel is identified by <code>StreamA</code> and a request on this tunnel is identified by <code>StreamB</code>, then the stream is known as <code>[StreamA, StreamB]</code> to the user. Gun then routes first to <code>StreamA</code>, a tunnel, which continues routing to <code>StreamB</code>.</p> +<p>A problem comes up if an intermediary is a SOCKS server, for example in the following scenario:</p> +<div class="listingblock"><div class="content"><pre>HTTP/2 proxy <-> SOCKS proxy <-> HTTP/1.1 origin</pre></div></div> +<p>The SOCKS protocol doesn't have a concept of stream, therefore when we refer to request to the origin server they are <code>[StreamA, StreamB]</code>, not <code>[StreamA, StreamB, StreamC]</code>. This is a problem for routing encoded/decoded TLS data to the SOCKS layer: we don't have a built-in way of referring to the SOCKS layer.</p> +<p>The solution is to have a separate <code>handle_continue_stream_ref</code> value that assigns a reference to the SOCKS layers. Gun will then be able to forward <code>handle_continue</code> message, and only them, to the appropriate layer.</p> +<p>Gun therefore has two different routing avenues, but the mechanism remains the same otherwise.</p> +<h2 id="_gun_tunnel">gun_tunnel</h2> +<p>In order to simplify the routing, the <code>gun_tunnel</code> module was introduced. For each intermediary (including the original CONNECT stream at the HTTP/2 layer) there is a <code>gun_tunnel</code> module.</p> +<p>Going back to the example above:</p> +<div class="listingblock"><div class="content"><pre>HTTP/2 proxy <-> SOCKS proxy <-> HTTP/1.1 origin</pre></div></div> +<p>In this case the modules involved to handle the request will be as follow:</p> +<div class="listingblock"><div class="content"><pre>gun_http2 <-> gun_tunnel <-> gun_tunnel <-> gun_http</pre></div></div> +<p>The <code>gun_http2</code> module doesn't do any routing, it just passes everything to the <code>gun_tunnel</code> module. The <code>gun_tunnel</code> module will then do the routing, which involves removing <code>StreamA</code> from <code>[StreamA, StreamB]</code> where appropriate.</p> +<p>The <code>gun_tunnel</code> module also receives the TLS encoded/decoded data and forwards it appropriately. When it comes to sending data, it will return a <code>send</code> command that allows the previous module to continue sending the data. The <code>gun_http2</code> module will ultimately wrap the data to be sent in a DATA frame and send it to the "inner socket".</p> + + + + + + + + + + + + <nav style="margin:1em 0"> + + <a style="float:left" href="https://ninenines.eu/docs/en/gun/2.0/guide/websocket/"> + Websocket + </a> + + + + <a style="float:right" href="https://ninenines.eu/docs/en/gun/2.0/guide/migrating_from_1.3/"> + Migrating from Gun 1.3 to 2.0 + </a> + + </nav> + + + + +</div> + +<div class="span3 sidecol"> + + +<h3> + Gun + 2.0 + + User Guide +</h3> + +<ul> + + <li><a href="/docs/en/gun/2.0/guide">User Guide</a></li> + + + <li><a href="/docs/en/gun/2.0/manual">Function Reference</a></li> + + +</ul> + +<h4 id="docs-nav">Navigation</h4> + +<h4>Version select</h4> +<ul> + + + + <li><a href="/docs/en/gun/2.0/guide">2.0</a></li> + + <li><a href="/docs/en/gun/1.3/guide">1.3</a></li> + + <li><a href="/docs/en/gun/1.2/guide">1.2</a></li> + + <li><a href="/docs/en/gun/1.1/guide">1.1</a></li> + + <li><a href="/docs/en/gun/1.0/guide">1.0</a></li> + +</ul> + +<h3 id="_like_my_work__donate">Like my work? Donate!</h3> +<p>Donate to Loïc Hoguin because his work on Cowboy, Ranch, Gun and Erlang.mk is fantastic:</p> +<form action="https://www.paypal.com/cgi-bin/webscr" method="post" style="display:inline"> +<input type="hidden" name="cmd" value="_donations"> +<input type="hidden" name="business" value="[email protected]"> +<input type="hidden" name="lc" value="FR"> +<input type="hidden" name="item_name" value="Loic Hoguin"> +<input type="hidden" name="item_number" value="99s"> +<input type="hidden" name="currency_code" value="EUR"> +<input type="hidden" name="bn" value="PP-DonationsBF:btn_donate_LG.gif:NonHosted"> +<input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!"> +<img alt="" border="0" src="https://www.paypalobjects.com/fr_FR/i/scr/pixel.gif" width="1" height="1"> +</form><p>Recurring payment options are also available via <a href="https://github.com/sponsors/essen">GitHub Sponsors</a>. These funds are used to cover the recurring expenses like food, dedicated servers or domain names.</p> + + + +</div> +</div> +</div> +</div> + + <footer> + <div class="container"> + <div class="row"> + <div class="span6"> + <p id="scroll-top"><a href="#">↑ Scroll to top</a></p> + <nav> + <ul> + <li><a href="mailto:[email protected]" title="Contact us">Contact us</a></li><li><a href="https://github.com/ninenines/ninenines.github.io" title="Github repository">Contribute to this site</a></li> + </ul> + </nav> + </div> + <div class="span6 credits"> + <p><img src="/img/footer_logo.png"></p> + <p>Copyright © Loïc Hoguin 2012-2018</p> + </div> + </div> + </div> + </footer> + + + <script src="/js/custom.js"></script> + </body> +</html> + + diff --git a/docs/en/gun/2.0/guide/introduction.asciidoc b/docs/en/gun/2.0/guide/introduction.asciidoc index a54a8d1d..097cf396 100644 --- a/docs/en/gun/2.0/guide/introduction.asciidoc +++ b/docs/en/gun/2.0/guide/introduction.asciidoc @@ -21,7 +21,7 @@ Gun is developed for Erlang/OTP 22.0 and newer. Gun uses the ISC License. ---- -Copyright (c) 2013-2019, Loïc Hoguin <[email protected]> +Copyright (c) 2013-2020, Loïc Hoguin <[email protected]> Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above diff --git a/docs/en/gun/2.0/guide/introduction/index.html b/docs/en/gun/2.0/guide/introduction/index.html index 6f8f0006..8a12d196 100644 --- a/docs/en/gun/2.0/guide/introduction/index.html +++ b/docs/en/gun/2.0/guide/introduction/index.html @@ -71,7 +71,7 @@ <p>Gun is developed for Erlang/OTP 22.0 and newer.</p> <h2 id="_license">License</h2> <p>Gun uses the ISC License.</p> -<div class="listingblock"><div class="content"><pre>Copyright (c) 2013-2019, Loïc Hoguin <[email protected]> +<div class="listingblock"><div class="content"><pre>Copyright (c) 2013-2020, Loïc Hoguin <[email protected]> Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above diff --git a/docs/en/gun/2.0/guide/migrating_from_1.3.asciidoc b/docs/en/gun/2.0/guide/migrating_from_1.3.asciidoc index a20a0582..0afd19eb 100644 --- a/docs/en/gun/2.0/guide/migrating_from_1.3.asciidoc +++ b/docs/en/gun/2.0/guide/migrating_from_1.3.asciidoc @@ -1,12 +1,16 @@ [appendix] == Migrating from Gun 1.3 to 2.0 -Gun 2.0 adds many features including graceful shutdown, -flow control for data messages, event handlers, support -for tunneling TLS connections through TLS proxies, Socks -proxy support, and much more. It has only a limited -number of breaking changes compared to the previous -version. +Gun 2.0 includes state of the art tunnel support. With +Gun 2.0 it is possible to make request or data go through +any number of proxy endpoints using any combination of +TCP or TLS transports and HTTP/1.1, HTTP/2 or SOCKS5 +protocols. All combinations of the scenario Proxy1 -> +Proxy2 -> Origin are tested and known to work. + +Gun 2.0 adds many more features such as Websocket over +HTTP/2, a built-in cookie store, graceful shutdown, flow +control for data messages, event handlers and more. Gun 2.0 greatly improves the HTTP/2 performance when it comes to receiving large response bodies; and when receiving @@ -63,21 +67,21 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. timeout options for each of the steps, and the transport options had to be split into `tcp_opts` and `tls_opts`. -* Gun can connect through any number of HTTP, HTTPS, Socks - or secure Socks proxies, including when Socks proxies are - located after the HTTP(S) proxies. The ultimate endpoint +* Gun now supports connecting through SOCKS proxies, + including secure SOCKS proxies. Both unauthenticated + and username/password authentication are supported. + +* Gun can connect through any number of HTTP, HTTPS, SOCKS + or secure SOCKS proxies, including SOCKS proxies + located after HTTP(S) proxies. The ultimate endpoint may be using any protocol, including plain TCP, TLS, HTTP/1.1 or HTTP/2. -* Gun now supports connecting through Socks5 proxies, - including secure Socks proxies. Both unauthenticated - and username/password authentication are supported. - * When specifying which protocols to use, options can now be provided specific to those protocols. It is now possible to have separate HTTP options for an HTTP proxy and the origin HTTP server, for example. - See the new `protocols()` type for details. + See the new `gun:protocols()` type for details. * Gun can now be used to send and receive raw data, as if it was just a normal socket. This can be @@ -91,6 +95,8 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. * Header names may now be provided as binary, string or atom. +* Gun now automatically lowercases provided header names. + * Many HTTP/2 options have been added, allowing great control over how Gun and the remote endpoint are using the HTTP/2 connection. They can be used to @@ -98,11 +104,11 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. example. * It is now possible to send many Websocket frames in - a single `gun:ws_send/2` call. + a single `gun:ws_send/3` call. -* Gun will now send Websocket ping frames automatically +* Gun may now send Websocket ping frames automatically at intervals determined by the `keepalive` option. It - defaults to 5 seconds. + is disabled by default. * A new `silence_pings` option can be set to `false` to receive all ping and pong frames when using Websocket. @@ -124,13 +130,13 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. connection without using Gun's supervisor. It defaults to `true`. -* Gun now automatically lowercases provided header names. - * Many improvements have been done to postpone or reject requests and other operations while in the wrong state (for example during state transitions when switching protocols or connecting to proxies). +* Update Cowlib to 2.10.0. + === Features removed * Gun used to reject operations by processes that were not @@ -143,8 +149,8 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. removed. It was previously deprecated in favor of `protocols`. * The `keepalive` timeout is now disabled by default - for HTTP/1.1. To be perfectly clear, this is unrelated - to the HTTP/1.1 keep-alive mechanism. + for HTTP/1.1 and HTTP/2. To be perfectly clear, this + is unrelated to the HTTP/1.1 keep-alive mechanism. === Functions added @@ -161,20 +167,32 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. === Functions modified -* The function `gun:info/1` now returns the owner of the connection. +* The function `gun:info/1` now returns the owner of the + connection as well as the cookie store. * The functions `gun:await/2,3,4`, `gun:await_body/2,3,4` and `gun:await_up/1,2,3` now distinguish the error types. They can be a timeout, a connection error, a stream error or a down error (when the Gun process exited while waiting). -* The functions `gun:await/2,3,4` will now receive upgrade and - Websocket messages and return them. +* The functions `gun:await/2,3,4` will now receive upgrades, + tunnel up and Websocket messages and return them. + +* Requests may now include the `tunnel` option to send the + request on a specific tunnel. * The functions `gun:request/4,5,6` have been replaced with `gun:headers/4,5` and `gun:request/5,6`. This provides a cleaner separation between requests that are followed by - a body separately from those that don't. + a body and those that don't. + +* The function `gun:ws_send/2` has been replaced with the + function `gun:ws_send/3`. The stream reference for the + corresponding Websocket upgrade request must now be given. + +=== Messages added + +* The `gun_tunnel_up` message has been added. === Messages modified @@ -189,6 +207,8 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. contribute to a response smuggling security vulnerability when Gun is used inside a proxy. +* Gun will now better detect connection closes in some cases. + * Gun will no longer send duplicate connection-wide `gun_error` messages to the same process. @@ -199,6 +219,9 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. CONNECT over TLS has been corrected. It was mistakenly not enabling HTTP/2. +* Protocol options provided for a tunnel destination were + sometimes ignored. This should no longer be the case. + * Gun will no longer send an empty HTTP/2 DATA frame when there is no request body. It was not necessary. @@ -218,7 +241,7 @@ Gun 2.0 requires Erlang/OTP 22.0 or greater. * Gun will no longer attempt to send empty data chunks. When using HTTP/1.1 chunked transfer-encoding this caused the - request body to terminated, even when `nofin` was given. + request body to end, even when `nofin` was given. * Gun now always retries connecting immediately when the connection goes down. diff --git a/docs/en/gun/2.0/guide/migrating_from_1.3/index.html b/docs/en/gun/2.0/guide/migrating_from_1.3/index.html index 6d9a22a9..65ea2bf5 100644 --- a/docs/en/gun/2.0/guide/migrating_from_1.3/index.html +++ b/docs/en/gun/2.0/guide/migrating_from_1.3/index.html @@ -62,7 +62,8 @@ <h1 class="lined-header"><span>Migrating from Gun 1.3 to 2.0</span></h1> -<p>Gun 2.0 adds many features including graceful shutdown, flow control for data messages, event handlers, support for tunneling TLS connections through TLS proxies, Socks proxy support, and much more. It has only a limited number of breaking changes compared to the previous version.</p> +<p>Gun 2.0 includes state of the art tunnel support. With Gun 2.0 it is possible to make request or data go through any number of proxy endpoints using any combination of TCP or TLS transports and HTTP/1.1, HTTP/2 or SOCKS5 protocols. All combinations of the scenario Proxy1 -> Proxy2 -> Origin are tested and known to work.</p> +<p>Gun 2.0 adds many more features such as Websocket over HTTP/2, a built-in cookie store, graceful shutdown, flow control for data messages, event handlers and more.</p> <p>Gun 2.0 greatly improves the HTTP/2 performance when it comes to receiving large response bodies; and when receiving response bodies from many separate requests concurrently.</p> <p>Gun now shares much of its HTTP/2 code with Cowboy, including the HTTP/2 state machine. Numerous issues were fixed as a result because the Cowboy implementation was much more advanced.</p> <p>The Gun connection process is now implemented using <code>gen_statem</code>.</p> @@ -78,11 +79,11 @@ </li> <li>In order to get separate events when connecting, the domain lookup, connection and TLS handshakes are now performed separately by Gun. As a result, there exists three separate timeout options for each of the steps, and the transport options had to be split into <code>tcp_opts</code> and <code>tls_opts</code>. </li> -<li>Gun can connect through any number of HTTP, HTTPS, Socks or secure Socks proxies, including when Socks proxies are located after the HTTP(S) proxies. The ultimate endpoint may be using any protocol, including plain TCP, TLS, HTTP/1.1 or HTTP/2. +<li>Gun now supports connecting through SOCKS proxies, including secure SOCKS proxies. Both unauthenticated and username/password authentication are supported. </li> -<li>Gun now supports connecting through Socks5 proxies, including secure Socks proxies. Both unauthenticated and username/password authentication are supported. +<li>Gun can connect through any number of HTTP, HTTPS, SOCKS or secure SOCKS proxies, including SOCKS proxies located after HTTP(S) proxies. The ultimate endpoint may be using any protocol, including plain TCP, TLS, HTTP/1.1 or HTTP/2. </li> -<li>When specifying which protocols to use, options can now be provided specific to those protocols. It is now possible to have separate HTTP options for an HTTP proxy and the origin HTTP server, for example. See the new <code>protocols()</code> type for details. +<li>When specifying which protocols to use, options can now be provided specific to those protocols. It is now possible to have separate HTTP options for an HTTP proxy and the origin HTTP server, for example. See the new <code>gun:protocols()</code> type for details. </li> <li>Gun can now be used to send and receive raw data, as if it was just a normal socket. This can be useful when needing to connect through a number of HTTP/Socks proxies, allowing the use of Gun's great proxying capabilities (including TLS over TLS) for any sort of protocols. This can also be useful when performing HTTP/1.1 Upgrade to custom protocols. </li> @@ -90,11 +91,13 @@ </li> <li>Header names may now be provided as binary, string or atom. </li> +<li>Gun now automatically lowercases provided header names. +</li> <li>Many HTTP/2 options have been added, allowing great control over how Gun and the remote endpoint are using the HTTP/2 connection. They can be used to improve performance or lower the memory usage, for example. </li> -<li>It is now possible to send many Websocket frames in a single <code>gun:ws_send/2</code> call. +<li>It is now possible to send many Websocket frames in a single <code>gun:ws_send/3</code> call. </li> -<li>Gun will now send Websocket ping frames automatically at intervals determined by the <code>keepalive</code> option. It defaults to 5 seconds. +<li>Gun may now send Websocket ping frames automatically at intervals determined by the <code>keepalive</code> option. It is disabled by default. </li> <li>A new <code>silence_pings</code> option can be set to <code>false</code> to receive all ping and pong frames when using Websocket. They are typically not needed and therefore silent by default. </li> @@ -106,17 +109,17 @@ </li> <li>A new option <code>supervise</code> can be used to start a Gun connection without using Gun's supervisor. It defaults to <code>true</code>. </li> -<li>Gun now automatically lowercases provided header names. -</li> <li>Many improvements have been done to postpone or reject requests and other operations while in the wrong state (for example during state transitions when switching protocols or connecting to proxies). </li> +<li>Update Cowlib to 2.10.0. +</li> </ul> <h2 id="_features_removed">Features removed</h2> <ul><li>Gun used to reject operations by processes that were not the owner of the connection. This behavior has been removed. In general the caller of a request or other operation will receive the relevant messages unless the <code>reply_to</code> option is used. </li> <li>The <code>connect_destination()</code> option <code>protocol</code> has been removed. It was previously deprecated in favor of <code>protocols</code>. </li> -<li>The <code>keepalive</code> timeout is now disabled by default for HTTP/1.1. To be perfectly clear, this is unrelated to the HTTP/1.1 keep-alive mechanism. +<li>The <code>keepalive</code> timeout is now disabled by default for HTTP/1.1 and HTTP/2. To be perfectly clear, this is unrelated to the HTTP/1.1 keep-alive mechanism. </li> </ul> <h2 id="_functions_added">Functions added</h2> @@ -128,13 +131,21 @@ </li> </ul> <h2 id="_functions_modified">Functions modified</h2> -<ul><li>The function <code>gun:info/1</code> now returns the owner of the connection. +<ul><li>The function <code>gun:info/1</code> now returns the owner of the connection as well as the cookie store. </li> <li>The functions <code>gun:await/2,3,4</code>, <code>gun:await_body/2,3,4</code> and <code>gun:await_up/1,2,3</code> now distinguish the error types. They can be a timeout, a connection error, a stream error or a down error (when the Gun process exited while waiting). </li> -<li>The functions <code>gun:await/2,3,4</code> will now receive upgrade and Websocket messages and return them. +<li>The functions <code>gun:await/2,3,4</code> will now receive upgrades, tunnel up and Websocket messages and return them. +</li> +<li>Requests may now include the <code>tunnel</code> option to send the request on a specific tunnel. </li> -<li>The functions <code>gun:request/4,5,6</code> have been replaced with <code>gun:headers/4,5</code> and <code>gun:request/5,6</code>. This provides a cleaner separation between requests that are followed by a body separately from those that don't. +<li>The functions <code>gun:request/4,5,6</code> have been replaced with <code>gun:headers/4,5</code> and <code>gun:request/5,6</code>. This provides a cleaner separation between requests that are followed by a body and those that don't. +</li> +<li>The function <code>gun:ws_send/2</code> has been replaced with the function <code>gun:ws_send/3</code>. The stream reference for the corresponding Websocket upgrade request must now be given. +</li> +</ul> +<h2 id="_messages_added">Messages added</h2> +<ul><li>The <code>gun_tunnel_up</code> message has been added. </li> </ul> <h2 id="_messages_modified">Messages modified</h2> @@ -144,12 +155,16 @@ <h2 id="_bugs_fixed">Bugs fixed</h2> <ul><li><strong>POTENTIAL SECURITY VULNERABILITY</strong>: Fix transfer-encoding precedence over content-length in responses. This bug may contribute to a response smuggling security vulnerability when Gun is used inside a proxy. </li> +<li>Gun will now better detect connection closes in some cases. +</li> <li>Gun will no longer send duplicate connection-wide <code>gun_error</code> messages to the same process. </li> <li>Gun no longer crashes when trying to upgrade to Websocket over a connection restricted to HTTP/1.0. </li> <li>The default value for the preferred protocols when using CONNECT over TLS has been corrected. It was mistakenly not enabling HTTP/2. </li> +<li>Protocol options provided for a tunnel destination were sometimes ignored. This should no longer be the case. +</li> <li>Gun will no longer send an empty HTTP/2 DATA frame when there is no request body. It was not necessary. </li> <li>Gun will no longer error out when the owner process exits. The error reason will now be a <code>shutdown</code> tuple instead. @@ -160,7 +175,7 @@ </li> <li>Hostnames can now be provided as atom as stated by the documentation. </li> -<li>Gun will no longer attempt to send empty data chunks. When using HTTP/1.1 chunked transfer-encoding this caused the request body to terminated, even when <code>nofin</code> was given. +<li>Gun will no longer attempt to send empty data chunks. When using HTTP/1.1 chunked transfer-encoding this caused the request body to end, even when <code>nofin</code> was given. </li> <li>Gun now always retries connecting immediately when the connection goes down. </li> @@ -192,8 +207,8 @@ <nav style="margin:1em 0"> - <a style="float:left" href="https://ninenines.eu/docs/en/gun/2.0/guide/websocket/"> - Websocket + <a style="float:left" href="https://ninenines.eu/docs/en/gun/2.0/guide/internals_tls_over_tls/"> + Internals: TLS over TLS </a> diff --git a/docs/en/gun/2.0/guide/protocols.asciidoc b/docs/en/gun/2.0/guide/protocols.asciidoc index d2529e38..cd6de2cf 100644 --- a/docs/en/gun/2.0/guide/protocols.asciidoc +++ b/docs/en/gun/2.0/guide/protocols.asciidoc @@ -65,10 +65,6 @@ cancellation mechanism which allows Gun to inform the server to stop sending a response for this particular request, saving resources. -It is not currently possible to upgrade an HTTP/2 connection -to Websocket. Support for this will be added in a future -release. - === Websocket Websocket is a binary protocol built on top of HTTP that @@ -76,12 +72,9 @@ allows asynchronous concurrent communication between the client and the server. A Websocket server can push data to the client at any time. -Websocket is only available as a connection upgrade over -an HTTP/1.1 connection. - -Once the Websocket connection is established, the only -operation available on this connection is sending Websocket -frames using `gun:ws_send/2`. +Once the Websocket connection is established over an HTTP/1.1 +connection, the only operation available on this connection +is sending Websocket frames using `gun:ws_send/3`. Gun will send a `gun_ws` message for every frame received. @@ -108,7 +101,7 @@ current protocol. | await_body | yes | yes | no | flush | yes | yes | no | cancel | yes | yes | no -| ws_upgrade | yes | no | no +| ws_upgrade | yes | yes | no | ws_send | no | no | yes |=== @@ -122,6 +115,6 @@ current protocol. | gun_data | yes | yes | no | gun_trailers | yes | yes | no | gun_error | yes | yes | yes -| gun_upgrade | yes | no | no +| gun_upgrade | yes | yes | no | gun_ws | no | no | yes |=== diff --git a/docs/en/gun/2.0/guide/protocols/index.html b/docs/en/gun/2.0/guide/protocols/index.html index ef0c58fd..ccf854d0 100644 --- a/docs/en/gun/2.0/guide/protocols/index.html +++ b/docs/en/gun/2.0/guide/protocols/index.html @@ -76,11 +76,9 @@ <p>The HTTP/2 interface is very similar to HTTP/1.1, so this section instead focuses on the differences in the interface for the two protocols.</p> <p>Gun will send <code>gun_push</code> messages for every push received. They will always be sent before the <code>gun_response</code> message. They can be ignored safely if they are not needed, or they can be canceled.</p> <p>The <code>gun:cancel/2</code> function will use the HTTP/2 stream cancellation mechanism which allows Gun to inform the server to stop sending a response for this particular request, saving resources.</p> -<p>It is not currently possible to upgrade an HTTP/2 connection to Websocket. Support for this will be added in a future release.</p> <h2 id="_websocket">Websocket</h2> <p>Websocket is a binary protocol built on top of HTTP that allows asynchronous concurrent communication between the client and the server. A Websocket server can push data to the client at any time.</p> -<p>Websocket is only available as a connection upgrade over an HTTP/1.1 connection.</p> -<p>Once the Websocket connection is established, the only operation available on this connection is sending Websocket frames using <code>gun:ws_send/2</code>.</p> +<p>Once the Websocket connection is established over an HTTP/1.1 connection, the only operation available on this connection is sending Websocket frames using <code>gun:ws_send/3</code>.</p> <p>Gun will send a <code>gun_ws</code> message for every frame received.</p> <h2 id="_summary">Summary</h2> <p>The two following tables summarize the supported operations and the messages Gun sends depending on the connection's current protocol.</p> @@ -157,7 +155,7 @@ </tr> <tr><td>ws_upgrade</td> <td>yes</td> -<td>no</td> +<td>yes</td> <td>no</td> </tr> <tr><td>ws_send</td> @@ -204,7 +202,7 @@ </tr> <tr><td>gun_upgrade</td> <td>yes</td> -<td>no</td> +<td>yes</td> <td>no</td> </tr> <tr><td>gun_ws</td> diff --git a/docs/en/gun/2.0/guide/websocket.asciidoc b/docs/en/gun/2.0/guide/websocket.asciidoc index 287b3f73..ba06e2c7 100644 --- a/docs/en/gun/2.0/guide/websocket.asciidoc +++ b/docs/en/gun/2.0/guide/websocket.asciidoc @@ -49,7 +49,7 @@ undocumented and must be set to `gun_ws_h`. .Upgrade to Websocket with protocol negotiation [source,erlang] ---- -gun:ws_upgrade(ConnPid, "/websocket", [] +StreamRef = gun:ws_upgrade(ConnPid, "/websocket", [] #{protocols => [{<<"xmpp">>, gun_ws_h}]}). ---- @@ -88,18 +88,18 @@ Once the Websocket upgrade has completed successfully, you no longer have access to functions for performing requests. You can only send and receive Websocket messages. -Use `gun:ws_send/2` to send messages to the server. +Use `gun:ws_send/3` to send messages to the server. .Send a text frame [source,erlang] ---- -gun:ws_send(ConnPid, {text, "Hello!"}). +gun:ws_send(ConnPid, StreamRef, {text, "Hello!"}). ---- .Send a text frame, a binary frame and then close the connection [source,erlang] ---- -gun:ws_send(ConnPid, [ +gun:ws_send(ConnPid, StreamRef, [ {text, "Hello!"}, {binary, BinaryValue}, close diff --git a/docs/en/gun/2.0/guide/websocket/index.html b/docs/en/gun/2.0/guide/websocket/index.html index 2d6b8c9c..b407ad4e 100644 --- a/docs/en/gun/2.0/guide/websocket/index.html +++ b/docs/en/gun/2.0/guide/websocket/index.html @@ -91,7 +91,7 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><tt><b><font color="#000000">gun:ws_upgrade</font></b>(<font color="#009900">ConnPid</font>, <font color="#FF0000">"/websocket"</font>, [] +<pre><tt><font color="#009900">StreamRef</font> <font color="#990000">=</font> <b><font color="#000000">gun:ws_upgrade</font></b>(<font color="#009900">ConnPid</font>, <font color="#FF0000">"/websocket"</font>, [] #{<font color="#0000FF">protocols</font> <font color="#990000">=></font> [{<font color="#990000"><<</font><font color="#FF0000">"xmpp"</font><font color="#990000">>></font>, <font color="#FF6600">gun_ws_h</font>}]})<font color="#990000">.</font></tt></pre> </div></div> <p>The upgrade will fail if the server cannot satisfy the protocol negotiation.</p> @@ -115,20 +115,20 @@ http://www.gnu.org/software/src-highlite --> </div></div> <h2 id="_sending_data">Sending data</h2> <p>Once the Websocket upgrade has completed successfully, you no longer have access to functions for performing requests. You can only send and receive Websocket messages.</p> -<p>Use <code>gun:ws_send/2</code> to send messages to the server.</p> +<p>Use <code>gun:ws_send/3</code> to send messages to the server.</p> <div class="listingblock"><div class="title">Send a text frame</div> <div class="content"><!-- Generator: GNU source-highlight 3.1.9 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><tt><b><font color="#000000">gun:ws_send</font></b>(<font color="#009900">ConnPid</font>, {<font color="#FF6600">text</font>, <font color="#FF0000">"Hello!"</font>})<font color="#990000">.</font></tt></pre> +<pre><tt><b><font color="#000000">gun:ws_send</font></b>(<font color="#009900">ConnPid</font>, <font color="#009900">StreamRef</font>, {<font color="#FF6600">text</font>, <font color="#FF0000">"Hello!"</font>})<font color="#990000">.</font></tt></pre> </div></div> <div class="listingblock"><div class="title">Send a text frame, a binary frame and then close the connection</div> <div class="content"><!-- Generator: GNU source-highlight 3.1.9 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><tt><b><font color="#000000">gun:ws_send</font></b>(<font color="#009900">ConnPid</font>, [ +<pre><tt><b><font color="#000000">gun:ws_send</font></b>(<font color="#009900">ConnPid</font>, <font color="#009900">StreamRef</font>, [ {<font color="#FF6600">text</font>, <font color="#FF0000">"Hello!"</font>}, {<b><font color="#000080">binary</font></b>, <font color="#009900">BinaryValue</font>}, <font color="#FF6600">close</font> @@ -165,8 +165,8 @@ http://www.gnu.org/software/src-highlite --> - <a style="float:right" href="https://ninenines.eu/docs/en/gun/2.0/guide/migrating_from_1.3/"> - Migrating from Gun 1.3 to 2.0 + <a style="float:right" href="https://ninenines.eu/docs/en/gun/2.0/guide/internals_tls_over_tls/"> + Internals: TLS over TLS </a> </nav> |