inets: httpd - Add chunk handling of client data

If the client uses PUT or POST to send a HTTP body the server
currently will gather the whole body in memory before passing it on to
the mod-callback. For legacy reasons it also converts the binary body
to a list, this is really bad for memory usage!

Add new option max_client_body_chunk to enable chunked handling of
content-length requests. Also make it possible for chunked-encoded data
to be propagated in portions to the mod-callback.

Conflicts:
	lib/inets/test/httpd_SUITE.erl

2 files changed, 59 insertions, 23 deletions

diff --git a/lib/inets/doc/src/httpd.xml b/lib/inets/doc/src/httpd.xml
index d74635fc01..edf8731a82 100644
--- a/lib/inets/doc/src/httpd.xml
+++ b/lib/inets/doc/src/httpd.xml
@@ -279,7 +279,18 @@
 	requests defined by <c>max_keep_alive_requests</c>, the server 
 	closes the connection. The server closes it even if there are 
 	queued request. Default is no limit.</p>
-      </item>   
+      </item>
+
+      
+      <tag><marker id="max_client_body_chunk"></marker>{max_client_body_chunk, integer()}</tag>
+      <item>
+	<p>Enforces chunking of a HTTP PUT or POST body data to be deliverd
+	to the mod_esi callback. Note this is not supported for mod_cgi. 
+	Default is no limit e.i the whole body is deliverd as one entity, which could
+	be very memory consuming. <seealso marker="mod_esi">mod_esi(3)</seealso>.
+	</p>
+      </item>
+      
     </taglist>
     
     <marker id="props_admin"></marker>
diff --git a/lib/inets/doc/src/mod_esi.xml b/lib/inets/doc/src/mod_esi.xml
index 46cc796c8a..e2f0758cdf 100644
--- a/lib/inets/doc/src/mod_esi.xml
+++ b/lib/inets/doc/src/mod_esi.xml
@@ -121,35 +121,61 @@
 
  <funcs>
    <func>
-     <name>Module:Function(SessionID, Env, Input)-> _ </name>
+     <name>Module:Function(SessionID, Env, Input)-> {continue, State} | _ </name>
      <fsummary>Creates a dynamic web page and returns it chunk by chunk 
      to the server process by calling <c>mod_esi:deliver/2</c>.</fsummary>
      <type>
        <v>SessionID = term()</v>
        <v>Env = env()</v>
-       <v>Input = string()</v>
+       <v>Input = string() | chunked_data()</v>
+       <v>chunked_data() = {first, Data::binary()} |
+       {continue, Data::binary(), State::term()} |
+       {last, Data::binary(), State::term()} </v>
+       <v>State = term()</v>
      </type>
       <desc>
         <p><c>Module</c> must be found in the code path and export
         <c>Function</c> with an arity of three. An <c>erlScriptAlias</c> must
         also be set up in the configuration file for the web server.</p>
-        <p>If the HTTP request is a 'post' request and a body is sent,
-        <c>content_length</c> is the length of the posted
-        data. If 'get' is used, <c>query_string</c> is the data after
-        <em>?</em> in the URL.</p>
-        <p><c>ParsedHeader</c> is the HTTP request as a key-value tuple
-        list. The keys in <c>ParsedHeader</c> are in lower case.</p>
-        <p><c>SessionID</c> is an identifier
-        the server uses when <c>deliver/2</c> is called. Do not
-        assume anything about the datatype.</p>
-        <p>Use this callback function to generate dynamic web
-        content dynamically. When a part of the page is generated, send the
-        data back to the client through <c>deliver/2</c>. Notice
-        that the first chunk of data sent to the client must at
-        least contain all HTTP header fields that the response
-        will generate. If the first chunk does not contain the
-        <em>end of HTTP header</em>, that is, <c>"\r\n\r\n",</c>
-        the server assumes that no HTTP header fields will be generated.</p>
+
+	<p><c>mod_esi:deliver/2</c> shall be used to generate the response
+	to the client and <c>SessionID</c> is an identifier that shall by used when
+	calling this function, do not assume anything about
+	the datatype. This function may be called
+	several times to chunk the the respons data. Notice that the
+	first chunk of data sent to the client must at least contain
+	all HTTP header fields that the response will generate. If the
+	first chunk does not contain the <em>end of HTTP header</em>,
+	that is, <c>"\r\n\r\n",</c> the server assumes that no HTTP
+	header fields will be generated.</p>
+
+	<p><c>Env</c> environment data of the request see description above.</p>
+		
+	<p><c>Input</c> is query data of a GET request or the body of
+	a PUT or POST request. The default behavior (legacy reasons)
+	for delivering the body, is that the whole body is gathered and
+	converted to a string. But if the httpd config parameter
+	<seealso
+	marker="httpd#max_client_body_chunk">max_client_body_chunk</seealso>
+	is set, the body will be delivered as binary chunks
+	instead. The maximum size of the chunks is either <seealso
+	marker="httpd#max_client_body_chunk">max_client_body_chunk</seealso>
+	or decide by the client if it uses HTTP chunked encoding
+	to send the body. When using the chunking
+	mechanism this callback must return {continue, State::term()}
+	for all calls where <c>Input</c> is <c>{first,
+	Data::binary()}</c> or <c>{continue, Data::binary(),
+	State::term()}</c>. When <c>Input</c> is <c>{last,
+	Data::binary(), State::term()}</c> the return value will be ignored.
+	<note><p>Note that if the body is
+	small all data may be delivered in only one chunk and then the
+	callback will be called with {last, Data::binary(), undefined}
+	without getting called with <c>{first,
+	Data::binary()}</c>.</p></note> The input <c>State</c> is
+	the last returned <c>State</c>, in it the callback can include
+	any data that it needs to keep track of when handling the chunks.
+	</p>
+	<p></p>
       </desc>
    </func>
 
@@ -159,14 +185,13 @@
      This function is deprecated and is only kept for backwards compatibility.</fsummary>
      <type>
        <v>Env = env()</v>
-       <v>Input = string()</v>
+       <v>Input = string() </v>
        <v>Response = string()</v>
      </type>
       <desc>
         <p>This callback format consumes much memory, as the
         whole response must be generated before it is sent to the
-        user. This function is deprecated and is only kept for backwards
-        compatibility.
+        user. This callback format is deprecated. 
         For new development, use <c>Module:Function/3</c>.</p>
       </desc>
    </func>