diff options
Diffstat (limited to 'archives/extend/attachments/20121221')
-rw-r--r-- | archives/extend/attachments/20121221/8bfb2f11/attachment.html | 44 | ||||
-rw-r--r-- | archives/extend/attachments/20121221/945f636e/attachment.html | 108 |
2 files changed, 152 insertions, 0 deletions
diff --git a/archives/extend/attachments/20121221/8bfb2f11/attachment.html b/archives/extend/attachments/20121221/8bfb2f11/attachment.html new file mode 100644 index 00000000..9650a876 --- /dev/null +++ b/archives/extend/attachments/20121221/8bfb2f11/attachment.html @@ -0,0 +1,44 @@ +<tt> +In my preliminary testing, I used Jmeter this morning since it's an easy GUI load testing app and this is what I am seeing:<div><br></div><div>With R15B03-01 [smp:4:4] [async-threads:4] [hipe] [kernel-poll:true], when I establish 1K concurrent connections via HTTPS, each connection takes up about 68K of memory.<br><br> +<div><br></div><div>Unfortunately, after about 1050-1200 connections, on my test server the Erlang scheduler jumps to 100% CPU utilization on all 4 schedulers, while up to that point the scheduler's load was oscillating up and down. Using the Observer, there is only 1 ssl_connection_sup in the ssl application, having to deal with 1000+ gen_fsm workers, so that might be the bottleneck. Since the ulimit on my server is 50000 I don't think I am hitting any type of file handler's limit.</div><br> +<div><br></div><div>Loïc and the group, am I missing some setting that is causing the scheduler to go to 100% CPU and the run que in observer to be 99?</div><div><br></div><div><table cellpadding="0" class="cf an5" id=":zc" style="border-collapse:collapse;outline:none;overflow:hidden;table-layout:fixed;width:202px;color:rgb(34,34,34);font-family:arial,sans-serif;text-align:start;background-color:rgb(255,255,255)"><br> +<tbody><tr><td class="anQ" style="margin:0px;overflow:hidden;text-overflow:ellipsis;vertical-align:top">Sincerely,<br><br>rambocoder</td></tr></tbody></table></div><div><br><br><div class="gmail_quote">On Fri, Dec 21, 2012 at 6:45 AM, Loïc Hoguin <span dir="ltr"><<a href="mailto:[email protected]" target="_blank">[email protected]</a>></span> wrote:<br><br> +<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 12/21/2012 04:34 AM, rambocoder wrote:<br><br> +<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br> +Does anybody know either from benchmarks or real world data what is the<br><br> +average memory footprint of each concurrent HTTPS connection to cowboy?<br><br> +</blockquote><br> +<br></div><br> +I don't have anything, sorry. I'm guessing it consumes a lot more than TCP though.<div class="im"><br><br> +<br><br> +<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br> +SSL app in Erlang reuses SSL session-ids so I am not sure if the Apache<br><br> +Bench I test with reuses the session id or it does not.<br><br> +</blockquote><br> +<br></div><br> +I wouldn't know, but I wouldn't trust Apache Bench doing the right thing. Any other benchmark tool usually works better in my experience.<div class="im"><br><br> +<br><br> +<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br> +BTW, what makes an erlang api "documented" vs "undocumented". For<br><br> +example ssl:session_info/1 function here (<br><br> +<a href="https://github.com/erlang/otp/blob/maint/lib/ssl/src/ssl.erl#L411" target="_blank">https://github.com/erlang/otp/<u></u>blob/maint/lib/ssl/src/ssl.<u></u>erl#L411</a> ) has<br><br> +a spec and a short doc, but session_info is not described<br><br> +<a href="http://www.erlang.org/doc/man/ssl.html" target="_blank">http://www.erlang.org/doc/man/<u></u>ssl.html</a> .ssl:session_info/1 is a useful<br><br> +function to be able to track if the load generator is reusing the SSL<br><br> +session_id or it is generating new one, because that would make all the<br><br> +difference during measurement due to Erlang caching SSL sessions by default.<br><br> +</blockquote><br> +<br></div><br> +The documentation is separate (they're not using edoc). It's perhaps not deemed useful enough for documenting it. I wouldn't worry about using it for measurements though.<br><br> +<br><br> +Try asking Ingela on the ML about it, perhaps they just forgot to document it.<span class="HOEnZb"><font color="#888888"><br><br> +<br><br> +-- <br><br> +Loďc Hoguin<br><br> +Erlang Cowboy<br><br> +Nine Nines<br><br> +<a href="http://ninenines.eu" target="_blank">http://ninenines.eu</a><br><br> +<br><br> +</font></span></blockquote></div><br></div></div><br> + +</tt> diff --git a/archives/extend/attachments/20121221/945f636e/attachment.html b/archives/extend/attachments/20121221/945f636e/attachment.html new file mode 100644 index 00000000..e10a9807 --- /dev/null +++ b/archives/extend/attachments/20121221/945f636e/attachment.html @@ -0,0 +1,108 @@ +<tt> +Long story short, I solved the problem by adding {max_connections, 50000} to cowboy:start_https because it default to 1024 at <a href="https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30">https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30</a><div><br> +<br></div><div>However, before I figured out that setting, I did run eprof and these are the function calls it was spending most of it's time on<br><div><div><br></div><div><br></div><div><div>FUNCTION CALLS % TIME [uS / CALLS]</div><br> +<div>-------- ----- --- ---- [----------]</div></div><div>dict:get_slot/2 174 1.73 1658 [ 9.53]</div><div><div>dict:on_bucket/3 171 1.77 1701 [ 9.95]</div><br> +<div>erlang:setelement/3 684 3.23 3098 [ 4.53]</div><div>dict:store_bkt_val/3 600 5.24 5028 [ 8.38]</div><div><br></div><div>Then I ran etop and it showed that ranch_acceptor:maybe_wait had the most reductions were, so I looked at the code in that <a href="https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72">https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72</a> and realized that like a newb I did not set the maximum connections for the listener :)</div><br> +<div><br></div><div>Problem solved. Looks like I won't need to put HAProxy in front of Cowboy after all.</div><div><br></div><div>Thank you,</div><div><br></div><div>rambocoder</div><br><div class="gmail_quote">On Fri, Dec 21, 2012 at 11:51 AM, Loïc Hoguin <span dir="ltr"><<a href="mailto:[email protected]" target="_blank">[email protected]</a>></span> wrote:<br><br> +<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can you try enabling eprof to see where the VM spends its time?<div class="im"><br><br> +<br><br> +On 12/21/2012 05:49 PM, rambocoder wrote:<br><br> +</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br> +In my preliminary testing, I used Jmeter this morning since it's an<br><br> +easy GUI load testing app and this is what I am seeing:<br><br> +<br><br> +With R15B03-01 [smp:4:4] [async-threads:4] [hipe] [kernel-poll:true],<br><br> +when I establish 1K concurrent connections via HTTPS, each connection<br><br> +takes up about 68K of memory.<br><br> +<br><br> +Unfortunately, after about 1050-1200 connections, on my test server the<br><br> +Erlang scheduler jumps to 100% CPU utilization on all 4 schedulers,<br><br> +while up to that point the scheduler's load was oscillating up and down.<br><br> +Using the Observer, there is only 1 ssl_connection_sup in the ssl<br><br> +application, having to deal with 1000+ gen_fsm workers, so that might be<br><br> +the bottleneck. Since the ulimit on my server is 50000 I don't think I<br><br> +am hitting any type of file handler's limit.<br><br> +<br><br> +Loïc and the group, am I missing some setting that is causing the<br><br> +scheduler to go to 100% CPU and the run que in observer to be 99?<br><br> +<br><br> +Sincerely,<br><br> +<br><br> +rambocoder<br><br> +<br><br> +<br><br> +<br><br> +On Fri, Dec 21, 2012 at 6:45 AM, Loïc Hoguin <<a href="mailto:[email protected]" target="_blank">[email protected]</a><br></div><div class="im"><br> +<mailto:<a href="mailto:[email protected]" target="_blank">[email protected]</a>>> wrote:<br><br> +<br><br> + On 12/21/2012 04:34 AM, rambocoder wrote:<br><br> +<br><br> + Does anybody know either from benchmarks or real world data what<br><br> + is the<br><br> + average memory footprint of each concurrent HTTPS connection to<br><br> + cowboy?<br><br> +<br><br> +<br><br> + I don't have anything, sorry. I'm guessing it consumes a lot more<br><br> + than TCP though.<br><br> +<br><br> +<br><br> + SSL app in Erlang reuses SSL session-ids so I am not sure if the<br><br> + Apache<br><br> + Bench I test with reuses the session id or it does not.<br><br> +<br><br> +<br><br> + I wouldn't know, but I wouldn't trust Apache Bench doing the right<br><br> + thing. Any other benchmark tool usually works better in my experience.<br><br> +<br><br> +<br><br> + BTW, what makes an erlang api "documented" vs "undocumented". For<br><br> + example ssl:session_info/1 function here (<br></div><br> + <a href="https://github.com/erlang/otp/__blob/maint/lib/ssl/src/ssl.__erl#L411" target="_blank">https://github.com/erlang/otp/<u></u>__blob/maint/lib/ssl/src/ssl._<u></u>_erl#L411</a><div class="im"><br><br> + <<a href="https://github.com/erlang/otp/blob/maint/lib/ssl/src/ssl.erl#L411" target="_blank">https://github.com/erlang/<u></u>otp/blob/maint/lib/ssl/src/<u></u>ssl.erl#L411</a>><br><br> + ) has<br><br> + a spec and a short doc, but session_info is not described<br></div><br> + <a href="http://www.erlang.org/doc/man/__ssl.html" target="_blank">http://www.erlang.org/doc/man/<u></u>__ssl.html</a><div class="im"><br><br> + <<a href="http://www.erlang.org/doc/man/ssl.html" target="_blank">http://www.erlang.org/doc/<u></u>man/ssl.html</a>> .ssl:session_info/1 is<br><br> + a useful<br><br> + function to be able to track if the load generator is reusing<br><br> + the SSL<br><br> + session_id or it is generating new one, because that would make<br><br> + all the<br><br> + difference during measurement due to Erlang caching SSL sessions<br><br> + by default.<br><br> +<br><br> +<br><br> + The documentation is separate (they're not using edoc). It's perhaps<br><br> + not deemed useful enough for documenting it. I wouldn't worry about<br><br> + using it for measurements though.<br><br> +<br><br> + Try asking Ingela on the ML about it, perhaps they just forgot to<br><br> + document it.<br><br> +<br><br> + --<br><br> + Loďc Hoguin<br><br> + Erlang Cowboy<br><br> + Nine Nines<br><br> + <a href="http://ninenines.eu" target="_blank">http://ninenines.eu</a><br><br> +<br><br> +<br><br> +<br><br> +<br></div><br> +______________________________<u></u>_________________<br><br> +Extend mailing list<br><br> +<a href="mailto:[email protected]" target="_blank">[email protected]</a><br><br> +<a href="http://lists.ninenines.eu:81/listinfo/extend" target="_blank">http://lists.ninenines.eu:81/<u></u>listinfo/extend</a><br><br> +<br><span class="HOEnZb"><font color="#888888"><br> +</font></span></blockquote><span class="HOEnZb"><font color="#888888"><br> +<br><br> +<br><br> +-- <br><br> +Loïc Hoguin</font></span><div class="HOEnZb"><div class="h5"><br><br> +Erlang Cowboy<br><br> +Nine Nines<br><br> +<a href="http://ninenines.eu" target="_blank">http://ninenines.eu</a><br><br> +<br><br> +</div></div></blockquote></div><br></div></div></div><br> + +</tt> |