Long story short, I solved the problem by adding {max_connections, 50000} to cowboy:start_https because it default to 1024 at <a href="https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30">https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30</a><div>
<br></div><div>However, before I figured out that setting, I did run eprof and these are the function calls it was spending most of it's time on<br><div><div><br></div><div><br></div><div><div>FUNCTION CALLS % TIME [uS / CALLS]</div>
<div>-------- ----- --- ---- [----------]</div></div><div>dict:get_slot/2 174 1.73 1658 [ 9.53]</div><div><div>dict:on_bucket/3 171 1.77 1701 [ 9.95]</div>
<div>erlang:setelement/3 684 3.23 3098 [ 4.53]</div><div>dict:store_bkt_val/3 600 5.24 5028 [ 8.38]</div><div><br></div><div>Then I ran etop and it showed that ranch_acceptor:maybe_wait had the most reductions were, so I looked at the code in that <a href="https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72">https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72</a> and realized that like a newb I did not set the maximum connections for the listener :)</div>
<div><br></div><div>Problem solved. Looks like I won't need to put HAProxy in front of Cowboy after all.</div><div><br></div><div>Thank you,</div><div><br></div><div>rambocoder</div><br><div class="gmail_quote">On Fri, Dec 21, 2012 at 11:51 AM, Loïc Hoguin <span dir="ltr"><<a href="mailto:essen@ninenines.eu" target="_blank">essen@ninenines.eu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can you try enabling eprof to see where the VM spends its time?<div class="im"><br>
<br>
On 12/21/2012 05:49 PM, rambocoder wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
In my preliminary testing, I used Jmeter this morning since it's an<br>
easy GUI load testing app and this is what I am seeing:<br>
<br>
With R15B03-01 [smp:4:4] [async-threads:4] [hipe] [kernel-poll:true],<br>
when I establish 1K concurrent connections via HTTPS, each connection<br>
takes up about 68K of memory.<br>
<br>
Unfortunately, after about 1050-1200 connections, on my test server the<br>
Erlang scheduler jumps to 100% CPU utilization on all 4 schedulers,<br>
while up to that point the scheduler's load was oscillating up and down.<br>
Using the Observer, there is only 1 ssl_connection_sup in the ssl<br>
application, having to deal with 1000+ gen_fsm workers, so that might be<br>
the bottleneck. Since the ulimit on my server is 50000 I don't think I<br>
am hitting any type of file handler's limit.<br>
<br>
Loïc and the group, am I missing some setting that is causing the<br>
scheduler to go to 100% CPU and the run que in observer to be 99?<br>
<br>
Sincerely,<br>
<br>
rambocoder<br>
<br>
<br>
<br>
On Fri, Dec 21, 2012 at 6:45 AM, Loïc Hoguin <<a href="mailto:essen@ninenines.eu" target="_blank">essen@ninenines.eu</a><br></div><div class="im">
<mailto:<a href="mailto:essen@ninenines.eu" target="_blank">essen@ninenines.eu</a>>> wrote:<br>
<br>
On 12/21/2012 04:34 AM, rambocoder wrote:<br>
<br>
Does anybody know either from benchmarks or real world data what<br>
is the<br>
average memory footprint of each concurrent HTTPS connection to<br>
cowboy?<br>
<br>
<br>
I don't have anything, sorry. I'm guessing it consumes a lot more<br>
than TCP though.<br>
<br>
<br>
SSL app in Erlang reuses SSL session-ids so I am not sure if the<br>
Apache<br>
Bench I test with reuses the session id or it does not.<br>
<br>
<br>
I wouldn't know, but I wouldn't trust Apache Bench doing the right<br>
thing. Any other benchmark tool usually works better in my experience.<br>
<br>
<br>
BTW, what makes an erlang api "documented" vs "undocumented". For<br>
example ssl:session_info/1 function here (<br></div>
<a href="https://github.com/erlang/otp/__blob/maint/lib/ssl/src/ssl.__erl#L411" target="_blank">https://github.com/erlang/otp/<u></u>__blob/maint/lib/ssl/src/ssl._<u></u>_erl#L411</a><div class="im"><br>
<<a href="https://github.com/erlang/otp/blob/maint/lib/ssl/src/ssl.erl#L411" target="_blank">https://github.com/erlang/<u></u>otp/blob/maint/lib/ssl/src/<u></u>ssl.erl#L411</a>><br>
) has<br>
a spec and a short doc, but session_info is not described<br></div>
<a href="http://www.erlang.org/doc/man/__ssl.html" target="_blank">http://www.erlang.org/doc/man/<u></u>__ssl.html</a><div class="im"><br>
<<a href="http://www.erlang.org/doc/man/ssl.html" target="_blank">http://www.erlang.org/doc/<u></u>man/ssl.html</a>> .ssl:session_info/1 is<br>
a useful<br>
function to be able to track if the load generator is reusing<br>
the SSL<br>
session_id or it is generating new one, because that would make<br>
all the<br>
difference during measurement due to Erlang caching SSL sessions<br>
by default.<br>
<br>
<br>
The documentation is separate (they're not using edoc). It's perhaps<br>
not deemed useful enough for documenting it. I wouldn't worry about<br>
using it for measurements though.<br>
<br>
Try asking Ingela on the ML about it, perhaps they just forgot to<br>
document it.<br>
<br>
--<br>
Loďc Hoguin<br>
Erlang Cowboy<br>
Nine Nines<br>
<a href="http://ninenines.eu" target="_blank">http://ninenines.eu</a><br>
<br>
<br>
<br>
<br></div>
______________________________<u></u>_________________<br>
Extend mailing list<br>
<a href="mailto:Extend@lists.ninenines.eu" target="_blank">Extend@lists.ninenines.eu</a><br>
<a href="http://lists.ninenines.eu:81/listinfo/extend" target="_blank">http://lists.ninenines.eu:81/<u></u>listinfo/extend</a><br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Loïc Hoguin</font></span><div class="HOEnZb"><div class="h5"><br>
Erlang Cowboy<br>
Nine Nines<br>
<a href="http://ninenines.eu" target="_blank">http://ninenines.eu</a><br>
<br>
</div></div></blockquote></div><br></div></div></div>