1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
|
<tt>
Long story short, I solved the problem by adding {max_connections, 50000} to cowboy:start_https because it default to 1024 at <a href="https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30">https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30</a><div><br>
<br></div><div>However, before I figured out that setting, I did run eprof and these are the function calls it was spending most of it's time on<br><div><div><br></div><div><br></div><div><div>FUNCTION CALLS % TIME [uS / CALLS]</div><br>
<div>-------- ----- --- ---- [----------]</div></div><div>dict:get_slot/2 174 1.73 1658 [ 9.53]</div><div><div>dict:on_bucket/3 171 1.77 1701 [ 9.95]</div><br>
<div>erlang:setelement/3 684 3.23 3098 [ 4.53]</div><div>dict:store_bkt_val/3 600 5.24 5028 [ 8.38]</div><div><br></div><div>Then I ran etop and it showed that ranch_acceptor:maybe_wait had the most reductions were, so I looked at the code in that <a href="https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72">https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72</a> and realized that like a newb I did not set the maximum connections for the listener :)</div><br>
<div><br></div><div>Problem solved. Looks like I won't need to put HAProxy in front of Cowboy after all.</div><div><br></div><div>Thank you,</div><div><br></div><div>rambocoder</div><br><div class="gmail_quote">On Fri, Dec 21, 2012 at 11:51 AM, Loïc Hoguin <span dir="ltr"><<a href="mailto:[email protected]" target="_blank">[email protected]</a>></span> wrote:<br><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can you try enabling eprof to see where the VM spends its time?<div class="im"><br><br>
<br><br>
On 12/21/2012 05:49 PM, rambocoder wrote:<br><br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
In my preliminary testing, I used Jmeter this morning since it's an<br><br>
easy GUI load testing app and this is what I am seeing:<br><br>
<br><br>
With R15B03-01 [smp:4:4] [async-threads:4] [hipe] [kernel-poll:true],<br><br>
when I establish 1K concurrent connections via HTTPS, each connection<br><br>
takes up about 68K of memory.<br><br>
<br><br>
Unfortunately, after about 1050-1200 connections, on my test server the<br><br>
Erlang scheduler jumps to 100% CPU utilization on all 4 schedulers,<br><br>
while up to that point the scheduler's load was oscillating up and down.<br><br>
Using the Observer, there is only 1 ssl_connection_sup in the ssl<br><br>
application, having to deal with 1000+ gen_fsm workers, so that might be<br><br>
the bottleneck. Since the ulimit on my server is 50000 I don't think I<br><br>
am hitting any type of file handler's limit.<br><br>
<br><br>
Loïc and the group, am I missing some setting that is causing the<br><br>
scheduler to go to 100% CPU and the run que in observer to be 99?<br><br>
<br><br>
Sincerely,<br><br>
<br><br>
rambocoder<br><br>
<br><br>
<br><br>
<br><br>
On Fri, Dec 21, 2012 at 6:45 AM, Loïc Hoguin <<a href="mailto:[email protected]" target="_blank">[email protected]</a><br></div><div class="im"><br>
<mailto:<a href="mailto:[email protected]" target="_blank">[email protected]</a>>> wrote:<br><br>
<br><br>
On 12/21/2012 04:34 AM, rambocoder wrote:<br><br>
<br><br>
Does anybody know either from benchmarks or real world data what<br><br>
is the<br><br>
average memory footprint of each concurrent HTTPS connection to<br><br>
cowboy?<br><br>
<br><br>
<br><br>
I don't have anything, sorry. I'm guessing it consumes a lot more<br><br>
than TCP though.<br><br>
<br><br>
<br><br>
SSL app in Erlang reuses SSL session-ids so I am not sure if the<br><br>
Apache<br><br>
Bench I test with reuses the session id or it does not.<br><br>
<br><br>
<br><br>
I wouldn't know, but I wouldn't trust Apache Bench doing the right<br><br>
thing. Any other benchmark tool usually works better in my experience.<br><br>
<br><br>
<br><br>
BTW, what makes an erlang api "documented" vs "undocumented". For<br><br>
example ssl:session_info/1 function here (<br></div><br>
<a href="https://github.com/erlang/otp/__blob/maint/lib/ssl/src/ssl.__erl#L411" target="_blank">https://github.com/erlang/otp/<u></u>__blob/maint/lib/ssl/src/ssl._<u></u>_erl#L411</a><div class="im"><br><br>
<<a href="https://github.com/erlang/otp/blob/maint/lib/ssl/src/ssl.erl#L411" target="_blank">https://github.com/erlang/<u></u>otp/blob/maint/lib/ssl/src/<u></u>ssl.erl#L411</a>><br><br>
) has<br><br>
a spec and a short doc, but session_info is not described<br></div><br>
<a href="http://www.erlang.org/doc/man/__ssl.html" target="_blank">http://www.erlang.org/doc/man/<u></u>__ssl.html</a><div class="im"><br><br>
<<a href="http://www.erlang.org/doc/man/ssl.html" target="_blank">http://www.erlang.org/doc/<u></u>man/ssl.html</a>> .ssl:session_info/1 is<br><br>
a useful<br><br>
function to be able to track if the load generator is reusing<br><br>
the SSL<br><br>
session_id or it is generating new one, because that would make<br><br>
all the<br><br>
difference during measurement due to Erlang caching SSL sessions<br><br>
by default.<br><br>
<br><br>
<br><br>
The documentation is separate (they're not using edoc). It's perhaps<br><br>
not deemed useful enough for documenting it. I wouldn't worry about<br><br>
using it for measurements though.<br><br>
<br><br>
Try asking Ingela on the ML about it, perhaps they just forgot to<br><br>
document it.<br><br>
<br><br>
--<br><br>
Loďc Hoguin<br><br>
Erlang Cowboy<br><br>
Nine Nines<br><br>
<a href="http://ninenines.eu" target="_blank">http://ninenines.eu</a><br><br>
<br><br>
<br><br>
<br><br>
<br></div><br>
______________________________<u></u>_________________<br><br>
Extend mailing list<br><br>
<a href="mailto:[email protected]" target="_blank">[email protected]</a><br><br>
<a href="http://lists.ninenines.eu:81/listinfo/extend" target="_blank">http://lists.ninenines.eu:81/<u></u>listinfo/extend</a><br><br>
<br><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote><span class="HOEnZb"><font color="#888888"><br>
<br><br>
<br><br>
-- <br><br>
Loïc Hoguin</font></span><div class="HOEnZb"><div class="h5"><br><br>
Erlang Cowboy<br><br>
Nine Nines<br><br>
<a href="http://ninenines.eu" target="_blank">http://ninenines.eu</a><br><br>
<br><br>
</div></div></blockquote></div><br></div></div></div><br>
</tt>
|