1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
<TITLE> [99s-extend] Cowboy HTTPS connection memory usage
</TITLE>
<LINK REL="Index" HREF="index.html" >
<LINK REL="made" HREF="mailto:extend%40lists.ninenines.eu?Subject=Re%3A%20%5B99s-extend%5D%20Cowboy%20HTTPS%20connection%20memory%20usage&In-Reply-To=%3CCAJ0zLRNud5vq9CnvZm8AgH1ZMPCzdkFe5tzdi5s13iP4HR6cuQ%40mail.gmail.com%3E">
<META NAME="robots" CONTENT="index,nofollow">
<style type="text/css">
pre {
white-space: pre-wrap; /* css-2.1, curent FF, Opera, Safari */
}
</style>
<META http-equiv="Content-Type" content="text/html; charset=us-ascii">
<LINK REL="Previous" HREF="000025.html">
<LINK REL="Next" HREF="000027.html">
</HEAD>
<BODY BGCOLOR="#ffffff">
<H1>[99s-extend] Cowboy HTTPS connection memory usage</H1>
<B>rambocoder</B>
<A HREF="mailto:extend%40lists.ninenines.eu?Subject=Re%3A%20%5B99s-extend%5D%20Cowboy%20HTTPS%20connection%20memory%20usage&In-Reply-To=%3CCAJ0zLRNud5vq9CnvZm8AgH1ZMPCzdkFe5tzdi5s13iP4HR6cuQ%40mail.gmail.com%3E"
TITLE="[99s-extend] Cowboy HTTPS connection memory usage">erlang at rambocoder.com
</A><BR>
<I>Fri Dec 21 20:25:10 CET 2012</I>
<P><UL>
<LI>Previous message: <A HREF="000025.html">[99s-extend] Cowboy HTTPS connection memory usage
</A></li>
<LI>Next message: <A HREF="000027.html">[99s-extend] [ANN] Ranch 0.6.0 Xmas Edition Released
</A></li>
<LI> <B>Messages sorted by:</B>
<a href="date.html#26">[ date ]</a>
<a href="thread.html#26">[ thread ]</a>
<a href="subject.html#26">[ subject ]</a>
<a href="author.html#26">[ author ]</a>
</LI>
</UL>
<HR>
<!--beginarticle-->
<PRE>Long story short, I solved the problem by adding {max_connections, 50000}
to cowboy:start_https because it default to 1024 at
<A HREF="https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30">https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30</A>
However, before I figured out that setting, I did run eprof and these are
the function calls it was spending most of it's time on
FUNCTION CALLS % TIME [uS /
CALLS]
-------- ----- --- ----
[----------]
dict:get_slot/2 174 1.73 1658 [
9.53]
dict:on_bucket/3 171 1.77 1701 [
9.95]
erlang:setelement/3 684 3.23 3098 [
4.53]
dict:store_bkt_val/3 600 5.24 5028 [
8.38]
Then I ran etop and it showed that ranch_acceptor:maybe_wait had the most
reductions were, so I looked at the code in that
<A HREF="https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72">https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72</A> and
realized that like a newb I did not set the maximum connections for the
listener :)
Problem solved. Looks like I won't need to put HAProxy in front of Cowboy
after all.
Thank you,
rambocoder
On Fri, Dec 21, 2012 at 11:51 AM, Loïc Hoguin <<A HREF="https://lists.ninenines.eu/listinfo/extend">essen at ninenines.eu</A>> wrote:
><i> Can you try enabling eprof to see where the VM spends its time?
</I>><i>
</I>><i>
</I>><i> On 12/21/2012 05:49 PM, rambocoder wrote:
</I>><i>
</I>>><i> In my preliminary testing, I used Jmeter this morning since it's an
</I>>><i> easy GUI load testing app and this is what I am seeing:
</I>>><i>
</I>>><i> With R15B03-01 [smp:4:4] [async-threads:4] [hipe] [kernel-poll:true],
</I>>><i> when I establish 1K concurrent connections via HTTPS, each connection
</I>>><i> takes up about 68K of memory.
</I>>><i>
</I>>><i> Unfortunately, after about 1050-1200 connections, on my test server the
</I>>><i> Erlang scheduler jumps to 100% CPU utilization on all 4 schedulers,
</I>>><i> while up to that point the scheduler's load was oscillating up and down.
</I>>><i> Using the Observer, there is only 1 ssl_connection_sup in the ssl
</I>>><i> application, having to deal with 1000+ gen_fsm workers, so that might be
</I>>><i> the bottleneck. Since the ulimit on my server is 50000 I don't think I
</I>>><i> am hitting any type of file handler's limit.
</I>>><i>
</I>>><i> Loïc and the group, am I missing some setting that is causing the
</I>>><i> scheduler to go to 100% CPU and the run que in observer to be 99?
</I>>><i>
</I>>><i> Sincerely,
</I>>><i>
</I>>><i> rambocoder
</I>>><i>
</I>>><i>
</I>>><i>
</I>>><i> On Fri, Dec 21, 2012 at 6:45 AM, Loïc Hoguin <<A HREF="https://lists.ninenines.eu/listinfo/extend">essen at ninenines.eu</A>
</I>>><i> <mailto:<A HREF="https://lists.ninenines.eu/listinfo/extend">essen at ninenines.eu</A>>> wrote:
</I>>><i>
</I>>><i> On 12/21/2012 04:34 AM, rambocoder wrote:
</I>>><i>
</I>>><i> Does anybody know either from benchmarks or real world data what
</I>>><i> is the
</I>>><i> average memory footprint of each concurrent HTTPS connection to
</I>>><i> cowboy?
</I>>><i>
</I>>><i>
</I>>><i> I don't have anything, sorry. I'm guessing it consumes a lot more
</I>>><i> than TCP though.
</I>>><i>
</I>>><i>
</I>>><i> SSL app in Erlang reuses SSL session-ids so I am not sure if the
</I>>><i> Apache
</I>>><i> Bench I test with reuses the session id or it does not.
</I>>><i>
</I>>><i>
</I>>><i> I wouldn't know, but I wouldn't trust Apache Bench doing the right
</I>>><i> thing. Any other benchmark tool usually works better in my experience.
</I>>><i>
</I>>><i>
</I>>><i> BTW, what makes an erlang api "documented" vs "undocumented". For
</I>>><i> example ssl:session_info/1 function here (
</I>>><i> <A HREF="https://github.com/erlang/otp/**__blob/maint/lib/ssl/src/ssl._**">https://github.com/erlang/otp/**__blob/maint/lib/ssl/src/ssl._**</A>
</I>>><i> _erl#L411<<A HREF="https://github.com/erlang/otp/__blob/maint/lib/ssl/src/ssl.__erl#L411">https://github.com/erlang/otp/__blob/maint/lib/ssl/src/ssl.__erl#L411</A>>
</I>>><i>
</I>>><i> <<A HREF="https://github.com/erlang/**otp/blob/maint/lib/ssl/src/**">https://github.com/erlang/**otp/blob/maint/lib/ssl/src/**</A>
</I>>><i> ssl.erl#L411<<A HREF="https://github.com/erlang/otp/blob/maint/lib/ssl/src/ssl.erl#L411">https://github.com/erlang/otp/blob/maint/lib/ssl/src/ssl.erl#L411</A>>
</I>>><i> >
</I>>><i> ) has
</I>>><i> a spec and a short doc, but session_info is not described
</I>>><i> <A HREF="http://www.erlang.org/doc/man/**__ssl.html<http://www.erlang.org/doc/man/__ssl.html">http://www.erlang.org/doc/man/**__ssl.html<http://www.erlang.org/doc/man/__ssl.html</A>>
</I>>><i>
</I>>><i> <<A HREF="http://www.erlang.org/doc/**man/ssl.html<http://www.erlang.org/doc/man/ssl.html">http://www.erlang.org/doc/**man/ssl.html<http://www.erlang.org/doc/man/ssl.html</A>>>
</I>>><i> .ssl:session_info/1 is
</I>>><i> a useful
</I>>><i> function to be able to track if the load generator is reusing
</I>>><i> the SSL
</I>>><i> session_id or it is generating new one, because that would make
</I>>><i> all the
</I>>><i> difference during measurement due to Erlang caching SSL sessions
</I>>><i> by default.
</I>>><i>
</I>>><i>
</I>>><i> The documentation is separate (they're not using edoc). It's perhaps
</I>>><i> not deemed useful enough for documenting it. I wouldn't worry about
</I>>><i> using it for measurements though.
</I>>><i>
</I>>><i> Try asking Ingela on the ML about it, perhaps they just forgot to
</I>>><i> document it.
</I>>><i>
</I>>><i> --
</I>>><i> Loďc Hoguin
</I>>><i> Erlang Cowboy
</I>>><i> Nine Nines
</I>>><i> <A HREF="http://ninenines.eu">http://ninenines.eu</A>
</I>>><i>
</I>>><i>
</I>>><i>
</I>>><i>
</I>>><i> ______________________________**_________________
</I>>><i> Extend mailing list
</I>>><i> <A HREF="https://lists.ninenines.eu/listinfo/extend">Extend at lists.ninenines.eu</A>
</I>>><i> <A HREF="http://lists.ninenines.eu:81/**listinfo/extend<http://lists.ninenines.eu:81/listinfo/extend">http://lists.ninenines.eu:81/**listinfo/extend<http://lists.ninenines.eu:81/listinfo/extend</A>>
</I>>><i>
</I>>><i>
</I>><i>
</I>><i> --
</I>><i> Loïc Hoguin
</I>><i>
</I>><i> Erlang Cowboy
</I>><i> Nine Nines
</I>><i> <A HREF="http://ninenines.eu">http://ninenines.eu</A>
</I>><i>
</I>><i>
</I>-------------- next part --------------
An HTML attachment was scrubbed...
URL: <<A HREF="http://lists.ninenines.eu/archives/extend/attachments/20121221/945f636e/attachment.html">http://lists.ninenines.eu/archives/extend/attachments/20121221/945f636e/attachment.html</A>>
</PRE>
<!--endarticle-->
<HR>
<P><UL>
<!--threads-->
<LI>Previous message: <A HREF="000025.html">[99s-extend] Cowboy HTTPS connection memory usage
</A></li>
<LI>Next message: <A HREF="000027.html">[99s-extend] [ANN] Ranch 0.6.0 Xmas Edition Released
</A></li>
<LI> <B>Messages sorted by:</B>
<a href="date.html#26">[ date ]</a>
<a href="thread.html#26">[ thread ]</a>
<a href="subject.html#26">[ subject ]</a>
<a href="author.html#26">[ author ]</a>
</LI>
</UL>
<hr>
<a href="https://lists.ninenines.eu/listinfo/extend">More information about the Extend
mailing list</a><br>
</body></html>
|
