[99s-extend] Cowboy load test

[Loïc Hoguin]

This just hit me after hitting send.

Another possibility is that you are using text frames which are required 
to be valid UTF-8. Cowboy checks that (as required by the RFC), and it's 
fairly expensive, while most other servers don't. Try with binary frames 
instead.

On 09/15/2013 09:02 PM, Loïc Hoguin wrote:
> This sounds like you have too many timers, or something else is using
> the CPU. 40k connections is literally nothing. Also make sure the
> clients are on a separate VM/machine.
>
> Either way 99% CPU for 10k sounds high, I've had that with 0 CPU (though
> on real hardware).
>
> Someone a couple years back got above 1 million after fixing 'too many
> timers issues' but I don't think it was on a medium instance.
>
> On 09/15/2013 07:01 PM, Joe Freeman wrote:
>> Hi,
>>
>> I've started work on a project using Clojure, but I was wondering
>> whether (and secretly hoping that) Erlang would be a better fit, so I've
>> been load testing a few web server frameworks. I'm particularly
>> interested in how the server can handle a large number of concurrent
>> WebSocket connections, and the test I've been running is similar to Eric
>> Moritz's [1].
>>
>> I've setup a simple Cowboy 'echo' server running on an EC2 instance
>> (m1.medium, as in Eric's test) which could comfortably handle 10k
>> concurrent WebSocket requests (as in Eric's results), while echoing
>> about 200 messages/second. The CPU usage of the VM at this point is
>> about 99%, but the server continues to handle up to 40k concurrent
>> connections with a consistent average response time (<30ms). Pushing the
>> test beyond this number results in a spike in response times and lots of
>> connection timeouts.
>>
>> 40k connections seems pretty good, but when comparing this to the same
>> test against a couple of Clojure/JVM-based frameworks (specifically
>> Aleph/Netty and http-kit) I find I can get higher numbers of concurrent
>> connections with slightly better average response times (100k
>> connections, <10ms response time) using much less CPU (~20%). In fact,
>> memory seems to be the limiting factor.
>>
>> So I have two questions:
>>
>> 1) Should I be concerned about the CPU usage in the Erlang/Cowboy test?
>> I have limited experience with Erlang so far, but 100% CPU feels like a
>> bad thing.
>>
>> 2) Is there any way to increase the performance of the cowboy server?
>> Are there any Erlang VM parameters I can change? The fact that the
>> Clojure/JVM tests (on the same machine) have managed to get to 100k
>> connections suggests that the limitation isn't being imposed by the
>> operating system (I've applied changes various changes to sysctl and
>> ulimit).
>>
>> (Perhaps an echo server isn't the best way to compare HTTP servers, but
>> it feels like a good starting point.)
>>
>> Thanks for any help.
>>
>> [1] https://github.com/ericmoritz/wsdemo/blob/results-v1/results.md -
>> the GitHub repo actually contains code for an Aleph server, but results
>> from this aren't included in the summary here.
>>
>>
>>
>> _______________________________________________
>> Extend mailing list
>> Extend at lists.ninenines.eu
>> http://lists.ninenines.eu:81/listinfo/extend
>>
>
>


-- 
Loïc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu