Erlang/OTP Forums

Author Message

<  Erlang questions mailing list  ~  http:request crash

spgarbet
Posted: Wed Aug 06, 2008 12:25 pm Reply with quote
Joined: 06 Aug 2008 Posts: 4
What am I doing wrong? The program is firing off several processes which make http requests (http:request) to do some load testing. It usually only happens when the concurrent process count is above 60. Should this be on the bug forum?

Several of the processes die a horrible death and read off the following monologue:

Code:

=ERROR REPORT==== 5-Aug-2008::21:42:16 ===
** Generic server <0.22109.0> terminating
** Last message in was {tcp_closed,#Port<0.21371>}
** When Server state == {state,
                            {request,#Ref<0.0.1.250636>,<0.21236.0>,0,http,
                                {"127.0.0.1",5984},
                                "/",[],get,
                                {http_request_h,undefined,"keep-alive",
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,"127.0.0.1",undefined,
                                    undefined,undefined,undefined,undefined,
                                    undefined,undefined,undefined,undefined,
                                    [],undefined,undefined,undefined,
                                    undefined,"0",undefined,undefined,
                                    undefined,undefined,undefined,undefined,
                                    []},
                                {[],[]},
                                {http_options,"HTTP/1.1",infinity,true,[],
                                    undefined,false},
                                "http://127.0.0.1:5984",[],none,[]},
                            {tcp_session,
                                {{"127.0.0.1",5984},<0.22109.0>},
                                false,http,#Port<0.21371>,1},
                            undefined,undefined,undefined,
                            {httpc_response,parse,[nolimit,false]},
                            {[],[]},
                            new,[],nolimit,nolimit,
                            {options,
                                {undefined,[]},
                                0,2,2,disabled,enabled,false},
                            {timers,[],undefined},
                            httpc_manager,undefined}
** Reason for termination ==
** session_remotly_closed
[/code]
View user's profile Send private message
spgarbet
Posted: Wed Aug 06, 2008 2:53 pm Reply with quote
Joined: 06 Aug 2008 Posts: 4
This was written by Jonathan Hicks, and myself.

I was calling it as follows against a couchdb installation on the same box:
> dominatrix:torquemada("http://127.0.0.1:5984", 10000, 60).

Code:

-module(dominatrix).
-export([torquemada/3]).
-export([strike/1,whip/2,inquisitor/0]).

%  Copyright 2008, Gnu Public License Version 3.0. Jonathan Hicks and Shawn Garbett

for(N, N, F) -> [F()];
for(I, N, F) -> [F()|for(I+1, N, F)].

strike(Site) ->
  case http:request(Site) of
    {ok, {{_, 200, _}, _, _}}  -> success;
    _                          -> failure
  end.

whip(Site, Inquisitor) ->
  receive
    stop -> true
  after 0 ->
    Inquisitor ! strike(Site),
    whip(Site, Inquisitor)
  end.
 
inquisitor() -> inquisitor({0,0}).
inquisitor({Successes, Failures}) ->
  receive
    success       -> inquisitor({Successes+1,Failures});
    failure       -> inquisitor({Successes,Failures+1});
    {results,Pid} -> Pid ! {Successes,Failures},
                     inquisitor({Successes,Failures});
    stop          -> true
  end.

%% Why does this die above 60 concurrent requests? This shouldn't happen!
torquemada(Site, Time, Concurrent) ->
  inets:start(),   
  Inquisitor = spawn(dominatrix, inquisitor, []),
  Torturers = for(1, Concurrent, fun() -> spawn(dominatrix, whip, [Site, Inquisitor]) end),
  timer:sleep(Time),
  Inquisitor ! {results, self()},
  lists:foreach(fun(Pid) -> Pid ! stop end, Torturers),
  receive
    {Confessions,Heretics} ->
        io:format("~w Successes and ~w Failures in ~w seconds with ~w concurrent users => ~w requests/second", [Confessions, Heretics, Time/1000, Concurrent, Confessions/Time*1000])
    end,
  Inquisitor ! stop.
View user's profile Send private message
spgarbet
Posted: Wed Aug 06, 2008 3:07 pm Reply with quote
Joined: 06 Aug 2008 Posts: 4
Quote:

mog: does this happen CyberGarp at lower numbers?
[10:00am] HunterXHunterXXX: that happaens to me too
[10:00am] CyberGarp: no
[10:00am] mog: whats your ulimit?
[10:01am] mog: this is a gnu/linux box i assume?
[10:01am] CyberGarp: Mac OSX
[10:01am] mog: ohhhh
[10:02am] CyberGarp: Running otp_src_R12B-3
[10:02am] mog: look up open file descriptors mac os x
[10:02am] CyberGarp: okay, sec
[10:02am] mog: im sure your exhausting that
[10:02am] mog: as this on local host you probably have 20,000 or so open file descriptors
[10:02am] mog: and macosx doesnt like that
[10:02am] mog: i know how to change it on gnu/linux boxes but not macs
[10:03am] CyberGarp: spgarbet$ ulimit
[10:03am] CyberGarp: unlimited
[10:04am] CyberGarp: Found it.
[10:04am] CyberGarp: ulimit -a
[10:04am] CyberGarp: open files (-n) 2560
[10:04am] mog: oh cool, its same thing as gnu/linux
[10:04am] mog: ya you cant open more than 2560 sockets right now
[10:05am] CyberGarp: It's basically BSD running a Mach kernal.
[10:05am] mog: you can up it to infiinity
[10:05am] mog: ya i know
[10:05am] CyberGarp: http://archives.postgresql.org/pgsql-admin/2007-11/msg00133.php
[10:05am] mog: but didnt know bsd had same tool for that
[10:05am] MisterN: lol i have 1024
[10:05am] mog: most gnu/linux default to that
[10:05am] CyberGarp: So I need to trap the error and print something intelligent?
[10:05am] CyberGarp: Then up the limit to actually do it.
[10:05am] mog: or up the ulimit before start
[10:05am] mog: ya
[10:06am] CyberGarp: cool
[10:06am] mog: well try uping it
[10:06am] CyberGarp: thx
[10:06am] mog: thats just my gues
[10:06am] mog: it could be something else as well
View user's profile Send private message
spgarbet
Posted: Wed Aug 06, 2008 3:57 pm Reply with quote
Joined: 06 Aug 2008 Posts: 4
While fiddling with the ulimit parameters I can get the amount to go higher. But why is it failing in the first place. 20 processes continually requesting through http:request shouldn't be hitting 10,000 open files?

Is there some option about how it uses the tcp/ip stack that will make it 20 open files?
View user's profile Send private message
danmil
Posted: Wed Oct 29, 2008 2:23 pm Reply with quote
Joined: 29 Oct 2008 Posts: 1
I ran into a very similar problem (lots of session_remotly_closed at high concurrency for inets/http client). Tried out the ibrowse http client, which seems to have solved the problem instantly. A bit hard to find the source -- had to check out all of the jungerl code collection from cvs at:

http://sourceforge.net/cvs/?group_id=73688

And then pull out the ibrowse source and build it separately.

-Dan Milstein
View user's profile Send private message

Display posts from previous:  

All times are GMT
Page 1 of 1
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum