| Author |
Message |
|
| Guest |
 Posted: Mon May 05, 2008 9:32 am |
 |
|
|
Guest
|
Hi guys, hope you can help me with this.
First - the problem is that we do something new with RabbitMQ that we probably shouldn't be doing - we just don't know what yet  |
|
|
| Back to top |
|
| Guest |
| Posted: Mon May 05, 2008 1:38 pm |
|
|
|
Guest
|
Michael,
Michael Arnoldus wrote:
> Anyway, the result is "too many processes".
A brute force way of checking on the currently running processes is
file:write_file("/tmp/processes.erl", io_lib:format("~p",
[[process_info(P) || P <- processes()]])).
Warning: this may include some message content and security credentials.
> So is there any way to ask RabbitMQ for the amount of connections?
> Channels?
Not easily. It's on our todo list.
Connects/disconnects are recorded in rabbit.log, so with some log
analysis you should be able to get an idea on the current connection
count. Plus of course there's the 'netstat' OS command.
> Will rabbit create a new process for each connection? Channel?
RabbitMQ creates about half a dozen processes per connection, and four
processes per channel.
> Any other suggestions on whats happening here?
Connections, channels and queues are the only entities performing
dynamic process creation, except for the various places that spawn
short-lived helper processes.
Are you sure that you haven't got stale queues lying around? I am asking
because of the mnesia errors you are seeing - connection and channel
creation does not result in mnesia writes, but queue creation does.
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon May 05, 2008 1:55 pm |
|
|
|
Guest
|
Matthias,
On May 5, 2008, at 15:38 , Matthias Radestock wrote:
> A brute force way of checking on the currently running processes is
>
> file:write_file("/tmp/processes.erl", io_lib:format("~p",
> [[process_info(P) || P <- processes()]])).
>
> Warning: this may include some message content and security
> credentials.
>
Thanks.
>> cSo is there any way to ask RabbitMQ for the amount of connections?
>> Channels?
>
> Not easily. It's on our todo list.
>
> Connects/disconnects are recorded in rabbit.log, so with some log
> analysis you should be able to get an idea on the current connection
> count. Plus of course there's the 'netstat' OS command.
All I get in the log is
=WARNING REPORT==== 5-May-2008::15:17:40 ===
Attempt by client to use invalid ticket 0
=WARNING REPORT==== 5-May-2008::15:17:40 ===
Lax ticket check mode: fabricating full ticket number 0
since we're using the qpid python client, but nothing else about
connections or channel open. Do I get the above message for each
connection or each channel open?
>
>
>> Will rabbit create a new process for each connection? Channel?
>
> RabbitMQ creates about half a dozen processes per connection, and
> four processes per channel.
Good to know! Thanks!
>
>
>> Any other suggestions on whats happening here?
>
> Connections, channels and queues are the only entities performing
> dynamic process creation, except for the various places that spawn
> short-lived helper processes.
Ok.
>
>
> Are you sure that you haven't got stale queues lying around? I am
> asking because of the mnesia errors you are seeing - connection and
> channel creation does not result in mnesia writes, but queue
> creation does.
Not sure - but approx 10 sec before the latest crash I had 12 queues
which had been there for at least 30 sec - and we have no 'make a lot
of queues very fast' function that might do something like this, so my
primary suspicions would go somewhere else. I'll work on it though.
Michael
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon May 05, 2008 2:08 pm |
|
|
|
Guest
|
Michael,
Michael Arnoldus wrote:
> On May 5, 2008, at 15:38 , Matthias Radestock wrote:
>> Connects/disconnects are recorded in rabbit.log, so with some log
>> analysis you should be able to get an idea on the current connection
>> count. Plus of course there's the 'netstat' OS command.
>
> All I get in the log is
>
> =WARNING REPORT==== 5-May-2008::15:17:40 ===
> Attempt by client to use invalid ticket 0
>
> =WARNING REPORT==== 5-May-2008::15:17:40 ===
> Lax ticket check mode: fabricating full ticket number 0
You should see messages like
=INFO REPORT==== 5-May-2008::08:37:48 ===
accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:53449
=INFO REPORT==== 5-May-2008::08:37:53 ===
closing TCP connection from 127.0.0.1:49606
for the opening and closing of a connection, respectively. In fact I
know you do - the above are copy&pasted from the log you sent earlier
Matthias
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon May 05, 2008 5:04 pm |
|
|
|
Guest
|
Matthias Radestock wrote:
> Michael Arnoldus wrote:
>> So is there any way to ask RabbitMQ for the amount of connections?
>> Channels?
>
> Not easily. It's on our todo list.
Actually, it's not that hard. In the shell define
F = fun (X) -> length([P || P <- processes(), case process_info(P,
current_function) of {current_function, {X, _, _}} -> true; _ -> false
end]) end.
and then call it like this:
F(rabbit_reader). %% number of connections
F(rabbit_framing_channel). %% number of channels
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| tonyg |
| Posted: Tue May 06, 2008 11:39 am |
|
|
|
User
Joined: 07 Nov 2006
Posts: 199
|
|
| Back to top |
  
|
| Guest |
| Posted: Tue May 06, 2008 1:45 pm |
|
|
|
Guest
|
On May 6, 2008, at 13:38 , Tony Garnock-Jones wrote:
> Michael Arnoldus wrote:
>> =ERROR REPORT==== 5-May-2008::08:46:22 ===
>> ** Too many db tables **
>
> This, in particular, is extremely weird. We don't create tables at any
> time after broker startup. (If nothing else is creating tables, then
> it's likely a symptom of some kind of overload that's being
> misreported.)
I see what you're saying - but to add to the weirdness when this error
happens we have to restart rabbit and it still doesn't work. The only
way we have found to make it work again is to delete the mnesia
directory. So it does seem like something is happening with mnesia.
Also after the restart this is in the logfile
=INFO REPORT==== 5-May-2008::08:50:05 ===
Exception: Channel 5, Reason {amqp,not_found,'basic.publish'}
If it is any help I can try to recreate the problem and send you a set
of corrupted mnesia files (or a textfile export).
Michael
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed May 07, 2008 5:01 am |
|
|
|
Guest
|
Michael,
Michael Arnoldus wrote:
>>> =ERROR REPORT==== 5-May-2008::08:46:22 ===
>>> ** Too many db tables **
> to add to the weirdness when this error happens we have to restart
> rabbit and it still doesn't work. The only way we have found to make
> it work again is to delete the mnesia directory. So it does seem like
> something is happening with mnesia.
I suspect that if rabbit ran out of processes before the restart then it
may have corrupted mnesia.
> Also after the restart this is in the logfile
> =INFO REPORT==== 5-May-2008::08:50:05 ===
> Exception: Channel 5, Reason {amqp,not_found,'basic.publish'}
That just means the server couldn't find the exchange during a
basic.publish, which isn't surprising if mnesia is playing up.
> If it is any help I can try to recreate the problem and send you a set
> of corrupted mnesia files (or a textfile export).
The most promising route of investigation is probably to look at what
processes are running before the system actually runs out of them.
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Wed May 07, 2008 5:35 am |
|
|
|
Guest
|
On May 7, 2008, at 7:00 , Matthias Radestock wrote:
>
> The most promising route of investigation is probably to look at
> what processes are running before the system actually runs out of
> them.
>
Ok, will do!
Michael
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
