Erlang/OTP Forums

Author Message

<  RabbitMQ mailing list  ~  rabbit chokes?

Guest
Posted: Wed Nov 04, 2009 4:08 pm Reply with quote
Guest
Hi all,
I've been running rabbitmq for some web services (limited intranet user
base) for about 6 months now and everything has been great... until
recently.

I started to notice that the rabbitmq service would randomly die.
Digging through the logs I started to see messages like the following:

=INFO REPORT==== 1-Nov-2009::09:13:41 ===
starting TCP connection <0.27459.0> from 127.0.0.1:47608

=INFO REPORT==== 1-Nov-2009::09:13:41 ===
closing TCP connection <0.27459.0> from 127.0.0.1:47608

=INFO REPORT==== 1-Nov-2009::09:13:43 ===
alarm_handler: {set,{system_memory_high_watermark,[]}}

=INFO REPORT==== 1-Nov-2009::09:13:44 ===
alarm_handler: {clear,system_memory_high_watermark}

=INFO REPORT==== 1-Nov-2009::09:13:51 ===
Rolling persister log to
"/var/lib/rabbitmq/mnesia/rabbit/rabbit_persister.LOG.previous"

=INFO REPORT==== 1-Nov-2009::09:14:16 ===
alarm_handler: {set,{system_memory_high_watermark,[]}}

=INFO REPORT==== 1-Nov-2009::09:14:17 ===
accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:47609

=INFO REPORT==== 1-Nov-2009::09:14:17 ===
starting TCP connection <0.27501.0> from 127.0.0.1:47609

=INFO REPORT==== 1-Nov-2009::09:14:17 ===
closing TCP connection <0.27501.0> from 127.0.0.1:47609

=INFO REPORT==== 1-Nov-2009::09:14:18 ===
alarm_handler: {clear,system_memory_high_watermark}

=INFO REPORT==== 1-Nov-2009::09:14:19 ===
accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:47612

=INFO REPORT==== 1-Nov-2009::09:14:19 ===
starting TCP connection <0.27508.0> from 127.0.0.1:47612

=INFO REPORT==== 1-Nov-2009::09:14:19 ===
closing TCP connection <0.27508.0> from 127.0.0.1:47612

=INFO REPORT==== 1-Nov-2009::09:14:19 ===
alarm_handler: {set,{system_memory_high_watermark,[]}}

=INFO REPORT==== 1-Nov-2009::09:14:22 ===
alarm_handler: {clear,system_memory_high_watermark}


The server that is running the web application, and the rabbitmq service
rarely feels any kind of strain, so I'm having a hard time believing
that it's suffering from resource starvation. Any ideas as to what I
can do to not have to jump start rabbitmq every couple days?

Regards,
Owen Nelson

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist
Guest
Posted: Thu Nov 05, 2009 9:39 am Reply with quote
Guest
Owen,

Owen Nelson wrote:
> =INFO REPORT==== 1-Nov-2009::09:13:43 ===
> alarm_handler: {set,{system_memory_high_watermark,[]}}
>
> =INFO REPORT==== 1-Nov-2009::09:13:44 ===
> alarm_handler: {clear,system_memory_high_watermark}

These messages indicate that rabbit is running low on memory and is
attempting to throttle producers.

I recommend checking the status of your rabbit using the 'rabbitmqctl
list_* ...' commands. Chances are you have lots of messages in queues,
or perhaps lots of queues, exchanges or bindings.


Regards,

Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist
Guest
Posted: Thu Nov 05, 2009 2:23 pm Reply with quote
Guest
Perhaps this is the culprit?

rabbitmqctl list_queues -p gmate_test
Listing queues ...
celerygraph 277558
celery 0
...done.

What's the best way to clear that out? Looking at the man page for
rabbitmqctl, the only thing I see that would potentially do it would be
to reset the node, then rebuild the vhosts/users. Is that the only way?


Matthias Radestock wrote:
> Owen,
>
> Owen Nelson wrote:
>> =INFO REPORT==== 1-Nov-2009::09:13:43 ===
>> alarm_handler: {set,{system_memory_high_watermark,[]}}
>>
>> =INFO REPORT==== 1-Nov-2009::09:13:44 ===
>> alarm_handler: {clear,system_memory_high_watermark}
>
> These messages indicate that rabbit is running low on memory and is
> attempting to throttle producers.
>
> I recommend checking the status of your rabbit using the 'rabbitmqctl
> list_* ...' commands. Chances are you have lots of messages in queues,
> or perhaps lots of queues, exchanges or bindings.
>
>
> Regards,
>
> Matthias.

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist
Guest
Posted: Thu Nov 05, 2009 3:48 pm Reply with quote
Guest
Hi Owen,

You've got two options for this. Either you can write a client app to do a queue purge, or you can install the BQL plugin which provides a console that you can execute commands on.

The page at http://www.rabbitmq.com/rabbitmq-bql.html#installation discusses installation of this plugin - once you've done that, you should be able to execute a command such as:
Guest
Posted: Thu Nov 05, 2009 3:54 pm Reply with quote
Guest
Unfortunately, its just been pointed out to me that you are using a non-default vhost which the BQL tool currently can't do. So perhaps it won't be useful in your situation at this point in time. Sorry about that.

On Thu, Nov 5, 2009 at 3:47 PM, Paul Jones <pauljones23@gmail.com (pauljones23@gmail.com)> wrote:
Quote:
Hi Owen,

You've got two options for this. Either you can write a client app to do a queue purge, or you can install the BQL plugin which provides a console that you can execute commands on.

The page at http://www.rabbitmq.com/rabbitmq-bql.html#installation discusses installation of this plugin - once you've done that, you should be able to execute a command such as:
Guest
Posted: Thu Nov 05, 2009 3:56 pm Reply with quote
Guest
Yeah, I use 2 separate vhosts, one for development and one for
production. Good to know!

Paul Jones wrote:
> Unfortunately, its just been pointed out to me that you are using a
> non-default vhost which the BQL tool currently can't do. So perhaps it
> won't be useful in your situation at this point in time. Sorry about that.
>
> On Thu, Nov 5, 2009 at 3:47 PM, Paul Jones <pauljones23@gmail.com
> <mailto:pauljones23@gmail.com>> wrote:
>
> Hi Owen,
>
> You've got two options for this. Either you can write a client app
> to do a queue purge, or you can install the BQL plugin which
> provides a console that you can execute commands on.
>
> The page at http://www.rabbitmq.com/rabbitmq-bql.html#installation
> discusses installation of this plugin - once you've done that, you
> should be able to execute a command such as:
> BQL> purge queue celerygraph;
>
> Thanks,
> Paul.
>
>
> On Thu, Nov 5, 2009 at 2:22 PM, Owen Nelson <onelson@ringling.edu
> <mailto:onelson@ringling.edu>> wrote:
>
> Perhaps this is the culprit?
>
> rabbitmqctl list_queues -p gmate_test
> Listing queues ...
> celerygraph 277558
> celery 0
> ...done.
>
> What's the best way to clear that out? Looking at the man
> page for
> rabbitmqctl, the only thing I see that would potentially do it
> would be
> to reset the node, then rebuild the vhosts/users. Is that the
> only way?
>
>
> Matthias Radestock wrote:
> > Owen,
> >
> > Owen Nelson wrote:
> >> =INFO REPORT==== 1-Nov-2009::09:13:43 ===
> >> alarm_handler: {set,{system_memory_high_watermark,[]}}
> >>
> >> =INFO REPORT==== 1-Nov-2009::09:13:44 ===
> >> alarm_handler: {clear,system_memory_high_watermark}
> >
> > These messages indicate that rabbit is running low on memory
> and is
> > attempting to throttle producers.
> >
> > I recommend checking the status of your rabbit using the
> 'rabbitmqctl
> > list_* ...' commands. Chances are you have lots of messages
> in queues,
> > or perhaps lots of queues, exchanges or bindings.
> >
> >
> > Regards,
> >
> > Matthias.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> <mailto:rabbitmq-discuss@lists.rabbitmq.com>
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist
Guest
Posted: Thu Nov 05, 2009 4:22 pm Reply with quote
Guest
On Nov 5, 2009, at 4:47 PM, Paul Jones wrote:

>
> celerygraph 277558


You're running celeryd with the --statistics option on.
http://ask.github.com/celery/reference/celery.bin.celeryd.html#cmdoption-celeryd-s

If you're going to use this feature, you have to flush the events from
time to time (consume it to generate
the statistics)

from celery.monitoring import StatsCollector
s = StatsCollector()
s.collect()
s.dump()

But chances are, if you didn't know this, you are probably using it by
accident.


--
Ask Solem
Opera Software, Web Team
askh@opera.com

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist
Guest
Posted: Thu Nov 05, 2009 4:29 pm Reply with quote
Guest
Great catch - thanks Solem. Disabling the stat output now.

Ask Solem wrote:
> On Nov 5, 2009, at 4:47 PM, Paul Jones wrote:
>
>
>> celerygraph 277558
>>
>
>
> You're running celeryd with the --statistics option on.
> http://ask.github.com/celery/reference/celery.bin.celeryd.html#cmdoption-celeryd-s
>
> If you're going to use this feature, you have to flush the events from
> time to time (consume it to generate
> the statistics)
>
> from celery.monitoring import StatsCollector
> s = StatsCollector()
> s.collect()
> s.dump()
>
> But chances are, if you didn't know this, you are probably using it by
> accident.
>
>
>

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist
wuji
Posted: Wed Aug 22, 2012 7:01 am Reply with quote
User Joined: 10 Aug 2012 Posts: 654
budgets have led to continued job losses, although the pace pace jordan 6s pace of decline appears to have slowed relative to last
he said.Mass. Man Pays Off Mortgage With PenniesHow Many Pennies Pennies [h4]jordan 6s[/h4] Pennies Does It Take to Pay Off a Home?By LYNEKA
6, 2012— A Milford, Mass., man saved his pennies to to replica designer *beep* to pay off his mortgage--literally. He carted more than 62,000
to the bank to make his last payment."Thirty-five years ago ago [h2]cheap polo ralph lauren[/h2] ago when my wife and I took out our mortgage
View user's profile Send private message

Display posts from previous:  

All times are GMT
Page 1 of 1
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum