| Author |
Message |
|
| Guest |
 Posted: Wed Sep 23, 2009 9:54 pm |
 |
|
|
Guest
|
| I wanted to get some feedback on what I am noticing with Rabbit and memory use. |
|
|
| Back to top |
|
| Guest |
| Posted: Wed Sep 23, 2009 9:58 pm |
|
|
|
Guest
|
Attached are Alice screenshots.
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 24, 2009 8:56 am |
|
|
|
Guest
|
Gavin,
Gavin M. Roy wrote:
> I wanted to get some feedback on what I am noticing with Rabbit and
> memory use. In this test case, I loaded up 4 million messages that were
> roughly 12 bytes or so of data, not counting protocol overhead. At
> various stages of processing I had everything from 0 consumers to 80
> consumers running against RabbitMQ using basic_consume. The machine
> that Rabbit is running on is dedicated to Rabbit and Alice
> (http://github.com/auser/alice)
>
> Some odd things:
>
> * In the 6am hour where you see swap spike (red in graph), consumers
> stopped consuming. I could not find out at the time if it was
> client or Rabbit that stopped dispatching messages. Restarting
> consumers resumed the previous rates of message consumption.
Next time this happens I suggest you gather some diagnostics with
rabbitmqctl list_queues name messages_ready messages_unacknowledged
consumers
If the consumer count is 0 then there are no consumers. If you have a
consumer count above 0 and a messages_ready count above 0, and the
broker appears to be idle, then that is a good indication that the
consumers are there but stuck and the tcp connection to them is backlogged.
> * At the time where you see the last major purple graph spike to 10G
> free, we had consumed all of the messages in the broker. The
> broker is currently sitting idle with no messages, but still has
> 2+GB of ram allocated.
>
> I did notice there seems to be some sort of cleanup worker that runs
> roughly every 30 minutes, but it's not reclaimed any of the 2+GB of
> allocation since the broker has gone idle. Is this expected behavior?
It is quite common for a gc'ed VM to hold on to large chunks of memory
instead of handing them back to the OS.
Are you monitoring queues with rabbitmqctl or Alice? If that happens
more frequently than once per second then the queue processes are kept
active, which delays the freeing up of memory.
To get some diagnostics on where the memory has gone, shell into rabbit
(erl -sname shell -remsh rabbit@<hostname>) and run
memory().
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 24, 2009 1:36 pm |
|
|
|
Guest
|
On Thu, Sep 24, 2009 at 4:57 AM, Matthias Radestock <matthias@lshift.net (matthias@lshift.net)> wrote:
Quote: Gavin,
Next time this happens I suggest you gather some diagnostics with
|
|
|
| Back to top |
|
| Guest |
| Posted: Thu Sep 24, 2009 2:10 pm |
|
|
|
Guest
|
Gavin,
Gavin M. Roy wrote:
> Is it safe to assume the VM will make use of what it's holding on to and
> it's not a leak of some sort?
I doubt it is a leak, but the VM can hold on to free memory in the
anticipation of future need which may never materialise.
> (rabbit@mq07)1> memory().
> [{total,354514080},
So that's just ~350MB there, which means the rest of the VM process'
memory is technically free but hasn't been returned to the OS, unless
there's a leak in the VM itself.
The docs for erlang:memory/0 mention various caveats regarding the
accuracy of the reported figures, and ways to improve that accuracy,
which is something you may want to look into if you want to investigate
this further.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 6:47 pm |
|
|
|
Guest
|
Is there a way to place a memory restriction on rabbitmq? I am getting a bit tired of it locking up all the available memory when there are a significant amount of items in the queue since it won't write them to disk
Suhail
On Thu, Sep 24, 2009 at 7:11 AM, Matthias Radestock <matthias@lshift.net (matthias@lshift.net)> wrote:
Quote: Gavin,
Gavin M. Roy wrote:
> Is it safe to assume the VM will make use of what it's holding on to and
> it's not a leak of some sort?
I doubt it is a leak, but the VM can hold on to free memory in the
anticipation of future need which may never materialise.
> (rabbit@mq07)1> memory().
> [{total,354514080},
So that's just ~350MB there, which means the rest of the VM process'
memory is technically free but hasn't been returned to the OS, unless
there's a leak in the VM itself.
The docs for erlang:memory/0 mention various caveats regarding the
accuracy of the reported figures, and ways to improve that accuracy,
which is something you may want to look into if you want to investigate
this further.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com (rabbitmq-discuss@lists.rabbitmq.com)
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 7:17 pm |
|
|
|
Guest
|
Suhail,
Suhail Doshi wrote:
> Is there a way to place a memory restriction on rabbitmq? I am getting a
> bit tired of it locking up all the available memory when there are a
> significant amount of items in the queue since it won't write them to disk
RabbitMQ stores all messages in memory. Messages that are marked
'persistent' are stored on disk *as well* as in memory.
That will change when the new persister gets released, at which point
you should indeed be able to restrict RabbitMQ's memory usage.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 7:42 pm |
|
|
|
Guest
|
I know it's being released when it's ready but is that occurring in the next 3 months?
Suhail
On Fri, Sep 25, 2009 at 12:17 PM, Matthias Radestock <matthias@lshift.net (matthias@lshift.net)> wrote:
Quote: Suhail,
Suhail Doshi wrote:
Quote: Is there a way to place a memory restriction on rabbitmq? I am getting a bit tired of it locking up all the available memory when there are a significant amount of items in the queue since it won't write them to disk
RabbitMQ stores all messages in memory. Messages that are marked 'persistent' are stored on disk *as well* as in memory.
That will change when the new persister gets released, at which point you should indeed be able to restrict RabbitMQ's memory usage.
Regards,
Matthias.
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 9:11 pm |
|
|
|
Guest
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
