| Author |
Message |
|
| Guest |
 Posted: Thu Oct 15, 2009 7:24 pm |
 |
|
|
Guest
|
One of our systems overfed the rabbit, and it's now a bit angry. The
strange problem that we're having, though, is that it seems that the
commit after the thousandth ack'd message is triggering the persister
log rollover, which is then causing beam to run out of memory. The
only thing we're doing is basic_get'ing messages, acking them, and
then committing (the channel is transactional). We've also tried
committing every thousand messages; either way, the first commit after
the thousandth ack'd message is causing the persister log rollover,
followed immediately by beam crashing due to being out of RAM.
Does anybody know why rolling the persister log would cause the system
to run out of memory? It seems like a strange place to need to
allocate a lot of RAM, but I'm not at all familiar with how the
persister rollover works. I'm going to next try just doing basic_gets
without being in a transaction to see if that prevents the crash, but
I'd like a better understanding of how the log rollover works, and why
it seems to need a lot of memory to succeed.
Also, will the future (1.8?) persister still do the process described
in https://dev.rabbitmq.com/wiki/RabbitPersisterDesign ? Writing the
entire rabbit state to disk every so-often doesn't seem like it would
work terribly well when storing huge amounts of data, or am I
understanding how the persister log works?
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 19, 2009 12:31 pm |
|
|
|
Guest
|
Tsuraan,
tsuraan wrote:
> One of our systems overfed the rabbit, and it's now a bit angry. The
> strange problem that we're having, though, is that it seems that the
> commit after the thousandth ack'd message is triggering the persister
> log rollover, which is then causing beam to run out of memory. The
> only thing we're doing is basic_get'ing messages, acking them, and
> then committing (the channel is transactional). We've also tried
> committing every thousand messages; either way, the first commit after
> the thousandth ack'd message is causing the persister log rollover,
> followed immediately by beam crashing due to being out of RAM.
The logic for when the persister log is rolled over is quite complex,
involving both time-based and message-count based heuristics.
As an aside, why are you using transactions on the consuming side? I
have yet to come across a use case where that is genuinely required.
> Does anybody know why rolling the persister log would cause the system
> to run out of memory? It seems like a strange place to need to
> allocate a lot of RAM
When rolling the persister log, rabbit writes a snapshot of all
currently persisted messages, and it needs to allocate memory for that
entire snapshot.
> Also, will the future (1.8?) persister still do the process described
> in https://dev.rabbitmq.com/wiki/RabbitPersisterDesign ?
No.
> Writing the entire rabbit state to disk every so-often doesn't seem
> like it would work terribly well when storing huge amounts of data
Indeed.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 19, 2009 4:11 pm |
|
|
|
Guest
|
> As an aside, why are you using transactions on the consuming side? I
> have yet to come across a use case where that is genuinely required.
Our consumers are almost always also producers, so we do the message
ack and the creation of new messages in a transaction. I'm not sure
that it's necessary; I'm used to databases, where I tend to do
everything in a transaction to ensure that everything is consistent
when I'm done, but our messaging stuff is designed to handle multiple
copies of the same message, so it's probably a bit overkill to also
use transactions.
I seem to remember that I had some problem when I was first looking at
rabbit that I could publish a stream of messages in non-transactional
mode, and not all the messages would be delivered (or enqueued). I
think I may have been doing something stupid like exiting my program
without closing my channel, but moving to transactional mode fixed the
problem, and I think I've been a bit religious about transactions
since then.
> When rolling the persister log, rabbit writes a snapshot of all
> currently persisted messages, and it needs to allocate memory for that
> entire snapshot.
Ok, that explains it.
>> Also, will the future (1.8?) persister still do the process described
>> in https://dev.rabbitmq.com/wiki/RabbitPersisterDesign ?
>
> No.
Glad to hear it 
Thanks!
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Oct 19, 2009 4:54 pm |
|
|
|
Guest
|
Tsuraan,
tsuraan wrote:
>> As an aside, why are you using transactions on the consuming side? I
>> have yet to come across a use case where that is genuinely required.
>
> Our consumers are almost always also producers, so we do the message
> ack and the creation of new messages in a transaction. I'm not sure
> that it's necessary; I'm used to databases, where I tend to do
> everything in a transaction to ensure that everything is consistent
> when I'm done, but our messaging stuff is designed to handle multiple
> copies of the same message, so it's probably a bit overkill to also
> use transactions.
>
> I seem to remember that I had some problem when I was first looking at
> rabbit that I could publish a stream of messages in non-transactional
> mode, and not all the messages would be delivered (or enqueued). I
> think I may have been doing something stupid like exiting my program
> without closing my channel, but moving to transactional mode fixed the
> problem, and I think I've been a bit religious about transactions
> since then.
Transactions are currently the only way for a client to get an
acknowledgment from the broker that sent messages have not only made it
to the broker (a normal connection/channel.close, or indeed any other
synchronous AMQP operation, is sufficient for that) but also will
survive a broker restart (if they were marked as persistent).
So use of tx mode on the sending side is quite common.
Use of tx mode on the receiving side, for acks, is much less common,
since the worst thing that can happen if an ack gets lost is that the
message will get delivered again. That can happen in other situations
anyway, e.g. client or connection failure, so apps need to be able to
cope with it already.
If your producers are also senders, and you want the aforementioned
guarantees on sending, I would recommend using two channels - one in tx
mode for sending and one on non-tx mode for receiving/ack'ing.
Regard,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
