| Author |
Message |
|
| Guest |
 Posted: Mon Dec 14, 2009 10:14 pm |
 |
|
|
Guest
|
Hi,
Can you please help me understand what is going on when I lose a member of a cluster?
I'm seeing an odd behaviour that makes me worry. This queue is supposed to hold
credit card transactions, so we _really_ want to avoid double-processing messages.
Cluster consists of 2 members: (rabbit@mq1, rabbit@mq2).
1) client connects to mq1, creates durable queue, pushes 100 persistent messages in
|
|
|
| Back to top |
|
| Guest |
| Posted: Mon Dec 14, 2009 10:35 pm |
|
|
|
Guest
|
I apologize for not answering your Rabbit question. What you are
describing is why you need an application-layer notion of idempotence.
It is at best frustrating and at worst dangerous to rely on a lower
level component, especially one written by someone else without this
exact use case in mind, to implement and enforce this sort of policy.
Best of luck,
b
On 12/14/09 2:13 PM, Robert Borkowski wrote:
> Hi,
>
> Can you please help me understand what is going on when I lose a member of a
> cluster?
> I'm seeing an odd behaviour that makes me worry. This queue is supposed to
> hold
> credit card transactions, so we _really_ want to avoid double-processing
> messages.
>
> Cluster consists of 2 members: (rabbit@mq1, rabbit@mq2).
>
> 1) client connects to mq1, creates durable queue, pushes 100 persistent
> messages in
> root@mq1:~# rabbitmqctl list_queues
> Listing queues ...
> testqueue 100
> ...done.
>
> 2) mq1 rabbitmq is stopped
>
> 3) client connects to mq2, creates durable queue, pushes 100 persistent
> messages in
> root@mq2:~# rabbitmqctl list_queues
> Listing queues ...
> testqueue 100
> ...done.
>
> 4) mq1 rabbitmq is started
> root@mq2:~# rabbitmqctl list_queues
> Listing queues ...
> testqueue 200
> ...done.
>
> 5) client drains the queue
> rborkows@mq1:~/client/bunny$ ./bunny-subscriber
> Drained 200 messages in 0.475371 seconds
>
> 6) queue is now empty
> root@mq2:~# rabbitmqctl list_queues
> Listing queues ...
> testqueue 0
> ...done.
>
> 7) mq1 rabbitmq is restarted, 100 credit card transactions pop back into
> existence, unhappy customers, etc.
> root@mq1:~# /etc/init.d/rabbitmq-server restart
> Restarting rabbitmq-server: SUCCESS
> rabbitmq-server.
> root@mq1:~# rabbitmqctl list_queues
> Listing queues ...
> testqueue 100
> ...done.
>
> The expected behaviour is that after step 5 the messages are _gone_ from the
> queue, never to be seen again.
> What's going on here?
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Dec 14, 2009 10:57 pm |
|
|
|
Guest
|
Agreed, and that's why I'm testing failure scenarios.
It would be nice to not to have to re-invent the wheel 
Isn't this a somewhat common use-case? A work queue filled with
tasks that can only be run once, running on real hardware which
means that at some point parts of the system WILL fail.
--
Robert Borkowski
On Mon, Dec 14, 2009 at 5:34 PM, Benjamin Black <b@b3k.us (b@b3k.us)> wrote:
Quote: I apologize for not answering your Rabbit question. |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Dec 14, 2009 11:27 pm |
|
|
|
Guest
|
On 12/14/09 2:57 PM, Robert Borkowski wrote:
> Agreed, and that's why I'm testing failure scenarios.
>
> It would be nice to not to have to re-invent the wheel
>
> Isn't this a somewhat common use-case? A work queue filled with
> tasks that can only be run once, running on real hardware which
> means that at some point parts of the system WILL fail.
>
No doubt, great candidate for a plugin. Not something appropriate for
core of Rabbit, imo.
b
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Dec 14, 2009 11:31 pm |
|
|
|
Guest
|
Robert Borkowski wrote:
> Agreed, and that's why I'm testing failure scenarios.
>
> It would be nice to not to have to re-invent the wheel
>
> Isn't this a somewhat common use-case? A work queue filled with
> tasks that can only be run once, running on real hardware which
> means that at some point parts of the system WILL fail.
"at-least once" and "at-most once" execution are quite easy to do.
"exactly once" is generally impossible. In case that is not obvious,
think about what precisely it is that you want to happen exactly once.
Then try to figure out how you'd perform that action *and record the
fact that it has happened* in one atomic action.
By far the easiest way to get around that is to make the desired action
idempotent. Then it is sufficient to guarantee "at-least once"
execution, which is easy to do in AMQP by ack'ing messages after the
actions that they trigger have been completed.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Mon Dec 14, 2009 11:34 pm |
|
|
|
Guest
|
So what is going on in my test case?
I restart rabbitmq and I suddenly have a bunch of old messages in the queue. Surely that's not expected
--
Robert Borkowski
On Mon, Dec 14, 2009 at 6:29 PM, Matthias Radestock <matthias@lshift.net (matthias@lshift.net)> wrote:
Quote: Robert Borkowski wrote:
Quote: Agreed, and that's why I'm testing failure scenarios.
It would be nice to not to have to re-invent the wheel
Isn't this a somewhat common use-case? A work queue filled with
tasks that can only be run once, running on real hardware which
means that at some point parts of the system WILL fail.
"at-least once" and "at-most once" execution are quite easy to do. "exactly once" is generally impossible. In case that is not obvious, think about what precisely it is that you want to happen exactly once. Then try to figure out how you'd perform that action *and record the fact that it has happened* in one atomic action.
By far the easiest way to get around that is to make the desired action idempotent. Then it is sufficient to guarantee "at-least once" execution, which is easy to do in AMQP by ack'ing messages after the actions that they trigger have been completed.
Regards,
Matthias.
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Tue Dec 15, 2009 12:23 am |
|
|
|
Guest
|
Robert,
Robert Borkowski wrote:
> So what is going on in my test case?
>
> I restart rabbitmq and I suddenly have a bunch of old messages in the
> queue. Surely that's not expected
Things actually went wrong way before then, namely when the client
created a queue with the same name on mq2. Rabbit is a bit schizophrenic
as to whether it considers that queue to be the same queue or a
different queue.
That is something we've known about for a while. See, for example, Jason
William's post and my follow-up at
http://old.nabble.com/Clustering-Question-td23888965.html#a23932367
What I didn't realise then is that the acknowledgements for the messages
from the queue on the first node will go astray, which is why those
message pop up again when that node is restarted. That clearly is
perplexing, so I have raised the priority of this bug in our issue tracker.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
