| Author |
Message |
|
| Guest |
 Posted: Wed Oct 31, 2007 6:45 pm |
 |
|
|
Guest
|
Hello all,
My scenario might be common and I am hoping someone might be working on a similar issue.
I set up a cluster of 2 nodes - rabbit@node1 and rabbit@node2. Several producers will
connect to either node and send a series of messages with topic hierarchy as routing key
(for example, "a.b.c.d.e.f"). Consumers will also connect to either node and will register
to receive some subset of what is being sent - in other words, for example, some of them
will request "a.#" while others might want to get only "#.e.f". My goal is to make it
happen if at least one of the nodes in the cluster is up and running. Both producers and
consumers will know about existence of both cluster nodes, but they need to be able to
randomly connect to either.
Any thoughts on how this setup can be accomplished? These requirements are somewhat
flexible as long as I achieve a design with no single points of failure.
One scenario that I can think of is that I can force all clients (both producers and
consumers) to connect to the same node. They will re-connect to another node only if the
first node went down. This will achieve active-passive redundancy but is not as flexible...
Thanks,
Dmitriy
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| 0x6e6562 |
| Posted: Thu Nov 01, 2007 7:22 am |
|
|
|
User
Joined: 12 Jul 2007
Posts: 250
|
Dmitriy,
> I set up a cluster of 2 nodes - rabbit@node1 and rabbit@node2. Several producers will
> connect to either node and send a series of messages with topic hierarchy as routing key
> (for example, "a.b.c.d.e.f"). Consumers will also connect to either node and will register
> to receive some subset of what is being sent - in other words, for example, some of them
> will request "a.#" while others might want to get only "#.e.f". My goal is to make it
> happen if at least one of the nodes in the cluster is up and running. Both producers and
> consumers will know about existence of both cluster nodes, but they need to be able to
> randomly connect to either.
As far as I know, message queues are not yet fully clustered in
rabbitmq, meaning that you cannot yet transparently failover or
migrate a queue to a different node in the cluster. These queues are
visible from any node in the cluster however.
HTH,
Ben
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
 
|
| Guest |
| Posted: Sat Nov 03, 2007 10:33 am |
|
|
|
Guest
|
Dmitriy,
Ben Hood wrote:
> Dmitriy,
>
>> I set up a cluster of 2 nodes - rabbit@node1 and rabbit@node2.
>> Several producers will connect to either node and send a series of
>> messages with topic hierarchy as routing key (for example,
>> "a.b.c.d.e.f"). Consumers will also connect to either node and will
>> register to receive some subset of what is being sent - in other
>> words, for example, some of them will request "a.#" while others
>> might want to get only "#.e.f". My goal is to make it happen if at
>> least one of the nodes in the cluster is up and running. Both
>> producers and consumers will know about existence of both cluster
>> nodes, but they need to be able to randomly connect to either.
> As far as I know, message queues are not yet fully clustered in
> rabbitmq, meaning that you cannot yet transparently failover or
> migrate a queue to a different node in the cluster. These queues are
> visible from any node in the cluster however.
Ben is right about the (lack of) queue clustering. Whether or not that
matters depends on what guarantees you expect on message delivery: Can
you afford to lose messages? Is it ok for a message to be delivered to a
consumer more than once?
You can guarantee exactly-once delivery if you use transactions, durable
queues and exchanges, and persistent messages, but only as long as any
failing node eventually recovers.
If you can afford to lose the odd message then the simplest setup is to
just have consumers detect when connections goes down, connect to the
other node and recreate their queues. Similarly, producers just
re-connect to the second node on failure of the first node, and start
sending messages to the same exchange as before.
If message loss is bad but duplication is ok then you can could set up
two parallel routing topologies with producers sending messages to both
nodes and each consumer consuming from two queues - one on each node.
There are various other options; which one is best really depends on the
specific requirements of your application.
All this is going to improve significantly in AMQP 0-10, where there is
a notion of a "session" whose lifecycle is independent from that of the
transport connection. Sessions can transparently reconnect to other
nodes and have built-in support for exactly-once delivery. We'd still
need to add queue clustering to RabbitMQ, but that's on our roadmap
already anyway.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| alexis |
| Posted: Sat Nov 03, 2007 11:22 am |
|
|
|
User
Joined: 06 Sep 2007
Posts: 80
Location: London
|
On 11/3/07, Matthias Radestock <matthias@lshift.net> wrote:
>
> You can guarantee exactly-once delivery if you use transactions, durable
> queues and exchanges, and persistent messages, but only as long as any
> failing node eventually recovers.
>
> If you can afford to lose the odd message then the simplest setup is to
> just have consumers detect when connections goes down, connect to the
> other node and recreate their queues. Similarly, producers just
> re-connect to the second node on failure of the first node, and start
> sending messages to the same exchange as before.
>
> If message loss is bad but duplication is ok then you can could set up
> two parallel routing topologies with producers sending messages to both
> nodes and each consumer consuming from two queues - one on each node.
>
> There are various other options; which one is best really depends on the
> specific requirements of your application.
All true.
Another MQ-like pattern that works for 0-8/0-9, for some cases, is
applicable when message loss is bad, and 'eventual delivery' is
desired. If you have a local disk, then run a second broker on the
client machine, in persistent mode. Then, write a consumer which
copies messages from the local broker on the client, to the broker on
the server. This may feel clunky but it lets the client application
'hand off' delivery to the local queue reliably, provided that the
local disk can be trusted. Then delivery to the server and hence to
the ultimate message recipient will eventually occur assuming there is
a network.
As Matthias pointed out these patterns for reliability, though
possible today, will be supported better in 0-10.
alexis
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
