|
|
| Author |
Message |
|
| 0x6e6562 |
 Posted: Thu Nov 08, 2007 6:59 am |
 |
|
|
User
Joined: 12 Jul 2007
Posts: 250
|
On 8 Nov 2007, at 03:36, Brian Granger wrote:
> On Nov 7, 2007 5:08 PM, Ben Hood <0x6e6562@gmail.com> wrote:
>>> Now you may wonder why we would want this. We have two types of
>>> messages: 1) messages that need to be sent to a particular client
>>> and
>>> 2) messages that can be sent to any one client. In this second
>>> case,
>>> we typically want to send the message to the least loaded client.
>>> Our
>>> messages are packets of work that the clients (they are workers)
>>> perform. We use the shared queue to perform dynamical load
>>> balancing
>>> for work that can be performed by any qualified worker. But that
>>> work
>>> still has to go through the private queues to preserve ordering.
>>>
>>> Does this make a bit more sense?
>>
>> Do your clients do anything contextually specific to the producer or
>> is it just that the work needs to get load balanced dynamically
>> according to the resources available to any given consumer?
>
> I am not quite sure what you mean here. But when clients submit work
> to the shared queue, it really is just for load balancing purposes.
> Can you give me an example of what "contextually specific" would mean
> in this case?
What I am trying to get at is find out whether the load balancing
could just be a concern of the queue infrastructure rather than
implementing it on the application layer. I am unsure why your
producers *need* to know about the existence of any consumers to
achieve your business requirements. Hence I was asking whether it had
to be producer driven rather than consumer driven. The reason that I
was imagining why you might want to make it producer driven is if
there was some kind of conversational state between a producer and a
consumer. Do you have different message types that can only be
consumed by certain consumers?
Ben
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
 
|
| Guest |
| Posted: Thu Nov 08, 2007 7:30 am |
|
|
|
Guest
|
> What I am trying to get at is find out whether the load balancing
> could just be a concern of the queue infrastructure rather than
> implementing it on the application layer.
At some level, I would like the load balancing to be handled in the
queue infrastructure, but I am not sure this is possible given our
other requirements....keep reading...
> I am unsure why your
> producers *need* to know about the existence of any consumers to
> achieve your business requirements.
This question definitely makes sense. In our system some types of
messages can only be consumed by particular workers. These message
bypass any load balancing. For these types of messages the producers
do need to be able to do things like 1) discover which consumers are
currently available 2) send messages to specific consumers and 3) get
back results from those consumers.
> Hence I was asking whether it had
> to be producer driven rather than consumer driven.
It is definitely producer driven in the sense the the producer is the
one making decisions about where to send what types of messages.
> The reason that I
> was imagining why you might want to make it producer driven is if
> there was some kind of conversational state between a producer and a
> consumer.
There is definitely conversational state between producers and
particular consumers. The consumer's also maintain a lot of internal
state. You could even look at our consumers as state machines that
change state in response to the messages.
> Do you have different message types that can only be
> consumed by certain consumers?
Yep.
Brian
>
> Ben
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
| 0x6e6562 |
| Posted: Thu Nov 08, 2007 8:43 am |
|
|
|
User
Joined: 12 Jul 2007
Posts: 250
|
On 8 Nov 2007, at 07:30, Brian Granger wrote:
Quote: This question definitely makes sense. |
|
|
| Back to top |
|
| alexis |
| Posted: Fri Nov 09, 2007 11:48 am |
|
|
|
User
Joined: 06 Sep 2007
Posts: 80
Location: London
|
Brian
Very interesting stuff 
I'm addressing a subset of your questions since Ben has sensibly
forked off the conversation about conversational messaging.
On Nov 7, 2007 8:42 PM, Brian Granger <ellisonbg.net@gmail.com> wrote:
>
> We are using this architecture for parallel computing in Python. Our
> workers are basically Python interpreters exposed to the network. Our
> "clients" are also python scripts that submit python code/objects to
> the workers to be performed. As you can imagine I am looking at the
> qpid python amqp client. We have run already with up to 256 workers
> and we already see scaling problems.
Is this with RabbitMQ as is? Can you shed a little more light on this please.
> Our stuff works just fine on
> supercomputers, so we would like to scale up to.....2**10, 2**11,
> 2**12, etc. We hit scaling problems in a number of ways:
>
> 1) # of file descriptors per process.
> 2) broadcasting/multicasting to all (or large subsets) of workers.
> 3) Latency - the latency determines the grain size of work, which we
> would like to be as small as possible.
If you could share some of the numbers with the list, that would be
very interesting.
> 4) Large messages. In some cases, the messages can be very large
> (files or large data sets). It is not uncommon for people to want to
> distribute 100 MB messages to subsets of workers. Our current
> solution generates a short lived temporary copy of the large objects
> for each worker they are sent to. This means if we have 128 workers,
> we use 128*msg_size memory for a short period of time.
Eek. And does each worker need to have a local copy of the whole
message, every time, to work from?
(As an aside one fun idea we have looked at is bittorrent style
sharing of data between broker peers, essentially implementing a DHT).
> We know how to
> fix this in Python, but it still doesn't solve the other scalability
> problems. All that to say, any solution that we come up with must
> also be able to handle large messages well. Does rabbitmq have any
> streaming support?
I shall leave this one for Tony.
> I would love to avoid the lower level stuff, but I also realize we
> have a somewhat unusual set of design requirements (scalability,
> low-latency && ability to handle large messages). In various
> prototypes we even implemented custom protocols on top of TCP. Once
> you start doing that Erlang becomes a "high-level" solution
Indeed
I still think you should be ok staying above the waterline and coding
to RabbitMQ, at least for now.
> > Cool. Please let us know if you need help, eg with the deregistration methods.
>
> Ahh yes, the deregistration methods could be subtle. The main concern
> there is if a worker dies without notifying the server that they are
> going to die. What facilities does rabbitmq/amqp have to handle this
> type of stuff?
Do you mean in the sense of leasing? I'll see what we can say about that.
One point to note in passing is that Erlang/OTP provides the notion of
a 'supervisor', see eg here:
http://www.computer.org/portal/site/dsonline/menuitem.9ed3d9924aeb0dcd82ccc6716bbe36ec/index.jsp?&pName=dso_level1&path=dsonline/2007/12&file=w6tow.xml&xsl=article.xsl&;jsessionid=Hp7BjTtbB6JYdhP2H6p6v61x4GFCfp0CF6lXF59Y7dTvjCdGFLPG!1017941068
alexis
--
Alexis Richardson
+44 20 7617 7339 (UK)
+44 77 9865 2911 (cell)
+1 650 206 2517 (US)
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post recived from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
