| Author |
Message |
|
| Guest |
 Posted: Thu Oct 01, 2009 8:05 pm |
 |
|
|
Guest
|
Does anyone know what rabbitmq throughput is from items sitting in the queue to sending them to a consumer?
Lets say we're doing 200 items in the queue a second with 50 consumers waiting to process the item, usually they do in < 30ms. Watching our consumer logs, it feels like they don't get items fast enough from rabbitmq, is that possible or is it me?
I am going to try adding another rabbit queue today to distribute it and see if it churns through faster.
Suhail
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 01, 2009 8:32 pm |
|
|
|
Guest
|
Suhail,
Suhail Doshi wrote:
> Does anyone know what rabbitmq throughput is from items sitting in the
> queue to sending them to a consumer?
>
> Lets say we're doing 200 items in the queue a second with 50 consumers
> waiting to process the item, usually they do in < 30ms.
So that's ~7kHz, which doesn't look unreasonable, but there are many
factors at play here which we'd need to know about in order to make a
better assessment:
- are these messages persistent or transient?
- how big are the messages?
- do the consumer consume in no-ack mode, or ack mode?
- are the consumers sharing a connection (or even channel) or do they
all have their separate connection/channel?
- where are the consumers in the network topology in relation to the server?
- what's the hardware spec of the server machine?
- what version of Erlang are you running?
- what AMQP client are you using?
Also, ...
> Watching our consumer logs, it feels like they don't get items fast
> enough from rabbitmq, is that possible or is it me?
Can you instrument the consumers to measure how long they are blocked
waiting for messages from RabbitMQ?
Also, with 200 messages shared across 50 consumers each consumer will
only get four items, which is a rather small figure to do any meaningful
statistical analysis with.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 01, 2009 8:41 pm |
|
|
|
Guest
|
On Thu, Oct 1, 2009 at 1:32 PM, Matthias Radestock <matthias@lshift.net (matthias@lshift.net)> wrote:
Quote: Suhail,
Suhail Doshi wrote:
Quote: Does anyone know what rabbitmq throughput is from items sitting in the queue to sending them to a consumer?
Lets say we're doing 200 items in the queue a second with 50 consumers waiting to process the item, usually they do in < 30ms.
So that's ~7kHz, which doesn't look unreasonable, but there are many factors at play here which we'd need to know about in order to make a better assessment:
- are these messages persistent or transient? Persistent |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 01, 2009 8:57 pm |
|
|
|
Guest
|
Turns out I already output info to basically notice if the consumer is blocking:
2009-10-01 20:55:27,935 INFO [id: 99704402] Processing complete, acknowledged.
2009-10-01 20:55:27,939 INFO [id: 99704402] Received queue item, processing...
2009-10-01 20:55:29,215 INFO [id: 99704402] Processing complete, acknowledged.
2009-10-01 20:55:29,215 INFO [id: 99704402] Received queue item, processing...
You'll notice there's a gap between receiving and processing so that means the consumer is just waiting for an item. So it really does look like rabbitmq is not able to get items to me fast enough with lots of items backed in the queue.
Suhail
On Thu, Oct 1, 2009 at 1:40 PM, Suhail Doshi <suhail@mixpanel.com (suhail@mixpanel.com)> wrote:
Quote:
On Thu, Oct 1, 2009 at 1:32 PM, Matthias Radestock <matthias@lshift.net (matthias@lshift.net)> wrote:
Quote: Suhail,
Suhail Doshi wrote:
Quote: Does anyone know what rabbitmq throughput is from items sitting in the queue to sending them to a consumer?
Lets say we're doing 200 items in the queue a second with 50 consumers waiting to process the item, usually they do in < 30ms.
So that's ~7kHz, which doesn't look unreasonable, but there are many factors at play here which we'd need to know about in order to make a better assessment:
- are these messages persistent or transient? Persistent |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 01, 2009 9:19 pm |
|
|
|
Guest
|
Suhail,
Suhail Doshi wrote:
> Persistent, ack mode
in which case 7KHz outbound is quite reasonable. For comparison, I
suggest you try transient messages, and no-ack mode.
> So it really does look like rabbitmq is not able to get items to me
> fast enough with lots of items backed in the queue.
You mentioned 200 items, which really isn't a lot.
How fast is "fast enough"? You have *50* consumers draining from a
single queue. No matter how fast rabbit is, if you add enough consumers
eventually rabbit won't be able to keep them all busy.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 01, 2009 11:04 pm |
|
|
|
Guest
|
On Thu, Oct 1, 2009 at 2:19 PM, Matthias Radestock <matthias@lshift.net (matthias@lshift.net)> wrote:
Quote: Suhail,
Suhail Doshi wrote:
|
|
|
| Back to top |
|
| Guest |
| Posted: Fri Oct 02, 2009 1:45 am |
|
|
|
Guest
|
>> How fast is "fast enough"? You have *50* consumers draining from a single
>> queue. No matter how fast rabbit is, if you add enough consumers
>> eventually
>> rabbit won't be able to keep them all busy.
>>
>> My expectation is that rabbit should push a new item to my consumer once
> that consumer has ACKed a message in less than 100ms
Out of curiosity, are you using basic_get, or basic_consume? I've
found that basic_get has performance like that, whereas basic_consume
is at least 10x faster. I'm also using persistent messages with
explicit ack (in transactional mode), and using Barry's python library
I get in the 1k - 10k messages/second range on hardware that isn't any
better than yours.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Oct 02, 2009 1:57 am |
|
|
|
Guest
|
On Thu, Oct 1, 2009 at 9:44 PM, tsuraan <tsuraan@gmail.com (tsuraan@gmail.com)> wrote:Quote: Out of curiosity, are you using basic_get, or basic_consume? |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Oct 02, 2009 4:32 am |
|
|
|
Guest
|
|
| Back to top |
|
| Guest |
| Posted: Fri Oct 02, 2009 7:27 am |
|
|
|
Guest
|
Suhail,
Suhail Doshi wrote:
> 200 items per second isn't a lot?
I misread your original post. I thought you said there were 200 items in
the queue that the 50 consumers are then draining in 30ms, whereas
actually you were saying that messages get published to the queue at a
rate of 200Hz (which still isn't a lot) and you have 50 consumers which
take <30ms to process each item. Right?
> My expectation is that rabbit should push a new item to my consumer once
> that consumer has ACKed a message in less than 100ms
Are you using qos? Only with qos will rabbit wait for the acks before
sending more messages; otherwise it will just keep sending them.
Also, are you using tx mode on either the publishing or consuming side?
Depending on the size of the transactions that can slow things down
significantly since a tx commit is forced to wait for a disk fsync.
Also, how long does the queue get?
As others have said, if you have things set up correctly you should be
able to see a throughput of thousands of Hz.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
