| Author |
Message |
|
| Guest |
 Posted: Fri Sep 25, 2009 7:01 pm |
 |
|
|
Guest
|
I was running some tests against RabbitMQ (1.6 on Linux)
One of the test was to run steady load (about 500 messages per second) over long period of time.
My client is using Java library.
I noticed that performance steadily going down to 100-200 messages/sec, resetting client (kill/start process) restores performance.
I don’t see any issues with either disk io, memory or CPU – it is all stays same all the time on client, only client accepts less and less messages.
I’m using basicConsume with QoS 10 messages.
|
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 7:21 pm |
|
|
|
Guest
|
Philippe,
Philippe Kirsanov wrote:
> One of the test was to run steady load (about 500 messages per second)
> over long period of time.
>
> I noticed that performance steadily going down to 100-200 messages/sec,
> resetting client (kill/start process) restores performance.
can you post your code so we can try to reproduce this?
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 9:29 pm |
|
|
|
Guest
|
The jar includes all dependencies.
I can't send source code, only jar. But this is a code:
Publisher:
messageProperties.messageId = utils.generateUUIDasBase64 //random
UUID
messageProperties.expiration = System.currentTimeMillis.toString
channel.basicPublish ( exchName, routingKey, messageProperties , msg )
Consumer:
private val tag = channel.basicConsume ( queueName, noAck,
new DefaultConsumer ( channel ) {
override def handleDelivery ( tag : String, env :
Envelope, props : AMQP.BasicProperties, body : Array [ Byte ] ) = {
if ( brokerConfig.trackMsg ) {
val now = System.currentTimeMillis
val than = props.expiration.toLong
actor {
statTracker.update ( now - than
)
}
}
cons ! newMessage ( body, env, props )
}
}
)
Ack happens in the Actor that handles message before message is
processed
This Actor Ack message and send it for processing to another Actor
react {
case msg @ newMessage ( m, e, p ) => {
a ! msg // send message for processing in different
Actor(Thread)
try {
channel.basicAck ( e.getDeliveryTag, false )
}
catch
{
case e : AlreadyClosedException => exit
case e : java.net.SocketException => exit
}
act
}
}
And this is actual worker that handles message, message should be
already ack by this time:
receive {
case msg @ newMessage ( m, e, p ) => handler ( m, e, p )
}
I will send jar directly
-----Original Message-----
From: Matthias Radestock [mailto:matthias@lshift.net]
Sent: Friday, September 25, 2009 15:21
To: Philippe Kirsanov
Cc: rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Steady performance degradation
Philippe,
Philippe Kirsanov wrote:
> One of the test was to run steady load (about 500 messages per second)
> over long period of time.
>
> I noticed that performance steadily going down to 100-200
messages/sec,
> resetting client (kill/start process) restores performance.
can you post your code so we can try to reproduce this?
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 9:41 pm |
|
|
|
Guest
|
Philippe,
Philippe Kirsanov wrote:
> The jar includes all dependencies.
> I can't send source code, only jar. But this is a code:
The code looks fine. And there is no slowdown when running code very
similar to this, e.g. try the MulticastMain example that ships with the
Java client and set the rate with "-r 500".
So I strongly suspect the problem is somewhere in the client code, which
is going to be difficult for us to track down if a) we can't see it, or
b) it's tangled up in a bigger code base with non-obvious control flow.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 9:52 pm |
|
|
|
Guest
|
I'm quite sure it is client code, but I was thinking it is Java lib. My
code don't do anything right now - just receives a message and
calculates statistics (2 variable, no arrays).
If you're sure it's not Java lib I'll keep looking. Strange thing is
that no recourse exhaustion of any kind, just fewer packets arrived to
client. My measure is between basicPublish and handleDelivery, with no
my code involvement at all.
Thanks for your help.
-----Original Message-----
From: Matthias Radestock [mailto:matthias@lshift.net]
Sent: Friday, September 25, 2009 17:42
To: Philippe Kirsanov
Cc: rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Steady performance degradation
Philippe,
Philippe Kirsanov wrote:
> The jar includes all dependencies.
> I can't send source code, only jar. But this is a code:
The code looks fine. And there is no slowdown when running code very
similar to this, e.g. try the MulticastMain example that ships with the
Java client and set the rate with "-r 500".
So I strongly suspect the problem is somewhere in the client code, which
is going to be difficult for us to track down if a) we can't see it, or
b) it's tangled up in a bigger code base with non-obvious control flow.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Fri Sep 25, 2009 10:07 pm |
|
|
|
Guest
|
I noticed that in example you create connection for every
consumer/producer.
I'm using just one connection and different channels for every consumer.
I read somewhere that it's best practice to have one connection/multiple
channels per server. Is this true or it's better to have separate
connection for every channel? This might result in lots of connection to
broker.
-----Original Message-----
From: Matthias Radestock [mailto:matthias@lshift.net]
Sent: Friday, September 25, 2009 15:21
To: Philippe Kirsanov
Cc: rabbitmq-discuss@lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] Steady performance degradation
Philippe,
Philippe Kirsanov wrote:
> One of the test was to run steady load (about 500 messages per second)
> over long period of time.
>
> I noticed that performance steadily going down to 100-200
messages/sec,
> resetting client (kill/start process) restores performance.
can you post your code so we can try to reproduce this?
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Sat Sep 26, 2009 4:07 am |
|
|
|
Guest
|
Philippe,
Philippe Kirsanov wrote:
> I'm quite sure it is client code, but I was thinking it is Java lib. My
> code don't do anything right now - just receives a message and
> calculates statistics (2 variable, no arrays).
> If you're sure it's not Java lib I'll keep looking.
I can't be sure of anything, given that I haven't seen the complete
code. But the order of suspects is: the application code, the
system/network setup, the Java client lib, the server.
> I noticed that in example you create connection for every
> consumer/producer.
> I'm using just one connection and different channels for every consumer.
> I read somewhere that it's best practice to have one connection/multiple
> channels per server. Is this true or it's better to have separate
> connection for every channel? This might result in lots of connection to
> broker.
If client or server resource consumption is a concern, then few
connections & lots of channels per connection is preferable to lots of
connections & few channels per connection If, OTOH, resources are
plentiful and you want to maximise performance then the decision is
reversed.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Sat Sep 26, 2009 4:27 am |
|
|
|
Guest
|
Philippe,
Matthias Radestock wrote:
> I can't be sure of anything, given that I haven't seen the complete
> code. But the order of suspects is: the application code, the
> system/network setup, the Java client lib, the server.
Also, the following would be on my list of things to check:
1) Is the test exhausting some server resources, in particular CPU or
network bandwidth?
2) Is the measurement code correct?
3) You mentioned the use of a basic.qos prefetch limit - do you get the
same results w/o that?
4) Are messages marked as persistent? If so, do you get the same results
w/o that?
5) Are you using tx at all? If so, do you get the same results w/o that?
I would also recommend that you simplify your test code as much as
possible. If you can condense it to something along the lines of the
shipped examples then a) you can send it to us, and b) it will be much
easier to track down where the problem may be.
Regards,
Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
