| Author |
Message |
|
| Guest |
 Posted: Tue Oct 13, 2009 10:57 am |
 |
|
|
Guest
|
|
| Back to top |
|
| Guest |
| Posted: Tue Oct 13, 2009 11:20 am |
|
|
|
Guest
|
On Tue, Oct 13, 2009 at 12:56:56PM +0200, Lynton Grice wrote:
> A long time ago I heard Mnesia has a 2GB storage limit.
That information is not correct. Mnesia has several different ways of
operating a table:
1. ram_copies. The table is stored in an ets table in memory and is
never written to disk. The only limit is the amount of available memory.
No 2GB limit here.
2. disc_copies. The table is stored in an ets table in memory, and is
periodically written to disk using the disk_log module (glad to see even
within Erlang's only libraries, they can't decide how to spell disk).
Again, there is no limit in size.
3. disc_only_copies. The table is stored in a dets table only on disk.
Dets tables have 32-bit internal pointers within the file, so you are
limited to 2GB.
Wonderfully, this means that if you happen to have a table in
disc_copies mode which is > 2GB, and you then decide you want to save
memory and force it to disc_only_copies, you blow up the table as dets
can't cope with tables that big.
> My question is what
> happens in the case were a company wanting to use RabbitMQ needs larger
> persistence? Say like 8GB?
The persister does not use Mnesia in any way. This is true of the
current persister in 1.7 and the future persister. We are not using
mnesia for the persister entirely because of its limitations.
The persister in 1.7 keeps all messages in memory, so you're entirely
bound by the amount of RAM available. The new persister will not have
this limitation and will be able to scale much better than the current
persister. I realise that we've been making noises about the "new"
persister since April, and we had hoped to get it out with 1.7, but hey,
software development is never a predictable activity!
> I have just installed the 1.7 release and it is super, great work guys!!
Good to hear! 
Matthew
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Tue Oct 13, 2009 12:16 pm |
|
|
|
Guest
|
Hi Matthew,
Thanks for the detailed response, and I really look forward to what the
persister will be doing in future releases, sounds great.
The SYNC transactions are not my concern, I do not really mind if those
messages get lost, it is the ASYNC ones that I want 100% guarantee they will
never get lost.
You said " With the persister in 1.7, messages are kept in RAM but are also
written out to disk." - are the messages written to disk by default as well?
As in the choice between "ram_copies", "disk_copies" and "disc_only_copies"?
Or are the messages simple written anyway in the 1.7 persister?
Using transactions sounds like the answer for me for now, thanks.
Last question: Do you ever forsee a day when there is no 2GB limit to DETS?
Thanks again 
Lynton
-----Original Message-----
From: Matthew Sackman [mailto:matthew@lshift.net]
Sent: 13 October 2009 02:03 PM
To: Lynton Grice
Subject: Re: [rabbitmq-discuss] RabbitMQ 1.7 and persistence limit?
Hi Lynton,
Please keep rabbitmq-discuss copied in unless you have specific reason
not to, that way others can benefit from discussions.
On Tue, Oct 13, 2009 at 01:36:28PM +0200, Lynton Grice wrote:
> Forgive me if I am wrong here but in my case I want the stuff to happily
> survive a system crash ..so:
>
> - "ram_copies" - in a system crash the messages will be lost
> - "disc_copies" - much better with the ETS and disk_log stuff BUT still
the
> *possibility* that message that are ONLY in ETS tables will get "lost" on
a
> server crash
Yes, there is the possibility that mnesia dirty operations will be lost,
but provided you use sync_transactions, you still get the guarantee that
they've been written to disk. The main gain from keeping stuff in RAM is
for speed of lookups, and also writes can be batched in some places.
> - "disc_only_copies" - although probably a little slower that the "memory
> messaging", this is the one I like, but it does have the 2GB limit ;-(
Oh, dets is *very* slow indeed. In fact it's awful - I've never seen
dets drive a disk at above 10MB/s. The new persister can easily generate
enough IO in sensible patterns to be able to totally saturate disk
bandwidth - 70+MB/s sustained is very achievable.
> You say " The persister in 1.7 keeps all messages in memory..." - then may
I
> ask a stupid question, what happens if you have 10 000 message with the
new
> persister in memory and the server bounces?
Hang on, what I'm referring to by the "new" persister, is the persister
that we're in the process of developing and is not in 1.7. With the
persister in 1.7, messages are kept in RAM but are also written out to
disk. If you exhaust RAM then yes, you will blow up RabbitMQ, though
RabbitMQ does try quite hard these days to use flow control to stop
producers when RAM gets really tight, so it should be reasonably
resiliant. However, see, eg Brian Whitman's post:
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-October/005095.htm
l
Here, he is talking about a very experimental version of the new,
unreleased persister, which demonstrates that it is able to scale vastly
beyond the amount of available RAM now.
> I am a HUGE RabbitMQ fan, I am just trying to see how safe these "memory
> messages" really are from System failure?
Pretty much the only way of guaranteeing that a message has been sent to
disk and stored safely is to put the channel in transactional mode and
do a commit. When you get back the commit-ok, you know it's either
already been delivered to a client and acknowledged, or that it's been
written to disk. In all other cases, there are very few hard guarantees
offered - writing a persistent message to a queue without using
transactions means that there will be a window after the client has
published the message during which the message has not been fsync'd on
disk and at that point, a crash *may* result in messages being lost. As
ever, it's a tradeoff - if you use transactions, the extra fsync's all
over the place will hurt performance quite badly, but you get guarantees
that you otherwise wouldn't get.
What we tend to recommend is that you really think about how much
message loss you can cope with - most applications can cope with, say, a
minute's worth of messages being lost in the event of system failure as
it's possible the messages can be regenerated through other means.
Matthew
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Tue Oct 13, 2009 12:34 pm |
|
|
|
Guest
|
Lynton,
On Tue, Oct 13, 2009 at 02:15:46PM +0200, Lynton Grice wrote:
> The SYNC transactions are not my concern, I do not really mind if those
> messages get lost, it is the ASYNC ones that I want 100% guarantee they will
> never get lost.
I'm completely confused by what you mean by this. That is just not
possible. If you send an async message, eg basic.publish, there are
probably a good few dozen buffers in your local computer, the language
you're writing in, the OS network stack, the network card itself, and so
forth, all in which the message can be stored and lost in the event of
power failure. Not to mention buffers in network hubs/switches/routers,
and then in Rabbit itself, all of which can hold the message
temporarily, after it's been "sent" and before it's been written to
disk.
Unless you get some sort of reply from Rabbit, which says that "yes,
I've received the message and it's been written to disk", there is *no*
guarantee that any message you send has even been received by Rabbit,
let alone safely stored.
> You said " With the persister in 1.7, messages are kept in RAM but are also
> written out to disk." - are the messages written to disk by default as well?
> As in the choice between "ram_copies", "disk_copies" and "disc_only_copies"?
> Or are the messages simple written anyway in the 1.7 persister?
By definition, persistent messages are written to disk. They also happen
to be stored in RAM in the persister in 1.7 (and before), for a variety
of reasons. This has *nothing* to do with the various modes of operation
of mnesia, as the persister does not use mnesia at all.
> Using transactions sounds like the answer for me for now, thanks.
Right, and thus this is effectively synchronous messaging as you get the
reply back from Rabbit - the commit-ok - when you commit on the channel.
> Last question: Do you ever forsee a day when there is no 2GB limit to DETS?
It's possible that the Erlang/OTP folks will get round to rewriting it
to make it 64-bit safe, I don't know. Certainly it's not something we're
looking at doing.
Matthew
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Tue Oct 13, 2009 12:59 pm |
|
|
|
Guest
|
Hi there,
I am an SAP PI / XI consultant by trade so I guess my "message
acknowledgement" from a "broker" can also been seen as your SYNC whereby the
sending client gets an "ok" back from RabbitMQ.....I understand now
...thanks
And yes, you are 100% correct on the "buffers" along the way...
But most importantly is that you have cleared my thoughts that Mnesia was
used for persistence....great....and that it is actually the Erlang standard
DETS stuff that is the "problem"....I hope they make it 64 bit one day
You have helped a lot and got my brain around a few things
Chat later
Lynton
-----Original Message-----
From: rabbitmq-discuss-bounces@lists.rabbitmq.com
[mailto:rabbitmq-discuss-bounces@lists.rabbitmq.com] On Behalf Of Matthew
Sackman
Sent: 13 October 2009 02:34 PM
To: 'rabbitmq'
Subject: Re: [rabbitmq-discuss] RabbitMQ 1.7 and persistence limit?
Lynton,
On Tue, Oct 13, 2009 at 02:15:46PM +0200, Lynton Grice wrote:
> The SYNC transactions are not my concern, I do not really mind if those
> messages get lost, it is the ASYNC ones that I want 100% guarantee they
will
> never get lost.
I'm completely confused by what you mean by this. That is just not
possible. If you send an async message, eg basic.publish, there are
probably a good few dozen buffers in your local computer, the language
you're writing in, the OS network stack, the network card itself, and so
forth, all in which the message can be stored and lost in the event of
power failure. Not to mention buffers in network hubs/switches/routers,
and then in Rabbit itself, all of which can hold the message
temporarily, after it's been "sent" and before it's been written to
disk.
Unless you get some sort of reply from Rabbit, which says that "yes,
I've received the message and it's been written to disk", there is *no*
guarantee that any message you send has even been received by Rabbit,
let alone safely stored.
> You said " With the persister in 1.7, messages are kept in RAM but are
also
> written out to disk." - are the messages written to disk by default as
well?
> As in the choice between "ram_copies", "disk_copies" and
"disc_only_copies"?
> Or are the messages simple written anyway in the 1.7 persister?
By definition, persistent messages are written to disk. They also happen
to be stored in RAM in the persister in 1.7 (and before), for a variety
of reasons. This has *nothing* to do with the various modes of operation
of mnesia, as the persister does not use mnesia at all.
> Using transactions sounds like the answer for me for now, thanks.
Right, and thus this is effectively synchronous messaging as you get the
reply back from Rabbit - the commit-ok - when you commit on the channel.
> Last question: Do you ever forsee a day when there is no 2GB limit to
DETS?
It's possible that the Erlang/OTP folks will get round to rewriting it
to make it 64-bit safe, I don't know. Certainly it's not something we're
looking at doing.
Matthew
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 15, 2009 5:35 am |
|
|
|
Guest
|
>
> The persister does not use Mnesia in any way. This is true of the
> current persister in 1.7 and the future persister. We are not using
> mnesia for the persister entirely because of its limitations.
>
> The persister in 1.7 keeps all messages in memory, so you're entirely
> bound by the amount of RAM available. The new persister will not have
> this limitation and will be able to scale much better than the current
> persister. I realise that we've been making noises about the "new"
> persister since April, and we had hoped to get it out with 1.7, but hey,
> software development is never a predictable activity!
>
>
Understable indeed 
I have a related question:
I'm not versed to Erlang enough to understand how/when it releases
memory, it has grabbed. So if there is a sudden burst of messages into
RabbitMQ, the memory usage will grow but will it be released eventually?
Cheers,
- Sylvain
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 15, 2009 4:38 pm |
|
|
|
Guest
|
> I'm not versed to Erlang enough to understand how/when it releases
> memory, it has grabbed. So if there is a sudden burst of messages into
> RabbitMQ, the memory usage will grow but will it be released eventually?
Yup. In practice, I've seen rabbit grow to over 1.5G, and then shrink
back down to ~200MB as messages are drained. I don't think I've ever
seen it shrink down to the memory usage of a totally new rabbit
server, but it definitely does free up RAM as it runs.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Oct 15, 2009 7:25 pm |
|
|
|
Guest
|
|
| Back to top |
|
| wuji |
| Posted: Mon Aug 27, 2012 7:12 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
if the guy's full of crap or not. You know know [h3]cheap authentic air jordans[/h3] know what I'm saying? Go look in the man's eyes,
you can tell.""He don't forget where he comes from," Maupin Maupin [h4]cheap Ralph Lauren[/h4] Maupin added. "If he does become VP, I can probably
on his door."Bin Laden Anniversary Triggers Law Enforcement SurgeUS Watches Watches cheap authentic jordans Watches for Potential Al Qaeda Plot Against Aviation TargetsBy BRIAN
RICHARD ESPOSITO and RHONDA SCHWARTZApril 26, 2012 — While U.S. U.S. authentic jordans U.S. officials say publicly there is no specific threat of
terror attack, behind the scenes law enforcement officials tell ABC ABC [h2]knockoff designer *beep*[/h2] ABC News there are plans for a major security surge |
|
|
| Back to top |
 
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
