| Author |
Message |
|
| bazil |
 Posted: Mon Dec 17, 2007 4:50 pm |
 |
|
|
Joined: 17 Dec 2007
Posts: 2
|
Hello every one.
Well, we are testing Mnesia DB under huge load, and we have a problem - with simple transaction, reading and writing, working beam process eats every seconds by one or two megabytes, until memory ends.
After replacing transaction with a dirty operations, situation becomes little better - memory grows not so fast, but grows.
But the main thing, is that after the end of heavy loading, memory usage amount is quickly decrease.
It makes me think that Erlang's garbage collector does not free unused memory under load, but after it - he does.
Is anyone have the same problems ?
Also, I post here two links to testing .erl files - you can run ones, to see the problem.
You can find tutorial of how to use them inside of them.
http://yanovsky.agenstvo.com/wp-content/uploads/2007/12/write_testerl.txt
http://yanovsky.agenstvo.com/wp-content/uploads/2007/12/match_testerl.txt
We are using the last OTP version - R12B-0.
The problem shows and under Windows and under the Linux. |
|
|
| Back to top |
 
|
| dvader |
| Posted: Mon Dec 17, 2007 5:16 pm |
|
|
|
Joined: 17 Dec 2007
Posts: 9
|
| It looks like a memory leak. Memory is never released while mnesia is running. Carbage collector never free this memory even at fullsweep_after = 0. |
|
|
| Back to top |
|
| bazil |
| Posted: Mon Dec 17, 2007 9:09 pm |
|
|
|
Joined: 17 Dec 2007
Posts: 2
|
I am just don't believe, that it could be a memory leak - this is very simple task. Everyone who have been used Mnesia should do this test.
Amazon.com, some famous banks, and other - no, I am not believe )
I think it is normal behavior, and could be tuned by some options. |
|
|
| Back to top |
|
| dvader |
| Posted: Fri Dec 21, 2007 9:49 am |
|
|
|
Joined: 17 Dec 2007
Posts: 9
|
| This mnesia feature makes almost impossible to use mnesia for tasks like search engines or directory indexing, or there is something we don't know. IMHO, mnesia should have a function or other way to free unused memory even at the cost of perfomance. Does the way exist? |
|
|
| Back to top |
|
| rbucker881 |
| Posted: Fri Dec 21, 2007 1:56 pm |
|
|
|
User
Joined: 11 Oct 2007
Posts: 17
|
Looking at the DOC I see the definition of RAM_COPIES so I am not convinced that this is a leak. More study is required before labeling.
[quote]ram_copies A table can be replicated on a number of Erlang nodes. The ram_copies property specifies a list of Erlang nodes where RAM copies are kept. These copies can be dumped to disc at regular intervals. However, updates to these copies are not written to disc on a transaction basis. [/quote] |
|
|
| Back to top |
|
| dvader |
| Posted: Fri Dec 21, 2007 10:09 pm |
|
|
|
Joined: 17 Dec 2007
Posts: 9
|
Yes, it is not a leak. It's awkward "feature". Memory is released after some minutes of idle state.
However, the memory consumption increases in DISC_COPIES too, more slowly, but it increases. It doesn't increases at DISC_ONLY_COPIES.
I do not quite understand the statement about replication, because the memory consumption rises fast not on updates only, but on match_object\read lookups  , and on dirty_ops too. And there is no other node, why replication is relevant in that case?
May be its' kind of undeleted inmemory log?
Crawlers, indexers, many other applications may update database frequently, so they memory consumption may be an important question.
It would be great if there was some kind of function mnesia:cleanup or somth. to do cleanup, wouldn't? |
|
|
| Back to top |
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
