|
|
| Author |
Message |
< Erlang ~ Erlang VM consumes over 50GB of ram (huge) |
| vsp |
 Posted: Thu Aug 19, 2010 5:47 pm |
 |
|
|
User
Joined: 03 Mar 2010
Posts: 14
|
We have a mnesia table with 700,000 rows each row has 12 string fields. Every string on average is 12 characters long. The table is partitioned across 32 fragments. The table is described and to be ram disk copy
We need to produce an XML file out of those rows.
The produced XML file is about 28 Mb.
We are running latest (june 16, 2010) release of Erlang 64 bit on Linux.
Erlang VM is started with the following relevant parameters
-shared
+K true
+P 250000
-env ERL_FULLSWEEP_AFTER 0
-env ERL_MAX_ETS_TABLES 1000
+sbtu
The simple approach that we took (hoping that Erlang 'knows' how to deal with the data) is the following
Do a mnesia read (dirty) read. Put all the data into Array of tuples compatible with Xmerl's simple output function.
Then pass that array to XMerl. Have it construct a xml string, and then convert the string into a binary and save it in a file.
When Erlang starts up, it takes about 5GB of memory (there are few other ramdisk tables there of similar size). Not sure if this 'unusual' or not.
Then when we run our process once, the VM grows to 12 GB, run again to 20GB and a few more times -- will get it to over 50GB.
This is very unreasonable.
Memory does not appear to get released at all.
We would be ok if it took say additional 1GB while we doing this, but additional 40GB -- does not make sense.
We are now forced to rethink the process and deal with chunks of data through out. Which is just way more programming effort than we thought it would take.
Therefore would like to know the following
a) How much memory erlang allocates per one byte in a string
b) are erlang arrays automatically managed by VM such that if array grows after a certain size, tempoary files are used -- or are the arrays always forced to be in RAM
c) is XMerl capable of using say ets or dts as an input to create the output xml file
d) is Xmerl (or may be another library ) capable of
writing out an XML incrementally to a file instead of returning a string
e) are there command line options to VM that would force it not grow this large and instead resort to using temporary on disk files to manage memory
f) are there any suggestions to resolve this short of rewritting our code to do incremental reads, outputs, concatenating chunks of strings together/etc. |
|
|
| Back to top |
 
|
| zajda |
| Posted: Mon Aug 23, 2010 9:56 pm |
|
|
|
User
Joined: 22 Aug 2009
Posts: 83
|
a) In erlang there are no Strings. So called string is the list of ints. Here is an integer spec http://www.erlang.org/doc/efficiency_guide/advanced.html#id2265992
To determine real size of your objects do the math with strings or use erts_debug:flat_size(Obj).
b) hmm.. erlang VM allocates memory in RAM (also for ETS) and if it is not enough, OS swaps it with disk or system crashes.
Why do you set ERL_FULLSWEEP_AFTER = 0? It basically may disable garbage collector. Set it for i.e. 10, so it forces GS even under heavy system load.
Here is some doc to it:
http://www.erlang.org/doc/man/erlang.html#system_flag-2
Do not store any strings, switch to binaries. Operations on binaries are very cheap, because there is no coping, only passing references. |
|
|
| Back to top |
|
| vsp |
| Posted: Tue Aug 31, 2010 6:52 pm |
|
|
|
User
Joined: 03 Mar 2010
Posts: 14
|
hi, thank you for a reply
on b) We are looking for a way for Erlang to not consume all the available memory on our server (when it does we have to reboot the server)
So we want to limit what it allocates in RAM to 5 GB max.
That's why I was asking if Erlang can manage data in arrays and ETS such that portions of them are on disk
as far as strings -- XMerl returns back strings (arrays of characters) and accepts strings. We do not have a way around it I think... Unless somebody knows different.
We have worked around the problem I posted earlier by basically going the 'difficult' way and modifying our code to deal with all the data in chunks.
But still wanted to know if Erlang has built in way to deal with data 'larger' then can be put into RAM, without relying on OS swapping from virtual memory. Similar to C++'s http://stxxl.sourceforge.net/
thank you |
|
|
| Back to top |
|
| wuji |
| Posted: Tue Sep 18, 2012 7:51 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
way society perceives unhealthy behaviors, like smoking, drinking and drug drug designer replica *beep* drug use, which used to fall under the general perception
a rebellious "cool."Still, what gives? Why is Slickster Danny Zuko Zuko replica designer *beep* Zuko from "Grease" out and Justin Bieber, the popstar with
boy-next-door charm in?"[Being] cool by definition requires a reference point-- point-- cheap real jordans point-- what is boring, normal, or even uncool," said Lewis.
day culture stops changing is the day our notions of of cheap designer *beep* of coolness will also be frozen in time."Unemployment Unchanged at
Pct, 80,000 Jobs AddedAnother So-So Jobs ReportBy SUSANNA KIM and and cheap designer *beep* and BILL McGUIREJuly 6, 2012— Employers added 80,000 jobs in
and the unemployment rate remained unchanged at 8.2 percent, the the [h4]discount designer *beep*[/h4] the Labor Department announced Friday, in another so-so report on
U.S. economy that promises to frame the debate for the the cheap authentic jordans the fall presidential election.Economists had expected that employers added around
jobs in June, higher than the revised 77,000 jobs added added [h2]discount designer *beep*[/h2] added in May, but lower than what is needed for |
|
|
| Back to top |
|
| dongdongwu |
| Posted: Wed Sep 19, 2012 2:39 am |
|
|
|
User
Joined: 19 Sep 2012
Posts: 236
|
| Girls would refuse to even leave their replicas behind. specifically when the product beats the complete meaning of the Christian Louboutin men outlet; in conditions of good quality and detailing.If Cinderella was residing in your twenty primary century as opposed to the aged a single then there would not be any 'Happy actually after'. properly appears like girls nowadays are as well fond of the strong;Christian Louboutin Men Shoes to leave them in your center of nowhere.Christian Louboutin for men Shoes would arrive true handy being a excellent handbag using the glimpse and really feel belonging to the authentic but at a very much lesser price tag adding as very much as types picture in your process. |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
