| Author |
Message |
|
| pegesund |
 Posted: Thu Nov 11, 2010 10:13 am |
 |
|
|
Joined: 09 May 2010
Posts: 5
|
Hi, I have a small program with lots of memory-updates which I try to
run in Erlang.
The same algorithm works fine in both Java and Perl, but fails in
Erlang because the program runs out of memory - and I can not figure
out why. Frustrating, as my Erlang-versjon seems to be the easiest to
scale as well as being the most readable.
The program is threaded and each thread writes to a ets-table which is
created at the beginning of the thread. When the thread dies I try to
do a ets:delete(Table), like described in the manual, but the memory
used by the thread never seems to be released.
Some facts:
- The memory usage of each thread is rather constant. This is
confirmed when I use ets:i() to show info about memory usage.
- The number of threads are constant - confirmed by both running top
and writing out the number of threads regularly. When a thread dies, I
create a new one.
- I have tried to end the thread by sending a exit-signal as the last
statement. This helps some, but does not solve the leak.
- I put small lists of size 3-4 integers into the ets as values, the
keys are list of same size as well.
- I garbage-collect each thread before it dies, as well as doing
regular global garbage-collects. No help.
- Information from ets:i() about memory when I sum usage by each
thread, is much lower than stated by memory() when i run
erlang:memory(). This might indicate something? Does not seem logical
to me, at least.
- Info from erlang:memory is about half of what top/the os tells.
- I am running on ubuntu, 64-bit, 14A but I have tried 14B as well.
Any clues? Dump from ets:i() and erlang:memory() is like below.
Cheers,
Petter
--- dump ---
eNumber of processes: 27
ets:i():
id name type size mem owner
----------------------------------------------------------------------------
13 code set 261 10692 code_server
4110 code_names set 58 7804 code_server
6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0>
7022018584 the_synapses ordered_set 15143493 310909950 <0.48.0>
7774416922 the_synapses ordered_set 8794649 182005810 <0.49.0>
ac_tab ac_tab set 6 848 application_controller
file_io_servers file_io_servers set 0 302 file_server_2
global_locks global_locks set 0 302 global_name_server
global_names global_names set 0 302 global_name_server
global_names_ext global_names_ext set 0 302 global_name_server
global_pid_ids global_pid_ids bag 0 302 global_name_server
global_pid_names global_pid_names bag 0 302 global_name_server
inet_cache inet_cache bag 0 302 inet_db
inet_db inet_db set 29 571 inet_db
inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db
inet_hosts_byname inet_hosts_byname bag 0 302 inet_db
inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302 inet_db
inet_hosts_file_byname inet_hosts_file_byname bag 0 302 inet_db
neurone_counter neurone_counter set 258394 1846182 entity_server
neurone_group_counter neurone_group_counter set 6 344
entity_group_server
neurone_group_name neurone_group_name set 6 426 entity_group_server
neurone_group_name_reverse neurone_group_name_reverse set 6
426 entity_group_server
neurone_name neurone_name set 258394 11824602 entity_server
neurone_name_reverse neurone_name_reverse set 258394 11824602 entity_server
memory(): [{total,5568669792},
{processes,1138936},
{processes_used,1128120},
{system,5567530856},
{atom,349769},
{atom_used,336605},
{binary,82704},
{code,3046365},
{ets,5562163256}]
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
 
|
| Guest |
| Posted: Thu Nov 11, 2010 10:16 am |
|
|
|
Guest
|
On Sun, Nov 7, 2010 at 10:15 PM, Petter Egesund
<petter.egesund@gmail.com> wrote:
> Full source code? It is unfortunately to long and to data-bound to
> make sense. Still hoping for a clue - ets:i() tells me that total ets
> should be less than to 1 gb total, but memory() says it is using more
> than 5 gb for ets.
ets:i/0 reports words.
erlang:memory/0 reports bytes.
So that should not be surprising that they differ by a factor of 4.
--
J.
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| pegesund |
| Posted: Thu Nov 11, 2010 10:20 am |
|
|
|
Joined: 09 May 2010
Posts: 5
|
Hi, and thanks for answering!
Yes, the memory should be reclaimed - I think the problem might be
that the garbage collector is failing?!
Summing up ets-memory-usage for each thread gives reasonable numbers,
but erlang:memory() shows ets is using much more ram - calling
ets:delete before the process finishes does not help.
No, I am only keeping integers in the ets, no strings  The program
is using binaries a lot, but these seems to be reclaimed from the gb
without problems.
Yes, I know the difference between processes/threads, only a couple
small ets-tables are global and shared between processes. The large
ones, which I think are the cause of the problem, is only written
to/read by one process.
Full source code? It is unfortunately to long and to data-bound to
make sense. Still hoping for a clue - ets:i() tells me that total ets
should be less than to 1 gb total, but memory() says it is using more
than 5 gb for ets.
Cheers,
Petter
On Sun, Nov 7, 2010 at 9:37 PM, Ryan Zezeski <rzezeski@gmail.com> wrote:
>
>
> On Sun, Nov 7, 2010 at 9:49 AM, Petter Egesund <petter.egesund@gmail.com>
> wrote:
>>
>> Hi, I have a small program with lots of memory-updates which I try to
>> run in Erlang.
>>
>> The same algorithm works fine in both Java and Perl, but fails in
>> Erlang because the program runs out of memory - and I can not figure
>> out why. Frustrating, as my Erlang-versjon seems to be the easiest to
>> scale as well as being the most readable.
>>
>> The program is threaded and each thread writes to a ets-table which is
>> created at the beginning of the thread. When the thread dies I try to
>> do a ets:delete(Table), like described in the manual, but the memory
>> used by the thread never seems to be released.
>>
>> Some facts:
>>
>> - The memory usage of each thread is rather constant. This is
>> confirmed when I use ets:i() to show info about memory usage.
>> - The number of threads are constant - confirmed by both running top
>> and writing out the number of threads regularly. When a thread dies, I
>> create a new one.
>> - I have tried to end the thread by sending a exit-signal as the last
>> statement. This helps some, but does not solve the leak.
>> - I put small lists of size 3-4 integers into the ets as values, the
>> keys are list of same size as well.
>> - I garbage-collect each thread before it dies, as well as doing
>> regular global garbage-collects. No help.
>> - Information from ets:i() about memory when I sum usage by each
>> thread, is much lower than stated by memory() when i run
>> erlang:memory(). This might indicate something? Does not seem logical
>> to me, at least.
>> - Info from erlang:memory is about half of what top/the os tells.
>> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well.
>>
>> Any clues? Dump from ets:i() and erlang:memory() is like below.
>>
>> Cheers,
>>
>> Petter
>>
>> --- dump ---
>>
>> eNumber of processes: 27
>> ets:i():
>> |
|
|
| Back to top |
|
| pegesund |
| Posted: Thu Nov 11, 2010 10:30 am |
|
|
|
Joined: 09 May 2010
Posts: 5
|
Sure
The most important part of my main loop works like this:
Spawn up to a given number of threads. When a thread dies it sends a
msg back to the main loop, so I can spawn a new one. And so on... The
problem, as it seems to me, is that ets-garbage from dead threads
never are collected?
I have also tried to do this from my initial call:
erlang:system_flag(fullsweep_after, 0), but without any change.
Code:
parse_dir_loop(ThreadCounter, MaxThreads, Dir, OldSpawnFiles) ->
io:fwrite("Number of processes: ~p ~n", [length(processes())]),
ets:i(),
ThreadCounterNew = case ThreadCounter < MaxThreads of
true -> {ok, RemainingFiles} = file:list_dir(Dir),
case RemainingFiles of
[] -> SpawnFiles = [], ThreadCounter;
Files -> SpawnFilesNum = min(MaxThreads -
ThreadCounter, length(Files -- OldSpawnFiles)),
SpawnFiles = lists:sublist(lists:sort(Files --
OldSpawnFiles), 1, SpawnFilesNum),
Receiver = self(),
lists:foreach(fun(File) -> spawn_link(fun() ->
parse_file(Dir ++ File),
%% Receiver ! {done, File},
exit({kill_me, File})
end)
end, SpawnFiles),
ThreadCounter + SpawnFilesNum
end;
false -> SpawnFiles = [], ThreadCounter
end,
NewSpawnFiles = OldSpawnFiles ++ SpawnFiles,
receive
{done, File} ->
io:fwrite("Done with a file ~n"),
parse_dir_loop(ThreadCounterNew - 1, MaxThreads, Dir,
NewSpawnFiles -- File);
{'EXIT', Pid, {kill_me, File}} ->
io:fwrite("Should be dead by now: ~p ~n", [Pid]),
parse_dir_loop(ThreadCounterNew - 1, MaxThreads, Dir,
NewSpawnFiles -- File);
{'EXIT', Pid, normal} ->
io:fwrite("Should be dead by now: ~p ~n", [Pid]),
garbage_collect(Pid),
parse_dir_loop(ThreadCounter, MaxThreads, Dir, NewSpawnFiles);
Signal -> io:fwrite("Got unknown signal in parse_dir_loop ~p~n", [Signal])
after 5000 -> case file:list_dir(Dir) of
{ok, []} -> io:fwrite("Parsing directory is empty - job
done: ~p~n", [Dir]);
{ok, _} -> parse_dir_loop(ThreadCounterNew, MaxThreads,
Dir, NewSpawnFiles)
end
end.
parse_dir(ThreadCounter, MaxThreads, Dir) ->
%% probably fix path for Windows?!
parse_dir_loop(ThreadCounter, MaxThreads, Dir, []).
On Sun, Nov 7, 2010 at 4:27 PM, Bengt Kleberg
<bengt.kleberg@ericsson.com> wrote:
> Greetings,
>
> Could you include the program, or preferably a small subset of it, that
> runs out of memory?
>
>
> bengt
>
> On Sun, 2010-11-07 at 15:49 +0100, Petter Egesund wrote:
>> Hi, I have a small program with lots of memory-updates which I try to
>> run in Erlang.
>>
>> The same algorithm works fine in both Java and Perl, but fails in
>> Erlang because the program runs out of memory - and I can not figure
>> out why. Frustrating, as my Erlang-versjon seems to be the easiest to
>> scale as well as being the most readable.
>>
>> The program is threaded and each thread writes to a ets-table which is
>> created at the beginning of the thread. When the thread dies I try to
>> do a ets:delete(Table), like described in the manual, but the memory
>> used by the thread never seems to be released.
>>
>> Some facts:
>>
>> - The memory usage of each thread is rather constant. This is
>> confirmed when I use ets:i() to show info about memory usage.
>> - The number of threads are constant - confirmed by both running top
>> and writing out the number of threads regularly. When a thread dies, I
>> create a new one.
>> - I have tried to end the thread by sending a exit-signal as the last
>> statement. This helps some, but does not solve the leak.
>> - I put small lists of size 3-4 integers into the ets as values, the
>> keys are list of same size as well.
>> - I garbage-collect each thread before it dies, as well as doing
>> regular global garbage-collects. No help.
>> - Information from ets:i() about memory when I sum usage by each
>> thread, is much lower than stated by memory() when i run
>> erlang:memory(). This might indicate something? Does not seem logical
>> to me, at least.
>> - Info from erlang:memory is about half of what top/the os tells.
>> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well.
>>
>> Any clues? Dump from ets:i() and erlang:memory() is like below.
>>
>> Cheers,
>>
>> Petter
>>
>> --- dump ---
>>
>> eNumber of processes: 27
>> ets:i():
>> |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Nov 11, 2010 10:45 am |
|
|
|
Guest
|
Hi
Are you sure that your ets memory information is in bytes, and not in words?
Morten.
On 11/7/10 10:15 PM, Petter Egesund wrote:
> Hi, and thanks for answering!
>
> Yes, the memory should be reclaimed - I think the problem might be
> that the garbage collector is failing?!
>
> Summing up ets-memory-usage for each thread gives reasonable numbers,
> but erlang:memory() shows ets is using much more ram - calling
> ets:delete before the process finishes does not help.
>
> No, I am only keeping integers in the ets, no strings The program
> is using binaries a lot, but these seems to be reclaimed from the gb
> without problems.
>
> Yes, I know the difference between processes/threads, only a couple
> small ets-tables are global and shared between processes. The large
> ones, which I think are the cause of the problem, is only written
> to/read by one process.
>
> Full source code? It is unfortunately to long and to data-bound to
> make sense. Still hoping for a clue - ets:i() tells me that total ets
> should be less than to 1 gb total, but memory() says it is using more
> than 5 gb for ets.
>
> Cheers,
>
> Petter
>
>
>
>
>
> On Sun, Nov 7, 2010 at 9:37 PM, Ryan Zezeski<rzezeski@gmail.com> wrote:
>>
>> On Sun, Nov 7, 2010 at 9:49 AM, Petter Egesund<petter.egesund@gmail.com>
>> wrote:
>>> Hi, I have a small program with lots of memory-updates which I try to
>>> run in Erlang.
>>>
>>> The same algorithm works fine in both Java and Perl, but fails in
>>> Erlang because the program runs out of memory - and I can not figure
>>> out why. Frustrating, as my Erlang-versjon seems to be the easiest to
>>> scale as well as being the most readable.
>>>
>>> The program is threaded and each thread writes to a ets-table which is
>>> created at the beginning of the thread. When the thread dies I try to
>>> do a ets:delete(Table), like described in the manual, but the memory
>>> used by the thread never seems to be released.
>>>
>>> Some facts:
>>>
>>> - The memory usage of each thread is rather constant. This is
>>> confirmed when I use ets:i() to show info about memory usage.
>>> - The number of threads are constant - confirmed by both running top
>>> and writing out the number of threads regularly. When a thread dies, I
>>> create a new one.
>>> - I have tried to end the thread by sending a exit-signal as the last
>>> statement. This helps some, but does not solve the leak.
>>> - I put small lists of size 3-4 integers into the ets as values, the
>>> keys are list of same size as well.
>>> - I garbage-collect each thread before it dies, as well as doing
>>> regular global garbage-collects. No help.
>>> - Information from ets:i() about memory when I sum usage by each
>>> thread, is much lower than stated by memory() when i run
>>> erlang:memory(). This might indicate something? Does not seem logical
>>> to me, at least.
>>> - Info from erlang:memory is about half of what top/the os tells.
>>> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well.
>>>
>>> Any clues? Dump from ets:i() and erlang:memory() is like below.
>>>
>>> Cheers,
>>>
>>> Petter
>>>
>>> --- dump ---
>>>
>>> eNumber of processes: 27
>>> ets:i():
>>> id name type size mem owner
>>>
>>> ----------------------------------------------------------------------------
>>> 13 code set 261 10692 code_server
>>> 4110 code_names set 58 7804 code_server
>>> 6746271765 the_synapses ordered_set 5425194 113336012<0.47.0>
>>> 7022018584 the_synapses ordered_set 15143493 310909950<0.48.0>
>>> 7774416922 the_synapses ordered_set 8794649 182005810<0.49.0>
>>> ac_tab ac_tab set 6 848
>>> application_controller
>>> file_io_servers file_io_servers set 0 302 file_server_2
>>> global_locks global_locks set 0 302
>>> global_name_server
>>> global_names global_names set 0 302
>>> global_name_server
>>> global_names_ext global_names_ext set 0 302
>>> global_name_server
>>> global_pid_ids global_pid_ids bag 0 302
>>> global_name_server
>>> global_pid_names global_pid_names bag 0 302
>>> global_name_server
>>> inet_cache inet_cache bag 0 302 inet_db
>>> inet_db inet_db set 29 571 inet_db
>>> inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db
>>> inet_hosts_byname inet_hosts_byname bag 0 302 inet_db
>>> inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302
>>> inet_db
>>> inet_hosts_file_byname inet_hosts_file_byname bag 0 302
>>> inet_db
>>> neurone_counter neurone_counter set 258394 1846182 entity_server
>>> neurone_group_counter neurone_group_counter set 6 344
>>> entity_group_server
>>> neurone_group_name neurone_group_name set 6 426
>>> entity_group_server
>>> neurone_group_name_reverse neurone_group_name_reverse set 6
>>> 426 entity_group_server
>>> neurone_name neurone_name set 258394 11824602 entity_server
>>> neurone_name_reverse neurone_name_reverse set 258394 11824602
>>> entity_server
>>> memory(): [{total,5568669792},
>>> {processes,1138936},
>>> {processes_used,1128120},
>>> {system,5567530856},
>>> {atom,349769},
>>> {atom_used,336605},
>>> {binary,82704},
>>> {code,3046365},
>>> {ets,5562163256}]
>>>
>>>
>> Hi Peter, ETS tables are not garbage collected. Each ETS table has _one_
>> owner (a process). When that owner dies the table is deleted and it's
>> memory is reclaimed. You can also delete a table (and reclaim the memory)
>> by calling ets:delete/1. Looking at your memory result, your ETS tables are
>> taking up ~5.2GB of data. However, you binary usage is very low so I'm
>> going to take a guess that you are sotring a list of strings? If so you
>> should note that on a 64-bit system *each character* in a string will use 16
>> bytes of memory! I highly recommend using binaries where possible when
>> dealing with a large amount of data; your program will not only be more
>> space efficient but also faster. I've written a non-trivial Erlang
>> application for work and I deal with CSV files that get up to 18 million
>> rows. I make heavy use of binaries and the binary module to parse these
>> files and write entries to ETS--you'd be surprised how fast it is! If you'd
>> like I could provide an example.
>> When you say "thread" do you mean "process?" You do realize that an OS
>> thread and Erlang process are two completely different things. IIRC, the VM
>> spawn's an OS thread per scheduler (along w/ some other threads for I/O and
>> such). Erlang processes are extremely cheap...don't be afraid to make
>> thousands or even tens-of-thousands of them.
>> You should not have to perform manual garbage collection, that seems like a
>> code smell to me. When a process dies it's heap will be reclaimed. Each
>> process has it's own isolated heap.
>> Do you have multiple processes all writing to the same ETS table? If so
>> there are some improvements that were made to ETS (and Erlang in general)
>> for concurrent writing/reading of an ETS table in 14B that you might want to
>> look at.
>> Finally, it would be helpful to see the full source code. There is a good
>> chance your solution is not optimal for Erlang. By that, I mean that if
>> your translation follows closely from your Java and Perl solutions than
>> chances are it's not an optimal Erlang program as the paradigms are vastly
>> different.
>> -Ryan
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
>
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| pegesund |
| Posted: Thu Nov 11, 2010 11:00 am |
|
|
|
Joined: 09 May 2010
Posts: 5
|
Yes, the words vs bytes theory might explain something
I know about delete - I allready do that, the ets-tables which appear
in the list are expected. They belong to not-finished processes.
Some math then:
Summing the ets:i() info gives about 631747158 words which corresponds
to 5562163256 bytes, meaning that each word is using 8 bytes, which
makes sense on a 64-bit operating system to me.
Using this line as an example shows that each entry in ets (entry is
key/value, both tuples with three integers), show that each integer
needs 113336012 * 8 / 5425194 = 167 bytes (or 27 bytes pr. integer):
6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0>
This is what tricked me - erlang is using much space serializing data
than the other languages - I will try to store my data as binary and
see if that helps.
Thanks!
Petter
On Sun, Nov 7, 2010 at 10:28 PM, Ryan Zezeski <rzezeski@gmail.com> wrote:
>
>
> On Sun, Nov 7, 2010 at 4:21 PM, Jesper Louis Andersen
> <jesper.louis.andersen@gmail.com> wrote:
>>
>> On Sun, Nov 7, 2010 at 10:15 PM, Petter Egesund
>> <petter.egesund@gmail.com> wrote:
>>
>> > Full source code? It is unfortunately to long and to data-bound to
>> > make sense. Still hoping for a clue - ets:i() tells me that total ets
>> > should be less than to 1 gb total, but memory() says it is using more
>> > than 5 gb for ets.
>>
>> ets:i/0 reports words.
>> erlang:memory/0 reports bytes.
>>
>> So that should not be surprising that they differ by a factor of 4.
>>
>> --
>> J.
>
> Jesper beat me to it...the memory reported by ets:info and ets:i is in
> words...not bytes.
> Petter, I don't think you're actually deleting these tables, which is proven
> by the fact that they show up in ets:i/0. |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Nov 11, 2010 11:10 am |
|
|
|
Guest
|
On Sun, Nov 7, 2010 at 9:49 AM, Petter Egesund <petter.egesund@gmail.com>wrote:
> Hi, I have a small program with lots of memory-updates which I try to
> run in Erlang.
>
> The same algorithm works fine in both Java and Perl, but fails in
> Erlang because the program runs out of memory - and I can not figure
> out why. Frustrating, as my Erlang-versjon seems to be the easiest to
> scale as well as being the most readable.
>
> The program is threaded and each thread writes to a ets-table which is
> created at the beginning of the thread. When the thread dies I try to
> do a ets:delete(Table), like described in the manual, but the memory
> used by the thread never seems to be released.
>
> Some facts:
>
> - The memory usage of each thread is rather constant. This is
> confirmed when I use ets:i() to show info about memory usage.
> - The number of threads are constant - confirmed by both running top
> and writing out the number of threads regularly. When a thread dies, I
> create a new one.
> - I have tried to end the thread by sending a exit-signal as the last
> statement. This helps some, but does not solve the leak.
> - I put small lists of size 3-4 integers into the ets as values, the
> keys are list of same size as well.
> - I garbage-collect each thread before it dies, as well as doing
> regular global garbage-collects. No help.
> - Information from ets:i() about memory when I sum usage by each
> thread, is much lower than stated by memory() when i run
> erlang:memory(). This might indicate something? Does not seem logical
> to me, at least.
> - Info from erlang:memory is about half of what top/the os tells.
> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well.
>
> Any clues? Dump from ets:i() and erlang:memory() is like below.
>
> Cheers,
>
> Petter
>
> --- dump ---
>
> eNumber of processes: 27
> ets:i():
> id name type size mem owner
>
> ----------------------------------------------------------------------------
> 13 code set 261 10692 code_server
> 4110 code_names set 58 7804 code_server
> 6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0>
> 7022018584 the_synapses ordered_set 15143493 310909950 <0.48.0>
> 7774416922 the_synapses ordered_set 8794649 182005810 <0.49.0>
> ac_tab ac_tab set 6 848
> application_controller
> file_io_servers file_io_servers set 0 302 file_server_2
> global_locks global_locks set 0 302 global_name_server
> global_names global_names set 0 302 global_name_server
> global_names_ext global_names_ext set 0 302
> global_name_server
> global_pid_ids global_pid_ids bag 0 302 global_name_server
> global_pid_names global_pid_names bag 0 302
> global_name_server
> inet_cache inet_cache bag 0 302 inet_db
> inet_db inet_db set 29 571 inet_db
> inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db
> inet_hosts_byname inet_hosts_byname bag 0 302 inet_db
> inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302
> inet_db
> inet_hosts_file_byname inet_hosts_file_byname bag 0 302
> inet_db
> neurone_counter neurone_counter set 258394 1846182 entity_server
> neurone_group_counter neurone_group_counter set 6 344
> entity_group_server
> neurone_group_name neurone_group_name set 6 426
> entity_group_server
> neurone_group_name_reverse neurone_group_name_reverse set 6
> 426 entity_group_server
> neurone_name neurone_name set 258394 11824602 entity_server
> neurone_name_reverse neurone_name_reverse set 258394 11824602
> entity_server
> memory(): [{total,5568669792},
> {processes,1138936},
> {processes_used,1128120},
> {system,5567530856},
> {atom,349769},
> {atom_used,336605},
> {binary,82704},
> {code,3046365},
> {ets,5562163256}]
>
>
>
Hi Peter, ETS tables are not garbage collected. Each ETS table has _one_
owner (a process). When that owner dies the table is deleted and it's
memory is reclaimed. You can also delete a table (and reclaim the memory)
by calling ets:delete/1. Looking at your memory result, your ETS tables are
taking up ~5.2GB of data. However, you binary usage is very low so I'm
going to take a guess that you are sotring a list of strings? If so you
should note that on a 64-bit system *each character* in a string will use 16
bytes of memory! I highly recommend using binaries where possible when
dealing with a large amount of data; your program will not only be more
space efficient but also faster. I've written a non-trivial Erlang
application for work and I deal with CSV files that get up to 18 million
rows. I make heavy use of binaries and the binary module to parse these
files and write entries to ETS--you'd be surprised how fast it is! If you'd
like I could provide an example.
When you say "thread" do you mean "process?" You do realize that an OS
thread and Erlang process are two completely different things. IIRC, the VM
spawn's an OS thread per scheduler (along w/ some other threads for I/O and
such). Erlang processes are extremely cheap...don't be afraid to make
thousands or even tens-of-thousands of them.
You should not have to perform manual garbage collection, that seems like a
code smell to me. When a process dies it's heap will be reclaimed. Each
process has it's own isolated heap.
Do you have multiple processes all writing to the same ETS table? If so
there are some improvements that were made to ETS (and Erlang in general)
for concurrent writing/reading of an ETS table in 14B that you might want to
look at.
Finally, it would be helpful to see the full source code. There is a good
chance your solution is not optimal for Erlang. By that, I mean that if
your translation follows closely from your Java and Perl solutions than
chances are it's not an optimal Erlang program as the paradigms are vastly
different.
-Ryan
Post received from mailinglist |
|
|
| Back to top |
|
| Guest |
| Posted: Thu Nov 11, 2010 11:10 am |
|
|
|
Guest
|
On Sun, Nov 7, 2010 at 4:21 PM, Jesper Louis Andersen <
jesper.louis.andersen@gmail.com> wrote:
> On Sun, Nov 7, 2010 at 10:15 PM, Petter Egesund
> <petter.egesund@gmail.com> wrote:
>
> > Full source code? It is unfortunately to long and to data-bound to
> > make sense. Still hoping for a clue - ets:i() tells me that total ets
> > should be less than to 1 gb total, but memory() says it is using more
> > than 5 gb for ets.
>
> ets:i/0 reports words.
> erlang:memory/0 reports bytes.
>
> So that should not be surprising that they differ by a factor of 4.
>
> --
> J.
>
Jesper beat me to it...the memory reported by ets:info and ets:i is in
words...not bytes.
Petter, I don't think you're actually deleting these tables, which is proven
by the fact that they show up in ets:i/0. I'll say it again, they are not
garbage collected...they don't live on the process heap. THere are two ways
to delete an ETS table: 1) call ets:delete or 2) the owner dies and there is
no heir. When they are deleted the memory is reclaimed.
Here is a erl session showing what I mean...
1> ets:i().
id name type size mem owner
----------------------------------------------------------------------------
13 code set 257 10711 code_server
4110 code_names set 57 7754 code_server
8207 shell_records ordered_set 0 112 <0.25.0>
ac_tab ac_tab set 6 878
application_controller
file_io_servers file_io_servers set 0 322 file_server_2
global_locks global_locks set 0 322 global_name_server
global_names global_names set 0 322 global_name_server
global_names_ext global_names_ext set 0 322 global_name_server
global_pid_ids global_pid_ids bag 0 322 global_name_server
global_pid_names global_pid_names bag 0 322 global_name_server
inet_cache inet_cache bag 0 322 inet_db
inet_db inet_db set 29 706 inet_db
inet_hosts_byaddr inet_hosts_byaddr bag 0 322 inet_db
inet_hosts_byname inet_hosts_byname bag 0 322 inet_db
inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 322 inet_db
inet_hosts_file_byname inet_hosts_file_byname bag 0 322 inet_db
ok
2> ets:new(test, [public, named_table]).
test
3> ets:i().
id name type size mem owner
----------------------------------------------------------------------------
13 code set 257 10711 code_server
4110 code_names set 57 7754 code_server
8207 shell_records ordered_set 0 112 <0.25.0>
ac_tab ac_tab set 6 878
application_controller
file_io_servers file_io_servers set 0 322 file_server_2
global_locks global_locks set 0 322 global_name_server
global_names global_names set 0 322 global_name_server
global_names_ext global_names_ext set 0 322 global_name_server
global_pid_ids global_pid_ids bag 0 322 global_name_server
global_pid_names global_pid_names bag 0 322 global_name_server
inet_cache inet_cache bag 0 322 inet_db
inet_db inet_db set 29 706 inet_db
inet_hosts_byaddr inet_hosts_byaddr bag 0 322 inet_db
inet_hosts_byname inet_hosts_byname bag 0 322 inet_db
inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 322 inet_db
inet_hosts_file_byname inet_hosts_file_byname bag 0 322 inet_db
test test set 0 322 <0.31.0>
ok
4> ets:delete(test).
true
5> ets:i().
id name type size mem owner
----------------------------------------------------------------------------
13 code set 257 10711 code_server
4110 code_names set 57 7754 code_server
8207 shell_records ordered_set 0 112 <0.25.0>
ac_tab ac_tab set 6 878
application_controller
file_io_servers file_io_servers set 0 322 file_server_2
global_locks global_locks set 0 322 global_name_server
global_names global_names set 0 322 global_name_server
global_names_ext global_names_ext set 0 322 global_name_server
global_pid_ids global_pid_ids bag 0 322 global_name_server
global_pid_names global_pid_names bag 0 322 global_name_server
inet_cache inet_cache bag 0 322 inet_db
inet_db inet_db set 29 706 inet_db
inet_hosts_byaddr inet_hosts_byaddr bag 0 322 inet_db
inet_hosts_byname inet_hosts_byname bag 0 322 inet_db
inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 322 inet_db
inet_hosts_file_byname inet_hosts_file_byname bag 0 322 inet_db
ok
Post received from mailinglist |
|
|
| Back to top |
|
| axel |
| Posted: Thu Nov 11, 2010 11:15 am |
|
|
|
User
Joined: 03 Mar 2005
Posts: 271
|
Greetings,
Could you include the program, or preferably a small subset of it, that
runs out of memory?
bengt
On Sun, 2010-11-07 at 15:49 +0100, Petter Egesund wrote:
> Hi, I have a small program with lots of memory-updates which I try to
> run in Erlang.
>
> The same algorithm works fine in both Java and Perl, but fails in
> Erlang because the program runs out of memory - and I can not figure
> out why. Frustrating, as my Erlang-versjon seems to be the easiest to
> scale as well as being the most readable.
>
> The program is threaded and each thread writes to a ets-table which is
> created at the beginning of the thread. When the thread dies I try to
> do a ets:delete(Table), like described in the manual, but the memory
> used by the thread never seems to be released.
>
> Some facts:
>
> - The memory usage of each thread is rather constant. This is
> confirmed when I use ets:i() to show info about memory usage.
> - The number of threads are constant - confirmed by both running top
> and writing out the number of threads regularly. When a thread dies, I
> create a new one.
> - I have tried to end the thread by sending a exit-signal as the last
> statement. This helps some, but does not solve the leak.
> - I put small lists of size 3-4 integers into the ets as values, the
> keys are list of same size as well.
> - I garbage-collect each thread before it dies, as well as doing
> regular global garbage-collects. No help.
> - Information from ets:i() about memory when I sum usage by each
> thread, is much lower than stated by memory() when i run
> erlang:memory(). This might indicate something? Does not seem logical
> to me, at least.
> - Info from erlang:memory is about half of what top/the os tells.
> - I am running on ubuntu, 64-bit, 14A but I have tried 14B as well.
>
> Any clues? Dump from ets:i() and erlang:memory() is like below.
>
> Cheers,
>
> Petter
>
> --- dump ---
>
> eNumber of processes: 27
> ets:i():
> id name type size mem owner
> ----------------------------------------------------------------------------
> 13 code set 261 10692 code_server
> 4110 code_names set 58 7804 code_server
> 6746271765 the_synapses ordered_set 5425194 113336012 <0.47.0>
> 7022018584 the_synapses ordered_set 15143493 310909950 <0.48.0>
> 7774416922 the_synapses ordered_set 8794649 182005810 <0.49.0>
> ac_tab ac_tab set 6 848 application_controller
> file_io_servers file_io_servers set 0 302 file_server_2
> global_locks global_locks set 0 302 global_name_server
> global_names global_names set 0 302 global_name_server
> global_names_ext global_names_ext set 0 302 global_name_server
> global_pid_ids global_pid_ids bag 0 302 global_name_server
> global_pid_names global_pid_names bag 0 302 global_name_server
> inet_cache inet_cache bag 0 302 inet_db
> inet_db inet_db set 29 571 inet_db
> inet_hosts_byaddr inet_hosts_byaddr bag 0 302 inet_db
> inet_hosts_byname inet_hosts_byname bag 0 302 inet_db
> inet_hosts_file_byaddr inet_hosts_file_byaddr bag 0 302 inet_db
> inet_hosts_file_byname inet_hosts_file_byname bag 0 302 inet_db
> neurone_counter neurone_counter set 258394 1846182 entity_server
> neurone_group_counter neurone_group_counter set 6 344
> entity_group_server
> neurone_group_name neurone_group_name set 6 426 entity_group_server
> neurone_group_name_reverse neurone_group_name_reverse set 6
> 426 entity_group_server
> neurone_name neurone_name set 258394 11824602 entity_server
> neurone_name_reverse neurone_name_reverse set 258394 11824602 entity_server
> memory(): [{total,5568669792},
> {processes,1138936},
> {processes_used,1128120},
> {system,5567530856},
> {atom,349769},
> {atom_used,336605},
> {binary,82704},
> {code,3046365},
> {ets,5562163256}]
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
>
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questions-unsubscribe@erlang.org
Post received from mailinglist |
|
|
| Back to top |
|
| smithjolin |
| Posted: Wed Aug 15, 2012 8:32 am |
|
|
|
User
Joined: 25 Jul 2012
Posts: 11
|
Hublot King Electricity Monaco F1 PVDLE Black Asian 7750
Hublot King Power Monaco F1 PVD LE Black Asian 7750
$488.00
Movement Asian 7750 Valjoux Functioning Chronograph motion, 28800bph, Engraved Adorned Insignia Rotor Event DIAMETER forty eight.5mm THICKNESS 17mm DIAL Coloration Black Graphical Texture Dial and Subdials, Red Increase Metal Sticks Hour Markers Circumstance Substance Good 316 chrome steel, Thick PVD Black Coating. BRACELET Leather strap with Insignia Deployant Clasp Front GLASS Sapphire crystal, Ambigu sided AR coated Back again GLASS Display situation back with depth Engraved case rims BEZEL Black Rubber Clad Bezel HACK Movement Indeed CLASP Style Deployant Folding Clasp Day INDICATOR 4.thirty Adjustment via Crown (Up to date New Font Datwheel) GMT HAND
OTHER REMARKS
bottega veneta Water Resistant Thoroughly Practical Chronograph Leading pusher starts stops chronograph Bottom pusher resets chronograph
At topareplica , we focus on leading excellent replica watches.Swiss engineering,precision constructed timepieces.We've been suppliers and sellers of such top good quality reproduction watches,we can easily offer you affordable rolex with factory price tag and also the greatest superior products.Our replica rolex are not low-priced imitations,they may be real replicas with the real merchandise.All our reproduction watches have alike search and truly feel in the first merchandise.Putting bottega veneta bags on these highly-priced shopping reproduction watches is prestigious,they make a statement at bottegaveneta perform and at enjoy.
Welcome to negotiate business and set up long-term welcoming cooperation relationships with us within the in close proximity to long term.
We specialist investing company that specializes in top rated quality clothing, jerseys,Footwear,purses, jewelry,Pen,sunglasses and swiss watches
Reproduction purses
Gucci *beep*,Louis Vuitton Purse,Burberry *beep*,Chanel Purses,Balenciaga *beep*,Barbarabui Purses,Chloe *beep*,Mentor Purses,Christian Dior Purses,Hermes *beep*,Dolce & Gabbana *beep*,Prada Purses,Jimmy Choo Purses,Juicy Couture Purses,Fendi Purses,Marc Jacobs Purses,Miu Miu Purses,Mulberry Purses,Thomas Wylde *beep*,YSL *beep*,Givenchy *beep*,Versace *beep*,Loewe Purses,Valentino *beep*,Mens *beep*,Wallet *beep*,BELT
Perfect swiss watches
Rolex for Man Watches,Rolex for Lady Watches,Rolex Mid Size Watches,Officine Panerai Watches,Omega Watches,Hublot Watches,Anonimo Watches, A Lange & Sohne Watches,Alain Silberstein Watches,Audemars Piguet Watches,Baume & Mercier Watches,B.R.M Watches,Breitling Watches,Breguet Watches,Blancpain Watches,Bvlgari Watches,Burberry Watches,Bell Ross Watches,Bedat & CO Watches,Chaumet Watches,Carl F Bucherer Watches,Chronoswiss Watches,Cartier Watches,Corum Watches,Concord Watches,Christian Dior Watches,Chopard Watches,Chanel Watches,De Witt Watches,Ebel Watches,Equip Watches,Fendi Watches,Franck Muller Watches,Ferragamo Watches,Ferrari Watches,Glashutte Watches,Graham Watches,Gucci Watches,Hysek Watches,Hermes Watches,Harry Winston Watches,IWC Watches,Jaeger-LeCoultre Watches,Jacob & Co Watches,Longines Watches,Louis Vuitton Watches,Lamborghini Watches,Maurice LeCroix Watches,Mont Blanc Watches,Montega Watches,Movado Watches,MB&F Watches,Oakley Watches,Oris Watches,Porsche Design Watches,Prada Watches,Piaget Watches,Paul Picot Watches,Patek Philippe Watches,Pierre Kunz Watches,Parmigiani Fleurier Watches,Roger Dubuis Watches,Romain Jerome Watches,Richard Mille Watches,Sinn Watches,Tag Heuer Watches,Tudor Watches,Ulysse Nardin Watches,U-Boat Watches,Vacheron Constantin Watches,Versace Watches,Zenith Watches,$98 Japan Watches
Reproduction footwear
FENDI Footwear,Bally Sneakers,Hogan Footwear,Coach Shoes,Burberry Shoes,LV Shoes,Bottega Veneta Sneakers,Chloe Footwear,CHANEL Shoes,Christian Louboutin Sneakers,Versace Shoes,D&G Sneakers,Salvatore Ferragamo Footwear,Gucci Shoes,JIMMY CHOO Footwear,Prada Sneakers,MARC JACOBS Shoes,Hermes Shoes,JC Juicy Couture Footwear,YSL Shoes,TORY BURCH Footwear,TODS Sneakers,Roger Vivier Sneakers,Zegna Footwear
Replica Clothes
Marlboro Classics Clothes,FRED PERRY Clothes,TOMMY Clothes,Dior Clothes,Bikembergs Clothes,Versace Clothes,Paul Smith Clothes,Paul Shark Clothes,Childrens Clothes,Pierre Cardin Clothes,Ralph lauren Clothes,Lacoste Clothes,Louis Vuitton Clothes,Gucci Clothes,Armani Clothes,Hugo www.bottegavenetabagstore.com Boss Clothes,Burberry Clothes,Abercrombie & Fitch Clothes,D&G Clothes,Ties Gift Set Clothes Scarf Clothes
Fashion Jewelry
Bvlgari Jewelry,Cartier Jewellery,Gucci Jewelry,Tiffany & Co Jewelry,Juicy Jewellery
Fashion Pen
Mont Blanc Pen,S T DUPONT Pen,Cartier Pen
We supply perfect reproduction luxury watches,
see mroe welcome to our web
topareplica
msn replicatop@hotmail
Gmail 002REPLICA@Gmail
Find Cheap replica watches of prime excellent
Read More Reviews on designer watches for Shopping Reference.
Find Everything about swiss watches,*beep*,sneakers
Related Articles - Hublot, Big Bang King Electrical power 48mm,
Email this Article to a Friend!
Receive Articles like this one direct to your email box!Subscribe for free today!
amazines Sales article_detail.cfm 2411021?articleid=2411021 |
|
|
| Back to top |
   
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
