| Author |
Message |
|
| datacompboy |
 Posted: Mon Jun 25, 2007 12:21 pm |
 |
|
|
User
Joined: 21 Sep 2006
Posts: 69
Location: Novosibirsk, Russia
|
I have small telephony system...
As for now, i have not imported all info from old system, but already
$ du -hs /var/lib/ps40
456M /var/lib/ps40
and (the most problematic)
grep -E beam|PID
Code: PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND
11988 pts/4 14873 131293137 1012 1820571 455396 - 1364428 - - - \_ /usr/lib/erlang/erts-5.5.2/bin/beam -- -root /usr/lib/erlang -progname erl -- -home /home/ps40
eaten LOT of RAM.
and -- system loads VERY long... about 10 minutes before it become ready to operation.
i'm think about move some tables (*_log) to disk_only, will that degrade perfomance on access to it?
or, may be, need to move archive data into separate disk_only tables ? this will increase complexity of code... but will that help?
Code: mnesia:info().
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
ps_order_log : with 1912 records occupying 96011 words of mem
ps_client_log : with 85312 records occupying 4829393 words of mem
ps_order_schedule: with 0 records occupying 68 words of mem
ps_queue : with 116 records occupying 2542 words of mem
ps_client : with 21325 records occupying 3553898 words of mem
ps_session_log : with 520469 records occupying 118981016 words of mem
ps_order : with 335 records occupying 39953 words of mem
ps_session_cache: with 10 records occupying 499 words of mem
ps_line_use : with 4 records occupying 339 words of mem
ps_adviser_runned: with 4 records occupying 343 words of mem
ps_soap_sessions: with 480856 records occupying 34291822 words of mem
ps_adviser : with 577 records occupying 170481 words of mem
ps_line_log : with 78820 records occupying 1787208 words of mem
ps_adviser_greeting: with 371 records occupying 172662944 bytes on disc
ps_phonebase : with 0 records occupying 275 words of mem
ps_session_run : with 8 records occupying 467 words of mem
ps_order_runned: with 0 records occupying 275 words of mem
ps_session : with 39122 records occupying 6726860 words of mem
ps_line : with 321 records occupying 58037 words of mem
ps_session_functions: with 46 records occupying 1717 words of mem
schema : with 26 records occupying 3308 words of mem
ps_adviser_log : with 156290 records occupying 5563342 words of mem
ps_sessstatcache: with 364784 records occupying 32194202 words of mem
ps_adviser_stats: with 577 records occupying 7199 words of mem
ps_queue_run : with 0 records occupying 275 words of mem
ps_sequence : with 5 records occupying 335 words of mem
===> System info in version "4.3.3", debug level = none <===
opt_disc. Directory "/var/lib/ps40" is used.
use fallback at restart = false
running db nodes = [nonode@nohost]
stopped db nodes = []
master node tables = []
remote = []
ram_copies = [ps_adviser_stats,
ps_line_use,
ps_order_runned,
ps_queue,
ps_queue_run,
ps_session_functions,
ps_session_run]
disc_copies = [ps_adviser,
ps_adviser_log,
ps_adviser_runned,
ps_client,
ps_client_log,
ps_line,
ps_line_log,
ps_order,
ps_order_log,
ps_order_schedule,
ps_phonebase,
ps_sequence,
ps_session,
ps_session_cache,
ps_session_log,
ps_sessstatcache,
ps_soap_sessions,
schema]
disc_only_copies = [ps_adviser_greeting]
[{nonode@nohost,disc_copies}] = [ps_sequence,
ps_sessstatcache,
ps_adviser_log,
schema,
ps_line,
ps_session,
ps_phonebase,
ps_line_log,
ps_adviser,
ps_soap_sessions,
ps_adviser_runned,
ps_session_cache,
ps_order,
ps_session_log,
ps_client,
ps_order_schedule,
ps_client_log,
ps_order_log]
[{nonode@nohost,disc_only_copies}] = [ps_adviser_greeting]
[{nonode@nohost,ram_copies}] = [ps_queue_run,
ps_adviser_stats,
ps_session_functions,
ps_order_runned,
ps_session_run,
ps_line_use,
ps_queue]
537954 transactions committed, 1721 aborted, 0 restarted, 365946 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
|
_________________
--- suicide proc near\n call death\n suicide endp |
|
| Back to top |
    
|
| anderst |
| Posted: Mon Jun 25, 2007 6:53 pm |
|
|
|
User
Joined: 21 Nov 2006
Posts: 37
|
http://www.planeta.toliman.pl/?p=101 is a blog which talks about memory usage in ejabberd when migrating from Mnesia to Postgres. The results speak for themselves. Use Mnesia only for live session data, not for archiving or logging. For session data, however, you get speed and reliability.
The fact that it takes so long for you to load the table (I assume you are using wait_for_tables/1) is probably due to the fact that Mnesia is fixing a badly closed file. The algorithm is not one of the most efficient ones.
Do you get the same effect when you shut down your application correctly (eg application:stop(mnesia)?
Hilsen,
Anders |
|
|
| Back to top |
|
| datacompboy |
| Posted: Tue Jun 26, 2007 8:15 am |
|
|
|
User
Joined: 21 Sep 2006
Posts: 69
Location: Novosibirsk, Russia
|
anderst wrote: http://www.planeta.toliman.pl/?p=101 is a blog which talks about memory usage in ejabberd when migrating from Mnesia to Postgres. The results speak for themselves. Use Mnesia only for live session data, not for archiving or logging. For session data, however, you get speed and reliability.
ohh... well... then, I should now try to migrate from mnesia too...
anderst wrote:
Do you get the same effect when you shut down your application correctly (eg application:stop(mnesia)?
if application closed correctly, it starts up in about 3 minutes.. that also too much.
But in common, i need to fast start after problem :)
ok, i'm understand: need to migrate to psql. will do... |
_________________
--- suicide proc near\n call death\n suicide endp |
|
| Back to top |
|
| anderst |
| Posted: Wed Jun 27, 2007 7:30 am |
|
|
|
User
Joined: 21 Nov 2006
Posts: 37
|
You need to ask yourself when to use and when not to use Mnesia. Une mnesia only for live, session data and as a cache. If your tables get big with inactive user data or statistics, then it it time to migrate.
Ejabberd uses Mnesia to store the images of users (Big binaries!) as well as off line messages, probably remains of initial prototyping activities. This is the perfect example of when to NOT use mnesia.
So before migrating your tables, see where you can sacrifice speed for a system using less memory, and where you can use mnesia as a cache vs a persistent database.
Most applications are fine with mnesia on its own without over engineering them with other RDBMS. It depends on the context in which they are used.
Good luck,
Anders |
|
|
| Back to top |
|
| datacompboy |
| Posted: Thu Jun 28, 2007 5:06 am |
|
|
|
User
Joined: 21 Sep 2006
Posts: 69
Location: Novosibirsk, Russia
|
anderst wrote: You need to ask yourself when to use and when not to use Mnesia. Une mnesia only for live, session data and as a cache. If your tables get big with inactive user data or statistics, then it it time to migrate.
Currently i'm have in mnesia everything: and live data, and archive, and blobs.
When i'm start development, there was plans to extend system with spare server, with DB replicas...
For now that plan are quite far, so i can migrate from mnesia.
anderst wrote: Ejabberd uses Mnesia to store the images of users (Big binaries!) as well as off line messages, probably remains of initial prototyping activities. This is the perfect example of when to NOT use mnesia.
Also, i'm store in mnesia adviser greetings. Big binaries. But there no problem -- that are disk_only table, that works fast (since i'm read only on primary key, and only if on telephony server no file in cache).
I have two types of tables: logs, where are session activity are logged. It are "bag" tables, from where I'm read on key everything, sort, and build information summary.
Processed logs are saved into main session archive table. In that table are all sessions saved. And live, and archive. From that table I'm load logs for some day/period, and collect some statistics -- that are most slow point in system. In SQL that a bit better...
So, I think, I should move to SQL archive. At least, move there data older than one month, otherwise mnesia "set" tables for logs and archive gets more and more bigger, and system starts more and more slowly :( |
_________________
--- suicide proc near\n call death\n suicide endp |
|
| Back to top |
|
| anderst |
| Posted: Thu Jun 28, 2007 1:42 pm |
|
|
|
User
Joined: 21 Nov 2006
Posts: 37
|
Quote: Currently i'm have in mnesia everything: and live data, and archive, and blobs. When i'm start development, there was plans to extend system with spare server, with DB replicas...
Pick a database with duplication. Berkely DB, My SQL.. I think postgres should have it as well. Else, you just need a thin layer on top of them, handling the duplication in Erlang.
Quote: Also, i'm store in mnesia adviser greetings. Big binaries. But there no problem -- that are disk_only table, that works fast (since i'm read only on primary key, and only if on telephony server no file in cache).
Store the big binaries as separate files, with a reference to the file name and location in Mnesia. Much cleaner. I remember reading about this in a post somewhere, but could not find the reference to it. Others have tried and solved it this way. The problem with dets is if the file is not closed correctly, you will have problems.
Hilsen,
Anders |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
