| Author |
Message |
|
| samkes |
 Posted: Sun Mar 01, 2009 8:50 pm |
 |
|
|
Joined: 25 Feb 2009
Posts: 3
Location: Sweden
|
The loading time in Mnesia is really starting to bug me. I can't figure out why it's taking nearly 4 minutes to load a table containing less than 100.000 rows (6.2 Mb on disk).
The table is properly closed before I shut down the VM so there is no repairing going on that I'm aware of. And from what I've seen Mnesia prints a little info in the shell when it starts repairing a table and there is none of that at least.
The record defining the table is:
Code:
-record(property, {id, key, value}).
The table is a disc_copies table with an index on key. It's not distributed or fragmented. I'm not using any parameters other than '-mnesia dir' when starting the VM. I've tried adding '+A 128' but that changes nothing on the load times (+A 10, 64, 256, 3... makes no difference).
Anybody have any insights as to what Mnesia might be up to that's taking so long? |
|
|
| Back to top |
  
|
| uwiger |
| Posted: Mon Mar 02, 2009 11:53 am |
|
|
|
User
Joined: 03 Jul 2006
Posts: 604
Location: Sweden
|
samkes wrote: The loading time in Mnesia is really starting to bug me. I can't figure out why it's taking nearly 4 minutes to load a table containing less than 100.000 rows (6.2 Mb on disk).
Have you enabled the thread pool?
This is something that tends to improve mnesia's performance rather drastically.
For example:
(I don't know if there's an optimal number. You can try to tune it if you want.)
BR,
Ulf W |
|
|
| Back to top |
|
| samkes |
| Posted: Mon Mar 02, 2009 12:32 pm |
|
|
|
Joined: 25 Feb 2009
Posts: 3
Location: Sweden
|
Yes I tried enabling the thread pool. Sorry about mixing that information in what all the text when I wrote my post earlier.
I read another post where you had recommended enabling the thread pool, but it made no difference at all.
Could it have something to do with the index on 'key'? I use the following when I create the table:
Code: create_table(Table) ->
Attr = [{attributes, record_info(fields, property)},
{type, bag},
{record_name, property},
{index, [key]},
{disc_copies, node()}],
mnesia:create_table(Table, Attr).
And again, the record definition is
Code: -record(property, {id, key, value}).
Could it be the index? Is it rebuilt on every load. Even if it is, the table shouldn't require 4 minutes to load?
Any help is greatly appreciated. I don't have the time to start digging in to Mnesia's code until a month or so (small business with too many clients at once). |
|
|
| Back to top |
|
| uwiger |
| Posted: Mon Mar 02, 2009 1:54 pm |
|
|
|
User
Joined: 03 Jul 2006
Posts: 604
Location: Sweden
|
samkes wrote: Could it be the index? Is it rebuilt on every load. Even if it is, the table shouldn't require 4 minutes to load?
Yes, the index is rebuilt on every load (except for disc_only_copies).
If you turn on verbose logging for mnesia (-mnesia debug verbose), you might be able to learn whether this is the culprit.
BR,
Ulf W |
|
|
| Back to top |
|
| samkes |
| Posted: Mon Mar 02, 2009 4:00 pm |
|
|
|
Joined: 25 Feb 2009
Posts: 3
Location: Sweden
|
Well it turns out it is the index taking 4 minutes to build. I removed the index from all tables and can't see any noticeable difference in performance yet (except load times being good).
There are some places I can use dirty calls to speed things up if the need arises as well. Mnesia is quite a different sort of database. Takes some getting used to, especially when using it for things it was never meant for.
Thanks for the help. I should have tried removing the index before posting, but I guess some other erlang beginner might read this thread and find something useful. |
|
|
| Back to top |
|
| nina |
| Posted: Fri Oct 16, 2009 2:09 am |
|
|
|
Joined: 16 Oct 2009
Posts: 1
|
I am facing the same problem as samkes. Does this mean that if we want to use indexes, we just have to accept really long load times at start up time? Or is there some way to minimise these as well?
I've done some testing with a table with 100000 records, and it takes about 15 minutes to load, vs no time at all when there are no indexes.
The only reason I would like indexes is to do queries like index_read/3 as I can't work out how to do that with list comprehension.
As an example, I do:
Code: mnesia:dirty_index_read(Table, Value, Field)
but to do this with list comprehension, would require something like
Code: qlc:e(qlc:q([E || E <- mnesia:table(Table), E#Table.Field == Value]))
which obviously incorrect syntax, but this is what I would like to do.
If anyone has any tips on how to minimise the load time of indexed tables OR doing index_read/3 with list comprehension, would be fantastic!
Thanks
Nina |
|
|
| Back to top |
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
