Erlang/OTP Forums

Author Message

<  Advanced Erlang/OTP  ~  Mensia log-based statistics.

datacompboy
Posted: Sat Jun 02, 2007 3:07 pm Reply with quote
User Joined: 21 Sep 2006 Posts: 69 Location: Novosibirsk, Russia
Assume we have such database:
Code:

-record(user, {uid, login, password}).
-record(line, {lid, phone, state}).
-record(call, {cid, uid, lid, date, time, log}).
  mnesia:create_table(user, [{disc_copies, [node()]}, {attributes, record_info(fields, user)}, {index, [login]}]),
  mnesia:create_table(line, [{disc_copies, [node()]}, {attributes, record_info(fields, line)}, {index, [phone]}]),
  mnesia:create_table(call, [{disc_copies, [node()]}, {attributes, record_info(fields, call)}, {index, [date,uid,lid]}]).


and such statistics calculator:
Code:

% find N top callers for period F-T
plain_usertop(N,{FD,FT},{TD,TT}) ->
    FM=FD+FT, TM=TD+TT,
    Logs = loadLogs(call, #call.date, FD, TD),
    CalcStat = fun
      (#call{uid=Id, date=D, time=T, log={success, Dur}}, Stat)
        when (D+T=<TM) andalso (D+T>=FM) ->
          orddict:update(Id, fun(O)->O+Dur end, Dur, Stat);
      (_, Stat) -> Stat
    end,
    Stat = lists:foldl(CalcStat, orddict:new(), Logs),
    StatL = orddict:to_list(Stat),
    StatSort = lists:keysort(2, StatL),
    if N > length(StatSort) -> lists:reverse(StatSort);
       ?ELSE -> lists:reverse(lists:nthtail(length(StatSort)-N,StatSort))
    end.

loadLogs(Table, Key, FstDate, LstDate) ->
    loadLogs(Table, Key, FstDate, LstDate, undefined, []).
loadLogs(Table, Key, FstDate, LstDate, SortField) ->
    loadLogs(Table, Key, FstDate, LstDate, SortField, []).
loadLogs(Table, Key, FstDate, LstDate, SortField, CurLogs) ->
    Logs = mnesia:dirty_index_read(Table, FstDate, Key),
    if (SortField=/=undefined) -> SLogs = lists:keysort(SortField, Logs); ?ELSE -> SLogs = Logs end,
    NewLogs = CurLogs++SLogs,
    if FstDate < LstDate ->
      loadLogs(Table, Key, FstDate+1000000*?SECONDS_PER_DAY, LstDate, SortField, NewLogs);
    ?ELSE ->
      NewLogs
    end.


is there way to not loading all logs at once for all period?
does mnemosyne will help there?

i'm can't write that part in mnemosyne style, since never use it :(

and what with speed? will it slower, or faster after writing on mnemosyne?

Sorry, but looks like i'm very stupid when get cold :roll:

_________________
--- suicide proc near\n call death\n suicide endp
View user's profile Send private message Visit poster's website MSN Messenger ICQ Number
lestat
Posted: Thu Jun 07, 2007 1:22 pm Reply with quote
User Joined: 07 Jun 2007 Posts: 14
Hi!

The answer is Query List Comprehension and cursors.
(It can be used only inside transactions, but if everything else is dirty performance shouldn't be a problem).

Something like this should work:
Code:
loadLogs(...) ->
Handle = qlc:q([X || X <- mnesia:table(call),
          X#call.date>=FstData,
          X#call.date=<LstData]),
    {atomic, Cursor} =
           mnesia:transaction(fun() ->qlc:cursor(Handle) end),
Cursor.


After that you can get data as much as you like in one go:

Code:
qlc:next_answers(Cursor[, Num]). %%default 10
View user's profile Send private message
datacompboy
Posted: Fri Jun 15, 2007 4:03 am Reply with quote
User Joined: 21 Sep 2006 Posts: 69 Location: Novosibirsk, Russia
lestat wrote:
Code:
loadLogs(...) ->
Handle = qlc:q([X || X <- mnesia:table(call),
          X#call.date>=FstData,
          X#call.date=<LstData]),
    {atomic, Cursor} =
           mnesia:transaction(fun() ->qlc:cursor(Handle) end),
Cursor.



cursor and next_answers should be inside ONE transaction, isn't it?

_________________
--- suicide proc near\n call death\n suicide endp
View user's profile Send private message Visit poster's website MSN Messenger ICQ Number
lestat
Posted: Fri Jun 15, 2007 4:46 pm Reply with quote
User Joined: 07 Jun 2007 Posts: 14
datacompboy wrote:
lestat wrote:
Code:
loadLogs(...) ->
Handle = qlc:q([X || X <- mnesia:table(call),
          X#call.date>=FstData,
          X#call.date=<LstData]),
    {atomic, Cursor} =
           mnesia:transaction(fun() ->qlc:cursor(Handle) end),
Cursor.



cursor and next_answers should be inside ONE transaction, isn't it?


You can use it in as many transactions as you want it.
Just don't forget to delete it when you don't need it anymore.


Mnesia documentation:
Quote:
This function executes the functional object Fun with arguments Args as a transaction.

The code which executes inside the transaction can consist of a series of table manipulation functions. If something goes wrong inside the transaction as a result of a user error or a certain table not being available, the entire transaction is aborted and the function transaction/1 returns the tuple {aborted, Reason}.

If all is well, {atomic, ResultOfFun} is returned where ResultOfFun is the value of the last expression in Fun.


So you can give the cursor back in ResultOfFun.

QLC documentation:
Quote:
Creates a query cursor and makes the calling process the owner of the cursor. The cursor is to be used as argument to next_answers/1,2 and (eventually) delete_cursor/1. Calls erlang:spawn_opt to spawn and link a process which will evaluate the query handle. The value of the option spawn_options is used as last argument when calling spawn_opt. The default value is [link].


The cursor is a process connected to the process which created it.

As long as you want to use the cursor in the same process it doesn't matter if it is in a different transaction.
View user's profile Send private message

Display posts from previous:  

All times are GMT
Page 1 of 1
This forum is locked: you cannot post, reply to, or edit topics.

Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum