| Author |
Message |
|
| forcewill |
 Posted: Fri Apr 09, 2010 11:01 am |
 |
|
|
User
Joined: 12 Feb 2010
Posts: 19
|
I've been asked to do dump of a mnesia table at regular intervals (most probably one time perday) of a mnesia table to a text file and then clear that mnesia table
The idea is to have an daily dump to an human readable text file which can later be used to answer some functional questions about our application. (whether or not in day X the record with Key Y was processed etc). |
|
|
| Back to top |
 
|
| jwatte |
| Posted: Fri Apr 09, 2010 5:52 pm |
|
|
|
User
Joined: 10 Feb 2010
Posts: 34
|
Do you have a specific question?
The obvious problem with the description as it is written is that there is a race condition. If you first dump to text, and then clear the table, any record entered between dump and clear will be lost.
One possible solution is to tag each entry in the table with a "generation count" or date stamp, and "clear" the table by removing only terms stored before the dump operation was initiated. |
|
|
| Back to top |
|
| forcewill |
| Posted: Fri Apr 09, 2010 8:20 pm |
|
|
|
User
Joined: 12 Feb 2010
Posts: 19
|
Indeed you're right, my question is that due to my inexperience
with mnesia i am having a hard time doing that transaction in an efficient way,
if by one hand i acquire a write lock on the table,
and use mnesia:foldl to accomplish the task
i kill my CPU spikes to 100% during the transaction :s
Edit: forgot to mention that the table in question as a size of 40 000 records |
Last edited by forcewill on Fri Apr 09, 2010 9:09 pm; edited 1 time in total |
|
| Back to top |
|
| jwatte |
| Posted: Fri Apr 09, 2010 9:06 pm |
|
|
|
User
Joined: 10 Feb 2010
Posts: 34
|
I think you'll generate CPU spikes no matter what, unless you want to "trickle" out the report. As long as those spikes don't actually impact the other responsiveness of the system, that shouldn't matter. If they do, then you'll have to sleep for a little while for every, say, 100 rows that you report out.
What I would do is mark each term with a "generation count."
When time comes to report, I would, in a transaction, read the generation count, then increment it by one. This means that any new data will be marked with the new transaction count.
Then I would do a query or fold or whatever over the table to find all terms with the old generation count, and write those out to the report.
Then I would delete all terms matching the old generation count.
If you absolutely can't live with the CPU spikes, then build a second node, explicitly for reporting, and replicate the table onto there. Then you can run the report, with 100% CPU, without affecting the source system. |
|
|
| Back to top |
|
| zajda |
| Posted: Sat Apr 10, 2010 10:40 am |
|
|
|
User
Joined: 22 Aug 2009
Posts: 83
|
The other way is to mantain two mnesia tables.
Lets say A and B. To write data to table use some wrapper (like an api) and when you want to do dump change 'current' table from A to B (managed behind universal api). Data in A is safe and you have whole day to make a backup and new data goes to B. This should also level CPU utilization.
Michał Zajda
------------------
Erlang Solutions Ltd |
|
|
| Back to top |
|
| forcewill |
| Posted: Mon Apr 12, 2010 9:58 am |
|
|
|
User
Joined: 12 Feb 2010
Posts: 19
|
thanks for the help i am going for zajda solution,
i have however one more question this one regards the transaction to dump the current table: i am doing it using this "algorithm"
1º acquire a write lock on the current table
2º do a mnesia foldl
2.1) use the Acc to know the total count of actually written recs
2.2) delete the record in question
2.3) this is my question
doing a timer:sleep(5) to allow the CPU to rest
my question is if its safe to do a timer:sleep inside a transaction.
Once again thanks for all help |
|
|
| Back to top |
|
| zajda |
| Posted: Mon Apr 12, 2010 10:48 am |
|
|
|
User
Joined: 22 Aug 2009
Posts: 83
|
|
| Back to top |
|
| Michal Ptaszek |
| Posted: Mon Apr 12, 2010 10:53 am |
|
|
|
User
Joined: 01 May 2008
Posts: 35
Location: Krakow
|
If you decided to store data from whole day in a single, separate table, why don't you use the following scenario:
- acquire write lock to table
- use mnesia:dump_to_textfile/1
- use mnesia:clear_table/1
within the same transaction?
I think putting anything that has some side-effects, including timer:sleep/1, is not a good idea for transaction. If, for whatever reason, the transaction would be rerun several times, you will get some serious timeouts. |
|
|
| Back to top |
|
| forcewill |
| Posted: Mon Apr 12, 2010 12:12 pm |
|
|
|
User
Joined: 12 Feb 2010
Posts: 19
|
Michal Ptaszek wrote:
- acquire write lock to table
- use mnesia:dump_to_textfile/1
- use mnesia:clear_table/1
within the same transaction?
Yes the first thing i considered was dump_to_textfile however it dumps all schema and all tables and i only need to dump one.... |
|
|
| Back to top |
|
| babyjing |
| Posted: Thu Jan 12, 2012 2:45 am |
|
|
|
Guest
|
| Popular nfl jerseys cheap are going to be scarce over the next three months.Then they're going to be more expensive.The most popular teams will experience the worst shortages, and the James Jones Jersey are breaking all sorts of popularity records this year, from on-field performance to stock sales to merchandise sales.To lure the customer towards their service, providers are leaving no stone unturned in order to provide several plans to make Josh Sitton Jersey.Thats why your website's critics are so vital, and ofcourse why companies must be alerted at all times.With the sport's new passing-crazed style of play challenging the traditional notion that Buffalo Sabres jerseys is a fundamental ingredient for postseason success, these Nick Collins Jersey will provide an intriguing look at the way to win in the NFL when the stakes are highest. |
|
|
| Back to top |
|
| wuji |
| Posted: Sat Aug 11, 2012 1:08 am |
|
|
|
User
Joined: 10 Aug 2012
Posts: 654
|
property.Once you get these internal government documents, you should scrutinize scrutinize [h3]designer replica *beep*[/h3] scrutinize them, looking for errors that give you a great
for your appeal. There are really only three basic basic cheap replica *beep* basic arguments you can make to appeal your tax assessment.
keeps it nice and simple. See if one applies:1. The The replica designer *beep* The assessor made a mistake in describing your house. This
covers situations where the assessor made simple math mistakes. Maybe Maybe cheap Ralph Lauren Polo Maybe he got the square footage wrong (only heated, livable
should be counted), or stated that you have five bedrooms bedrooms cheap designer *beep* bedrooms when you have three. If you uncover these errors,
you – you've just upped your chances of a slam slam replica designer bags for sale slam dunk win. All you have to do is gather
and describe the errors in writing or in person.2. You You cheap replica *beep* You just bought the house for less. If you recently
your home for less than the assessed value would indicate, indicate, cheap Christian Louboutin indicate, lucky you. If the assessment is wrong, and grossly
you're likely to win your appeal. (If the assessment is is red bottom shoes is right, you will lose, but it means you got |
|
|
| Back to top |
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
