|
|
| Author |
Message |
< Advanced Erlang/OTP ~ Re: Measuring Function Execution Time |
| bluefly |
 Posted: Fri Jan 11, 2008 4:03 pm |
 |
|
|
User
Joined: 06 Jan 2008
Posts: 10
|
I am wondering if the article on Measuring Function Execution Time ought to at least give a nod towards the "First Run" metric.
This is an aspect of performance benchmarking that a lot of developers overlook. Unfortunately for us benchmarkers, the OS caches, the runtime caches, the compiler optimizes, and the very first iteration in the exact same operation is sometimes longer than the remaining iterations and an important consideration for improving the performance. Sometimes a wild first iteration will only reveal itself the first time it is performed after the hardware boots.
Examples of lists of returned metrics with the "First Run" bias are [4, 3, 3, 3, 3, 3] and [12, 5, 6, 5, 5, 7, 5, 4, 5, 5].
For averaging purposes, one should generally not average in the first iteration, reporting it separately. The real outliers will reveal themselves in the remaining iterations, as in [3, 3, 3, 8, 3, 3] -- how did we get the jump to 8? Context switch? Starved process? And finally, now that caching is not involved on that first iteration, we can take a look at optimizing its algorithm or consciously choosing not to. |
|
|
| Back to top |
 
|
| francesco |
| Posted: Wed Jan 16, 2008 5:03 pm |
|
|
|
User
Joined: 07 Jul 2006
Posts: 249
Location: London
|
The tutorial is written on a wiki for the very purpose you quote. If you feel it is important, you are very welcome to mention and add teh First Run metric.
In regards to Erlang, what comes to my mind is memory allocation in the first iteration, and triggering teh GC in the subsequent ones.
Regards,
Francesco
--
http://www.erlang-consulting.com |
|
|
| Back to top |
|
|
|
All times are GMT
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
|
|
|
Digg It
Del.icio.us
Reddit
Facebook
Stumble Upon
Technorati
