tag:blogger.com,1999:blog-51710987273643952422024-03-18T05:41:59.219+00:00Psychosomatic, Lobotomy, SawIt's X, you'll need Y, I'll get ZNitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.comBlogger70125tag:blogger.com,1999:blog-5171098727364395242.post-14601473072899740892018-07-11T09:36:00.000+01:002018-07-11T09:50:55.670+01:00How Inlined Code Makes For Confusing Profiles<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiVUc_EYxyYHOsM3-SjxYotCla33ZP1gq969ba7EoLE9CEhpIXTjFnP1nDY1tvVKD4J4qZQScMGpxCJiDfzb4evd1r0Rk90RxceWQ6R8XkVElITAyKl6zekTXIZ1MU85tBCrbWe9hTnaA/s1600/confusion.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="675" data-original-width="1024" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiVUc_EYxyYHOsM3-SjxYotCla33ZP1gq969ba7EoLE9CEhpIXTjFnP1nDY1tvVKD4J4qZQScMGpxCJiDfzb4evd1r0Rk90RxceWQ6R8XkVElITAyKl6zekTXIZ1MU85tBCrbWe9hTnaA/s320/confusion.jpg" width="320" /></a>Inlining is a powerful and common optimization technique used by compilers. <br />
But inlining combines with other optimizations to transform your code to such an extent that other tooling becomes confused, namely profilers.<br />
<div>
<ol>
<ol>
</ol>
</ol>
<h3>
Mommy, What is Inlining?</h3>
<a href="https://en.wikipedia.org/wiki/Inline_expansion" target="_blank">Inlining</a> is the mother of all optimizations (to quote C. Click), and mothers are awesome as we all know. Inlining is simply the practice of replacing a method call with the method body. This doesn't sound like much of an improvement, but it's potentially a massive win for a couple of reasons:<br />
<ul>
<li>Expanding the scope of other optimizations! calling a method with a constant parameter may just end up getting folded into a constant for instance. Most compilers take a view that you either know everything about a method, because it's the method you are compiling ATM, or you know next to nothing about it. Calling another method may end up blocking, or having some unknown side effect, or bring the world to an end. Inlining expands the horizons of all available optimizations into the inlined code. These benefits can be quite dramatic, eliminating much of the inlined code (callee) on the one hand and assisting in supporting assumptions about the compilation unit at hand (caller).</li>
<li>Very small methods call overhead may actually exceed the method cost. Importantly, and counter intuitively perhaps, the cost of calling a method is not fixed. The cost will depend for example on calling convention (whose job is it to keep track of registers), pre/post conditions (e.g. ordering and safepoints).</li>
</ul>
But Inlining is not without it's potential issues:<br />
<ul>
<li>"code explosion". If I have a method of size 128b and that method cannot be reduced in any form by the context of it's caller, every time I inline it I increase my code size by 128b (a bit less, as I can remove the calling related prologue/epilogue). This may lead to a very bloated application, which will put pressure on my instruction cache as well as the JVM code cache. </li>
<li>Increased CPU pressure from the compiler itself, being challenged with larger and larger compilation units will lead to higher compile times, increased CPU usage, delayed compilation etc.</li>
<li>Increasingly complex compilation units may result in less optimal code than simpler ones. This is a real world limitation, though I would think in theory the more context the compiler has to work with, and assuming an infinite amount of cpu/memory/monkeys to throw at the problem, the better overall result we should get. In practice I have seen several cases where limiting inlining can improve results.</li>
</ul>
Picking the right boundaries is challenging, but luckily we got them bright compiler engineers to worry about that. <br />
<br />
<h3>
How Much Inlining Can We Do?</h3>
<div>
The amount of inlining a compiler does will depend on many factors. I'm not going to get very far if I try and explain the heuristics involved in great detail but some basic dimensions to the problem are:</div>
<div>
<ul>
<li>Who is the callee? In Java almost all method calls are 'virtual'. This means that at runtime any number of implementations could possibly exist for a particular call. This is traditionally where you'd give up on inlining. To make inlining happen anyway the compiler must convert a virtual call into a static call before it can be inlined. The same code in a different runtime may induce different compiler decisions on this. JIT compilation allows the compiler to tailor the decision in a way that AOT compilation cannot (in a dynamic environment). </li>
<li>How big is the callee? Most compilers put some upper limit on how big a caller method can grow as well as how big can a callee method be and still get inlined. A very large caller will be blocked from further inlining and even a medium sized callee might be blocked from being inlined. What is considered big will depend on the compiler, configuration and the profile. Limitations on compilation time for JIT compilers also drive this decision, at least partially.</li>
<li>How deep is the already inlined stack? How many veils can the compiler pierce? What about recursion?</li>
</ul>
So, that's the general shape of the problem: Virtual x Size x Depth. And if you were gonna try and write an inlining strategy for a compiler you'd need allot more details, but we can keep going with this for now.</div>
<div>
<br /></div>
<h3>
How Much Inlining Does Java Actually Do?</h3>
<div>
And by Java I mean the Oracle JDK 8u161. These things change whimsically from run to run, release to release, JVM to JVM etc, so stick a big YMMV on this shit.<br />
We can have the JVM helpfully share with us compilation details via a bunch of flags, but most importantly for this feature we will need: <span style="font-family: "courier new" , "courier" , monospace;">-XX:+PrintInlining -XX:+PrintCompilation</span> (output is a bit hard to reason about because concurrent logging...). <br />
<span style="font-family: inherit;">For the sake of demonstration, here's the output of the offer method of SpscArrayQueue from a throughput benchmark I run regularly for JCTools:</span></div>
<script src="https://gist.github.com/nitsanw/c104c92287ea785c8d30325ba18157dc.js"></script>
<br />
Some observations:<br />
<ul>
<li>Tiny methods and some intrinsics are inlined even by the early compilation.</li>
<li>The <i>offerSlowPath</i> method which is not inlined by the first compilation is inlined by the C2 compiler at a second pass. </li>
</ul>
This is not a particularly challenging piece of code, but I picked it because it is easy enough for a human to follow. Small methods get inlined, even if they are nested a bit in other tiny methods.<br />
There's a performance related nugget to dig here:<br />
<ul>
<li>I split the <i>offerSlowPath</i> method with the intent of not having it inlined (or at least leaving this decision to the compiler). The compiler decided it is better off inlined. Who's right? has the compiler defeated my puny human brain? am I as worthless as my mother in law suspects? these questions will be answered some other day.</li>
</ul>
<h3>
Where Do Profilers Come In? </h3>
In traditional Java profilers no attempt is made at distinguishing between inlined frames (calls to methods which have been inlined) and 'real' frames (real method calls on the stack). Telling the 2 apart can be useful:</div>
<div>
<ul>
<li>Inlining is often a good thing, so failure to inline a method can be worth looking into. It's also instructive to see just how much the compiler can inline, and how many layers of distraction it must cut through to get to where the actual code you want to run is. Excessive "Manager/Abstract/Adapter/Proxy/Wrapper/Impl/Dimple/Pimpl" layering however can still prevent inlining and may also lead to erectile dysfunction, early onset dementia and being knifed by your colleagues.</li>
<li>Top of stack hot methods which are not inlined present obvious curios. Perhaps callers are too large, or the method itself?</li>
<li>Multiple implementors of same methods which are not inlined indicate a megamorphic callsite. If one of these is significantly hotter than the rest you might be able to special case for it (peel off the callsite with an <i>instanceof</i> check, effectively rolling your own conditional inlining).</li>
</ul>
But missing out on some good opportunities is not the issue I'm concerned with in this post. The point I'd like to get across is that <b>Inlining makes an already confusing reality much much more confusing</b>.</div>
<div>
<br /></div>
<div>
<h3>
A Confusing Reality</h3>
CPUs execute instructions, these instructions are not our code but the product of a series of compilation and optimization steps. It follows that if we hope to learn where the bottleneck in our code/application is we must be able to translate back from CPU instructions (the product of C1/C2/Other JIT compilers) to bytecode (the product of javac/JVM language compiler/bytecode generating agent/Other) to Code coordinates (file + line or class + method).<br />
This is not a straight forward mapping because:<br />
<ul>
<li>Some instructions do not map to any bytecode: this can happen when the JVM generates code for it's own accounting purposes.</li>
<li>Some instructions map to many bytecodes: the compiler can be quite clever in substituting many operations with one specialized instruction, Or when the result of one computation can be reused elsewhere.</li>
<li>Some bytecodes do not map to Code: this can happen due to bytecode generation.</li>
<li>Sometimes the compiler just fails in it's book keeping :-(. </li>
</ul>
A modern JIT compiler employs a significant number of optimizations to any given method, allowing it to deliver us the speed we love, but making reverse engineering the instructions into code a challenge. The mapping rules employed by profiling tools trying to bridge this gap, and by the compilers trying to give them a usable index boil down to:</div>
<div>
<ul>
<li>We assume instruction X translates to at most 1 bytecode location.</li>
<li>When a mapping does not exist, find the nearest instruction Y which has a mapping and use that.</li>
</ul>
This is already confusing, as the imperfect mapping, coupled with the compiler's license to reorder code combine to make "nearest instruction" possibly be "not nearest line of code".<br />
The problem is made worse by the inaccuracy of the profiler observations themselves. It is a known issue, largely unsolved, that the reported instruction from which the mapping starts is often a few (in theory many) instructions away from the intended sample location. This can have a knock on effect on the ultimately blamed/mapped code. This issue was touched on in a <a href="http://psy-lob-saw.blogspot.com/2016/06/the-pros-and-cons-of-agct.html" target="_blank">previous post on AGCT profilers</a>.<br />
When looking at a single method this is often a pesky, but ultimately managable problem. Sure, the profiler pointed to line A, but any fool can see the problem is line B... silly little hobbit.<br />
<br />
<h3>
MOAR, BIGGER, CONFUSION</h3>
So, given that:</div>
<div>
<ul>
<li>The instruction reported is a few instructions off.</li>
<li>The instructions do not map perfectly to lines of code.</li>
<li>The order of instructions does not have to match the order of the code.</li>
</ul>
We can see how even a small inaccuracy can translate into a large distance in lines of code between the cost we are trying to profile and where we end up. This is made exponentially worse by inlining.</div>
<div>
Where we had one method, we now have many methods, from many classes. As the code gets pulled in it is subjected to the same compiler "remixing", but at an expanded horizon. For instance, a computation repeated in the caller and callee can now be done once. Considering that inlining 4-6 layers of indirection is very common the "LOC distance" between "near instructions" can end up being very confusing.</div>
<div>
I should really elaborate with some clear example, but I do not have a repeatable experiment to hand and already stalled with this article for a few months while distracted elsewhere. An interesting example of how this could work out is available <a href="https://github.com/elastic/elasticsearch/issues/26339#issuecomment-328079161" target="_blank">here</a>. In this issue we have the wrong instruction being blamed for the cost of a cache-miss in a loop. The cache miss originates in the bowels of an inlined LinkedHashMap::get. The nearest instruction with a mapping however is from Field::binaryValue(line 441) it's easy to see how this would end up a very misleading profile if considered from a code perspective. When observed at the assembly level the instruction slip is much easier (as much as reading assembly is easy) to see through, but profiling on the assembly level is something very few people do and very few tools support.<br />
<br />
<h3>
Summary</h3>
Inlining is good, and profiling is good. But profiling modern compiled inlined code presents us challenges which are not easily solvable by looking at the common output of most tools. The more sense the compiler makes of our code, it's context, it's execution tree, the less obvious the mapping between instructions and code.<br />
So, what can we do? I think we should have our tools approach the problem in 2 ways:<br />
<ol>
<li>Fuzzier profiles: What would happen if instead of assigning cost to a single instruction (and it's code coordinates) we distributed the cost on a range of instructions? We could have more or less sophisticated models, but even a naive bell curve around the sampled instruction would bring in more context to the profile, and may help. This is hypothetical, as I am not aware of any tools which attempt this ATM.</li>
<li>Differentiate between real/inlined frames: This is something the JVM is certainly capable of doing, but most APIs for profiling do not support. AFAIK only <a href="http://www.oracle.com/technetwork/server-storage/developerstudio/downloads/index.html" target="_blank">Oracle Studio</a> and perf/bcc + <a href="https://github.com/jvm-profiling-tools/perf-map-agent" target="_blank">perf-map-agent</a> support views on real vs. inlined. I tend to use the latter more than the former. There's talk of adding <a href="https://github.com/jvm-profiling-tools/async-profiler/issues/66" target="_blank">support for this into async-profiler</a>.</li>
</ol>
<br />
Many thanks to <a href="https://twitter.com/VladimirSitnikv" target="_blank">V. Sitnikov</a> for reviewing :-).<br />
<br />
This post is one of a few on the topic of profiling, check the other ones out:<br />
<ul>
<li><a class="K3JSBVB-c-g" href="http://psy-lob-saw.blogspot.com/2016/02/why-most-sampling-java-profilers-are.html">Why (Most) Sampling Java Profilers Are Fucking Terrible</a></li>
<li><a class="K3JSBVB-c-g" href="http://psy-lob-saw.blogspot.com/2016/06/the-pros-and-cons-of-agct.html">The Pros and Cons of AsyncGetCallTrace Profilers</a> </li>
<li><a class="K3JSBVB-c-g" href="http://psy-lob-saw.blogspot.com/2017/02/flamegraphs-intro-fire-for-everyone.html">Java Flame Graphs Introduction: Fire For Everyone!</a></li>
<li><a href="http://psy-lob-saw.blogspot.com/2015/07/jmh-perfasm.html">JMH perfasm explained: Looking at False Sharing on Conditional Inlining</a> </li>
</ul>
</div>
<div>
<ul>
</ul>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com2tag:blogger.com,1999:blog-5171098727364395242.post-4734216856329012582018-01-05T17:27:00.000+00:002018-01-06T10:41:08.959+00:00What a difference a JVM makes?<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif9hyphenhyphenAXdekVmYk9zg3ramxrfrtSOkuK69ToapHARi4uxRlSEcoflDEJIDL8rdZgbmY8waU7uZsb018aQ0JRvuiamudrUIBRMLDomsq0abkKriIw-P27iVUK4UGjKyAKnEihEKgbwTkRUs/s1600/the_good__the_bad_and_the_ugly_by_felonland-d46b6az.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1125" data-original-width="900" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif9hyphenhyphenAXdekVmYk9zg3ramxrfrtSOkuK69ToapHARi4uxRlSEcoflDEJIDL8rdZgbmY8waU7uZsb018aQ0JRvuiamudrUIBRMLDomsq0abkKriIw-P27iVUK4UGjKyAKnEihEKgbwTkRUs/s200/the_good__the_bad_and_the_ugly_by_felonland-d46b6az.png" width="160" /></a>JDK 9 is out! But as a library writer, this means change, and change can go either way... Once we've satisfied that JCTools works with JDK9, what other observations can we make? Well, one of the main motivations for using JCTools is performance, and since the code has been predominantly tested and run with JDK8, is it even better with JDK9? is it worse?<br />
<br />
<h3>
A JVM gets into trouble</h3>
I started my comparison (and we will not cover anything else cause the fuckers broke early) with the simplest queue, the SpscArrayQueue:<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ taskset -c 4-7 java -jar jctools-benchmarks/target/microbenchmarks.jar throughput.QueueThroughputBackoffNone -p qType=SpscArrayQueue -p qCapacity=131072 -jvmArgs="-Xmx1g -Xms1g" -i 5 -wi 15 -r 5 -w 1 -f 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Oracle8u144:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.020 ± 0.020 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.093 ± 0.102 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 361.161 ± 4.126 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Oracle9.0.1:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.065 ± 0.269 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 5.987 ± 2.788 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 26.182 ± 2.273 ops/us</span><br />
<div>
<br /></div>
<div>
Some explanations on method and results:<br />
<ul>
<li>This is running on my beefy laptop, lots of memory to spare, Xeon(R) CPU E3-1505M v6 @ 3.00GHz. I set the gov'nor to "userspace" and frequency to 2.8GHz to avoid CPU frequency scaling and turbo boosting while benchmarking. I use taskset above to pin the JVM process 1 logical core on each physical core, so no 2 threads share a core. Easier than disabling HT, and sufficient for this exploration.</li>
<li>The <span style="font-family: "courier new" , "courier" , monospace;">QueueThroughputBackoffNone</span> benchmark is an all out throughput benchmark for queues where producers and consumers spin to offer/poll (<a href="http://psy-lob-saw.blogspot.com/2014/07/poll-me-maybe.html" target="_blank">with j.u.Queue semantics</a>). Failures are recorded and the throughput observed is the rate of successful polls per <b>microsecond</b>(so millions per second if you prefer that figure). The benchmark is run with a single producer and single consumer thread, and is known to be sensitive to producer/consumer speed balance as contending on empty/full queue can lead to degraded performance. <a href="http://psy-lob-saw.blogspot.com/2016/12/linked-array-queues-part-2-spsc.html" target="_blank">See some discussion of the benchmark here</a>.</li>
</ul>
</div>
<div>
<div>
<span style="font-family: inherit;">Back to the results. They is not good :(</span><br />
<span style="font-family: inherit;">Why would this happen? HOW COULD THIS HAPPEN!!!</span></div>
<div>
<span style="font-family: inherit;">I profiled this miserable little bastard only to find that </span><span style="font-family: inherit;">on the offer side of there's a previously unobserved bottleneck:</span></div>
</div>
<div>
<span style="font-family: inherit;"><script src="https://gist.github.com/nitsanw/732b691191c32576d5a441638007ce96.js"></script></span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
What's happening? The whole point of SPSC is that there's no need for a strong memory barrier, no need for lock add or CAS, just some careful ordering (<a href="http://psy-lob-saw.blogspot.co.za/2016/12/what-is-lazyset-putordered.html" target="_blank">using putOrdered/lazySet</a>). But here we got this LOCK ADDL (line 34) instruction spoiling all the fun and eating all the cycles (but the next line is getting all the blame, typical).<br />
Where did it come from?<b> I didn't put it there. </b>Note that this is add 0 to the base of the stack(-0x40), which is how a StoreLoad barrier is implemented (<a href="https://shipilev.net/blog/2014/on-the-fence-with-dependencies/" target="_blank">see interesting post on barrier implementation details</a>). But there's no volatile store in sight.</div>
</div>
<div>
<div>
<br />
<h3>
A JVM gets out of trouble</h3>
</div>
<div>
TBH, even before bothering to look at the assembly I reached for a big "usual suspect" for performance differences with JDK9: G1GC is the new default GC!</div>
<div>
I quickly verified this theory (that G1GC fucked this up for me) by re-running the same benchmark on JDK9 with the JDK8 default GC (-XX:+UseParallelGC). Got similar results to JDK8. Awesome-ish:<br />
<span style="font-family: "courier new" , "courier" , monospace;">Oracle9.0.1 (-XX:+UseParallelGC)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.059 ± 0.133 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.147 ± 0.251 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 356.100 ± 6.305 ops/us</span><br />
<div>
<br /></div>
<div>
The bottleneck I've hit? it's G1GC<a href="http://psy-lob-saw.blogspot.co.za/2014/10/the-jvm-write-barrier-card-marking.html" target="_blank"> card marking! I even wrote a post about it.</a> G1GC write barrier is quite different from CMS/Parallel. You'll notice that the barrier discussed there is a little different from the one we see here. Times, they are a changing...</div>
<div>
So... the G1GC write barrier is ANGRY. Why so angry?</div>
</div>
<div>
<script src="https://gist.github.com/nitsanw/c25e6e3969ac1b1575da71a79802035a.js"></script></div>
<div>
<br /></div>
<div>
What this means is that the <span style="background-color: white; white-space: pre-wrap;"><span style="font-family: "courier new" , "courier" , monospace;">g1_write_barrier_post </span><span style="font-family: inherit;">is pissed because:</span></span><br />
<ul>
<li><span style="white-space: pre-wrap;">buffer (the backing array for the queue) and element (being offered) are from different regions</span></li>
<li><span style="white-space: pre-wrap;">element is not null</span></li>
<li><span style="white-space: pre-wrap;">card (for the buffer) is not young</span></li>
</ul>
<span style="white-space: pre-wrap;">Confusingly, when playing around with this issue I moved from a small machine (8gb) to a bigger one (32gb) and when running with a larger heap size specified the issue became allot less pronounced. This is because, if we set the size of the heap to 1g we get 1mb regions. </span><span style="white-space: pre-wrap;">If we set the size of the heap to 8g (set both mx and ms) however we get 4mb regions.</span><span style="white-space: pre-wrap;"> We can demonstrate this is the issue by running again with G1 and setting the region size:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; white-space: pre-wrap;">Oracle9.0.1 (-XX:+UseG1GC -Xms1g -Xmx1g -XX:G1HeapRegionSize=4m)
offersFailed | 0.009 ± 0.033 ops/us
pollsFailed | 0.252 ± 0.257 ops/us
pollsMade | 183.827 ± 16.650 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace; white-space: pre-wrap;"><br /></span></div>
<div>
<span style="background-color: white; white-space: pre-wrap;"><span style="font-family: inherit;">So, still not so brilliant, but much better. This however implies that the improvement is due to the buffer and the element being allocated from the same region. This is purely a product of the benchmark and the chosen queue size, and is not typical of normal applications. It follows therefore (I think) that the behaviour we see here is quite likely to manifest pathologically for writes into long standing data structures, such as caches and queues. And indeed if we increase the queue capacity the situation reverts back.</span></span><br />
All this messing around with JDK9 vs 8, runtime and GC impact comparison got me thinking of a couple of other regional collectors which might exhibit interesting behaviours here. Namely, Zing C4 and Shenandoah.<br />
<h3>
</h3>
<h3>
Put some Zing in that thing</h3>
I no longer work at Azul, so I had to get an <a href="https://www.azul.com/zing-oss-open-source-developer-access/" target="_blank">Open Source developer licence</a>, which was a breeze. I had to get some help with getting it all to work on Ubuntu 17.10, but 16.04 works off the bat (just follow the instructions). Starting with same parameters I got:<br />
<span style="font-family: "courier new" , "courier" , monospace;">Zing8u17.12:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.016 ± 0.010 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.013 ± 0.022 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 302.288 ± 3.602 ops/us</span><br />
<br /></div>
<div>
So Zing is 20% slower than Oracle ParallelGC here, but significantly better than G1 (either case). The JMH perfasm profiler does not work with Zing, though Azul does have a matching tool if you ask their support. To look at the profile I can either use ZVision, or Oracle Studio. I went with the latter, just because.<br />
The profile is hard to read, so I might go into it another time, but the question that seems obvious is: "Is Zing slower than Oracle+ParallelGC because of read/write barrier costs?"<br />
Zing after all is not a GC add-on to OpenJDK, but a completely different runtime+GC+compiler. In particular, Zing has recently switched from their old C2 like compiler (they forked paths many moons ago, but share a parent in Cliff Click), to an LLVM compiler which is now the default called Falcon. Testing that quickly by forcing Zing to use the C2 compiler yields the following results:<br />
<span style="font-family: "courier new" , "courier" , monospace;">Zing8u17.12 (-XX:+UseC2):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.034 ± 0.055 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.010 ± 0.017 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 198.067 ± 31.983 ops/us</span></div>
<div>
<br />
OK, so Falcon is a big win for Zing here, that's not the issue. Can we take the read and write barriers out of the picture?<br />
Sure we can! Zing supports 2 exciting GCs, you may have heard of the C4 GC, but Zing also supports the NoGC (it's very Zen), which is exactly what it sounds like. Running with no GC however may remove some positive effects GC has (e.g. not crashing when you've allocated more than your heap size and never collected, but also compacted relocated data), so we need to run NoGC with barriers, and NoGC with no barriers:<br />
<span style="font-family: "courier new" , "courier" , monospace;">Zing8u17.12 (-XX:+GPGCNoGC -XX:+UseSVBs -XX:+UseLVBs):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.035 ± 0.053 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.022 ± 0.043 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 302.433 ± 2.675 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: inherit;">So, turning GC off makes no difference at all for this benchmark. That's great, as we can consider the removal of the barriers in isolation:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Zing8u17.12 (-XX:+GPGCNoGC -XX:-UseSVBs -XX:-UseLVBs):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.099 ± 0.070 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.027 ± 0.048 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 314.498 ± 17.872 ops/us</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Note that we see:</span><br />
<ol>
<li><span style="font-family: inherit;">Some improvement when barriers are removed.</span></li>
<li><span style="font-family: inherit;">Increased variance in results. This was due to large run to run variance which requires further digging.</span></li>
</ol>
So, while we can certainly see a difference here which is due to read/write barriers, that is not the whole story(maybe another day). My gut feeling is that Falcon is over inlining in this instance.<br />
<br />
<h3>
Oh Shenandoah</h3>
<div>
For Shenandoah I grabbed <a href="https://builds.shipilev.net/openjdk-shenandoah-jdk9/" target="_blank">one of the builds provided by the benevolent Shipilev here</a>. The build I got is this one: <span style="font-family: "courier new" , "courier" , monospace;">build 1.8.0-internal-jenkins_2017_11_12_03_35-b00</span></div>
<div>
Running with same heap size, but remembering that these builds don't default to Shenandoah:</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Shenandoah (-XX:+UseShenandoahGC):</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed | 0.031 ± 0.024 ops/us</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.009 ± 0.025 ops/us</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 143.165 ± 3.172 ops/us</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">Note that for Shenandoah there's not a massive imbalance between the offer/poll side, which indicates the issue is not pathological to one method or the other. For G1GC the problem was very clearly on the offer side. I had a peek at the assembly, but it's going to take me some time to get to grips with what's going on as it's a completely new set of tricks and quirks to look at. To get a high level view though, it's interesting to compare the HW counters for ParallelGC/Zing/G1/Shenandoah:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> | PGC | Zing | G1 | Shenandoah</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 368.055 | 303.039 | 194.538 | 140.236</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">CPI | 0.244 | 0.222 | 0.352 | 0.355 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">cycles | 7.629 | 9.319 | 14.764 | 19.998 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">instructions | 31.218 | 41.972 | 42.002 | 56.294 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">branches | 6.052 | 11.672 | 7.735 | 9.017 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">L1-dcache-loads | 10.057 | 10.936 | 14.320 | 26.707 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">L1-dcache-load-misses | 0.067 | 0.149 | 0.162 | 0.100 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">L1-dcache-stores | 3.553 | 3.042 | 4.057 | 5.064 </span></div>
<div>
<br /></div>
</div>
<div>
<ul>
<li><span style="font-family: inherit;">This is a tiny workload, with 30-50 instructions per operation. I want to make it clear that it is very easy to have large differences between JVMs in such specific scenarios. This workload is all about loading and storing references in/out of an array. The data is all L1 resident, this is NOT A REPRESENTATIVE COMPARISON OF PERFORMANCE. If you want to know how these JVMs/GCs can help your application, run a </span>representative<span style="font-family: inherit;"> workload with your application.</span></li>
<li><span style="font-family: inherit;">Seriously, let's not start a "my JVM/GC is better than thou" war here, OK people?</span></li>
<li><span style="font-family: inherit;">For simplicity of comparison I've run PGC/G1/Shenandoah out of the Shenandoah build that includes all 3. This makes for a simpler comparison as they all share the same compiler, but is not comparing with the relevant Oracle build (though it should be pretty much the same).</span></li>
<li>Zing has better CPI than PGC, but 10 more instructions per operation. These include 1 extra load, and 6 more branches. There are no branch misses in this workload, so the branches are just extra pressure on the branch predictor and more instructions. The 10 instructions difference translates into a 1.6 cycle difference, this is indicative of the success of the branch predictor in reducing the impact of the branches. These extra branches are the Zing LVB or read barrier. Each reference load costs at extra branch. Zing is doing less stores here, this is due to it's defensive approach to card marking.</li>
<li>G1 is given here with the good case (same region), as we already covered the bad case. We see G1 increasing the loads by 4 but using less branches than Zing, only 2 extra branches. These are related to the write barrier. We also see an extra store.</li>
<li>Shenandoah is using 3 extra instructions, and 16 extra loads. This is more than I expected. Since Shenandoah is using a Brooks-Pointer you would expect an extra load for each reference load. If we estimate from the Zing branch increase that we have 6 reference loads, I'd expect to have 6 extra loads on the Shenandoah side. I assume the other loads are related to card marking but I will need to learn more about this collector to say anything. <a href="http://cr.openjdk.java.net/~shade/shenandoah/jctools-QueueThroughputBackoffNone.txt" target="_blank">Shipilev has expanded on my rudimentary analysis here</a>. His conclusion: <span style="font-family: Times, Times New Roman, serif;"><u>"read and write barriers around Unsafe intrinsics are very active. C2 handling on Unsafe intrinsics uses CPUOrder membars a lot (http://hg.openjdk.java.net/shenandoah/jdk10/file/1819ee64325f/src/hotspot/share/opto/library_call.cpp#l2631),which may inhibit some barrier optimizations. The workload is also tied up in a very unlucky volatile-predicated loop that prevents barrier hoisting.</u></span><pre style="white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: Times, Times New Roman, serif;"><u>Pending codegeneration improvements alleviate barrier costs even when they are not optimized."</u></span></pre>
</li>
</ul>
</div>
<span style="font-family: inherit;">Since I fully expect all the crusaders to get hot and bothered about the above I thought I'd throw in...</span><br />
<h3>
<span style="font-family: inherit;"><br /></span></h3>
<h3>
<span style="font-family: inherit;">A Holy Graal!</span></h3>
<div>
<span style="font-family: inherit;">Graal is the next gen compiler for HotSpot. Coming out of Oracle Labs and already running in Twitter production, I thought we should add another compiler dimension to this mix. To run Graal you can use: </span><span style="font-family: "courier new" , "courier" , monospace;">"<span style="background-color: white;">-XX:+UnlockExperimentalVMOptions -XX:+</span><wbr style="background-color: white;"></wbr><span style="background-color: white;">EnableJVMCI -XX:+UseJVMCICompiler"</span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: inherit;">Comes included in the Java 9 package!!! So how does it do?</span></span></div>
<div>
<span style="background-color: white;"></span><br />
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace;">Oracle9.0.1 (-XX:+UseParallelGC -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI -XX:+UseJVMCICompiler)</span></span></div>
<span style="background-color: white;">
</span>
<br />
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace;">offersFailed | ≈ 0 ops/us</span></span></div>
<span style="background-color: white;">
</span>
<div>
<span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace;">pollsFailed | 0.347 ± 0.441 ops/us</span></span></div>
<span style="background-color: white;">
<div>
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade | 51.657 ± 3.568 ops/us</span></div>
<div style="font-family: inherit;">
<br /></div>
<div>
<span style="font-family: inherit;">WHHHHHHHHYYYYYYYYYYYY!!!!!!!!</span></div>
<div>
<span style="font-family: inherit;"><script src="https://gist.github.com/nitsanw/db29b0193ac92136e0ce01e727bc6c1a.js"></script></span></div>
<div>
<span style="font-family: inherit;">Why can't we have nice things? Well... It turns out the good folks at Oracle Labs have not yet implemented putOrdered as nicely as C2 and Falcon has, and have thus competely buggered up my SPSC queue :(</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">Summary: There Are Many JVMs In My Father's House</span></h3>
</div>
</span></div>
Variety is the spice of life as they say. In the narrow narrow usecase presented above we saw different JVMs/GCs/JITs throwing up all different behaviours. Some good, some less so. For this workload there's very little happening, no GC, no exciting vectorization opportunities, it's very limited. But, being limited has the benefit of simplicity and the opportunity to contrast.<br />
Also note that by profiling and refining this code on C2 I have perhaps overfitted it to one compiler at the expense of others, I certainly had more opportunity to eliminate any issues for the C2+ParallelGC scenario....<br />
<br />
I hope you enjoyed to tour, I encourage you to take these new friends home and play with them ;-).</div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com12tag:blogger.com,1999:blog-5171098727364395242.post-16412729563539753232017-02-14T17:14:00.001+00:002017-02-14T17:14:59.065+00:00Java Flame Graphs Introduction: Fire For Everyone!<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZBUQecR1L1vNVpko_1rao3eynN3zVSvAs-1QGYK-wxjRGP_jWDesLrhvby6FKZFusuKm7jmQhLaOYGWMJeUirblmZiAl-Nev_1H7ByvlGwLpCCSif_bDGUkjCNE9ySez-k7y1SEv-SbE/s1600/fire-heart-961194_1920.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZBUQecR1L1vNVpko_1rao3eynN3zVSvAs-1QGYK-wxjRGP_jWDesLrhvby6FKZFusuKm7jmQhLaOYGWMJeUirblmZiAl-Nev_1H7ByvlGwLpCCSif_bDGUkjCNE9ySez-k7y1SEv-SbE/s320/fire-heart-961194_1920.jpg" width="320" /></a><a href="https://github.com/brendangregg/FlameGraph" target="_blank">FlameGraphs</a> are superawesome. If you've never heard of FlameGraphs and want to dive straight in the deep end, you should run off and check out the many many good resources provided by <a href="https://twitter.com/brendangregg" target="_blank">Brendan Greg</a> in his <a href="http://www.brendangregg.com/flamegraphs.html" target="_blank">one stop shop page here</a>. This post will give a quick intro and some samples to get you started with collecting profiles for all JVMs everywhere. I'm taking a slightly different tack then Brendan in presenting the topic, so if it turns out my explanations suck you should see if his make more sense.<br />
<br />
<h3>
What's the big deal?</h3>
<div>
If you've ever used a profiler to look at your code you will have seen 2 profile reports typically:</div>
<div>
<ol>
<li>Flat profile: This is often presented as the "top X" methods/classes/packages where <i>time </i>(or samples, or ticks or whatever) is spent. This is useful as it immediately shows up common bottlenecks across your code, but these are shown out of context. Sometimes this is enough, but often in larger application profiles context is significant. This representation is very useful when a method with a high overall impact is called from many callsites, making each callsite cheap but the method itself significant.</li>
<li>Tree profile: This profile will present you with a call tree where each method is a node with a total and self <i>time</i> quantity. The self measure implies the amount of time spent in the method itself(the amout of samples in which the method is the leaf), and total is for the total number of samples in which it shows up (leaf and node).</li>
</ol>
The problem with the tree view is that it is very unpleasant to navigate. Click click click click and as the stack deepens it becomes harder to look at and ingest. Enter FlameGraphs.</div>
<div>
FlameGraph represents a tree profile in a single interactive SVG where:</div>
<blockquote class="tr_bq">
<span style="background-color: white; text-align: justify;"><span style="font-family: "courier new" , "courier" , monospace;"><b>The x-axis shows the stack profile population, sorted alphabetically (it is not the passage of time), and the y-axis shows stack depth. Each rectangle represents a stack frame. The wider a frame is is, the more often it was present in the stacks. The top edge shows what is on-CPU, and beneath it is its ancestry.</b></span></span></blockquote>
Like most visualisations, it makes sense when you see it rather than explain it. Let start with data sets we can easily grasp and see what they look like.<br />
<br />
<h3>
Synthetic Samples For Starters</h3>
For instance, what does a single stack sample look like? The FlameGraphs SVG generating script takes as it's input a "collapsed stacks" file which has a dead simple format, frames separated by semi-colons followed by the number of times this stack was sampled. Here's a dummy handwritten example of a single sample file (call it sample.cstk):<br />
<span style="background-color: #cccccc;"><span style="font-family: "courier new" , "courier" , monospace;">main;0;1;2;3;4;5;6;7;8;9;10 1</span></span><br />
<br />
We can feed this to the flames (now is a good time to clone this repo and try shit out):<br />
<span style="background-color: #eeeeee; font-family: "courier new" , "courier" , monospace;">flamegraph.pl single-stack.cstk > single-stack.svg</span><br />
<br />
Here's what a single stack trace looks like:<br />
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBTFFGMEp6UllGN28" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object>
<br />
<br />
But a single stack trace is just one data point, not a profile. What if we had 1M samples of this same stack?<br />
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBbTNEX1R3eTJ0VG8" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object>
<br />
<br />
Well.. it would look pretty much the same, but if you hover over it will tell you it got 1M samples. It looks the same because we still have 100% the same stack for the whole profile. It's the same profile.<br />
"<b>BUT!</b>" I hear you say, "But, colours?". Yes the colors mean nothing at this point, but will become interesting later. The default colour palate is red and the choice of colors is random, hence the different colour selection changes from run to run. Just forget colors for a second, OK?<br />
Right, next we want to look at a set of samples with a few more stacks:<br />
<span style="background-color: #cccccc; font-family: "courier new" , "courier" , monospace;">
main;0;1;2;3;4;5 1<br />
main;0;1;2;3;4;5;6 2<br />
main;0;1;2;3;4;5;6;7 3<br />
main;0;1;2;3;4;5;6;7;8 4<br />
main;0;1;2;3;4;5;6;7;8;9 5</span><br />
<br />
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBYnIzRWVrUmJ6eE0" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object>
<br />
<br />
Now you can also get a feel for what clicking around does and how you zoom in and out.<br />
By now I hope you get the picture for how a bunch of stacks and frequencies look with a simple data sets. Last synthtic example to look at has several root frames and a little more varied stacks. Lets try this:<br />
<span style="background-color: #cccccc; font-family: "courier new" , "courier" , monospace;">
main;a;1;2;3;4;5 1<br />main;c;1;2;3;4;5;6;7 4<br />main;a;1;2;3;4;5;6 2<br />main;c;1;2;3;4;5;6 4<br />
main;c;1;2;3;4;5;6;8 4<br />
main;b;1;2;3;4;5;6;7 3<br />
main;b;1;2;3;4;5;6;8 3<br />
main;b;1;2;3;4;5;6;9 3<br />
main;d;1;2;3;4;5;6;7;8;9 5</span><br />
<br />
And we get this:<br />
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBaHhGQXpaTmJFWk0" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object>
<br />
<div>
<span style="background-color: #cccccc; font-family: "courier new" , "courier" , monospace;"><br /></span></div>
We see here that stacks are sorted alphabetically and ordered from left to right. The ordering has nothing to do with the order in the file. The collapsed stacks format is itself an aggregation with no view on timing. So the order from left to right is only about merging, not time or anything else. We can see that stacks which share a common parent naturally aggregate under that parent. The width of each frame is it's relative <i>total-time</i> share. It's <i>self-time</i> share is it's top exposure, or how much of it is not covered by it's callees, the frames on top of it.<br />
<br />
<h3>
Tree View vs Flames</h3>
Now that we got the hang of this flamy thing, lets take a look at the same profile using 2 presentations. The venerated tree-view and this new hipsterish whatever flame thing. The following is a profile collected using <a href="https://github.com/RichardWarburton/honest-profiler" target="_blank">honest-profiler</a> for a <a href="https://github.com/netty/netty" target="_blank">netty</a> benchmark:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlxf9fn2rFjnpSQah87KjQGcGCRMz9e7ZmAUtwZN1uU8M3Jizb7fQWu5pEmUEpkCkAZ15Se7qGj6yFzAmDY0hvbha0IMTenV4pAw8gnG3vLqEOynwOwfSNum2-gnXEjeI-cPJEzCtHLDs/s1600/Screenshot+from+2017-02-10+17-36-33.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlxf9fn2rFjnpSQah87KjQGcGCRMz9e7ZmAUtwZN1uU8M3Jizb7fQWu5pEmUEpkCkAZ15Se7qGj6yFzAmDY0hvbha0IMTenV4pAw8gnG3vLqEOynwOwfSNum2-gnXEjeI-cPJEzCtHLDs/s640/Screenshot+from+2017-02-10+17-36-33.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
In typical workflow I step further and further into the hot stack, but this pushes out the big picture out of my view. I would now have to go back up and fold it to see what hides under other hot branches in the tree. It's a familiar and annoying experience if you've ever used a profiler with this kind of view. The problem is that Java class and method names are typically long, and stacks are quite deep. This is a simple application and I quickly run out of room.<br />
Here's the FlameGraph for the same profile (I chose green, because later it makes sense):<br />
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBaTh6XzJtN19xWHM" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object>
<br />
<br />
<b><i>NOTE: I made all the flame graphs in this post narrow so they fit the layout. They don't have to be this narrow. You can set the width to whatever you like, I used "--width=700" for the graphs in this post.</i></b><br />
We can see the root frames quickly break out to main activities, with the deep netty stack now visible upfront. We can click and zoom easily. I also find the search ability which colors matching strings useful to highlight class/package participation in the profile. Prominent flat-tops indicate hot leaf methods we might want to look at.<br />
It's perhaps a matter of taste, but I love it. I've been using flame graphs for a while and they are pure genius IMO. I find the image itself is intuitively more approachable, and with the ability to quickly zoom in/out and search I can usually quickly work through a profile without losing sight of the full picture.<br />
So how do you get one?<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJbSIVGsSzvQ6XyhSbLOW92-9XGZgU_tP77ps-Jzk8mHUB2IsPR2Q9T8nuE9P-cBGWwjcfyuXpS8FmaxCbkKyKh7mMpIlOUtzAJ87br_ThpU2k1fuEAX5Jl2QzgbYJB_etTFKzOKV4bcw/s1600/bob-wont-hurt-you.gif" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="239" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJbSIVGsSzvQ6XyhSbLOW92-9XGZgU_tP77ps-Jzk8mHUB2IsPR2Q9T8nuE9P-cBGWwjcfyuXpS8FmaxCbkKyKh7mMpIlOUtzAJ87br_ThpU2k1fuEAX5Jl2QzgbYJB_etTFKzOKV4bcw/s320/bob-wont-hurt-you.gif" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">It's Bob! yay?</td></tr>
</tbody></table>
<h3>
Everybody Gets A FlameGraph!</h3>
<div>
Yes, even you poor suckers running JDK 1.3 on Windows XP! I don't <a href="http://psy-lob-saw.blogspot.co.za/2016/02/why-most-sampling-java-profilers-are.html" target="_blank">recommend this method of profiling</a> if you have a more recent JVM, or if your JVM supports AsyncGetCallTrace, but if your deployment is stuck in the past you can still be in on this. This is because ALL JVMs must support JVMTI and AFAIK allow you to hit them with jstack/JVisualVM/hprof. It's a terrible way to profile, there's allot of overhead, and usually you can find a better way, but this is universally available. Collecting a sample via jstack is (a terrible idea) quite easy. Just find the <i>pid</i> of the process you want to profile using <i style="font-weight: bold;">jps </i>and then do something like:<br />
<span style="font-family: "courier new" , "courier" , monospace;">for i in {1..100}; do</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> jstack <pid> >> iloveoldshit.jstk;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> sleep 0.1;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">done</span></div>
<div>
And Bob is your relative (which is a good thing apparently).</div>
<div>
Once you've collected a large enough sample for your application you can go on and feed flame graphs:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">cat </span><span style="font-family: "courier new" , "courier" , monospace;">iloveoldshit</span><span style="font-family: "courier new" , "courier" , monospace;">.jstk | ./stackcollapse-jstack.pl | ./flamegraph.pl --color=green > jstack-flames.svg</span></div>
<div>
And you get:</div>
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBX1l4aG5OY3BaUG8" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object>
<br />
<div>
<br /></div>
<div>
This is the same benchmark from before, but different profile with the safepoint bias. You can compare the two by scrolling up and down. OR you can use FlameGraphs to diff the 2, in a moment.<br />
FlameGraphs supports converting <b>jstack </b>output into collapsed stacks (as above). Efforts exist on GitHub to convert the <b>hprof </b>format which JVisualVM produces (as well as other profilers) into collapsed stack format.<br />
So for all my poor readers, struggling under the oppression of JVMs for which better means to spill out stack samples do not exist, I got your backs, you too can be looking at flame graphs!<br />
But seriously, just move on.<br />
<br />
<h3>
Level Up: AsyncGetCallTrace or JFR</h3>
</div>
<h3>
</h3>
Now, if you want a better profiler, which does not result in bringing your application to safepoint and pausing ALL your threads at each sample AND your are either running a 1.6 or higher JDK (OpenJDK/Oracle/recent Zing) on Linux/Mac/BSD you can use <a href="https://github.com/RichardWarburton/honest-profiler" target="_blank">Honest-Profiler</a> to collect your profile. If you got Oracle JDK 1.7u40 or higher you can use Java Flight Recorder (if you use it in production you need to pay for the licence). These <a href="http://psy-lob-saw.blogspot.co.za/2016/06/the-pros-and-cons-of-agct.html" target="_blank">profilers rely on AsyncGetCallTrace</a> to record the Java stack safely on interrupt(from a signal handler, not at safepoint).<br />
To collect with Honest-Profiler I start my JVM with the following parameter:<br />
<span style="font-family: "courier new" , "courier" , monospace;">-agentpath:$HONEST_PROFILER_HOME/liblagent.so=host=localhost,port=4242,logPath=$PWD/netty.hpl</span><br />
Then, when I feel the time is right, I can start and stop the profile collection<br />
<span style="font-family: "courier new" , "courier" , monospace;">echo start | nc localhost 4242</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">echo stop | nc localhost 4242</span><br />
<span style="font-family: inherit;">To convert the binary format into collapsed stacks I need to use a helper class in the honest-profiler.jar:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">java -cp $HONEST_PROFILER_HOME/honest-profiler.jar com.insightfullogic.honest_profiler.ports.console.FlameGraphDumperApplication netty.hpl netty.cstk</span><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZvIyiln8DgC7XvB5-rLbTwiMeklsWR8bVSSrRL4d1_XRkQcTPS1eAe4qnh2kV1yr_ospTQ8oHHKp_ElQKsRNM38cmrqQKpas5ioCzxq2w6xSnuj-p7aKzLujKUd0DDCSRta2SA5V69xI/s1600/larry.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="113" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZvIyiln8DgC7XvB5-rLbTwiMeklsWR8bVSSrRL4d1_XRkQcTPS1eAe4qnh2kV1yr_ospTQ8oHHKp_ElQKsRNM38cmrqQKpas5ioCzxq2w6xSnuj-p7aKzLujKUd0DDCSRta2SA5V69xI/s200/larry.jpg" width="200" /></a><span style="font-family: inherit;">I can then feed the flamegraph script the collapsed stacks file and get the result which we've already seen.</span><br />
<span style="font-family: inherit;">To <a href="http://isuru-perera.blogspot.co.za/2015/09/java-mixed-mode-flame-graphs.html" target="_blank">convert JFR recordings to flame graphs see this post</a>. But remember children, you must pay Oracle if you use it in production, or Uncle Larry might come for you.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;">Bonus: Diff Profiles!</span></h3>
<div>
A nice benefit of having 2 different profilers produce (via some massaging) a unified format for flame graphs is that we can now diff profiles from 2 profilers. Not something that is generally useful, granted. But diffing profiles is an established capability in many profilers, and is usually used to do a before/after comparison. Flame Graphs support this via the same visualization. Once you have converted your profiles into the collapsed stacks format you can produce a diff file and graph it:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">./difffolded.pl -n A.cstk B.cstk | ./flamegraph.pl > A-B-diff.svg</span></div>
<div>
Diffing the Honest-Profiler and jstack profiles gives us the following:</div>
<div>
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBY3JuSHBqU3U4ZEE" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object> </div>
<div>
<br />
The white squares are not interesting, the red/pink squares highlight the <i>delta of self samples as a percentage of total samples </i>(not so intuitive)<i>.</i> I admit it may seem a tad confusing at first, but at least it draws your eyes to the right places. <a href="http://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html" target="_blank">More on differential flame graphs here</a>.<br />
<b>Note</b>:<i> to make this diff work I had to shave off the thread names from the jstack collected collapsed stacks file.</i><br />
<br />
<h3>
Further Bonus: Icicle Graph</h3>
</div>
<div>
Some times the bottleneck is not a particular call stack plateau, but rather a particular method being called from many call sites. This kind of bottleneck will not show well in a flame graph as the different stacks with similar tops will be split and may not stand out. This is really where a flat profile is great, but we can also flip the flame graph view to highlight the top method merging:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">cat netty.cstk | ./flamegraph.pl --reverse --invert --color=green > netty-icicles.svg</span></div>
<div>
<span style="font-family: inherit;">I've filtered the stacks to show only a relevant portion:</span></div>
<div>
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBY2ZyQ3pSU0g5a28" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object><br />
This is not dissimilar to functionality offered by other profiler GUIs which allow the drill down direction to start from hot methods and into their callers in a tree view presentation.<br />
<br />
<h3>
Level UP++: Java Perf Flame Graphs FTW!</h3>
</div>
<div>
If you are so fortunate as to:</div>
<div>
<ol>
<li>Be running OpenJDK/Oracle 1.8u60 or later(this functionality is coming to Zing in a near future release, fingers crossed)</li>
<li>Running on Linux</li>
<li>Got permissions to make this run</li>
</ol>
You can get amazing visibility into your system by using a combination of:</div>
<div>
<ol>
<li>Java with: -XX:+PreserveFramePointer (also recommended -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints so unfolded frame are more accurate)</li>
<li><a href="https://github.com/jrudolph/perf-map-agent/" target="_blank">perf-map-agent</a>: This attachable agent dumps a symbol mapping file for all runtime generated code in the JVM, enabling perf to correctly resolve addresses to methods. You'll need to clone and build.</li>
<li>perf: You'll need permissions and you'll need to install it. I assume you are root of you own machine for simplicity.</li>
</ol>
With the above working you can collect a perf profile of your Java process(by itself or as part of whole system). This results in a <i>perf.data</i> file and a <i>perf-<pid>.map</i> file in your /tmp folder. You can then proceed to generate a collapsed stack profile from that file, the simplest way to get this going is by using a script packed with <i>perf-map-agent</i>:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">perf-java-flames <pid></span></div>
<div>
<span style="font-family: inherit;">This will ask for password as it needs to sudo a few things. Be the sudo you want to see in the world. After a suspenseful wait of 15 seconds you'll get this:</span><br />
<div>
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBU2FWRWhRWFNPN2c" type="image/svg+xml">
Please Use modern Browser(e.g. recent chrome?) to see this SVG!
</object>
</div>
<i style="font-family: inherit;">Note: Netty deploys it's native lib into tmp, loads it and deletes it, which means perf gets lost looking for it. I deleted it from the benchmarks jar and loaded it directly using LD_LIBRARY_PATH to resole this visibility issue. It doesn't make a huge difference, but in case you try this out.</i><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The green frames are Java, and the rest are all the magic which happens to help Java along. Here's what happened:</span><br />
<br />
<ul>
<li>Red frames are C library or Kernel code. We can see that the socket writes in the original profile actually go into native land. We now have further visibility down the hole. Importantly this illustrates where hot Java methods are in fact not Java methods at all and so looking at their Java code for optimisation ops is futile.</li>
<li>Yellow frames are C++. We can see the call chain leading into the interpreter and ultimately into compiled Java code.</li>
<li>Green frames are Java. BUT if you compare the previously presented profiles will this one you will notice there are some intermediate frames missing here. This is because the frames in this profile are "real" frames, or rather they map to stack frame. Inlined methods in Java do not have their own stack frames, so we can't see them (for now, we'll sort this out in a second). Further more, the keen observer will notice the familiar "Thread.run" bottom of the stack is missing, replaced by the "interpreter". As is often the case, the run method did not get compiled in this benchmark so it is not a proper compiled method for which we have a mapping. Methods in the interpreter are opaque in this profile.</li>
<li>Some stacks are broken, which can be confusing. In the example above we can see the 13.8 unknown chunk which leads to some JVM activities, but also to some Java code. More on that later.</li>
</ul>
So, it would seem that we have gained something in visibility into the native/OS/JVM CPU utilization, but lost allot of information we had in the Java side. When is this still useful:<br />
<br />
<ul>
<li>This profile is super useful if you are writing Netty and trying to workout which system calls you end up with from your JNI code, or where time is spent in that code (netty implements it's own native epoll selector, very cool). If you are writing an application which utilizes JNI libraries this profile will give you visibility across the divide. The alternative here would be to use 2 profilers and try and correlate them. Solaris Studio also offers some help here, I will one day write a post on Solaris Studio.</li>
<li>This in not a good example of a profile dominated by JVM threads, but in many profiles the GC activity will show up. This is very useful, as GC and compiler CPU utilization can get in the way of application threads using the available CPU. A Java only profiler leaves you to correlate GC/compilation logs and application profile to figure out who ate the pie. It's also an interesting view into which part of the GC is to blame.</li>
<li>Some JVM <a href="https://en.wikipedia.org/wiki/Intrinsic_function" target="_blank">intrinsics</a> are confusing to AsyncGetCallTrace, and invisible to safepoint profilers. The biggest culprit I see is array copy. Array copies will show up as failed samples on AGCT profilers (unless, like JMC they just fail to tell you about failed samples all together). They show up in this profile (search above for <i>arraycopy</i>), but only a tiny bit.</li>
<li>This profile can be collected system wide, allowing you to present a much wider picture and expose machine wide issues. This is important when you are looking at machine level analysis of your application to improve configuration/setup.</li>
<li>In depth view of OS calls can inform your configuration.</li>
</ul>
<blockquote class="tr_bq">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIHggJt_981-ZaXR4fjQQNbVTxLGfJahGFE37oaKGyhQWxOw3w8cohM1ruIeD57QHL9BzBR4upIvXq88MfkYmKY9XdPtutonxEEaaEdqPMPKME4w_k9Pw2-eevYhHcEMoQrH1iccWkpgo/s1600/cat-in-hat.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIHggJt_981-ZaXR4fjQQNbVTxLGfJahGFE37oaKGyhQWxOw3w8cohM1ruIeD57QHL9BzBR4upIvXq88MfkYmKY9XdPtutonxEEaaEdqPMPKME4w_k9Pw2-eevYhHcEMoQrH1iccWkpgo/s320/cat-in-hat.jpg" width="224" /></a><br />
<blockquote class="tr_bq" style="text-align: center;">
<i>'look at me! </i><i>look at me now!' said the cat.</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>'with a cup and a cake </i><i>on the top of my hat!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>I can hold up TWO books!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>I can hold up the fish!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>and a little toy ship!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>and some milk on a dish!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>and look!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>I can hop up and down on the ball!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>but that is not all!</i></blockquote>
<blockquote class="tr_bq" style="text-align: center;">
<i>oh, no. T</i><i>hat is not all...</i></blockquote>
</blockquote>
<br />
<br />
<h3>
Bonus: Inlined Frames! Threads! COLOR!</h3>
We can win back the inlined frames information by asking perf-map-agent to create a more detailed map file with inlining data. This leads to larger map files, but should be worth it.<br />
You can further tweak the command line to color kernel frames differently and control sample duration and frequency. And while we're a-tweakin' lets also have threads info.<br />
Here's what you run:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">PERF_COLLAPSE_OPTS="--kernel --tid" PERF_RECORD_FREQ=99 PERF_RECORD_SECONDS=10 PERF_MAP_OPTIONS=unfoldall perf-java-flames <pid></span></div>
<div>
And the graph now looks like this:<br />
<div>
<object data="https://drive.google.com/uc?export=view&id=0B4jDFCGuuSqBU04yZkU1cF9ZR2s" type="image/svg+xml">
Please Use modern Browser (e.g. recent chrome?) to see this SVG!
</object>
</div>
<br />
<br />
<ul>
<li>The Java frames are now green and aqua. Aqua frames are inlined frames and green are "real". This information is not presented at all by most profilers, and is pretty inaccessible in others. Here we can instantly see some interesting inlining challenges in the tall towers of same sized frames. The compiler inlines through many, but maybe eventually gives up, maybe there's something to be won by simplifying the abstraction here?</li>
<li>Thread id is added as a base frame. This is helpful in this particular example because there are only 2 interesting threads and I very much want to see this split. It also helps bring back some broken stacks into the fold. Now I can tell these frames belong to the Netty epoll thread. Yay.</li>
<li>Orange frames are kernel frames.</li>
<li>Having the thread id highlights that the none Java frames on the left are from a variety of threads. If we had more GC/Compiler work happening this may become interesting.</li>
<li>Profiling a large application with thread pools this separation by thread may not be what you want... but sometimes it is very helpful, like above. In this benchmark I have a thread generating load and a thread I want to profile, so telling them apart works great. At the moment there's no mapping of threads to thread names, but in future we may be able to easily group thread pools for more meaningful views.</li>
</ul>
<h3>
Bonus: Hack It All Up</h3>
<div>
There's very little code in perf-map-agent, and the scripts would take you 5 minutes to read through. You don't have to use the scripts, you can write your own. You can add or enhance features, it's easy to participate or customize. Dig in, have fun :-)</div>
<div>
The FlameGraph scripts are nice and tidy, and the pipeline separation of [profile -> collapsed stacks -> graph] means you can read through them and tweak as you like the bits you care about without caring too much about the rest. While working on this post I played with the diff presentation a bit. It was my first ever interaction with Perl, and I'm not so very bright, and I managed to get what I wanted. Surely someone as fine as yourself can do better.</div>
<div>
If you look at Brenden's updates page you'll see many many people are jumping in and tweaking and sharing and making funky things. Go for it!</div>
<div>
<br /></div>
<h3>
Summary And Credits</h3>
<div>
So you can have flame graphs, all of you. And you can feed these with inputs from several sources, each with their own set of pros and cons. It makes a great tool in your tool box and may give you that extra perspective you are missing in your profiling.</div>
<div>
There's a few people who deserve mention in the context of the tools above, look them up, they all helped make it happen:</div>
<div>
<ul>
<li><a href="http://brendangregg.com/" target="_blank">Brendan Gregg</a>: FlameGraphs proud daddy. Brendan has written allot about FlameGraphs, work through his posts and you'll learn plenty.</li>
<li><a href="https://github.com/jrudolph" target="_blank">Johannes Rudolph</a>: Author and maintainer of perf-map-agent.</li>
<li><a href="https://github.com/tjake" target="_blank">Jake Lucianni</a>: Contributed flamegraphs support for inlined frames.</li>
<li><a href="https://github.com/RichardWarburton" target="_blank">Richard Warburton</a>: Author and maintainer of honest-profiler.</li>
</ul>
Thanks for reading :-) next up some JVM profile analysis</div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com2tag:blogger.com,1999:blog-5171098727364395242.post-39603964351224405042016-12-20T08:33:00.001+00:002016-12-20T20:10:04.806+00:00What do Atomic*::lazySet/Atomic*FieldUpdater::lazySet/Unsafe::putOrdered* actually mean?<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1hqLa0xDE0oy0GV-ibJfsNuM_NHXtmu8pdV_jVPtF7OU4kBxyPC2Z1ixkC9tOlVI5qk-o56Oh1t9ixx-uU18RJJfZ5w5sJDxhyphenhyphen8JvLKDAZInqhTk85M4k9q2BwVX9otbay_vB7efNaW4/s1600/ShelbyBlondell-RoadToHell.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1hqLa0xDE0oy0GV-ibJfsNuM_NHXtmu8pdV_jVPtF7OU4kBxyPC2Z1ixkC9tOlVI5qk-o56Oh1t9ixx-uU18RJJfZ5w5sJDxhyphenhyphen8JvLKDAZInqhTk85M4k9q2BwVX9otbay_vB7efNaW4/s200/ShelbyBlondell-RoadToHell.jpeg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Paved with well intended definitions it is.</td></tr>
</tbody></table>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">lazySet/putOrdered (or an <b>ordered store</b>) was added as a bit of a rushed/non-commital afterthought after the JMM was done, so it's description is subject to many debates on the mailing lists, stack overflow and watercoolers the world over. This post merely tries to provide a clear definition with references to relevant/reputable sources.</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">
</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Definition:</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>An ordered store is a <u>weakened volatile store[2][5]</u>. It prevents preceding stores and loads from being reordered with the store[1][3][6], but does not prevent subsequent stores and loads from being reordered with it[2][4].</b></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If there was a JMM cookbook entry for </span><b style="font-family: 'helvetica neue', arial, helvetica, sans-serif;">ordered store</b><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> defining it with barriers in mind it would seem that the consensus is that </span><b style="font-family: 'helvetica neue', arial, helvetica, sans-serif;">ordered stores</b><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"> are preceded by a StoreStore AND a LoadStore barrier[4][6].</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><b style="font-family: 'helvetica neue', arial, helvetica, sans-serif;">Ordered store </b><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">is practically the same as a C++ memory_release_store[5][7].</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">
</span></pre>
<div style="text-align: right;">
</div>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">References:</span><span style="font-family: inherit;"><span style="font-family: inherit;">
</span><b><span style="font-family: "courier new" , "courier" , monospace;">[1] Original bug:</span></b></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: inherit;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: inherit;">"</span><span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">lazySet provides a preceding store-store barrier (which is either a no-op or very cheap on current platforms), but no store-load barrier</span><span style="font-family: inherit;">"</span></span><span style="font-family: inherit;">
</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">See here: <a href="http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6275329" style="color: #663366;">http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6275329</a>
</span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; white-space: pre-wrap;">
</span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: "courier new" , "courier" , monospace; white-space: pre-wrap;"><b>[2] java.util.concurrent docs</b>:</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: inherit;"><span style="white-space: pre-wrap;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: inherit;">"</span><span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">lazySet has the memory effects of writing (assigning) a volatile variable except that it permits reorderings with subsequent (but not previous) memory actions that do not themselves impose reordering constraints with ordinary non-volatile writes.</span><span style="font-family: inherit;">"</span></span><span style="font-family: inherit;">
</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">See here: </span></span><span style="color: #663366; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><a href="https://docs.oracle.com/javase/8/docs/api/?java/util/concurrent/package-summary.html" style="color: #663366; white-space: pre-wrap;">https://docs.oracle.com/javase/8/docs/api/?java/util/concurrent/package-summary.html</a></span><span style="font-family: inherit; white-space: pre-wrap;"><span style="font-family: inherit;">
</span></span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: inherit;">
</span></pre>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJtrblVmAjzkvanbFYcF0QnCw0Jvoe-OCgdkrvYA_RRD6qdsmmCfsWZQijo7Tg0adCWMO-vGvGIibOWb1L6rqRSCUwvPvI8lHI2ydxO1TUykBj-CfGBdCStvfK3Xzq-PpAhofx11fzU0w/s1600/cookbook.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="212" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJtrblVmAjzkvanbFYcF0QnCw0Jvoe-OCgdkrvYA_RRD6qdsmmCfsWZQijo7Tg0adCWMO-vGvGIibOWb1L6rqRSCUwvPvI8lHI2ydxO1TUykBj-CfGBdCStvfK3Xzq-PpAhofx11fzU0w/s320/cookbook.jpg" width="320" /></a></div>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: inherit;"><span style="white-space: pre-wrap;"><b><span style="font-family: "courier new" , "courier" , monospace;">[3] JMM cookbook:</span></b><span style="font-family: inherit;"> </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Defining barriers meaning here: </span></span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; white-space: pre-wrap;"><a href="http://g.oswego.edu/dl/jmm/cookbook.html">http://g.oswego.edu/dl/jmm/cookbook.html</a></span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><b style="white-space: pre-wrap;"><span style="font-family: inherit;">
</span></b></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;"><b style="white-space: pre-wrap;">[4] concurrency-interest Q&A with Doug Lea, October 2011</b><span style="white-space: pre-wrap;">:</span><span style="white-space: pre-wrap;"><span style="font-family: inherit;">
</span></span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: "courier new" , "courier" , monospace; white-space: pre-wrap;"><span style="font-family: inherit;">"[Ruslan]:</span><span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">... If it happens (== we see spin-wait loop finished) -- does it mean,that all writes preceding lazySet are also done, committed, and visible to thread 2, which finished spin-wait loop?
</span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: inherit;"><span style="white-space: pre-wrap;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">[Doug]: Yes, although technically, you cannot show this by reference to the Synchronization Order in the current JLS.
...
lazySet basically has the properties of a TSO store</span><span style="font-family: inherit;">"</span></span><span style="font-family: inherit;">
</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">See here: </span></span><a href="http://cs.oswego.edu/pipermail/concurrency-interest/2011-October/008296.html" style="color: #663366; white-space: pre-wrap;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">http://cs.oswego.edu/pipermail/concurrency-interest/2011-October/008296.html</span></a><span style="white-space: pre-wrap;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">
</span><b style="font-family: inherit;"><u>
</u></b></span></span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="width: 50em;"><span style="font-family: inherit;"><span style="white-space: pre-wrap;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">The discussion is REALLY worth reading, involving Hans, Doug, Vitaly, Ruslan and other such respectable members of this excellent mailing list. Go on, I'll wait.
The discussion on that thread concludes the following is required:</span><span style="font-family: inherit;">
</span></span><span style="white-space: pre-wrap;">...
<span style="font-family: inherit;">LoadStore + </span>StoreStore
st [Y],X // store X into memory address Y
...
</span><span style="white-space: pre-wrap;">
</span></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Outcome: Stores before and after are now prevented from floating across the barrier. Loads before the barrier are also prevented from floating down. Later loads are free to float up. Note that <b>st</b> may in theory be delayed indefinitely, certainly other loads and stores are allowed to float up between it and the barrier.</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: inherit;">
</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;"><b>[5] concurrency-interest Q&A with Aleksey Shipilev, May 2016</b>:</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">"</span>putOrdered is a release in disguise, most of the C++11 std::atomic(...,</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;">mem_order_release) reasoning applies here."</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">"</span>acquire/release are the relaxations from the usual volatile</span></pre>
<pre style="white-space: pre-wrap;"><span style="font-family: "courier new" , "courier" , monospace;">rules -- while producing happens-before-s, they drop from total
</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;">synchronization order, thus breaking sequential consistency."</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: inherit;">
</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">And adds some fine examples:</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;">"Safe publication still works:</span></pre>
<pre style="white-space: pre-wrap;"><span style="font-family: "courier new" , "courier" , monospace;">
int x; volatile int y;
-----------------------------------------------------------------------
put(x, 1); | r1 = get{Acquire|Volatile}(y);
put{Release|Volatile}(y, 2); | r2 = get(x);
(r1, r2) = (2, 0) is forbidden.
But anything trickier that requires sequential consistency fails. IRIW
fails, because no consistent write order observed by all threads. Dekker
fails, because release stores followed by loads may or may not be
visible in program order:
volatile int x; volatile int y;
-----------------------------------------------------------------------
putRelease(x, 1); | putRelease(y, 1);
r1 = getAcquire(y); | r2 = getAcquire(x);
(r1, r2) = (0, 0) is allowed. Forbidden if all ops are volatile.
Safe construction still <b>does not work</b> (even for volatiles!):
A global;
-----------------------------------------------------------------------
A a = <alloc>; | A a = global;
put{Release|Volatile}(a.x, 1); | r1 = get{Acquire|Volatile}(a.x);
global = a; |
(r1) = (0) is allowed."</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">See here: <a href="http://cs.oswego.edu/pipermail/concurrency-interest/2016-May/015104.html">http://cs.oswego.edu/pipermail/concurrency-interest/2016-May/015104.html</a></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">
</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><b><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">[6] </span>concurrency-interest Q&A with Aleksey Shipilev, March 2016:</span></b></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">"</span><span style="font-family: "courier new" , "courier" , monospace;">><i> int a, b;</i></span></pre>
<pre style="white-space: pre-wrap;"><span style="font-family: "courier new" , "courier" , monospace;">><i>
</i>><i> boolean tryLock() {
</i>><i> UNSAFE.putOrdered(a, 1); // Declare intent.
</i>><i>
</i>><i> // No StoreLoad here as store is not volatile.
</i>><i>
</i>><i> if (UNSAFE.getVolatile(b) == 1)) {
</i>><i> // Reset intent and return false;
</i>><i> }
</i>><i>
</i>><i> return true;
</i>><i> }
</i>
Even in the naive barrier interpretation that usually gives stronger
answers, you have:
[LoadStore|StoreStore]
a = 1;
r1 = b;
</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;"> [LoadLoad|LoadStore]</span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">"</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">See here: <a href="http://cs.oswego.edu/pipermail/concurrency-interest/2016-March/015037.html">http://cs.oswego.edu/pipermail/concurrency-interest/2016-March/015037.html</a></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;">
</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><b><span style="font-family: "courier new" , "courier" , monospace;">[7] C++ memory_order_release definition:</span></b></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "courier new" , "courier" , monospace;">"<span style="background-color: white; line-height: 14.08px;">A store operation with this memory order performs the </span><i style="background-color: white; line-height: 14.08px;">release operation</i><span style="background-color: white; line-height: 14.08px;">: no reads or writes in the current thread can be reordered after this store. All writes in the current thread are visible in other threads that acquire the same atomic variable (see </span><a href="http://en.cppreference.com/w/cpp/atomic/memory_order#Release-Acquire_ordering" style="background: none rgb(255, 255, 255); color: #0b0080; line-height: 14.08px; text-decoration: none;">Release-Acquire ordering</a><span style="background-color: white; line-height: 14.08px;"> below) and writes that carry a dependency into the atomic variable become visible in other threads that consume the same atomic (see </span><a href="http://en.cppreference.com/w/cpp/atomic/memory_order#Release-Consume_ordering" style="background: none rgb(255, 255, 255); color: #0b0080; line-height: 14.08px; text-decoration: none;">Release-Consume ordering</a><span style="background-color: white; line-height: 14.08px;"> below).</span>"</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">See here: <a href="http://en.cppreference.com/w/cpp/atomic/memory_order">http://en.cppreference.com/w/cpp/atomic/memory_order</a></span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Many thanks to <a href="https://twitter.com/shipilev" target="_blank">A. Shipilev</a>, <a href="https://twitter.com/mjpt777" target="_blank">M. Thompson</a>, <a href="https://twitter.com/switchology" target="_blank">D. Lawrie</a> and <a href="https://twitter.com/dj_begemot" target="_blank">C. Ruslan</a> for reviewing, any remaining errors are their own and they shall be most severely reprimanded for them.</span></pre>
<pre class="bz_comment_text" id="comment_text_3" style="white-space: pre-wrap; width: 50em;"></pre>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com9tag:blogger.com,1999:blog-5171098727364395242.post-62480795744545761642016-12-13T09:36:00.001+00:002016-12-13T10:21:35.315+00:00Linked Array Queues, part 2: SPSC Benchmarks<a href="https://github.com/JCTools/JCTools" target="_blank">JCTools</a> has a bunch of <a href="https://github.com/JCTools/JCTools/tree/master/jctools-benchmarks" target="_blank">benchmarks</a> we use to stress test the queues and evaluate optimizations. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdNqXTDmp1OGBN4R4Z6Yjzy4aUt2xaGQV3ujwui7P26mHMlesF4vkVOK7RzHINoDjH_sYkJF2A01pXxvB3OKZaZJJDzQ8_GgnUeNJdHIdD6yVxGslupIMnOb2euOFg2RIW3rvHJvVa89I/s1600/fire+lab3.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdNqXTDmp1OGBN4R4Z6Yjzy4aUt2xaGQV3ujwui7P26mHMlesF4vkVOK7RzHINoDjH_sYkJF2A01pXxvB3OKZaZJJDzQ8_GgnUeNJdHIdD6yVxGslupIMnOb2euOFg2RIW3rvHJvVa89I/s1600/fire+lab3.jpg" /></a></div>
These are of course not 'real' workloads, but serve to highlight imperfections and opportunities. While it is true that an optimization might work in a benchmark but not in the real world, a benchmark can work as a demonstration that there are at least circumstances in which it does work. All measurement is imperfect, but not as imperfect as claims made with no fucking evidence whatsoever, so here goes.<br />
<div>
How do these linked-array queues fare in the benchmarks? what can we learn here?</div>
<div>
The <a href="http://psy-lob-saw.blogspot.com/2016/10/linked-array-queues-part-1-spsc.html" target="_blank">linked array queues</a> are a hybrid of the array and linked queues. So it seems reasonable that we should compare them to both SpscArrayQueue and SpscLinkedQueue. We should also consider how the queues differ and see if we can flush out the differences via the benchmarks.<br />
If you crack under the pressure of boring details, skip to the summary, do not stop at interlude, do not collect a cool drink or get praise, just be on yer fuckin' merry way.<br />
<br />
<h3>
Setup:</h3>
Benchmarks are run on a quiet server class machine:
<br />
<ul>
<li>Xeon processor(Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz): 2 CPUs x 12 cores x 2 threads (HT)</li>
<li>CentOS</li>
<li>Oracle JDK8u101</li>
<li>All benchmarks are run taskset to cores on the same numa node, but such that threads cannot share the same physical core.</li>
<li>Turbo boost is off, the scaling governor is userspace and the frequency is fixed.</li>
<li>The <a href="https://github.com/JCTools/JCTools/tree/master/jctools-core/src/main/java/org/jctools/queues" target="_blank">code is on github</a></li>
</ul>
</div>
<h3>
Throughput benchmark: background and method</h3>
<div>
A throughput benchmark for queues is a tricky fucker. In particular the results change meaning depending on the balance between consumer and producer:</div>
<div>
<ul>
<li>If the consumer is faster than the producer we are measuring empty queue contention (producer/consumer hitting the same cache line for elements in the queue, perhaps sampling each other index). Empty queues are the expected state for responsive applications.</li>
<li>If the producer is faster than the consumer we are measuring full queue contention, which may have similar issues. For some queues which optimize for the healthy assumption that queues are mostly empty this may be a particularly bad place to be.</li>
<li>If the producer and consumer are well balanced we are testing a streaming use case which offers the most opportunities for progress for both consumer and producer. This should yield the best performance, but for most applications <b>may not be a realistic scenario at all</b>.</li>
</ul>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGRiHSD1GqoSX91m6qZV7UVFZilU0RCl8nHZSHoN8R2RuahLwY2VThMU7sHX6DSH1viSFnDyVU_7qiv0YeYtJCsdkLp2jumcyd78A-HR-DkENrt9xanlZ8I-OO8tYLRCKBjKAv0H390p0/s1600/lab+fire.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGRiHSD1GqoSX91m6qZV7UVFZilU0RCl8nHZSHoN8R2RuahLwY2VThMU7sHX6DSH1viSFnDyVU_7qiv0YeYtJCsdkLp2jumcyd78A-HR-DkENrt9xanlZ8I-OO8tYLRCKBjKAv0H390p0/s1600/lab+fire.jpg" /></a></div>
The JCTools throughput benchmark <b>does not resolve these issues</b>. It does however report results which give us an idea of poll/offer failure rates which are in turn indicative of which state we find ourselves in.<br />
A further challenge in managed runtime environments, which is unrelated to queues, is that garbage generating benchmarks will have GC state accumulate across measurement iterations. The implication is that each iteration is measuring from a different starting state. Naturally occurring GCs will leave the heap in varying states depending on the point at which they hit. We can choose to either embrace the noise in the measurement as an averaging of the cost/overhead of garbage or allocate a large enough heap to accommodate a single iteration worth of allocation and force a full GC per iteration, thus resetting the state per iteration. <b>The benchmarks below were run with 8g heap and a GC cycle between iterations.</b><br />
The <a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/jmh/throughput/QueueThroughputBackoffNone.java" target="_blank">benchmark I run here is the no backoff version of the throughput benchmark</a> where failure to offer/poll makes no attempt at waiting/yielding/tapping of foot and just tries again straight away. This serves to maximize contention and is not a recipe for happiness in real applications.<br />
JMH parameters common to all runs below:
<br />
<ul>
<li>-gc true -> GC cycle between iterations</li>
<li>-jvmArgs="-Xmx8g -Xms8g" -> 8g heap</li>
<li>-i 10 -r 1 -> 10 measurement iterations, 1 second each</li>
<li>-wi 20 -w 1 -> 20 warmup iterations, 1 second each</li>
<li>-f 5 -> five forks each to expose run to run variance</li>
</ul>
</div>
<div>
<br /></div>
<div>
<h3>
Throughput benchmark: baseline(JMH params: -bm thrpt -tu us)</h3>
Here's some baseline results, note the unit is ops/us equal to millions of ops per second:<br />
<span style="font-family: "courier new" , "courier" , monospace;">SpscArrayQueue (128k capacity)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed 0.005 ± 0.008 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersMade 252.201 ± 1.649 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed 0.009 ± 0.008 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade 252.129 ± 1.646 ops/us</span></div>
<div>
<br /></div>
<div>
So the SpscArrayQueue is offering great throughput, and seems pretty well balanced with failed offers/polls sort of cancelling out and low compared to the overall throughput.</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SpscLinkedQueue</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">offersFailed ≈ 0 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">offersMade 14.711 ± 5.897 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsFailed 12.624 ± 8.281 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pollsMade 14.710 ± 5.896 ops/us</span></div>
</div>
<div>
<br /></div>
<div>
For the SpscLinkedQueue we have no failed offers, since it's an unbounded queue. We do see a fair amount of failed polls. We expect the polls to be faster than the offers as offering pays for allocation of nodes on each element (24b overhead per element), while the poll simply leaves it to the GC to toss it all away.</div>
<div>
With this baseline we would expect linked arrays queues performance to be somewhere between the 2 data points above. Unlikely to hit the highs of the preallocated array queue, but hopefully much better than a linked queue.<br />
<br /></div>
<div>
<h3>
Throughput benchmark: growable</h3>
<div>
So assuming we let it grow to 128k, how does the SpscGrowableArrayQueue perform in this benchmark and how much does the initial size impact the performance? CNK here is the <b>initial</b> buffer size. The buffer will double in size when offer fills up a buffer until we hit the max size buffer.</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> CNK Score Error Units</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 offersFailed 0.006 ± 0.006 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 offersMade 183.720 ± 0.450 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 pollsFailed 0.003 ± 0.001 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 pollsMade 183.592 ± 0.450 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 offersFailed 0.003 ± 0.006 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 offersMade 184.236 ± 0.336 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 pollsFailed 0.003 ± 0.001 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 pollsMade 184.107 ± 0.336 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K offersFailed 0.001 ± 0.003 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K offersMade 183.113 ± 1.385 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K pollsFailed 0.003 ± 0.001 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K pollsMade 182.985 ± 1.385 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K offersFailed 0.007 ± 0.006 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K offersMade 181.388 ± 5.380 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K pollsFailed 0.004 ± 0.001 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K pollsMade 181.259 ± 5.380 ops/us</span></div>
<br /></div>
<div>
<ul>
<li>Under constant streaming pressure the Growable queue will keep growing until either full sized buffer is allocated (very likely) or a smaller buffer in which the throughput is sustainable is found (unlikely for this benchmark as all it takes is a single spike). If that was the case we would have no failing offers. Either way we expect transition to the last buffer to be a short phase after which the algorithm is very similar to SpscArrayQueue and no further allocations happen. The number of resizing events is small, as the buffer doubles each time (so log2(capacity/initial size), e.g. for initial capacity 16k: 16k -> 32k -> 64k -> 128k).</li>
<li>You may consider the slow down from SpscArrayQueue large at roughly 25%, but I don't think it too bad considering that with the throughputs in question we are looking at costs in the single digit nanoseconds where every extra instruction is going to show up (back of envelope: 250 ops/us -> ~4ns per offer/poll vs 180 ops/us -> ~5ns. 1ns = ~3 cycle ~= 12 instructions or 1 L1 load).</li>
</ul>
</div>
<div>
<h3>
Throughput benchmark: chunked</h3>
<div>
<div>
For Chunked we see the expected increase in throughput as we increase the chunk size (CNK is the <b>fixed chunk size</b>, the max size is 128K):</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> CNK Score Error Units</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 offersFailed ≈ 0 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 offersMade 43.665 ± 0.892 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 pollsFailed 9.160 ± 0.519 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 pollsMade 43.665 ± 0.892 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 offersFailed ≈ 10⁻⁴ ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 offersMade 151.473 ± 18.786 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 pollsFailed 0.380 ± 0.331 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 pollsMade 151.443 ± 18.778 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K offersFailed 0.309 ± 0.375 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K offersMade 149.351 ± 14.102 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K pollsFailed 0.112 ± 0.125 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K pollsMade 149.314 ± 14.120 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K offersFailed ≈ 10⁻⁸ ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K offersMade 175.408 ± 1.563 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K pollsFailed 0.038 ± 0.031 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K pollsMade 175.394 ± 1.563 ops/us</span></div>
<div>
<br />
<ul>
<li>Note the decline in throughput for smaller chunks is matched with an increase in poll failures indicating that the consumer is becoming faster than the producer as the chunk grows smaller requiring more frequent allocations by the produce.</li>
<li>Note also that even with 16 slot chunks this option is ~3 times faster than the linked alternative.</li>
<li>Under constant streaming pressure the Chunked queue will be pushed to it's maximum size, which means the producer will be constantly allocating buffers. The producer resize conditions are also slightly trickier and require sampling of the consumer index. The consumer will be slowed down by this sampling, and also slowed down by jumping to new buffers. This problem will be worse as more resizing happens, which is a factor of chunk size.</li>
<li>The benefit of larger chunks will cap out at some point, you could explore this parameter to find the optimum.</li>
<li>An exercise to readers: run the benchmark with the JMH GC profiler and compare the queues. Use it to verify the assumption that Growable produces a bounded amount of garbage, while Chunked continues to churn.</li>
<li>Max throughput is slightly behind Growable.</li>
</ul>
</div>
</div>
The main take aways for sizing here seem to me that tiny chunks are bad, but even with small/medium chunks you can have pretty decent throughput. The right size for your chunk should therefore depend on your expectations of average traffic on the one hand and desirable size when empty.<br />
<br /></div>
<div>
<h3>
Throughput benchmark: unbounded</h3>
</div>
<div>
For unbounded we see the expected increase in throughput as we increase the chunk size (CNK is the chunk size, the max size is infinity and beyond):</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> CNK Score Error Units</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 offersFailed ≈ 0 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 offersMade 56.315 ± 7.563 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 pollsFailed 10.823 ± 1.611 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16 pollsMade 56.315 ± 7.563 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 offersFailed ≈ 0 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 offersMade 135.119 ± 23.306 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 pollsFailed 1.236 ± 0.851 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 128 pollsMade 131.770 ± 21.535 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K offersFailed ≈ 0 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K offersMade 182.922 ± 3.397 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K pollsFailed 0.005 ± 0.003 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 1K pollsMade 176.208 ± 3.221 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K offersFailed ≈ 0 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K offersMade 177.586 ± 2.929 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K pollsFailed 0.031 ± 0.038 ops/us</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> 16K pollsMade 176.884 ± 2.255 ops/us</span></div>
<div>
<br />
<ul>
<li>The 16 chunk size is ~4 times faster than the linked list option, as chunk size increases it gets more efficient.</li>
<li>Max throughput is slightly behind growable.</li>
<li>Why is Chunked faster than Unbounded on 128 chunks, but slower on 1K? I've not looked into it, it's taken long enough to write this bloody post as it is. How about you check it out and let me know?</li>
</ul>
<div>
</div>
<br />
<h3>
Throughput benchmark: summary</h3>
<ul>
<li>Growable queue performs well regardless of initial size for this case.</li>
<li>For chunked and unbounded the chunk size has definite implications on throughput. Having said that throughput is very good even for relatively small chunks. </li>
<li>Note that the results for the same benchmark without a GC cycle between iterations were very noisy. The above result intentionally removes the variance GC induces by forcing GC and allowing a large heap. The GC impact of linked array queues when churning will likely be in increasing old generation pressure as the overflow chunks are likely to have been promoted before they get collected. This is assuming a load where overflow is not that frequent and other allocation is present.</li>
</ul>
<br />
<h3>
Interlude</h3>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT_tF54ZDNdVsOPEPfuXTN5GJKgKbBTZPzJRDQWa-tMygssLx9A-V9aQwCveH9UMA5AQMQftW8uhpcIh2JAfQPd6fVBvu8W1iZUd65C4BbgkdydcZQFkXne4Sxpho2Se1iwal2wC3B7IQ/s1600/lab+fire2.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT_tF54ZDNdVsOPEPfuXTN5GJKgKbBTZPzJRDQWa-tMygssLx9A-V9aQwCveH9UMA5AQMQftW8uhpcIh2JAfQPd6fVBvu8W1iZUd65C4BbgkdydcZQFkXne4Sxpho2Se1iwal2wC3B7IQ/s320/lab+fire2.jpg" width="320" /></a></div>
Go ahead, grab a beer, or a coffee, a mojito perhaps(Norman/Viktor, go on), or maybe order a large <a href="https://en.wikibooks.org/wiki/Bartending/Cocktails/Pan_Galactic_Gargle_Blaster" target="_blank">Pan Galactic Gargle Blaster</a>, you've earned it. I never thought you'd read this far, it's a tad dry innit? Well, it's not fun writing it either, but we're getting there, just need to look at one more benchmark...<br />
<br />
<h3>
Burst "cost"/latency benchmark: background and method</h3>
</div>
<div>
The burst cost benchmark is a more stable workload than the throughput one. The producer sends a burst of messages to a consumer. The consumer signals completion when the last message in the burst has arrived. The measurement is from first message sent and arrival of last message observed from the producer thread. It's a 'latency' benchmark, or rather an estimate of average communication cost via the particular thread. It's got bells on. It's a friend, and it's a companion, it's the only product you will ever need, follow these easy assembly instructions it never needs ironing.</div>
<div>
This is, I think, a better evaluation of queue characteristics than the throughput benchmark for most applications. Queue starts empty, is hit with a burst of traffic and the burst is drained. The cost measured is inclusive of return signal latency, but as scenarios go this is not too far fetched. Calling this queue latency is a damn sight better than PRETENDING THE BLOODY INVERSE OF THROUGHPUT IS LATENCY. <deep breath></div>
<div>
Same machine and JMH parameters used as above. All the measurements below are average time per operation in nanoseconds. The <a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/jmh/latency/QueueBurstCost.java" target="_blank">benchmark code can be found here</a>.</div>
<div>
<br /></div>
<h3>
Burst Cost benchmark: baseline</h3>
<div>
Testing first with SpscArrayQueue and SpscLinkedQueue to establish the expected baseline behaviour, BRST is the size of the burst:</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SpscArrayQueue (128k capacity)</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">BRST Score Error Units</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 284.709 ± 8.813 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 368.028 ± 6.949 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 914.150 ± 11.424 ns/op</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">Right, sending one message has the overhead of cache coherency making data visible to another core. Sending 10/100 messages we can see the benefits of the SpscArrayQueue in allowing consumer and producer to minimize cache coherency overhead per element. We see a satisfying drop in cost per element as the burst size grows (the per element cost is the cost of the burst divided by the number of elements sent, so we see here: 1 -> 284, 10 -> 36, 100 -> 9), but this DOES NOT MEAN THE FRIGGIN' LATENCY IS BLOOMIN' DOWN TO 9ns WHEN WE SEND 100 MESSAGES.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SpscLinkedQueue</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">BRST Score Error Units</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 378.043 ± 7.536 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 1675.589 ± 44.496 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 17036.528 ± 492.875 ns/op</span></div>
</div>
</div>
<div>
<br /></div>
<div>
For the linked queue the per element overheads are larger, as well as the cost of scanning through a linked list rather than an array as we poll data out. The gap between the it and SpscArrayQueue widens as the burst size grows. The linked queue fails to make the most of the batching opportunity offered by slack in the queue in other words.</div>
<div>
<br /></div>
<div>
<h3>
Burst Cost benchmark: growable</h3>
</div>
<div>
We expect the growable queue to grow to accommodate the size of the burst. The eventual buffer size will be a tighter fit around the burst size, which in theory might be a benefit as the array is more likely to fit in cache. Let's spin the wheel (CNK is the <b>initial </b>chunk size, the max size is 128K):</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">BRST CNK Score Error Units</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 16 327.703 ± 11.485 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 128 292.382 ± 9.807 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 1K 275.573 ± 6.230 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 16K 286.354 ± 6.980 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 16 599.540 ± 73.376 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 128 386.828 ± 10.016 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 1K 376.295 ± 8.009 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 16K 358.096 ± 6.107 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 16 1173.644 ± 28.669 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 128 1152.241 ± 40.067 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 1K 966.612 ± 9.504 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 16K 951.495 ± 12.425 ns/op</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
We have to understand the implementation to understand the results here, in particular:</div>
<div>
<ul>
<li>The growable queue buffer will grow to accommodate the burst in a power of 2 sized array. This in particular means that when the burst size is 100 the buffer for the initially smaller 16 chunk queue is also 128. The delta between the 2 configurations becomes marginal once that happens as we see in the 100 burst which forces the initially size 16 element buffer to grow to 128.</li>
<li>The queue tries to probe ahead within a buffer to avoid reading on each element.The read ahead step is a 25% of the buffer size. The smaller the buffer the more often we need to probe ahead (e.g. for a 16 element buffer we do this every 4 elements). This overhead is visible in the smaller buffers.</li>
<li>A burst which manages to fill more than 75% will fail to read ahead with the long probe described above and fall back to reading a single element ahead. This implies that buffers that fit too snugly to the burst size will have worse performance.</li>
<li>When the buffers are sufficiently large the costs closely match the costs observed for the SpscArrayQueue. Yay!</li>
</ul>
<br />
<div>
<h3>
Burst Cost benchmark: chunked</h3>
</div>
</div>
<div>
<div>
For Chunked we see a slight increase in base cost and a bummer when the burst size exceeds the chunk size (CNK is the chunk size, the max size is 128K):</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">BRST CNK Score Error Units</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 16 311.743 ± 11.613 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 128 295.987 ± 5.468 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 1K 281.308 ± 8.381 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 16K 281.962 ± 7.376 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 16 478.687 ± 52.547 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 128 390.041 ± 16.029 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 1K 371.067 ± 7.789 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 16K 386.683 ± 5.276 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 16 2513.226 ± 38.285 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 128 1117.990 ± 14.252 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 1K 969.435 ± 10.072 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 16K 939.010 ± 8.173 ns/op</span></div>
</div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">Results are overall similar to the growable, what stands out is:</span></div>
<div>
<ul>
<li>If the chunk is too little to accommodate the burst we see a large increase to cost. Still, comparing this to the SpscLinkedQueue shows a significant benefit. Comparing to the growable version we see the sense in perhaps letting the queue grow to a better size as a response to bursts.</li>
<li>If the chunk is large enough to accommodate the burst behaviour closely matches SpscGrowableArrayQueue. Yay!</li>
</ul>
<br />
<div>
<h3>
Burst Cost benchmark: unbounded</h3>
</div>
<div>
Final one, just hang in there. </div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">BRST CNK Score Error Units</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 16 303.030 ± 11.812 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 128 308.158 ± 11.064 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 1K 286.379 ± 6.027 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 1 16K 282.574 ± 10.886 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 16 554.285 ± 54.468 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 128 407.350 ± 11.227 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 1K 379.716 ± 9.357 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 10 16K 370.885 ± 12.068 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 16 2748.900 ± 64.321 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 128 1150.393 ± 26.355 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 1K 1005.036 ± 14.491 ns/op</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">100 16K 979.372 ± 13.369 ns/op</span></div>
</div>
<div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">What stands out is:</span></div>
<div>
<ul>
<li>If the chunk is too little to accommodate the burst we see a large increase to cost. Still, comparing this to the SpscLinkedQueue shows a significant benefit.</li>
<li>If the chunk is large enough to accommodate the burst and make the most of probing ahead the costs closely resemble the SpscArrayQueue for larger bursts. Yay!</li>
</ul>
<h3>
Burst Cost benchmark: summary</h3>
<div>
We see a pretty much expected result for these queues, which is to say that on the fast path they are the same and therefore if the fast path dominates they show the same costs as a plain SpscArrayQueue, which is good news. When chunks are too small and we have to allocate new chunks we start to see overheads.</div>
<div>
A more subtle observation here is that smaller buffers have some drawbacks as the slow path of the producer code is more likely to be executed. This reflects correctly the empty queue assumption that the JCTools queues rely on, but broken assumptions are... well... broken, so the cost goes up.</div>
<div>
A further consideration here for smaller buffer is the hot/cold structure of the code. It is intended that the producer code inlines the "offer" hot path, but as the cold path is rarely run it will fail to inline it. This is an intentional inlining fail. Inlining the cold path will make the "offer" larger and allot more complex, making the compilers job harder and may result in worse resulting code. When we run with burst/buffer sizes which systematically violate the hot/cold assumption we can trigger a bad inlining decision. This can be worked around by marking the cold methods as "dontinline" using the CompileCommand option or the compiler oracle file.</div>
<h3>
Mmmm... this is boring :(</h3>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg9o6kDh-Ut_bXIs4CQ4VR_0EIWmT8RrifUBo4IMNrVkVsuwl6gmPPXmKurbvLwqZhqcCWb1J3gohqp8HehGiNq-ukK_krp8O9c_Nw6walvXWuULOQTvj7u-GA-LYIhozxrDjXRWT3XfY/s1600/Isaac_Newton_Labratory_Fire-300x207.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgg9o6kDh-Ut_bXIs4CQ4VR_0EIWmT8RrifUBo4IMNrVkVsuwl6gmPPXmKurbvLwqZhqcCWb1J3gohqp8HehGiNq-ukK_krp8O9c_Nw6walvXWuULOQTvj7u-GA-LYIhozxrDjXRWT3XfY/s1600/Isaac_Newton_Labratory_Fire-300x207.jpg" /></a></div>
<div>
Yes... Nothing too surprising happened here, I did not emerge from the lab with my coat on fire, these things happen. One anecdote worth sharing here is that I originally run the benchmarks with only 2 threads allocated to the JVM, this resulted in noisier measurement as I effectively under provisioned the JVM with CPUs for compilation/GC or any OS scheduling contention/interrupts. When running on a 2 core laptop this is a reasonable compromise to fix the cross core topology of the benchmark, but on a server class machine it is easy enough to provision the same topology with more CPUs.</div>
<div>
Next part will feature the astounding extension of these queues to the MPSC domain and will be far more interesting! I promise.</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<ul></ul>
</div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com0tag:blogger.com,1999:blog-5171098727364395242.post-1439519077705911162016-10-24T07:57:00.000+01:002016-10-24T07:57:24.037+01:00Linked Array Queues, part 1: SPSC<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9yp7IHSFZ76szZbFkkiGBy6slWcUO_pEj8AAx5vfC2GE5FrSrb2eoTZfa-VTrXF1_jh4ulplILKXW6g-V7mWf5z_zHzrF_dlagnpDELTM7wTVCK72KhzvmjLoQXBANZA54w8DzCiVaKU/s1600/chunky+monkey.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9yp7IHSFZ76szZbFkkiGBy6slWcUO_pEj8AAx5vfC2GE5FrSrb2eoTZfa-VTrXF1_jh4ulplILKXW6g-V7mWf5z_zHzrF_dlagnpDELTM7wTVCK72KhzvmjLoQXBANZA54w8DzCiVaKU/s1600/chunky+monkey.jpg" /></a></div>
When considering concurrent queues people often go for either:<br />
<div>
<ol>
<li>An array backed queue (circular array/ring buffer etc.)</li>
<li>A linked list queue</li>
</ol>
</div>
<div>
The trade off in the Java world seems to be that array backed queues offer better throughput, but are always bounded and allocated upfront, and linked queues offer smaller footprint when empty, but worse throughput and higher overhead per element when full. Linked queues are also often unbounded.</div>
<div>
In <a href="https://github.com/JCTools/JCTools" target="_blank">JCTools</a> we have the above duality, but in later versions introduced a hybrid queue which attempts to offer a middle ground between the 2 extremes above. This hybrid is the linked array queue:</div>
<div>
<ol>
<li>Queue is backed by one or more arrays of references.</li>
<li>As long as the queue can fit in a single array no further arrays are allocated.</li>
<li>When empty the queue can naturally shrink to a single backing array.</li>
</ol>
For the SPSC case this has already been done in C++ Fast Flow with their uSPSC queues. In Java there are no other implementations that I know of (please let me know if I missed anything).</div>
<div>
In implementing these queues I have relied heavily on the feedback and support of <a href="https://github.com/akarnokd" target="_blank">@akarnokd</a>, <a href="https://github.com/pcholakov" target="_blank">@pcholakov</a> and others who contributed fixes, filed bugs, and so forth. Thanks guys!</div>
<div>
3 variations on linked array queues have been explored in JCTools:</div>
<div>
<ol>
<li>Chunked: Each backing array is a fixed chunk size. The queue is bounded.</li>
<li>Unbounded: Each backing array is a fixed chunk size. The queue is unbounded.</li>
<li>Growable: Chunk size doubles every time until the full blown backing array is used. The queue is bounded.</li>
</ol>
<div>
<br /></div>
<h3>
Hot Path offer/poll</h3>
The queues all share the same polling logic and on the fast path share the same offer logic as well:</div>
<div>
<script src="https://gist.github.com/nitsanw/45be60f4e051e4c2d3fecfe36a765726.js"></script></div>
<div>
If you've not read JCTools code before, or maybe you've forgotten, here's the convention:</div>
<div>
<ul>
<li>sv/lvXXX: Store/Load Volatile, same as a volatile field write/read</li>
<li>sp/lpXXX: Store/Load Plain, same as a normal field write/read</li>
<li>soXXX: Store Ordered, same as an AtomicXXX.lazySet</li>
</ul>
This code will need reconsidering in a post-JDK9 world, some other time perhaps.</div>
<div>
So what's the deal:</div>
<div>
<ul>
<li>As long as we are not passed the producerLimit, keep writing.</li>
<ul>
<li>If we have passed it go to slow path (where the money be)</li>
</ul>
<li>As long as there's stuff in the queue, read it.</li>
<ul>
<li>Unless it's JUMP, in which case read through to next array.</li>
</ul>
</ul>
The general idea here being that the common case is small and simple. This should have the following effects:</div>
<div>
<ol>
<li>offer/poll code is small enough to inline when hot.</li>
<li>offerColdPath/newBufferPoll are set up to either not inline or, when inlined be out of band code blocks. This should keep size on the hot path small and help the compiler focus on more important things.</li>
<li>offer/poll should perform similar to the SpscArrayQueue in the common case.</li>
</ol>
<span style="background-color: #fce5cd;">NOTE: The producer publishes element then index, but the consumer does the reverse. This ensures a consistent view on the consumer side where the following assertion must hold:</span><br />
<span style="background-color: #fce5cd;"> !queue.isEmpty() => queue.poll() != null</span></div>
<div>
<span style="background-color: #fce5cd;"><br /></span>
<span style="background-color: #fce5cd;">NOTE: In some early versions the new array was entered instead of the JUMP constant marker. This required an instanceof check for each element loaded and a compromise to either not allow Object[] to be an element of the queue or introduce some wrapper class. Comparing to a constant turned out to be much simpler and faster.</span><br />
<ol>
</ol>
<h3>
Cold Path poll</h3>
</div>
<div>
The consumer has hit a JUMP, which indicates the producer has linked a new array to this one. The new array is the last element of the current array. We go to newBufferPoll:</div>
<div>
<script src="https://gist.github.com/nitsanw/d59092ebbace552767466ba1fb1c97a5.js"></script></div>
<div>
<br /></div>
<div>
<span style="background-color: white;">The consumer therefore has to:</span></div>
<div>
<ol>
<li>Load new array from the last element of old array.</li>
<li>Null out reference from old to new (prevent nepotism).</li>
<li>Adjust consumer view on buffer and mask.</li>
<li>Poll (remove element and progress counter) from new buffer.</li>
</ol>
Note that peek similarly refreshes view of buffer on hitting the JUMP marker. This goes against the standard spirit on peek which is a view only operation.<br />
<br />
<h3>
Cold Path offer: Unbounded</h3>
</div>
<div>
This method is implemented differently for each of the use cases, unbounded is the simplest:</div>
<div>
<script src="https://gist.github.com/nitsanw/a0130d5113fb04064d2d41ffc9bec71e.js"></script></div>
<div>
In the unbounded case we care only about our ability to make progress inside the current buffer:<br />
<br />
<ol>
<li>Probe inside buffer to see if we have 'many' elements to go before full. If buffer is mostly empty (this is the case for most applications most of the time), than we have successfully saved ourselves loading elements from the queue before writing in. A successful probe will update the producer limit and write to the queue.</li>
<li>Probe failed, we check if we can write a single element. Remember we always need one free slot to JUMP with, so we look at the slot following the one we are on. If the next slot is empty we write to the current one, but we don't update the limit.</li>
<li>A single slot remains in the buffer. We allocate a new buffer and link to it.</li>
</ol>
This is not a large departure from the SpscArrayQueue, I leave the comparison as an exercise to the delightful reader.<br />
<br />
<h3>
Cold Path offer: Chunked</h3>
</div>
<div>
With chunked linked array queues we have a fixed chunk size, but an overall bound on the size. This complicates matters because on top of the buffer level limits put on the producer we must also consider the queue level limitation. In particular there might be space available in the current producer buffer, while the queue is in fact full. Here's the implementation:</div>
<div>
<script src="https://gist.github.com/nitsanw/840c50bfe0d01408a49b5a100cb5c7a7.js"></script></div>
<div>
Similar to the above but the difference lies in negotiating the buffer vs. queue limit.</div>
<div>
<br />
<h3>
Cold Path offer: Growable</h3>
</div>
<div>
The growable queue is similar in spirit to an ArrayList as it doubles it's backing array capacity when a buffer is full. This adds an interesting bit of information to the game, since:</div>
<div>
<ol>
<li>If we are not on the last buffer, we've not hit the queue limit,</li>
<li>If we're on the last buffer, and so is the consumer, we can revert to just checking for null in the buffer.</li>
<li>If we're on the last buffer, and the consumer isn't, we need to hang tight and let it pass. It's a temporary state.</li>
</ol>
The code for handling this is rather ugly and long, but since you've put up with the rest:</div>
<div>
<script src="https://gist.github.com/nitsanw/ca0b4dbed6a41947ad1c14e4f76f1320.js"></script></div>
<div>
The lookAheadStep is dynamically adjusted as the buffer grows, and also acts as an indicator for the transition period which the producer is on the last buffer and the consumer is trailing. It's a mess to look at, but sadly performs better than a neater alternative which builds on the Chunked variant... General idea:<br />
<br />
<ol>
<li>lookAheadStep is positive => we are either not on last buffer, or on it for both consumer and producer => it is enough to consider the elements in the producer buffer to determine if the queue is full. In particular if the buffer is full then we must resize unless we are on the last buffer in which case we are full. Note that we allow using the last buffer to the last element, since we don't need a JUMP anymore.</li>
<li>lookAheadStep is negative => we are waiting for consumer to get to the last buffer. We use the lookAheadStep to indicate how far behind the consumer is.</li>
</ol>
</div>
<div>
It's not complex, just messy, and if you got an elegant representation please ping me with your suggestions.<br />
<h3>
Performance?</h3>
</div>
<div>
GODDAMN it! this is the time consuming part! I've benchmarked on a few setups, but not kept track or clear records. I'll need to do it all again, might as well be another post since nobody reads this far into these things anyhow. La la la la la, performance is great, la la la la la, if I look now I might be disappointed, la la la la la, this is better than benchmarking, la la la la la.</div>
<div>
<br /></div>
<h2>
TO BE CONTINUED...</h2>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com1tag:blogger.com,1999:blog-5171098727364395242.post-75051928713883304252016-10-21T13:26:00.001+01:002016-10-21T13:26:19.981+01:004 Years Blog Anniversary<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLkmeqqjXgXv72uxCv-Zw2h65Xoka9swNmqXibBQoWVI_otWViNhEtKWU5zJX2TNWK2gHk9i0vpmlqvz2qUek3c6jsWaeH26w8U-xUhAcCirUTj8zZDgk7B1O7DHeCTotvsaWn3uEwLjM/s1600/jabberwocky.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLkmeqqjXgXv72uxCv-Zw2h65Xoka9swNmqXibBQoWVI_otWViNhEtKWU5zJX2TNWK2gHk9i0vpmlqvz2qUek3c6jsWaeH26w8U-xUhAcCirUTj8zZDgk7B1O7DHeCTotvsaWn3uEwLjM/s320/jabberwocky.jpg" width="215" /></a></div>
This has been a very slow year blogging wise... too much work, travel, and family. 4 years ago I aimed for 2 posts a month. I nearly hit it in 2013, failed by a little in 2014, and went on to not meeting at all the last 2 years...<br />
Coincidently I also have a delightfully busy 2.5 year old running around, I suspect she has something to do with how little time I have. My eldest has developed a terrible addiction to reading so I hardly see her these days, can't be blame for this one.<br />
Since I started the blog I've also embarked on developing and maintaining <a href="http://www.jctools.org/" target="_blank">JCTools</a>, <a href="http://psy-lob-saw.blogspot.co.za/p/talks.html" target="_blank">speaking at conferences</a>, and starting a local<a href="http://www.meetup.com/Cape-Town-Java-Meetup/" target="_blank"> JUG in Cape Town</a>. Sharing of one kind breeds another, but you don't seem to gain bonus hours in the day. Hmmm.<br />
If I sound like I'm making excuses it's because I'm still feeling bad about missing that 2 posts a month goal.<br />
Despite my decreased output, blog traffic has steadily increased teaching me a thing or 2 about search and web economics. I hope to write more in the next year as I plan to give conferences a rest for the next 6-8 months, will see how it goes.<br />
In any case,<b> thanks for reading :-)</b><br />
<br />Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com2tag:blogger.com,1999:blog-5171098727364395242.post-48305169621596752542016-07-18T09:20:00.000+01:002016-07-20T09:42:22.189+01:00Fixing Coordinated Omission in Cassandra Stress<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0WpwEICMKGeppXb0OX1GpBoOoqtDvUqO9R_tfoekzE3sLM5-itHCVTcQQEX97ciLuk8Cc0eEJlKE9ZZi7HYfrPpo3qe_pDMe2dOe69RhKLVPb1GJUJWBi9m7EiWqJDVGSrnLWQgwenfE/s1600/cassandra-logo.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0WpwEICMKGeppXb0OX1GpBoOoqtDvUqO9R_tfoekzE3sLM5-itHCVTcQQEX97ciLuk8Cc0eEJlKE9ZZi7HYfrPpo3qe_pDMe2dOe69RhKLVPb1GJUJWBi9m7EiWqJDVGSrnLWQgwenfE/s1600/cassandra-logo.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="background-color: #f9f9f9; color: #252525; font-family: sans-serif; line-height: 22.4px;"><span style="font-size: xx-small;">Copyright © 2016 Apache Software Foundation</span></span></td></tr>
</tbody></table>
I have it from reliable sources that incorrectly measuring latency can lead to losing ones job, loved ones, will to live and control of bowel movements. Out of my great love for fellow humans I have <a href="http://psy-lob-saw.blogspot.com/2015/03/fixing-ycsb-coordinated-omission.html" target="_blank">fixed the YCSB load generator</a> to avoid the grave danger that is Coordinated Omission, this was met with great joy and I now gone on to fixup <a href="http://cassandra.apache.org/" target="_blank">Cassandra</a> Stress in similar vein. The <a href="https://github.com/apache/cassandra/commit/89f275c6587150907b90ed03ca515b12c0c6ae11" target="_blank">fix is now merged</a> into <a href="https://github.com/apache/cassandra" target="_blank">trunk</a>, so expect to have it with the next Cassandra release (or just build it from source NOW).<br />
Before we start on the how and what, let us take a step back and consider the why:<br />
<ul>
<li>Coordinated Omission is a term coined by my esteemed colleague Gil Tene to describe a common measurement anti pattern where the measuring system inadvertently coordinates with the system under measurement (watch Gil's <a href="https://www.youtube.com/watch?v=lJ8ydIuPFeU" target="_blank">How NOT to Measure Latency talk</a>). This results in greatly skewed measurements in commonly used load generators, and has led to long discussions on <a href="https://groups.google.com/forum/#!msg/mechanical-sympathy/icNZJejUHfE/BfDekfBEs_sJ" target="_blank">Mechanical Sympathy mailing list</a> etc. I've given <a href="http://psy-lob-saw.blogspot.com/2015/03/fixing-ycsb-coordinated-omission.html" target="_blank">my own explanation in a previous post</a>, so go and have a read if you need a refresher on the subject.</li>
<li>Cassandra Stress (CS) is a tool which comes bundled with Cassandra to enable load testing of a given cluster. It allows the user to specify their own queries and schemas and is the prefered tool for Cassandra benchmarking as it gives better access to Cassandra features etc. CS allows 2 testing modes: throughput(default) or 'limit' where a given number of operations per second are thrown at the cluster (the fix discussed here does away with <b>limit</b> and replaces it with <b>throttle/fixed</b>, read on).</li>
</ul>
So coordinated omission is bad, and it is now fixed in Cassandra Stress. This post talks a bit on motivation (part I), a bit on implementation (part II) and a bit on what you can do with the new features (part III). Feel free to skip any and all parts, god knows those selfies don't take themselves.<br />
<br />
<h3>
PART I: Demonstrating The Issue</h3>
<div>
CS default mode is to hit the system under test as hard as it can. This is a common strategy in load generators and can give system writers an interesting edge case to work with. I run the benchmark on my laptop (no attempt at finding out real performance numbers here, I just care about measurement issue) with the example provided workload I can saturate my Cassandra server (I only gave it a single core to run on) pretty easily. CS tells me the following about my miserable little server:</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Results:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Op rate : 5,055 op/s [insert: 500 op/s, simple1: 4,556 op/s]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Partition rate : 17,294 pk/s [insert: 12,739 pk/s, simple1: 4,556 pk/s]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Row rate : 150,266 row/s [insert: 12,739 row/s, simple1: 137,527 row/s]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Latency mean : 4.5 ms [insert: 7.5 ms, simple1: 4.2 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Latency median : 3.0 ms [insert: 5.4 ms, simple1: 2.8 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Latency 95th percentile : 13.1 ms [insert: 20.1 ms, simple1: 11.9 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Latency 99th percentile : 23.8 ms [insert: 33.7 ms, simple1: 21.8 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Latency 99.9th percentile : 49.8 ms [insert: 55.0 ms, simple1: 49.0 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Latency max : 603.5 ms [insert: 598.7 ms, simple1: 603.5 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Total partitions : 1,000,000 [insert: 736,585, simple1: 263,415]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Total errors : 0 [insert: 0, simple1: 0]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Total GC count : 112</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Total GC memory : 34.850 GiB</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Total GC time : 1.0 seconds</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Avg GC time : 9.4 ms</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 StdDev GC time : 16.6 ms</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 15:02:46 Total operation time : 00:00:57</span></div>
</div>
<div>
With the other 3 cores on my laptop hitting it as hard as they could the median latency on this maxed out server was 3ms. That is pretty awesome. But also, it makes no sense.</div>
<div>
How can a maxed out server have a typical response time of 3ms? In reality when servers are maxed out they are unresponsive, the typical response time becomes worse as the load increases. What CS is reporting however is not response time. It is 'latency'. Latency is one of those terms people use to mean many things and in this case in particular it does not mean "response time" but rather "service time". Here's a definition of more specific terms to describe system responsiveness(see wiki on <a href="https://en.wikipedia.org/wiki/Response_time_(technology)" target="_blank">response time</a>):</div>
<div>
<div>
<ul>
<li>Response time: The time between the submission of a request and the completion of the response.</li>
<li>Service time: The time between the initiation and completion of the response to a request.</li>
<li>Wait time: The time between the submission of the request and initiation of the response.</li>
</ul>
</div>
</div>
<div>
In an all out test one could argue we want all the results as soon as possible, and given a magic load generator they would all launch instantaneously at the benchmark start. This will mean <b>submission time</b> for all requests is at the beginning of the run. Naturally the server will not be able to respond instantly to all requests and can only allow so many requests to be handled in parallel. If the max throughput is 5000 ops/sec, and we are measuring for 100,000 ops it would mean 95K ops have waited a second, so their response time would be at least 1 second (response time > wait time). By the end of the run we would have 5K ops which have patiently waited at least 19 seconds (so 99%ile should be at least 19 seconds).</div>
<div>
It follows that in an all out throughput benchmark response time is terrible by definition, and completely uninformative. It also follows that we should not expect the 'latency' above to be at all indicative of the sort of response time we would get from this server.</div>
<div>
The alternative to an all out throughput benchmark is a Responsiveness Under Load (RUL) benchmark. Using Cassandra Stress one can (or rather they could before this fix went in) use the '-rate <b>limit</b>=17000/s' option to benchmark under a load of 17k pks/sec.(pks = partition keys, each operation costs X keys, throughput limit is specified in pks not ops) Running this gives me a warm fuzzy feeling, now for sure I shall get a glimpse of the response time at max throughput:</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Results:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Op rate : 3,712 op/s [insert: 369 op/s, simple1: 3,343 op/s]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Partition rate : 12,795 pk/s [insert: 9,452 pk/s, simple1: 3,343 pk/s]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Row rate : 110,365 row/s [insert: 9,452 row/s, simple1: 100,913 row/s]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Latency mean : 1.0 ms [insert: 1.6 ms, simple1: 0.9 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Latency median : 0.7 ms [insert: 1.3 ms, simple1: 0.7 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Latency 95th percentile : 2.2 ms [insert: 3.4 ms, simple1: 2.0 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Latency 99th percentile : 4.6 ms [insert: 7.4 ms, simple1: 4.1 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Latency 99.9th percentile : 13.4 ms [insert: 23.8 ms, simple1: 12.1 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Latency max : 63.9 ms [insert: 59.9 ms, simple1: 63.9 ms]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Total partitions : 300,000 [insert: 221,621, simple1: 78,379]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Total errors : 0 [insert: 0, simple1: 0]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Total GC count : 33</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Total GC memory : 10.270 GiB</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Total GC time : 0.2 seconds</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Avg GC time : 7.5 ms</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 StdDev GC time : 2.5 ms</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">INFO 08:03:54 Total operation time : 00:00:23</span></div>
</div>
<div>
<span style="font-family: inherit;">This seems nice, and if I were not a suspicious man I might accept it. The thing is, I asked for 17k pks per second, but I only got </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">12,795 pk/s </span><span style="font-family: inherit;">so obviously the server could not meet the implied schedule. If it could not meet the schedule, response time should be terrible. But it's not, it's very similar to the result we got above. Because? Because again, latency here means <b>service time </b>and not <b>response time</b>. While response time is not informative in an all out test, in an RUL benchmark it is the whole purpose of the benchmark. I have a schedule in mind, requests come at a particular rate, which implies they have a known start time (request n will start at: T0 + n/rate, T0 being the start of the test). This is coordinated omission, lets fix it.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<h3>
Part II(a): It Puts The Measurements In The HdrHistogram or It Gets The Hose Again!</h3>
<div>
First off, I like to have <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> files to work with when looking at latency data (<a href="http://psy-lob-saw.blogspot.com/2015/02/hdrhistogram-better-latency-capture.html" target="_blank">made a whole post about it too</a>). They are usually tons better than whatever it is people hand roll, and they come with a bunch of tooling that makes data post processing easy. Importantly HdrHistograms are loss-less, configurable compact and support loss-less and compressed logging. Combine that with the high resolution of data and you have a great basis for post run analysis.<br />
Cassandra Stress had it own sampling latency collection mechanism which would randomly drop some samples as a means to improve memory footprint, so the replacement improved reporting accuracy and reduced the amount of code. A side effect of this change is that Cassandra perf regression testing stability has improved rather dramatically. As indicated by this graph:</div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOiorrYxCs1YiXFBx6run5mOt50KcTcxrfA6C_pcmSYamKLtxr6pTds67RKztRZhfm6QNpYxMP0ldQ4NkQsXntQmGZr1rOVYEaWD1nTkk6ekIUNGBADeeHXdrYlwWLUfl73p1vsoo08vY/s1600/99.9th_latency.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="531" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOiorrYxCs1YiXFBx6run5mOt50KcTcxrfA6C_pcmSYamKLtxr6pTds67RKztRZhfm6QNpYxMP0ldQ4NkQsXntQmGZr1rOVYEaWD1nTkk6ekIUNGBADeeHXdrYlwWLUfl73p1vsoo08vY/s640/99.9th_latency.png" width="640" /></a>
<br />
<div>
<div class="separator" style="clear: both; text-align: left;">
Because of the random sampling 99.9%ile reporting was unstable before the patch went in (May 28th), but smoooooth ever since. Ain't that nice?</div>
<div class="separator" style="clear: both; text-align: left;">
Here's the <a href="https://github.com/nitsanw/cassandra/commit/bf6c04a8948fc91c4eac7a5e86a21c4469c24e65" target="_blank">commit with the main changes</a> on the branch that became the PR, there were further changes introduced to enable histogram logging to files. The change is mostly tedious, but importantly it introduces the notion of <a href="https://github.com/nitsanw/cassandra/commit/bf6c04a8948fc91c4eac7a5e86a21c4469c24e65#diff-70988bd41ddcbb6b9ffe75e1fac6df8eR31" target="_blank">split response/service/wait histograms</a>. The data is <a href="https://github.com/nitsanw/cassandra/commit/bf6c04a8948fc91c4eac7a5e86a21c4469c24e65#diff-70988bd41ddcbb6b9ffe75e1fac6df8eR64" target="_blank">recorded as follows</a>:</div>
<div>
<script src="https://gist.github.com/nitsanw/e766fc23cc2e9dc91866ac4af076e9dc.js"></script></div>
<div>
The question is, what is the intended start time?</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<h3 style="clear: both; text-align: left;">
Part II(b): <a href="https://www.youtube.com/watch?v=-4v1fkylL3s" target="_blank">Where Do I Start?</a></h3>
<div>
Load generators, when not going all out, generally have some notion of schedule. It is more often than not quite simply a fixed rate of requests, though the notion of schedule holds regardless of how you come up with it. A schedule in this context means that for any event X there is a time when it should happen: st(X). That time is easily computed in a fixed rate schedule as: "st(n) = T0 + n/rate". Cassandra Stress however was using google's RateLimiter to provide it with the scheduling, and while battle tested and handy it does not have a notion of schedule. The replacement took place in 2 steps.</div>
<div>
First I refactored the existing code into hiding the details of how operations are scheduled and where they come from behind a<a href="https://github.com/nitsanw/cassandra/commit/8fb6f89e33ea6ebddf08df833b679dc44356d359" target="_blank"> blocking queue like interface.</a> The <a href="https://github.com/nitsanw/cassandra/commit/7c02c5ef2dde423607097dff56ef5e4fb48bfb30" target="_blank">next step</a> was to support a fixed rate stream of operations where the intended schedule is available so I can use it. This is what I ended up with (<a href="https://github.com/nitsanw/cassandra/commit/9e6cb583e2e6aeeab29de15d895ada0bd479d06a" target="_blank">further tweaked</a> to only start the clock when all the consumer initialization has happened):</div>
<div>
<script src="https://gist.github.com/nitsanw/72df7c3fea2be7353f3b316cf5236c91.js"></script></div>
<div>
<br /></div>
<div>
Now we're all set to benchmark with no Coordinated Omission!<br />
<br />
<h3>
Part II(c): Who Reports What?</h3>
</div>
<div>
The measurement collection now all set up we face the question of what should different load generating modes report. Since it turned out that 'limit' was a confusing name (hard limit? upper limit?) it was decided to replace it with 2 distinct modes, adding to a total of 3 modes:</div>
<div>
<ul>
<li>Throughput: latency == service time, response/wait time not recorded. Maximum throughput is an important test mode for flushing out CPU bottlenecks and contention. It may help in exposing IO configuration issues as well. It is not however a good predictor of response time distribution as no attempt is made to estimate independent request timing (i.e uncoordinated measurement). The maximum throughput number is valuable to batch processing etc, but I'd caution against using it for the sizing of a responsive system. If a system has responsiveness SLAs it is better to use the 'fixed' test mode and run for significant periods to determine the load under which response times SLA can be met.</li>
<li> Throttle ("-rate throttle=1000/s"): latency == service time, response/wait time recorded like in fixed. This mode is a compromise for people who liked the old 'limit' measurement and want to measure "service time under attempted load". The partitions/second parameter is attempted for, but the summary does not reflect the response time. If you like, this is a way to sneakily record the response time while keeping the summary as it is so people are not so startled by the difference. I don't see myself using it, but have a blast.</li>
<li>Fixed ("-rate fixed=1000/s"): latency == response time, service/response/wait time recorded. This mode is for benchmarking response time under load. The partitions/second parameter is used to determine the fixed rate scheduled of the requests. The hdr log can be used to visualize latencies over time or aggregate latencies for any segment down to the logging frequency. The logs contain response/wait/ service components that can be extracted and handled separately.</li>
</ul>
</div>
In all of the above you can choose to record the HDR histograms to a file on a fixed interval. The interval is the one used for summary reporting. To enable histogram logging use: " -log interval=10s hdrfile=cs.hdr"<br />
Note that the interval should be quite long, and the default of 1s is not actually achievable by Cassandra Stress at the moment. This is down to the terrible way the reporting thread synchronizes with the load generation threads, and while it is one my wish list ("I wish I had time to fix it...") it was outside the scope of fixing CO, so lived to taunt us another day. I've settled on 10 second intervals for now, you can have even longer ones, all depends on the sort of granularity you want in your reports.<br />
<br />
<h3>
Part III: What Is It Good For?</h3>
</div>
<div>
So we got 3 load generating modes, and we've broken up latency into 3 components, the numbers are in histograms, the histograms are in the logs... let's have a little worked example.</div>
<div>
I run a Cassandra 2.1 node on my laptop (old software, bad hardware... I don't care. This is not about absolute numbers, it's about the measurement features). To run this please:</div>
<div>
<ol>
<li>Clone the Cassandra trunk: <span style="font-family: "courier new" , "courier" , monospace;">git clone https://github.com/apache/cassandra.git</span></li>
<li>Build Cassandra and Cassandra Stress: <span style="font-family: "courier new" , "courier" , monospace;">ant clean build stress-build jar</span></li>
</ol>
</div>
<div>
<br /></div>
<div>
I use the example workload provided by the good Cassandra folks, and start with an all out stress test from cassandra/tools/bin:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">$ ./cassandra-stress user profile=../cqlstress-example.yaml duration=60s ops\(insert=1,simple1=9\) -mode native cql3 protocolVersion=2 -rate threads=3 -log interval=10s hdrfile=throughput.hdr</span></div>
<div>
<script src="https://gist.github.com/nitsanw/9e57e25c2fdddad575a44c0069079ac3.js"></script></div>
<div>
Joy, my laptop is giving me awesome throughput of 13,803 pk/s. The summary itself is pretty informative for throughput runs what do we win with the HDR log?<br />
The log we produce can be summarized, manipulated and graphed by a bunch of utilities. Here's what the log looks like (this is from the fixed mode run):<br />
<script src="https://gist.github.com/nitsanw/994f6bb6ba0e683caad342140b8685a1.js"></script><br />
Note that while we logged our latency in nanoseconds, the max column is in milliseconds. The nanosecond level measurements are still available in the compressed histogram to the right. Sadly it's not very friendly on the eye. HdrHistogram does include a log processing utility, but it offers quite basic facilities. I've put together a few utilities for histogram log management in <a href="https://github.com/nitsanw/HdrLogProcessing" target="_blank">HdrLogProcessing</a>. These allow you to split, union and summarize logs and work with the tags feature. Lets make them into handy aliases:<br />
<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">hdrsum=java -jar HdrLogProcessing-1.0-SNAPSHOT-jar-with-dependencies.jar SummarizeHistogramLogs</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">hdrspl=</span><span style="font-family: "courier new" , "courier" , monospace;">java -jar HdrLogProcessing-1.0-SNAPSHOT-jar-with-dependencies.jar SplitHistogramLogs</span></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">hdruni=</span><span style="font-family: "courier new" , "courier" , monospace;">java -jar HdrLogProcessing-1.0-SNAPSHOT-jar-with-dependencies.jar UnionHistogramLogs</span></li>
</ul>
<br />
The <span style="font-family: "courier new" , "courier" , monospace;">throughput.hdr</span> file we requested uses a recently added HdrHistogram feature which allows for the tagging of different histograms in one log. This makes it easy for applications logging histograms for different measurements to shove them all into a single file rather than many. Since we want to track 2 different operations with their separate response/service/wait times (so up to 6 files) this is quite handy. We can start by using the all in one HdrLogProcessing summary utility to add up the histogram data. The default mode of summary is percentiles and will produce a summary very similar to the one produced above by Cassandra Stress (using the parameters "<span style="font-family: "courier new" , "courier" , monospace;">-if throughput.hdr -ovr 1000000</span>" to summarize in milliseconds):<br />
<script src="https://gist.github.com/nitsanw/d33a7da6564aef4c7ea90ec80f4e597c.js"></script><br />
We can have the microsecond or nanosecond report by tweaking the <span style="font-family: "courier new" , "courier" , monospace;">outputValueUnitRatio(ovr). </span><span style="font-family: inherit;">We are also free to ignore tags and create an all inclusive summary by specifying the </span><span style="font-family: "courier new" , "courier" , monospace;">ignoreTag(it)</span><span style="font-family: inherit;"> parameter. Used together to create a total summary in microseconds we get (</span>"<span style="font-family: "courier new" , "courier" , monospace;">-if throughput.hdr -ovr 1000 -it</span>" <span style="font-family: inherit;">):</span><br />
<span style="font-family: inherit;"><script src="https://gist.github.com/nitsanw/bd2e118498c40ba679ce58c76b18d868.js"></script></span><br />
As a bonus, the summary tool allows us to process multiple files into the same result (logs from multiple benchmark runs for instance) and selecting periods within the logs to include.<br />
The summary tool also supports generating the HGRM format which allows us to produce the following graph (using "-if throughput.hdr -ovr 1000 -st hgrm -of tpt" and the <a href="http://hdrhistogram.github.io/HdrHistogram/plotFiles.html" target="_blank">google charts provided in HdrHistogram</a>):<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3ZWfj80oMvAxGfT1dObzDsrFMKSvwFGjqalmLUwHNnM6oAolkz7IKgp_boegVXLj3yEIG5MYxJu4u0d8shpvJNzdTJflX8s75KgSdfF-j0D08erhi1MmsU9U5piC4tQMor3yKvK_wzG8/s1600/Histogram+%25281%2529.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3ZWfj80oMvAxGfT1dObzDsrFMKSvwFGjqalmLUwHNnM6oAolkz7IKgp_boegVXLj3yEIG5MYxJu4u0d8shpvJNzdTJflX8s75KgSdfF-j0D08erhi1MmsU9U5piC4tQMor3yKvK_wzG8/s1600/Histogram+%25281%2529.png" /></a><br />
<br />
<br />
Now imagine we used 3 different machines to stress a single Cassandra node. Because the logs are additive and loss less there's no issue with using the union tool to aggregate them all into a single log and process it as such. Similarly you can use the splitting tool to split out the operation you are interested in and manipulate it in isolation. Indeed the sky is the limit.<br />
Now, with a stunning 13K partitions per second, an a 50ms 99.99%ile I might think it a reasonable idea to say my server can safely sustain a 10K pk/s rate and call it a day. I shall test this adventurous little estimate using the throttle mode:<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ ./cassandra-stress user profile=../cqlstress-example.yaml duration=60s ops\(insert=1,simple1=9\) -mode native cql3 protocolVersion=2 -rate <b>throttle</b>=10000/s threads=3 -log interval=10s hdrfile=throttle.hdr</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><script src="https://gist.github.com/nitsanw/db79d65215b2e735f3f7e8dfed7eaa66.js"></script></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
There according to this, we should have no trouble at all with 10K pk/s. I mean sure it's a bit close to the SLA, but surely no too bad? The throttle mode is in this case just a way to keep your head in the sand a bit longer, but it does record the response time in case you feel like maybe comparing them. Let's compare the service time histogram and the response time histogram for this run. To get both operations service time histograms I need to play with the log processing a bit:<br />
<br />
<ol>
<li>Split the throttle.hdr file:"<span style="font-family: "courier new" , "courier" , monospace;">hdrspl -if throttle.hdr</span>" -> This results in 6 different files <span style="font-family: "courier new" , "courier" , monospace;"><op>-<rt/st/wt>.throttle.hdr</span>.</li>
<li>Summarize the service time histograms:"<span style="font-family: "courier new" , "courier" , monospace;">hdrsum -if .*.-st.throttle.hdr -ovr 1000 -it -st hgrm -of throttle-st</span>" -> we get <span style="font-family: "courier new" , "courier" , monospace;">throttle-st.hgrm</span></li>
<li>Summarize the service time histograms:"<span style="font-family: "courier new" , "courier" , monospace;">hdrsum -if .*.-rt.throttle.hdr -ovr 1000 -it -st hgrm -of throttle-rt</span>" -> we get <span style="font-family: "courier new" , "courier" , monospace;">throttle-rt.hgrm</span></li>
<li><span style="font-family: inherit;">Load them into the <a href="http://hdrhistogram.github.io/HdrHistogram/plotFiles.html" target="_blank">google charts provided in HdrHistogram</a>.</span></li>
</ol>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXFCkU-NEGDSov03-dn2PLnq29I56K2_IBFywSaMk2XcSaazbuhkP4mt7NdHHAcxPKZAwM9j60-bAD1MXHkkBrko0dQ82aB8gcvyMUfRwM9xSTjAjX0IjnjeaV3ebxu-66ntz7_jpCdYo/s1600/Histogram+%25282%2529.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXFCkU-NEGDSov03-dn2PLnq29I56K2_IBFywSaMk2XcSaazbuhkP4mt7NdHHAcxPKZAwM9j60-bAD1MXHkkBrko0dQ82aB8gcvyMUfRwM9xSTjAjX0IjnjeaV3ebxu-66ntz7_jpCdYo/s1600/Histogram+%25282%2529.png" /></a></div>
<div>
Take the red pill and you can keep your estimate, take the blue pill and you might keep your job.<br />
This is a fine demonstration of just how far from schedule operations can get without service time measurement registering an issue. In other words the red line, service time is the measurement with coordinated-omission, the blue is measurement which does not suffer from it.<br />
If you are finally convinced that you should be looking at response time instead of service time you can skip the <b>throttle</b> mode and move on straight to the <span style="font-family: "courier new" , "courier" , monospace;"><b>fixed</b></span> mode. The difference is only in what view of the world you want to see in your summary.<br />
Finally, you can take the histogram logs and graph the latencies over time using <a href="https://github.com/ennerf/HdrHistogramVisualizer" target="_blank">HdrHistogramVisualizer</a>. It currently does not handle tags well, so you'll need to split your logs first, but after that you can generate graphs as funky as this one (this is a longer run with fixed 10K rate, plotting the maximum latency for inserts at the top):<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ ./cassandra-stress user profile=../cqlstress-example.yaml duration=360s ops\(insert=1,simple1=9\) -mode native cql3 protocolVersion=2 -rate <b>fixed</b>=10000/s threads=3 -log interval=10s hdrfile=fixed.hdr</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ hdrspl -if fixed.hdr</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8HKeSygeOWNP91RahfitXl7T-DKf1bYbq90KEx-Y6LKe5e28zudtpR6l_54m-BflkqrgLUFt3hQOKbyj47vx41eaI3DgNGhlrDciCdPaBamaX1uEX11KqisISsEqAQtQbiLnhItohKrU/s1600/fixed-insert-max.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8HKeSygeOWNP91RahfitXl7T-DKf1bYbq90KEx-Y6LKe5e28zudtpR6l_54m-BflkqrgLUFt3hQOKbyj47vx41eaI3DgNGhlrDciCdPaBamaX1uEX11KqisISsEqAQtQbiLnhItohKrU/s1600/fixed-insert-max.png" /></a><br />
This tells an interesting story, it seems as if the server was coping alright with the load to begin with, but hit a bump (GC? compaction?) after 225 seconds, and an even larger bump slightly later. That's lovely to know. I'm not much of a UI wiz and I'm sure some readers out there can make some valuable PRs to this project ;-)<br />
<br />
<h3>
Summary: What's new?</h3>
</div>
<div>
Given that with recent version of Cassandra Stress you can now test older versions of Cassandra as well (using the <span style="font-family: "courier new" , "courier" , monospace;">protocolVersion</span> parameter as demonstrated above), you can stop using your crummy old version of Cassandra Stress, build trunk from source to get the following benefits:</div>
<div>
<ul>
<li>HdrHistogram latency capturing and logging. Set the following options to get your very own histogram log: "<span style="font-family: "courier new" , "courier" , monospace;">-log interval=10s hdrfile=myawesomelog.hdr</span>".</li>
<li>Have a look at the HdrLogProcessing project for some handy utilities to help you slice and dice the data. They are pretty simple and you can feel free to build/contribute your own.</li>
<li>2 new load generation modes: <b>throttle</b> and <b>fixed</b> now replace the old <b>limit</b>. Use <b>fixed</b> to get a view on your cluster's response time under attempted constant load.</li>
</ul>
</div>
<div>
With HdrHistogram support now available in many languages, you can easily build post processing utilities for analysis in a language of your choice.<br />
Have fun storming the castle!<br />
<br />
Many many thanks to <a href="https://twitter.com/tjake" target="_blank">Jake Luciany</a> of the Apache Cassandra project for reviewing the PR, helping to shape it, and merging it it. Jake is an Open Source hero, buy that man a drink on sight!<br />
Both Jake and <a href="https://twitter.com/chbatey" target="_blank">Chris Batey</a> reviewed this post, if any errors remain it is down to their disgustingly sloppy and unprofessional ways, please let me know and I shall have words with their parents.<br />
<br /></div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com1tag:blogger.com,1999:blog-5171098727364395242.post-26789666646962788002016-06-23T12:45:00.000+01:002018-07-10T14:28:26.464+01:00The Pros and Cons of AsyncGetCallTrace Profilers<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsDR2GscFwL-nQITSst3lHF5lDXUuHrz_42GX76sDFXX2si-XVwVumI24L1Xy85rCs5aV1F2ZRb-evfPb8sAoAWiUWguZaQfDZVzz-lEBz309UQkj-y8m33jaBnd7mjh2jCAIhyz8KNEg/s1600/the-pros-and-cons-of-hitch-hiking-5088dafa7fc2f.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsDR2GscFwL-nQITSst3lHF5lDXUuHrz_42GX76sDFXX2si-XVwVumI24L1Xy85rCs5aV1F2ZRb-evfPb8sAoAWiUWguZaQfDZVzz-lEBz309UQkj-y8m33jaBnd7mjh2jCAIhyz8KNEg/s200/the-pros-and-cons-of-hitch-hiking-5088dafa7fc2f.jpg" width="200" /></a>So, going on from my <a href="http://psy-lob-saw.blogspot.co.za/2016/02/why-most-sampling-java-profilers-are.html" target="_blank">whingy post on safepoint bias</a>, where does one go to get their profiling kicks? One option would be to use an OpenJDK internal API call AsyncGetCallTrace to facilitate non-safepoint collection of stack traces.<br />
AsyncGetCallTrace is NOT official JVM API. It's not a comforting place to be for profiler writers, and was only implemented on OpenJDK/Oracle JVMs originally (Zing has recently started supporting AGCT to enable support for Solaris Studio and other profilers, I will write a separate post on Studio). It's original use case was for Solaris Studio, and it provides the following API (see <a href="http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/tip/src/share/vm/prims/forte.cpp" target="_blank">forte.cpp</a>, the name is a left over from the Forte Analyzer days). Here's what the API adds up to:<br />
<script src="https://gist.github.com/nitsanw/dfe809a21869ccbd0e8d7b56e0079432.js"></script><br />
Simple right? You give it a <span style="font-family: "courier new" , "courier" , monospace;"><i>ucontext</i></span> and it fills in the trace with call frames (or it gets the hose again).<br />
<br />
<h3>
A Lightweight Honest Profiler</h3>
The 'async' in the name refers to the fact that AGCT is safe to call in a signal handler. This is mighty handy in a profiling API as it means you can implement your profiler as follows:<br />
<ol>
<li>In a JVMTI agent, register a signal handler for signal X.</li>
<li><span style="font-family: inherit;"><a href="http://man7.org/linux/man-pages/man2/setitimer.2.html" target="_blank">Setup a timer</a>, triggering the signal X at the desired sample frequency. Honest Profiler is using the<span style="font-family: inherit;"> <b style="font-size: 16px;">ITIMER_PROF </b><span style="font-size: 16px;">option, which means we'll get signalled based on CPU time. The signal will be sent to the process and one of the running threads will end up being interrupted and calling into our signal handler. Note that this assumes the OS will distribute the signals fairly between threads so we will get a fair sample of all running threads.</span></span></span></li>
<li>From signal handler, call AGCT: Note that the interrupted thread (picked at 'random' from the threads currently executing on CPU) is now running your signal handler. The thread is NOT AT A SAFEPOINT. It may not be a Java thread at all.</li>
<li>Persist the call trace: Note that when running in a signal handler only 'async' code is legal to run. This means for instance that any blocking code is forbidden, including malloc and IO.</li>
<li>WIN!</li>
</ol>
The <span style="font-family: "courier new" , "courier" , monospace;"><i>ucontext</i></span> ingredient is the very same context handed to you by the signal handler (your signal handler is a callback of the signature <i><span style="font-family: "courier new" , "courier" , monospace;">handle(int signum, siginfo_t *info, void *context)</span></i>). From it AGCT will dig up the instruction/frame/stack pointer values at the time of the interrupt and do it's best to find out where the hell you landed.<br />
This exact approach was followed by Jeremy Manson, who explained the infrastructure and <a href="https://code.google.com/archive/p/lightweight-java-profiler/" target="_blank">open sourced a basic profiler</a> (in a proof of concept, non commital sort of way). His great series of posts on the matter:<br />
<ul>
<li><a href="http://jeremymanson.blogspot.co.za/2007/05/profiling-with-jvmtijvmpi-sigprof-and.html" target="_blank">Profiling with JVMTI/JVMPI, SIGPROF and AsyncGetCallTrace</a></li>
<li><a href="http://jeremymanson.blogspot.co.za/2007/06/more-about-profiling-with-sigprof.html" target="_blank">More about profiling with SIGPROF</a></li>
<li><a href="http://jeremymanson.blogspot.co.za/2007/06/more-thoughts-on-sigprof-jvmti-and.html" target="_blank">More thoughts on SIGPROF, JVMTI and stack traces</a></li>
<li><a href="http://jeremymanson.blogspot.co.za/2010/07/why-many-profilers-have-serious.html" target="_blank">Why Many Profilers have Serious Problems (More on Profiling with Signals)</a></li>
</ul>
The same code was then picked up by Richard Warburton and further improved and stabilized in <a href="https://github.com/RichardWarburton/honest-profiler" target="_blank">Honest-Profiler</a> (to which I have made some contributions). Honest Profiler is an effort to make that initial prototype production ready, and compliment it with some tooling to make the whole thing usable. The serialization in a signal handler issue is resolved by using a lock-free MPSC ring-buffer of pre-allocated call trace structs (pointing to preallocated call frame arrays). A post-processing thread then reads the call traces, collects some extra info (like converting BCI to line number, jmethodIds into class names/file names etc) and writes to a log file. See the projects wiki for more details.<br />
The log file is parsed offline (i.e. by some other process, on another machine, at some other time. Note that you can process the file while the JVM is still profiling, you need not wait for it to terminate.) to give you the hot methods/call tree breakdown. I'm going to use Honest-Profiler in this post, so if you want to try this out at home you're going to have to go and <a href="https://github.com/RichardWarburton/honest-profiler/wiki/How-to-build" target="_blank">build it</a> to experiment locally (works on <a href="http://openjdk.java.net/" target="_blank">OpenJDK</a>/<a href="https://www.azul.com/products/zulu/" target="_blank">Zulu</a> 6/7/8 + <a href="http://www.oracle.com/technetwork/java/javase/downloads/index.html" target="_blank">Oracle JVM</a>s + recent <a href="https://www.azul.com/products/zing/" target="_blank">Zing</a> builds). I'll do some comparisons for the same experiments with JMC, you'll need an Oracle JVM (1.7u40 and later) to try that out.<br />
<br />
<h3>
What Does AGCT Do?</h3>
<div>
By API we can say AGCT is a mapping between instruction/frame/stack pointer and a call trace. The call trace is an array of Java call frames (jmethodId, BCI). To produce this mapping the following process is followed:</div>
<div>
<ol>
<li>Make sure thread is in 'walkable' state, in particular not when:</li>
<ul>
<li>Thread is not a Java thread.</li>
<li>GC is active</li>
<li>New/uninitialized/just about to die. I.e. threads that are either before or after having Java code running on them are of no interest.</li>
<li>During a deopt</li>
</ul>
<li>Find the current/last Java frame (as in actual frame on the stack, revisit Operating Systems 101 for definitions of stacks and frames):</li>
<ul>
<li>The instruction pointer (commonly referred to as the PC - Program Counter) is used to look up a matching Java method (compiled/interpreter). The current PC is provided by the signal context.</li>
<li>If the PC is not in a Java method we need to find the last Java method calling into native code.</li>
<li>Failure is an option! we may be in an 'unwalkable' frame for all sorts of reasons... This is quite complex and if you must know I urge you to get comfy and dive into the maze of relevant code. Trying to qualify the top frame is where most of the complexity is for AGCT.</li>
</ul>
<li>Once we have a top frame we can fill the call trace. To do this we must convert the real frame and PC into:</li>
<ul>
<li>Compiled call frames: The PC landed on a compiled method, find the BCI (Byte Code Index) and record it and the jMethodId</li>
<li>Virtual call frames: The PC landed on an instruction from a compiled inlined method, record the methods/BCIs all the way up to the framing compiled method</li>
<li>Interpreted call frames</li>
<li>From a compiled/interpreted method we need to walk to the calling frame and repeat until we make it to the root of the Java call trace (or record enough call frames, whichever comes first)</li>
</ul>
<li>WIN!</li>
</ol>
Much like medication list of potential side effects, the error code list supported by a function can be very telling. AGCT supports the following reasons for not returning a call trace:<br />
<script src="https://gist.github.com/nitsanw/8a40494c54b3854aca2f62b20522408b.js"></script><br />
While this data is reported from AGCT, it is often missing from the reports based on it. More on that later.<br />
<br /></div>
<h3>
The Good News!</h3>
<div>
So, looking at the bright side, we can now see between safepoint polls!!! How awesome is that? Lets see exactly how awesome by running the benchmarks which we could not measure correctly with the safepoint biased profiler in the previous post.<br />
[<i>Note: Honest-Profiler reports (t X,s Y) which reflect the total % of stack trace samples containing this method+line vs. the self % of samples in which this method+line is the leaf. The output is sorted by self.</i>]:</div>
<div>
<script src="https://gist.github.com/nitsanw/2518888fa21ec6a891490c072b30c082.js"></script></div>
<div>
<br />
Sweet! Honest Profiler is naming the right methods as the hot methods, and nailing the right lines where the work is happening.</div>
<div>
Let's try the copy benchmark, this time comparing the Honest-Profiler result with the JMC result:</div>
<div>
<script src="https://gist.github.com/nitsanw/6ba4d4d6dd8b26c3a8379945268bf0dd.js"></script></div>
<div>
Note the difference in reporting switching on the "-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints" flags makes to JMC. Honest-Profiler now (<a href="https://github.com/RichardWarburton/honest-profiler/pull/133" target="_blank">recent change</a>, hot off the press) causes the flag to be on without requiring user intervention, but JMC does not currently do that (this is understandable when attaching after the process started, but should perhaps be default if the process is started with recording on).</div>
<div>
Further experimentation and validation exercises for the young and enthusiastic:</div>
<div>
<ul>
<li>Compare the Honest-Profiler results with JFR/JMC results on larger applications. What's different? Why do you think it's different?</li>
<li>Run with -XX:+PrintGCApplicationStoppedTime to see no extra safepoints are caused by the profiling (test with profiler on/off).</li>
<li>The extra-studious of y'all can profile the other cases discussed in previous post to see they similarly get resolved.</li>
</ul>
</div>
<div>
While this is a step forward, we still got some issues here... </div>
<h3>
Collection errors: Runtime Stubs</h3>
<div>
As we can see from the list of errors, any number of events can result in a failed sample. A particularly nasty type of failed sample is when you hit a runtime generated routine which AGCT fails to climb out of. Let see what happens when we don't roll our own array copy:<br />
<div>
<script src="https://gist.github.com/nitsanw/4696f30d663d8601a3fffd3b0d8938e5.js"></script>
<br />
<div>
<br />
Now, this great unknown marker is a recent addition to Honest-Profiler, which increases it's honesty about failed samples. It indicates that for 62.9% of all samples, AGCT could not figure out what was happening and returned <i><span style="font-family: "courier new" , "courier" , monospace;">ticks_unknown_java</span></i>. Given that there's very little code under test here we can deduce the missing samples all fall within System.arrayCopy (you can pick a larger array size to further prove the point, or use a native profiler for comparison).</div>
<div>
Profiling the same benchmarks under JMC will not report failed samples, and the profiler will divide the remaining samples as if the failed samples never happened. Here's the JMC profile for systemArrayCopy:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Number of samples(over 60 seconds) : 2617</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Method::Line Samples %</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">systemArrayCopy_avgt_jmhStub(...) :: 165 1,729 66.043</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">systemArrayCopy() :: 41 331 12.643</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">systemArrayCopy() :: 42 208 7.945</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">systemArrayCopy_avgt_jmhStub(...) :: 166 93 3.552</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">systemArrayCopy_avgt_jmhStub(...) :: 163 88 3.361</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: inherit;">JMC is reporting a low number of samples (the total is sort of available in the tree view as the number of samples in the root), but without knowing what the expected number of samples should be this is very easy to miss. This is particularly true for larger and noisier samples from real applications collected over longer period of time.</span></div>
<div>
<span style="font-family: inherit;">Is this phenomena isolated to System.arrayCopy? Not at all, here's crc32 and adler32 as a further comparison:</span></div>
<script src="https://gist.github.com/nitsanw/a96f79921fe2248ee8d0d10292abaece.js"></script>
<br />
<div>
<br /></div>
<div>
<span style="font-family: inherit;">What do Crc32 and System.arrayCopy have in common? They are both JVM intrinsics, replacing a method call (Java/native don't really matter, though in this case both are native) with a combination of inlined code and a call to a JVM runtime generated method. This method call is not guarded the same way a normal call into native methods is and thus the AGCT stack walking is broken.</span><br />
<span style="font-family: inherit;">Why is this important? The reason these methods are worth making into intrinsics is because they are sufficiently important bottlenecks in common applications. Intrinsics are like tombstones for past performance issues in this sense, and while the result is faster than before, the cost is unlikely to be completely gone.</span><br />
<span style="font-family: inherit;">Why did I pick CRC32? because I recently spent some time profiling Cassandra. Version 2 of Cassandra uses adler32 for checksum, while version 3 uses crc32. As we can see from the results above, it's potentially a good choice, but if you were to profile Cassandra 3 it would look better than it actually is because all the checksum samples are unprofilable. Profiling with a native profiler will confirm that the checksum cost is still a prominent element of the profile (of the particular setup/benchmark I was running).</span></div>
<div>
<span style="font-family: inherit;"><b>AGCT profilers are blind to runtime stubs (some or all, this may get fixed in future releases...). Failed samples are an indication of such a blind spot.</b></span><br />
<br />
Exercise to the reader:<br />
<br />
<ul>
<li>Construct a benchmark with heavy GC activity and profile. The CPU spent on GC will be absent from you JMC profile, and should show as <span style="font-family: "courier new" , "courier" , monospace; font-style: italic;">ticks_GC_active </span><span style="font-family: inherit;">in your Honest-Profiler profile.</span></li>
<li><span style="font-family: inherit;">Construct a benchmark with heavy compiler activity and profile. As above look for compilation CPU. There's no compilation error code, but you should see allot of </span><span style="font-family: "courier new" , "courier" , monospace;"><i>ticks_unknown_not_Java</i></span> which indicate a non-Java thread has been interrupted (<a href="https://github.com/RichardWarburton/honest-profiler/issues/138" target="_blank">this is a conflated error code, we'll be fixing it soon</a>).</li>
<li>Extra points! Construct a benchmark which is spending significant time deoptimising and look for the <i><span style="font-family: "courier new" , "courier" , monospace;">ticks_deopt</span></i> error in your Honest-Profiler profile.</li>
</ul>
<br />
<h3>
<b style="font-family: inherit;">Blind Spot: Sleeping code</b></h3>
</div>
<div>
Since sleeping code doesn't actually consume CPU it will not appear in the profile. This can be confusing if you are used to the JVisualVM style reporting where all stacks are reported including WAITING (or sometimes RUNNING Unsafe.park). Profiling sleeping or blocked code is not something these profilers cover. I mention this not because they promise to do this and somehow fail to deliver, but because it's a common pitfall. For Honest-Profiler <a href="https://github.com/RichardWarburton/honest-profiler/issues/126" target="_blank">this feature</a> should help highlight mostly off-CPU applications as such by comparing expected and actual samples (the delta is signals which were not delivered to the process as it was not running).</div>
<div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"></span><br />
<h3>
<span style="font-family: inherit;">Error Margin: Skidding + inlining</span></h3>
</div>
<div>
<span style="font-family: inherit;">I'll dive deeper into this one another time, since other profilers share this issue. In essence the problem is that instruction profiling is not accurate and we can often encounter a skid effect when sampling the PC(program counter). This in effect means the instruction reported can be a number of instructions after the instruction where the time is spent. Since AsyncGetCallTrace relies on PC sampling to resolve the method and code line we get the same inaccuracy reflected through the layer of mapping (PC -> BCI -> LOC) but where on the assembly level we deal with a single blob of instructions where the proximity is meaningful, in converting back to Java we have perhaps slipped from one inlined method to the next and the culprit is no longer in a line of code nearby.</span></div>
<div>
<span style="font-family: inherit;">To be continued...</span><br />
</div>
<h3>
Summary: What is it good for?</h3>
<div>
AsyncGetCallTrace is a step up from GetStackTraces as it operates at lower overheads and does not suffer from safepoint bias. It does require some mental model adjustments:</div>
<div>
<ol>
<li>-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints : if this is not on you are still hobbled by the resolution of the debug data. This is different from safepoint bias, since you actually can sample anywhere, but the translation from PC to BCI kills you. I've not seen these flags result in measurable overhead, so I'm not convinced the default value is right here, but for now this is how it is. Honest-Profiler takes care of this for you, with JMC you'll need to add it to your command line.</li>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDSCtHE_E73oWqcsAVfX0iuhIvVbAqC9q9tYyPTx4bJ3lZBG66L9-ATLxrexE7_JRx_uOmfMRItTmsrEvihvWtkFSVpB0MFfKcshISaXod5HCYWlvb-GxXo-WD0AWKWpbARASr_raZm3M/s1600/the_good__the_bad_and_the_ugly_by_felonland-d46b6az.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDSCtHE_E73oWqcsAVfX0iuhIvVbAqC9q9tYyPTx4bJ3lZBG66L9-ATLxrexE7_JRx_uOmfMRItTmsrEvihvWtkFSVpB0MFfKcshISaXod5HCYWlvb-GxXo-WD0AWKWpbARASr_raZm3M/s400/the_good__the_bad_and_the_ugly_by_felonland-d46b6az.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">But seriously guys, we should be using Fortran for this</td></tr>
</tbody></table>
<li>Each sample is for a <b>single, on CPU, thread:</b> This is very different from the GetStackTraces approach which sampled all threads. It means you are getting less traces per sample, and are completely blind to sleeping/starved threads. Because the overhead is that much lower you can compensate by sampling more frequently, or over longer periods of time. <b>This is a good thing</b>, sampling all the threads at every sample is a very problematic proposition given the number of threads can be very large.</li>
</ol>
<div>
AsyncGetCallTrace is great for profiling most 'normal' Java code, where hotspots are in your Java code, or rather in the assembly code they result in. This seems to hold in the face of most optimizations to reasonable accuracy (but may on occasion be off by quite a bit...).
</div>
<br />
AsyncGetCallTrace is of limited use when:</div>
<div>
<ol>
<li>Large numbers of samples fail: This can mean the application is spending it's time in GC/Deopts/Runtime code. Watch for failures. I think currently honest-profiler offers better visibility on this, but I'm sure the good folks of JMC can take a hint.</li>
<li>Performance issue is hard to glean from the Java code. E.g. see the <a href="http://psy-lob-saw.blogspot.com/2015/07/jmh-perfasm.html" target="_blank">issue discussed in a previous post using JMH perfasm</a> (false sharing on the class id in an object header making the conditional inlining of an interface call very expensive). </li>
<li>Due to instruction skid/compilation/available debug information the wrong Java line is blamed. This is potentially very confusing in the presence of inlining and code motion.</li>
</ol>
</div>
<div>
Using Honest-Profiler you can now profile Open/Oracle JDK6/7/8 applications on Linux and OS X. You can also use it to profile Zing applications on recent Zing builds (version 15.05.0.0 and later, all JDKs). Honest-Profiler is lovely, but I would caution readers that it is not in wide use, may contain bugs, and should be used with care. It's a useful tool, but I'm not sure I'd unleash it on my production systems just yet ;-).<br />
JMC/JFR is available on Oracle JVMs only from JDK7u40, but works on Linux, OS X, Windows and Solaris (JFR only). JMC/JFR is free for development purposes, but requires a licence to use in production. Note that JFR collects a wealth of performance data which is beyond the scope of this post, I whole heartedly recommend you give it a go.<br />
<br />
A big thank you to all the kind reviewers: <a href="https://twitter.com/jpbempel" target="_blank">JP Bempel</a>, <a href="https://twitter.com/switchology" target="_blank">Doug Lawrie</a>, <a href="https://twitter.com/hirt" target="_blank">Marcus Hirt</a> and <a href="https://twitter.com/RichardWarburto" target="_blank">Richard Warburton</a>, any remaining errors will be deducted from their bonuses directly.</div>
</div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com2tag:blogger.com,1999:blog-5171098727364395242.post-67945332562227546952016-03-21T10:16:00.001+00:002016-03-22T07:32:08.096+00:00GC 'Nepotism' And Linked QueuesI've just run into this issue this week, and it's very cute, so this here is a summary. Akka has their own MPSC linked queue implementation, and this week in was suggested they'd swap to using JCTools. The context in which the suggestion was made was a recently closed bug with the mystery title: <a href="https://github.com/akka/akka/issues/19216" target="_blank">AbstractNodeQueue suffers from nepotism</a><br />
An Akka user claimed to not be suffering from the bug because they were using JCTools, but as it turned out the issue was indeed present so I had to fix it etc... But what the hell is GC nepotism? You won't find it in your GC textbooks, but it's a catchy term for a very real problem.<br />
<br />
<h3>
Your Dead Olds Relatives Can Promote You!</h3>
<div>
In the open JDK a generational GCs, there are distinct generations, Old and New. The 2 generations get collected separately, but are not independent from each other. In particular, references from NewGen to OldGen will keep OldGen references alive through an OldGen collection cycle, and similarly references from OldGen to NewGen will keep objects alive through a NewGen collection. This is all quite straight forward, but has a surprising implication:</div>
<blockquote class="tr_bq">
Dead OldGen objects will keep dead NewGen objects they reference alive until collected by an OldGen.</blockquote>
The reverse is also true, but since the whole notion of generational GC is based on the assumption that new gen is collected far more often then the old gen the above use case is the real issue. How does this effect you? Is this even a real problem?<br />
Turns out it's real enough to have convinced Doug Lea to fix a particular manifestation of it, but also subtle enough to have slipped past him (and me, and the Akka guys, and Martin Thompson, and others no doubt) in the first place.<br />
<br />
<h3>
Linked Queues: Pathological Nepotism</h3>
<div>
Consider the following LinkedQueue implementation:</div>
<div>
<script src="https://gist.github.com/nitsanw/258b851242d9c7fd6665.js"></script></div>
<div>
The following invariants are maintained:</div>
<div>
<ul>
<li>head == tail <=> queue is empty</li>
<li>head.next == null</li>
<li>tail.value == null</li>
<li>In summary: queue is empty <=> (head == tail && tail.next == null && tail.value == null)</li>
</ul>
</div>
<div>
The nodes are created and linked together on offer, and dereferenced as we poll. The poll method clears MUST clear the value because the queue retains the last node. If we were to let the last node keep the value we will have effectively leaked this value reference.</div>
<div>
Ignoring the values, imagine the queue is part of the live set and we have enqueued a few elements, the arrow denotes what next references:<br />
<span style="font-family: "courier new" , "courier" , monospace;">head -> N3 -> null</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">tail -> N0 -> N1 -> N2 -> N3</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">If we now dequeue the elements we'll get back to empty, brackets denote live referenced from dead:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">head -> N3 -> null</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">tail -> N3 -> null</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">DEAD: </span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">N0 -> N1 -> N2 </span><span style="font-family: "courier new" , "courier" , monospace;">(-> N3)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">Dead objects are traditionally considered the GC problem, and <u>cleaning up about to be collected objects is considered an anti-pattern</u>. After all, what is the point of cleaning up garbage? you are just duplicating work. Indeed in a single generation collector this is a non-issue.</span></div>
<div>
<span style="font-family: inherit;">But consider for a moment the possibility that one of the above nodes was promoted into the OldGen. This is something which may not happen in short lived tests, or even 'lucky' long lived tests, but is obviously quite possible. We will have:</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">head -> N3 -> null</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">tail -> N3 -> null</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Zombie NEW: </span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">N1 -> N2 (-> N3)</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">DEAD OLD: N0 (-> </span><span style="font-family: "courier new" , "courier" , monospace;">N1 -> N2 (-> N3)</span><span style="font-family: "courier new" , "courier" , monospace;">)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">The young are alive, as far as the old are concerned. The old are dead, but their promotion gives life to their 'children' until they get collected. To demonstrate this issue I've written the following program: </span></div>
<div>
<script src="https://gist.github.com/nitsanw/fe9d5db85d035ecb331b.js"></script></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">If you look at the GC logs, or plug in Visual VM with the Visual GC plugin you will notice the young GC is entirely ineffective. Everything gets promoted to old, until an old GC clears the lot. After the old GC the problem disappears (in the test program, but is likely to repeat in real programs). This is 'pathological' because ANY promoted node will result in the promotion of ALL following nodes until a GC resolves the issue. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">At this point you might be shaking your head sadly and saying: "Ye thick old fart, surely this is a silly issue which NEVER happens in the real world". I've had my share of heads being shook at me and am not offended, but consider that this issue is real enough for other people:</span><br />
<br />
<ul>
<li><a href="https://github.com/akka/akka/issues/17547" target="_blank">AbstractNodeQueue potential GC leak - many objects get tenured if we have "bursts" of messages</a></li>
<li><a href="https://bugs.openjdk.java.net/browse/JDK-6806875" target="_blank">Young memory "leak" in LinkedBlockingQueue</a> : see also <a href="http://tech.puredanger.com/2009/02/11/linkedblockingqueue-garbagecollection/" target="_blank">this post</a> on the same with further background. See relevant comment on <a href="http://cs.oswego.edu/pipermail/concurrency-interest/2009-February/005836.html" target="_blank">concurrency-interest from Doug Lea</a>, I find the comment "it may be that the effects didn't show up on the tests, machines, VMs I was using. But these are subject to transient changes" in particular reflects the difficulty is exposing the issue in synthetic testing.</li>
<li>This <a href="https://www.youtube.com/watch?v=M9o1LVfGp2A&feature=youtu.be&t=1382" target="_blank">war story from Tony Printezis</a> referring to this as a recurring pattern in Java services (I wholeheartedly recommend you watch Tony's presentation from beginning to end)</li>
</ul>
<h3>
An Exercise To the Reader</h3>
<br />
<span style="font-family: inherit;">My plan was to walk through the way this issue has been fixed in JCTools vs Akka vs Agrona. Each solution reflecting different considerations, but sadly I'm quite short on time for a tidy summary and will give my observations which you can choose to read up on and argue with.</span><br />
<span style="font-family: inherit;">All three implementations follow the algorithm outlined by <a href="http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue" target="_blank">D. Vyukov</a>. Note that Agrona is ignoring the Queue interface requirement for poll(return null iff queue is empty).</span><br />
<br />
<ul>
<li>In Akka the next reference is nulled on poll. This solves the issue, but at the cost of further weakening the accuracy of the size() method.</li>
<li>In JCTools the next reference is set to point to the discarded node. This helps in telling the difference between a removed node and the head. This piece of information is used in the size method to skip gaps in the size counted chain when racing with the consumer thread. The result is a more accurate size estimate. The change does come at a cost to the size method however as we are no longer able to rely on the chain remaining whole throughout the size operation.</li>
<li>In Agrona, the next reference is nulled as in Akka. A further effort is made to reuse a special empty node to avoid repeating promotion of transient nodes in empty queues. The effort is arguably worth while on systems with large numbers of active but mostly empty queues.</li>
</ul>
<br /></div>
<h3>
Open questions/discussion:</h3>
<div>
<ul>
<li>How common is this issue for 'classic' data structures? it seems reasonable that a host of queues/stacks/tree data structures are similarly vulnerable. A scan of JDK sources suggests this is an known pitfall to them, but I would bet many third party frameworks writers didn't get the memo.</li>
<li>How common is this issue for idiomatic 'user' Java code? given the tendency towards composition rather than inheritance and the drive towards smaller objects, we may argue that user object graphs often approximate trees. If we have short-lived 'trees' a parent node can certainly drag a useless chunk of state to the old gen. If a reference bridges 2 trees we may have something approximating the pathological behaviour above. I'm not aware of any studies trying to determine the impact of nepotism on user code, part of the problem would be the definition of 'typical' user code.</li>
<li>Where does the term 'nepotism' come from? I made an attempt at tracing the origins, but got nowhere. The problem seems to be documented at least 8 years ago, but the term nepotism is not used by most writing. [UPDATE: it seems the term can be tracked down to a GC paper published in 1991:<a href="https://www.researchgate.net/publication/220404180_An_Adaptive_Tenuring_Policy_for_Generation_Scavengers" target="_blank">An Adaptive Tenuring Policy for Generation</a>]</li>
</ul>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com16tag:blogger.com,1999:blog-5171098727364395242.post-7086702037146446692016-02-24T07:07:00.000+00:002016-06-30T13:21:44.867+01:00Why (Most) Sampling Java Profilers Are Fucking Terrible <div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwdyc6zQY04Eq14p-Pd2ihFTWlF9-mFXdX22lmvZ4UCWDdeK3TCOhVhk1KjFf432dOFCIpaFPnXn4YMDhT0IGNe4bW2S72fwGgAwjdHC7qyJrJXaFRyEzUZlZVigXNdbmdIgPIuJnqBLc/s1600/sad-panda.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwdyc6zQY04Eq14p-Pd2ihFTWlF9-mFXdX22lmvZ4UCWDdeK3TCOhVhk1KjFf432dOFCIpaFPnXn4YMDhT0IGNe4bW2S72fwGgAwjdHC7qyJrJXaFRyEzUZlZVigXNdbmdIgPIuJnqBLc/s1600/sad-panda.jpg" /></a></div>
This post builds on the basis of a <a href="http://psy-lob-saw.blogspot.com/2015/12/safepoints.html" target="_blank">previous post on safepoints</a>. If you've not read it you might feel lost and confused. If you have read it, and still feel lost and confused, and you are certain this feeling is related to the matter at hand (as opposed to an existential crisis), please ask away.<br />
So, now that we've established what safepoints are, and that:<br />
<ol>
<li>Safepoint polls are dispersed at fairly arbitrary points (depending on execution mode, mostly at uncounted loop back edge or method return/entry).</li>
<li>Bringing the JVM to a global safepoint is high cost</li>
</ol>
We have all the information we need to conclude that profiling by sampling at a safepoint is perhaps a bit shite. This will not come as a surprise to some, but this issue is present in the most commonly used profilers. According to this <a href="http://pages.zeroturnaround.com/RebelLabs---All-Report-Landers_Developer-Productivity-Report-2015.html?utm_source=performance-survey-report&utm_medium=allreports&utm_campaign=rebellabs&utm_rebellabsid=97" target="_blank">survey</a> by RebelLabs this is the breakdown:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKApO4NZjNtutcp7Uuy8zd377UYTloJAnGWognTBbND-pM_UGYvHezVD7ces7Fmyyx3tpg6O0BTxSTn13U38Lkb8QlSE29tvbEknCYzZWdBy188jo0FHwYoP0t1veBCDgniV9hzDNZx2U/s1600/which-profiler.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKApO4NZjNtutcp7Uuy8zd377UYTloJAnGWognTBbND-pM_UGYvHezVD7ces7Fmyyx3tpg6O0BTxSTn13U38Lkb8QlSE29tvbEknCYzZWdBy188jo0FHwYoP0t1veBCDgniV9hzDNZx2U/s1600/which-profiler.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
VisualVM, NB Profiler(same thing), YourKit and JProfiler all provide a sampling CPU profiler which samples at a safepoint. Seeing how this is a rather common issue, lets dig into it.<br />
<br />
<h3>
How Sampling Execution Profilers Work (in theory)</h3>
<div>
Sampling profilers are supposed to approximate the 'time spent' distribution in our application by collecting samples of where our application is at different points in time. The data collected at each sample could be:</div>
<div>
<ul>
<li>current instruction</li>
<li>current line of code</li>
<li>current method</li>
<li>current stack trace</li>
</ul>
The data can be collected for a single thread or all threads at each sample. What do we need to hold for sampling to work?<br />
<blockquote class="tr_bq">
"However, for sampling
to produce results that are comparable to a full (unsampled) profile,
the following two conditions must hold.
<b>First, we must have a large number of samples to get statistically
significant results.</b> For example, if a profiler collects only a single
sample in the entire program run, the profiler will assign 100% of
the program execution time to the code in which it took its sample
and 0% to everything else. [...]<br />
<b>Second, the profiler should sample all points in a program run
with equal probability.</b> If a profiler does not do so, it will end up with bias in its profile. For example, let’s suppose our profiler can
only sample methods that contain calls. This profiler will attribute
no execution time to methods that do not contain calls even though
they may account for much of the program’s execution time." - from <a href="http://plv.colorado.edu/papers/mytkowicz-pldi10.pdf" target="_blank">Evaluating the Accuracy of Java Profilers</a>, we'll get back to this article in a bit</blockquote>
</div>
<div>
<ol>
</ol>
That sounds easy enough, right?</div>
<div>
Once we have lots of samples we can construct a list of hot methods, or even code lines in those methods (if the samples report it), we can look at the samples distributed on the call tree (if call traces are collected) and have an awesome time!</div>
<div>
<br /></div>
<h3>
How Do Generic Commercial Java Sampling Execution Profilers Work?</h3>
Well, I can sort of reverse engineer here from different solutions, or read through open source code bases, but instead I'll offer unsupported conjecture and you are free to call me out on it if you know better. Generic profilers rely on the JVMTI spec, which all JVMs must meet:<br />
<ul>
<li>JVMTI offers only safepoint sampling stack trace collection options (GetStackTrace for the calling thread doesn't require a safepoint, but that is not very useful to a profiler. On Zing GetStackTrace to another thread will bring only that thread to a safepoint.). It follows that vendors who want their tools to work on ALL JVMs are limited to safepoint sampling.</li>
<li>You hit a global safepoint whether you are sampling a single thread or all threads (at least on OpenJDK, Zing is slightly different but as a profiler vendor OpenJDK is your assumption.). All profilers I looked into go for sampling all threads. AFAIK they also do not limit the depth of the stack collected. This amounts to the following JVMTI call: <i><span style="font-family: "courier new" , "courier" , monospace;">JvmtiEnv::GetAllStackTraces(0, &stack_info, &thread_count)</span></i></li>
<li><span style="font-family: inherit;">So this adds up to: setup a timer thread which triggers at 'sampling_interval' and gathers all stack traces.</span></li>
</ul>
This is bad for several reasons, some of which are avoidable:<br />
<ol>
<li>Sampling profilers need samples, so it is common to set sampling frequency is quite high (usually 10 times a second, or every 100ms). It's instructive to set the -XX:+PrintGCApplicationStoppedTime and see what sort of pause time this introduces. It's not unusual to see a few milliseconds pause, but YMMV(depending on number of threads, stack depth, TTSP etc). A 5ms pause every 100ms will mean a 5% overhead (actual damage is likely worse than that) introduced by your profiler. You can usually control the damage here by setting the interval longer, but this also means you will need a longer profiling period to get a meaningful sample count.</li>
<li>Gathering <b>full</b> stack traces from <b>all</b> the threads means your safepoint operation cost is open ended. The more threads your application has (think application server, SEDA architecture, lots of thread pools...), and the deeper your stack traces (think Spring and Co.) the longer your application will wait for a single thread to go round taking names and filling forms. This was clearly demonstrated in the previous post. AFAIK, current profilers do nothing to help you here. If you are building your own profiler it would seem sensible to set a limit on either quantities so that you can box your overheads. The JVMTI functionality allows you to query the list of current threads, you could sample all if there's less than a 100 and otherwise pick a random subset of 100 to sample. It would make sense to perhaps bias towards sampling threads that are actually doing something as opposed to threads which spend all their time blocked.</li>
<li>As if all that was not bad enough, sampling at a safepoint is a tad meaningless.</li>
</ol>
Points 1 and 2 are about profiling overheads, which is basically about cost. In my previous post on safepoints I looked at these costs, so there's no point repeating the exercise. Cost may be acceptable for good profiling information, but as we'll see the information is not that great.<br />
Point 3 bears explaining, so off we go in the search for meaning.<br />
<br />
<h3>
Safepoint Sampling: Theory</h3>
<div>
So what does sampling at a safepoint mean? It means only the safepoint polls in the running code are visible. Given hot code is likely compiled by C1/C2 (client/server compilers) we have reduced our sampling opportunities to method exit and uncounted loop backedges. This leads to the phenomena called <b>safepoint bias</b> whereby the sampling profiler samples are biased towards the next available safepoint poll location (this fails the second criteria set out above "<b>the profiler should sample all points in a program run with equal probability</b>").<br />
This may not sound so bad at first, so lets work through a simple example and see which line gets the blame.<br />
<b>NOTE</b>: In all of the following examples I will be using <a href="http://psy-lob-saw.blogspot.co.za/p/jmh-related-posts.html" target="_blank">JMH</a> as the test harness and make use of the 'CompilerControl' annotation to prevent inlining. This will let me control the compilation unit limits, and may seem cruel and unusual, or at least unfair, of me. Inlining decisions in the 'wild' are governed by many factors and it is safe (in my opinion) to consider them arbitrary (in the hands of several compilers/JVM vendors/command line arguments etc.). Inlinining may well be the "mother of all optimizations", but it is a fickle and wily mother at that.<br />
Let's look at something simple:<br />
<script src="https://gist.github.com/nitsanw/d40a165cb1e6f02a30ea.js"></script><br />
This is an easy example to think about. We can control the amount of work in the method by changing the size of the array. We know the counted loop has no safepoint poll in it (verified by looking at the assembly output), so in theory the methods above will have a safepoint at method exit. The thing is, if we let the method above get inlined the end of method safepoint poll will disappear, and the next poll will be in the measurement loop:<br />
<script src="https://gist.github.com/nitsanw/3475ff92ef355de4f3dc.js"></script><br />
So, it would seem reasonable to expect the method to get blamed if it is not inlined, but if it does get inlined we can expect the measurement method to get blamed. Right? very reasonable, but a bit off.<br />
<br />
<h3>
Safepoint Sampling: Reality</h3>
<div>
Safepoint bias is discussed in this nice paper from 2010: <a href="http://plv.colorado.edu/papers/mytkowicz-pldi10.pdf" target="_blank">Evaluating the Accuracy of Java Profilers</a>, where the writers recognize that different Java profilers identify different hotspots in the same benchmarks and go digging for reasons. What they don't do is set up some benchmarks where the hotspot is known and use those to understand what it is that safepoint biased profilers see. They state:</div>
<blockquote class="tr_bq">
"If we knew the “correct” profile for a program run, we could evaluate
the profiler with respect to this correct profile. Unfortunately,
there is no “correct” profile most of the time and thus we cannot
definitively determine if a profiler is producing correct results."</blockquote>
<div>
So, if we construct a known workload... what do these profilers see?</div>
<div>
We'll study that with the JMH safepoint biased profiler "-prof stack". It is much like JVisualVM in the profiles it presents for the same code, and a damn sight more convenient for this study.<br />
<i>NOTE: In the following sections I'm using the term sub-method to describe a method which is being called into from another method. E.g. if method A calls method B, B is a sub method of A. Maybe a better terminology exists, but that's what I mean here.</i></div>
<div>
If we run the samples above we get 2 different hot lines of code (run with '-prof stack:detailLine=true'):<br />
<span style="font-family: "courier new" , "courier" , monospace;"># Benchmark: safepoint.profiling.SafepointProfiling.meSoHotInline</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">....[Thread state: RUNNABLE]...</span></div>
</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 99.6% 99.8% meSoHotInline_avgt_jmhStub:165</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"># Benchmark: safepoint.profiling.SafepointProfiling.meSoHotNoInline</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">....[Thread state: RUNNABLE]...</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 99.4% 99.6% meSoHotNoInline_avgt_jmhStub:163</span></div>
</div>
<div>
<br /></div>
<div>
Neither one in the actually hot method. It seems that the <u>method exit safepoint is not deemed indicative of it's own method, but rather of the line of code from which it was called</u>. So forcing the method under measurement to not inline implicated the calling line of code in the measurement loop, while leaving it to inline meant the back edge of the loop got the blame. It also appears that an uncounted loop safepoint poll is deemed indicative of it's own method.</div>
<div>
We may deduce(but we won't necessarily be correct) that when looking at this kind of profile without line of code data a hot method is indicative of:</div>
<div>
<ol>
<li>Some non-inlined sub method is hot*</li>
<li>Some code (own method? inlined sub method? non-inlined sub method?) in an uncounted loop is hot*</li>
</ol>
Having line of code data can help disambiguate the above cases, but it's not really very useful as line of code data. A hot line of code would be indicative of:</div>
<div>
<ol>
<li>Line has a method call: A method called from this line is hot (or maybe it's inlined sub-methods are)*</li>
<li>Line is a loop back edge: Some code (include inlined submethods) in this loop is hot*</li>
</ol>
<i><span style="font-size: x-small;"> *Does that seem useful? Don't get your hopes up.</span></i><br />
Because we usually have no idea which methods got inlined this can be a bit confusing (you can use -XX:+PrintInlining if you want to find out, but be aware that inlining decisions can change from run to run).<br />
<br /></div>
<h3>
Mind The Gap</h3>
If the above rules held true you could use a safepoint biased profile by examining the code under the blamed node in the execution tree. In other words, it would mean a hot method indicates the hot code lies somewhere in that code or in the method it calls. This would be good to know, and the profile could serve as a starting point for some productive digging. But sadly, these rules do not always hold true. They are missing the fact that the hot code can be anywhere between the indicated safepoint poll and the previous one. Consider the following example:<br />
<script src="https://gist.github.com/nitsanw/919dca8636e32d448916.js"></script><br />
Clearly, the time is spent in the loop before calling setResult, but the profile blames setResult. There's nothing wrong with setResult except that a method it calls into is not inlined, providing our profiler with a blame opportunity. This demonstrates the randomness with which the safepoint poll opportunities present themselves to user code, and shows that the hot code may be anywhere between the current safepoint poll and the previous one. This means that a hot method/line-of-code in a safepoint biased profile are potentially misleading without some idea of where the previous safepoint poll is. Consider the following example:<br />
<script src="https://gist.github.com/nitsanw/549b68bc42f13ea88588.js"></script><br />
The profiler implicates the caller to a cheap method 9 levels down the stack, but the real culprit is the loop at the topmost method. Note that inlining will prevent a method from showing, but not-inlined frames only break the gap between safepoints on return (on OpenJDK anyhow. This is vendor and whim dependent. e.g. Zing puts the method level safepoint on entry for instance and I'm not sure where J9 stands on this issue. This is not to say that one way is somehow better than the other, just that the location is arbitrary). This is why <i>setResult6</i> which is not inlined and higher up the stack doesn't show up.<br />
<br />
<h3>
Summary: What Is It Good For?</h3>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgum5Ks0rMO-jYJTTWltyEEyaUr7r2Vjs9dsJsuvU-_cgbunnhkRJVXBMY-gbtpA8jiw1PVW834bjariDh6kyOkDhLLJn5yD4lzMqy5ZoLP7vbo0SEsogAFlFv-4L2YcOu5wgc4v-uDrU/s1600/PS_0825W_ELEMENT_CONFUSION.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgum5Ks0rMO-jYJTTWltyEEyaUr7r2Vjs9dsJsuvU-_cgbunnhkRJVXBMY-gbtpA8jiw1PVW834bjariDh6kyOkDhLLJn5yD4lzMqy5ZoLP7vbo0SEsogAFlFv-4L2YcOu5wgc4v-uDrU/s320/PS_0825W_ELEMENT_CONFUSION.jpg" width="320" /></a>As demonstrated above, a safepoint sampling profiler can have a wildly inaccurate idea of where the hot code in your application is. This renders derived observations on "Running" threads pretty suspect, but they are at least correct observations on which threads are running. This doesn't mean they are entirely useless and sometimes all we need is a hint in the right direction for some good analysis to happen, but there's a potential for huge waste of time here. While samples of code running in the interpreter do not suffer safepoint bias, this is not very useful as hot code is quickly compiled. If your hot code is running in the interpreter you have bigger fish to fry than safepoint bias...</div>
<div>
The stack traces for blocked threads are accurate, and so the "Waiting" profile is useful to discover blocked code profile. If blocking methods are at the root of your performance issue than this will be a handy observation.</div>
<div>
There are better options out there! I'll get into some of them in following posts:</div>
<div>
- Java Mission Control</div>
<div>
- Solaris Studio</div>
<div>
- Honest-Profiler</div>
<div>
- Perf + perf-map-agent (or <a href="http://psy-lob-saw.blogspot.com/2015/07/jmh-perfasm.html" target="_blank">perfasm</a> if your workload is wrapped in a JMH benchmark)</div>
No tool is perfect, but all of the above are a damn sight better at identifying where CPU time is being spent.<br />
Big thanks to the many reviewers, if you find any issues please comment and I will punish them severely.<br />
<br />
<b>UPDATE: Read follow up post of Honest-Profiler/Java Flight Recorder type profilers in this next post - <a href="http://psy-lob-saw.blogspot.co.za/2016/06/the-pros-and-cons-of-agct.html" target="_blank">The Pros And Cons of AsyncGetCallTrace Profilers</a></b></div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com18tag:blogger.com,1999:blog-5171098727364395242.post-5164247164152020742016-02-01T07:56:00.000+00:002016-02-01T07:56:56.144+00:00Wait For It: Counted/Uncounted loops, Safepoints and OSR CompilationIn this <a href="http://psy-lob-saw.blogspot.com/2015/12/safepoints.html" target="_blank">previous post about Safepoints</a> I claimed that this here piece of code:<br />
<script src="https://gist.github.com/nitsanw/173f896c1549c07f5fe6.js"></script><br />
Will get your JVM to hang and will not exit after 5 seconds, as one might expect from reading the code. The reason this happens, I claimed, is because the compiler considers for loops limited by an int to be <b>counted</b>, and will therefore not insert a <b>safepoint poll</b> into the generated assembly code. This was the theory, and I tested it, and it definitely hung on my machine ('It Works On My Machine'™).<br />
To my great surprise, some people not only read the posts, they also run the code (it's like they don't trust me ;-) ). And 2 of those diligent readers got back to me in the comments saying that actually the JVM exists just fine. So I ran it again, and it hung, again. Then I ran it with different versions of the JVM, and it hung as I expected. Then I thought, maybe they are running the code I put on GitHub, so I tried that. That also hung when I ran it from Eclipse, but did not when I built it with Maven and ran it from the command line. EUREKA!<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB-_FWpNiVJLZvZzNTd0S1DLTlExpe_fV6wMXAkVaBoVOg29Ab9Ss_91qP0I8R0IYnmtLH_59fCav-2VmkWxc1v6-lXHDrFUqWI4Lt7ZH9114n0GjP7c2RImKYCdNwSBGsK1NzNFfAFog/s1600/jabberwocky.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB-_FWpNiVJLZvZzNTd0S1DLTlExpe_fV6wMXAkVaBoVOg29Ab9Ss_91qP0I8R0IYnmtLH_59fCav-2VmkWxc1v6-lXHDrFUqWI4Lt7ZH9114n0GjP7c2RImKYCdNwSBGsK1NzNFfAFog/s320/jabberwocky.jpg" width="215" /></a></div>
As it happens, Eclipse has it's own Java compiler (a fact well known to those who know it well, no doubt) and the bytecode it generated (the .class file) hung every time, but Maven uses javac and the .class file generated from that did not hang. When testing initially I was running my class inside Eclipse, or using the Eclipse .class file to run from the command line. Both .class files are valid, but different (same-same but different). We can use javap to print out class bytecode and compare ("javap -c -v -p <class file>"). For your pleasure you can find the original Java file, the 2 class files and their respective bytecode printouts on GitHub and come to your own conclusions before you continue reading.<br />
So... is my understanding (and the Eclipse javac) somehow out of date? can this problem of telling counted loops from uncounted loops be solved by the bytecode compiler?<br />
<br />
<h3>
How is the bytecode different?</h3>
Well, mostly not that different:<br />
<ol>
<li>An easy to spot difference which we can tell is semantically the same is the bytecode result for "System.out.println("How Odd:" + l);" you'll notice that Eclipse is using the constructor "java/lang/StringBuilder."<init>":(Ljava/lang/String;)" while javac is opting for "java/lang/StringBuilder."<init>":()V" followed by an append. Semantically the 2 choices are the same. We create a StringBuilder and append the String "How odd:" and the variable <b style="font-style: italic;">l</b>. The reality is that there are subtle differences, the Eclipse version will allocate a larger <i>char[]</i> for instance. It may also be that the Eclipse pattern will miss out on some String concatenation optimization because the OpenJDK implementers are likely to focus on patterns generated from the official javac.</li>
<li>The loop interpretation is different. Eclipse is checking the loop conditions at the end of the loop whereas javac is building a loop with the checks upfront and the loop backedge at the end. That seems like the important difference.</li>
</ol>
Let's zoom in on the relevant bytecode for the loop:<br />
<script src="https://gist.github.com/nitsanw/1e8e8bab731cf79b18c3.js"></script><br />
If we were to reverse translate the loop bytecode to ugly Java (which supported goto) we'd get:<br />
<div>
<script src="https://gist.github.com/nitsanw/3df7909578c24e10fd2d.js"></script></div>
<br />
Note that there's no special Java bytecode for loop control flows. So why so different at runtime? If we dump out the assembly for the run method we will see that the Eclipse version is missing the safepoint in the loop (the method has a <i style="font-weight: bold;">{poll_return}</i> but there's no <b><i>{poll}</i></b>), and the javac version has that safepoint there. Is it the bytecode difference that makes this happen?<br />
<br />
<h3>
Variations on bytecode</h3>
<div>
I tinkered with the class a bit and refactored the loop out of the lambda. This results in similar bytecode for the loop and same behaviour. I then made the limit of the loop a parameter of the method, this time it behaved like the javac version, but had the same funky loop structure:<br />
<script src="https://gist.github.com/nitsanw/57f136c1b85436394f8a.js"></script><br />
WTF? if this is a purely a bytecode subtlety then it's seems pretty weird. What sort of flaky compiler would swing like this?<br />
<br /></div>
<h3>
The compiler did it?</h3>
<div>
But... which compiler? so we looked at the above bytecode compiler results, this is but the first of many compilers to process your code on it's way to being executed. Traditionally when discussing Java code compilation we look at the following stages:<br />
<ol>
<li>Javac: From code to bytecode</li>
<li>Interpreter: Step through the bytecode</li>
<li>C1/Client: Short compilation time, does not generate the best optimized code. Intended for short lived applications.</li>
<li>C2/Server: Longer compilation time, better optimized code. Intended for long running applications.</li>
</ol>
And the compilation timeline would look something like this:<br />
<span style="font-family: "courier new" , "courier" , monospace;"> /->C1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">*.java->javac->*.class->interpreter->class+profile<</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> \->C2</span><br />
<span style="font-family: inherit;">The situation with OpenJDK8 is more complex with <a href="http://www.slideshare.net/maddocig/tiered" target="_blank">Tiered Compilation</a> allowing us a smooth transition from interpreter to C2 compilation via C1 compilations. </span><br />
<span style="font-family: inherit;">Given the behaviour we are after was supposed to manifest after a C1/C2 compilation we can run the code above to find which compiler compiled the method in question by adding the flag: -XX:+PrintCompilation:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 159 201 % 3 safepoint.hang.WhenWillItExitInt2::countOdds @ 13 (44 bytes)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 159 202 3 safepoint.hang.WhenWillItExitInt2::countOdds (44 bytes)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 159 203 % 4 safepoint.hang.WhenWillItExitInt2::countOdds @ 13 (44 bytes)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 162 201 % 3 safepoint.hang.WhenWillItExitInt2::countOdds @ -2 (44 bytes) made not entrant</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 142 200 % 3 safepoint.hang.WhenWillItExitInt2::countOdds @ -2 (44 bytes) made not entrant</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 1896 202 % 4 safepoint.hang.WhenWillItExitInt2::countOdds @ -2 (44 bytes) made not entrant</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 1896 203 % 3 safepoint.hang.WhenWillItExitInt2::countOdds @ 13 (44 bytes)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 1896 204 % 4 safepoint.hang.WhenWillItExitInt2::countOdds @ 13 (44 bytes)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 1899 203 % 3 safepoint.hang.WhenWillItExitInt2::countOdds @ -2 (44 bytes) made not entrant</span><br />
<span style="font-family: inherit;">The 3/4 printouts on the 4th column above indicate the method is compiled by C1 (profiled) and C2. But what is that <b>% </b>about? This is an indicator that the compilations are not 'normal' compilations but rather On Stack Replacement(OSR) compilations. OSR compilations are 'special', from the OpenJDK glossary:</span><br />
<blockquote class="tr_bq">
<b>on-stack replacement</b> <br />
Also known as 'OSR'. The process of converting an interpreted (or less optimized) stack frame into a compiled (or more optimized) stack frame. This happens when the interpreter discovers that a method is looping, requests the compiler to generate a special nmethod with an entry point somewhere in the loop (specifically, at a backward branch), and transfers control to that nmethod. A rough inverse to deoptimization.</blockquote>
I like the "rough inverse to deoptimization", it's a great mental qualification helper. OSR is the opposite of deoptimization because it is a mid-method replacement of code with compiled code, whereas a deoptimization is a mid-method replacement of code with interpreted code. Normal compilation unit replacements happen out of method, so while the method is off stack. OSR is specifically done to replace long running loops, like our countOdds method. As we can see from the printout above both C1 and C2 can produce OSR compilations.<br />
It is impossible to disable OSR directly in a product build (there's a developer option -XX:-CICompileOSR), but we can force upfront compilation by using the following flags: "-XX:-TieredCompilation -server/client -Xcomp". Running with these arguments all 3 variations (original, refactored loop, refactored parametrized loop) now hang both for the Eclipse and the javac generated bytecode.<br />
In both cases where the Eclipse versions hang we can see OSR compilations of the relevant code.<br />
So it seems:<br />
<ul>
<li>C1/C2 non-OSR compilations of the loop treat it as a counted loop, thus eliminating the safepoint poll and demonstrating the hung behaviour due to huge TTSP.</li>
<li>C1/C2 OSR compilations of the same code result in different results depending on the generated bytecode for the loop. I'm not going to dig further into the reasons, but I would welcome any feedback from OpenJDK compiler guys.</li>
</ul>
The OSR compilation decision to treat the loop as non-counted is quite reasonable. If we have a loop which remains on stack for very long periods we should perhaps consider it uncounted.<br />
<br />
<h3>
Killer Task</h3>
</div>
<div>
I started down this path looking for valid Java code which will result in a hung JVM due to huge TTSP. Since OSR got in the way, can we build such a task? Sure we can, we just need to warmup the loop until it is C2 compiled and then run it with a large limit, like so:</div>
<div>
<script src="https://gist.github.com/nitsanw/a8b20fc6ecbb9ae9c054.js"></script></div>
<div>
This fucker now hangs in both the Eclipse and javac versions. Joy!<br />
<br />
<h3>
Side note: OSR and JMH</h3>
</div>
<div>
I look at benchmarks people write quite a bit, and it is very common to see people rolling their own benchmark harnesses put their "code under measurement" in a counted loop, run it 100,000 times and derive their results. Often that means the code they are measuring is OSR compiled code, which is different (for better or worse, mostly worse) from the code they will be running normally. If you are one of these people, I beg you: cut that fucking shit out!</div>
<div>
JUST USE JMH!<br />
PLEASE?</div>
<div>
To a large extent it will cover your use case. Where it doesn't cover your use case, post a question on jmh-dev mailing list, it might be that you've missed something. If it turns out JMH cannot cover your usecase, make sure you've read the JMH samples before rolling your own harness.<br />
I'm not going to include a before and after example, because it just makes me sad. Look at SO for some fine examples.<br />
<br /></div>
<h3>
Counted Loops?</h3>
I keep saying 'counted loops' like it means something, so here are some examples for you to play with and <a href="https://www.youtube.com/watch?v=yy5THitqPBw" target="_blank">contemplate</a>:<br />
<script src="https://gist.github.com/nitsanw/c08ed6e17b667c383b0b.js"></script><br />
So, as you can see the definition ain't so clear and I would imagine some of the above behaviour will vary depending on JVM version/vendor/phase of the moon etc.<br />
<h3>
Summary</h3>
<div>
So, it seems my previous example was not as tested as it should have been. I apologize to anyone who has based life changing decisions on that sample. I hope this post helps in explaining away the subtleties which led to my downfall.</div>
<div>
The above examples help demonstrate that the line between counted and uncounted loops is a tad confusing and the classification may depend on differences in bytecode, compiler mode, and so on.</div>
<div>
<br /></div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com10tag:blogger.com,1999:blog-5171098727364395242.post-8311682357667705072015-12-14T11:49:00.001+00:002016-02-01T08:09:27.024+00:00Safepoints: Meaning, Side Effects and OverheadsI've been giving a couple of talks in the last year about profiling and about the JVM runtime/execution and in both I found myself coming up against the topic of Safepoints. Most people are blissfully ignorant of the existence of safepoints and of a room full of people I would typically find 1 or 2 developers who would have any familiarity with the term. This is not surprising, since safepoints are not a part of the Java language spec. But as safepoints are a part of every JVM implementation (that I know of) and play an important role, here's my feeble attempt at an overview.<br />
<br />
<h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-z3CCIhVG5WfGtsROF0nxKA85-aoa_pVtBBErX3kijvsGNJSCQUxfUzNaCU712j-CqfpaLYnNHUL9Hp_hNudvyQKGU-VGG_BxgPXorOkdoqKrsiT9JrW2H-KJD2gqyvwmtjP3F8xnNIE/s1600/safepineapples.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-z3CCIhVG5WfGtsROF0nxKA85-aoa_pVtBBErX3kijvsGNJSCQUxfUzNaCU712j-CqfpaLYnNHUL9Hp_hNudvyQKGU-VGG_BxgPXorOkdoqKrsiT9JrW2H-KJD2gqyvwmtjP3F8xnNIE/s200/safepineapples.jpg" width="200" /></a>
What's a Safepoint?</h3>
<div>
I was recently asked: "Are safepoints like safe words?". The short answer is "not really", but since I like the metaphor I'll run with it:<br />
Imagine if you will a JVM full of mutator threads, all busy, sweating, mutating the heap. Some of them have <gasp> shared mutable state. They're mutating each others state, concurrently, like animals. Some stand in corners mutating their own state (go blind they will). Suddenly a neon sign flashes the word PINEAPPLES. One by one the mutators stop their rampant heap romping and wait, sweat dripping. When the last mutator stops, a bunch of elves come in, empty the ashtrays, fill up all the drinks, mop up the puddles, and quickly as they can they vanish back to the north pole. The sign is turned off and the threads go back to it....<br />
<i><span style="color: orange;"><inner-dialogue>Too much? It's like 35c in the shade here, so sweating is like... no? never mind </inner-dialogue></span></i><br />
There are many references to Safepoints to be found online and what follows here is my attempt at a more nuanced use/definition, no sweating mutators from this point onwards.<br />
<blockquote class="tr_bq">
A <b>safepoint</b> is a range of execution where the state of the executing thread is well described. <b>Mutator</b> threads are threads which manipulate the JVM heap (all your Java Threads are <b>mutators</b>. Non-Java threads may also be regarded as <b>mutators </b>when they call into JVM APIs which interact with the heap).<br />
<b>At a safepoint</b> the <b>mutator</b> thread is at a known and well defined point in it's interaction with the heap. This means that all the references on the stack are mapped (at known locations) and the JVM can account for all of them. As long as the thread remains <b>at a safepoint</b> we can safely manipulate the heap + stack such that the thread's view of the world remains consistent when it leaves the safepoint.</blockquote>
This is particularly useful when the JVM wishes to examine or change the heap, for a GC or any one of many other reasons. If references on the stack were unaccounted for and the JVM was to run a GC it may miss the fact that some objects are still alive (referenced from the stack) and collect them, or it may move some object without updating it's reference on the stack leading to memory corruption.</div>
<div>
A JVM will therefore need means of bringing threads to safepoints (and keeping them there) so that all sorts of runtime magic can happen. Here's a partial list of activities which JVMs run only once all mutator threads are at a safepoint <u>and cannot leave it until released</u> (at a <b>global safepoint</b>), these are sometime called <b>safepoint operations</b>:</div>
<div>
<ul>
<li>Some GC phases (the Stop The World kind)</li>
<li>JVMTI stack sampling methods (not always a global safepoint operation for Zing))</li>
<li>Class redefinition</li>
<li>Heap dumping</li>
<li>Monitor deflation (not a global safepoint operation for Zing)</li>
<li>Lock unbiasing</li>
<li>Method deoptimization (not always)</li>
<li>And many more!</li>
</ul>
</div>
<div>
A <a href="https://www.parleys.com/tutorial/with-gc-solved-what-else-makes-jvm-pause" target="_blank">great talk by Azul's own John Cuthbertson from JavaOne 2014</a> goes into some background on Safepoints and allot of details about safepoint operations other than GC (we at Azul feel GC is a solved problem, so the talk goes into the remaining reasons to stop).<br />
Note that the distinction between requesting a <b>global safepoint </b>and a <b>thread safepoint</b> exists only for some JVM implementations (e.g. <b>Zing</b>, the Azul Systems JVM. <u>Reminder: I work for Azul</u>). Importantly it does <b>not </b>exist on OpenJDK/Oracle JVMs. This means that Zing can bring a single thread to a safepoint.</div>
<div>
To summarize:</div>
<div>
<ul>
<li>Safepoints are a common JVM implementation detail</li>
<li>They are used to put mutator threads on hold while the JVM 'fixes stuff up'</li>
<li>On OpenJDK/Oracle every safepoint operation requires a <b>global safepoint</b></li>
<li>All current JVMs have some requirement for <b>global safepoints</b></li>
</ul>
</div>
<h3>
When is my thread at a safepoint?<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDQqpdgpoZ-bijVpQ9GJsvhN3HZEfDMcdM-dKSqjCdwSjQFqVR-wDN8ClnVCnqTBlOTyki4LZABlxhcvzsYO8AENyQEjqpUYPctCO-BW70cXqiPtJc4CzjHWW9M3PPNkl2e8KOCMDFAQ4/s1600/By+Toutatis.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="153" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDQqpdgpoZ-bijVpQ9GJsvhN3HZEfDMcdM-dKSqjCdwSjQFqVR-wDN8ClnVCnqTBlOTyki4LZABlxhcvzsYO8AENyQEjqpUYPctCO-BW70cXqiPtJc4CzjHWW9M3PPNkl2e8KOCMDFAQ4/s320/By+Toutatis.jpg" width="320" /></a></div>
</h3>
<div>
So having threads at a safepoint allows the JVM to get on with it's managed runtime magic show, great! When is this groovy state happening?</div>
<div>
<ul>
<li>A Java thread is <b>at a safepoint</b> if it is blocked on a lock or synchronized block, waiting on a monitor, parked, or blocked on blocking IO. Essentially these all qualify as orderly de-scheduling events for the Java thread and as part of tidying up before put on hold the thread is brought to a safepoint.</li>
<li>A Java thread is <b>at a safepoint</b> while executing JNI code. Before crossing the native call boundary the stack is left in a consistent state before handing off to the native code. This means that the thread can still run while <b>at a safepoint</b>.</li>
<li>A Java thread which is executing bytecode <b>is NOT at a safepoint </b>(or at least the JVM cannot assume that it is at a safepoint).</li>
<li>A Java thread which is interrupted (by the OS) while not at a safepoint is not brought to a safepoint before being de-scheduled.</li>
</ul>
The JVM and your running Java threads have the following relationship around safepoints:</div>
<div>
<ul>
<li>The JVM cannot force any thread into a safepoint state.</li>
<li>The JVM can stop threads from leaving a safepoint state.</li>
</ul>
So how can the JVM bring all threads into a safepoint state? The problem is suspending a thread at a <u>known state</u>, not just interrupting i<u>t</u>. To achieve this goal JVMs have the Java threads suspend themselves at convenient spots if they observe a 'safepoint flag'.<br />
<br /></div>
<h3>
Bringing a Java Thread to a Safepoint</h3>
Java threads poll a 'safepoint flag' (global or thread level) at 'reasonable' intervals and transition into a safepoint state (thread is blocked at a safepoint) when they observe a 'Go to safepoint' flag. This is simple enough, but since we don't want to spend all of our time checking if we need to stop the C1/C2 compilers (-client/-server JIT compilers) try and keep safepoint polls to a minimum. On top of the cost of the flag check itself, maintaining a 'known state' adds significant complexity to the implementation of certain optimizations and so keeping safepoints further apart opens up a wider scope for optimization. These considerations combined lead to the following locations for <b>safepoint polls:</b><br />
<ul>
<li>Between any 2 bytecodes while running in the interpreter (effectively)</li>
<li>On 'non-counted' loop back edge in C1/C2 compiled code</li>
<li>Method entry/exit (entry for Zing, exit for OpenJDK) in C1/C2 compiled code. Note that the compiler will remove these safepoint polls when methods are inlined.</li>
</ul>
If you are the sort of person who looks at assembly for fun (or profit, or both) you'll find safepoint polls in the -XX:+PrintAssembly output by looking for:<br />
<ul>
<li><b>'{poll}'</b> or '<b>{poll return}' </b>on OpenJDK, this will be in the instructions comments</li>
<li>'<b>tls.pls_self_suspend</b>' on Zing, this will be the flag examined at the poll operation</li>
</ul>
This mechanism is implemented differently on different VMs (on x86 + Linux, I've not looked at other architectures):<br />
<ul>
<li>On Oracle/OpenJDK a blind TEST of an address on a special memory page is issued. Blind because it has no branch following it so it acts as a very unobtrusive instruction (usually a TEST is immediately followed by a branch instruction). When the JVM wishes to bring threads to a safepoint it protects the page causing a SEGV which is caught and handled appropriately by the JVM. There is one such special page per JVM and therefore to bring any thread to a safepoint we must bring all threads to a safepoint.</li>
<li>On Zing the safepoint flag is thread local (hence the tls prefix). Threads can be brought to a safepoint independently.</li>
</ul>
See related post: <a href="https://psy-lob-saw.blogspot.com/2014/03/where-is-my-safepoint.html" target="_blank">Dude, Where's My Safepoint?</a> for more details on polls.<br />
Once a thread detects the safepoint flag it will perform the safepoint action that the poll triggers. This normally means blocking on some sort of JVM level lock, to be released when the safepoint operation is over. Think of it as a locking mechanism where:<br />
<br />
<ul>
<li>Threads can lock themselves out (e.g. by calling into JNI, or blocking at a safepoint).</li>
<li>Threads can try and re-enter (when returning from JNI), but if the lock is held by the JVM they will block.</li>
<li>The <b>safepoint operation </b>requests the lock and blocks until it own it (when all the mutators have locked themselves out)</li>
</ul>
<br />
<h3>
All Together Now</h3>
Now we have all the moving parts of the safepoint process:<br />
<ul>
<li>Safepoint triggers/operations: reasons to request, and stuff to do which requires one or all threads to be at a safepoint</li>
<li>Safepoint state/lock: into which Java threads can voluntarily check in, but can only leave when the JVM is done executing the safepoint operation</li>
<li>Safepoint poll: the voluntary act by a Java thread to go into a safepoint if one is needed</li>
</ul>
Let's see how it all hangs together, for example when a profiler is sampling the stack using JVMTI::GetStackTrace:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPUB1OvwpKSyn-263emaV1gCMIYoKTpu2mzkXGUxnMRkjyEYyEeHLSnRWqFIoJ1I1KkGo_hs09HycuC2gHNyFNbdBZamQtU41YEsDqa89x4I5x1yznWQ-4ktQgP7-BAqm7ejuxGYw6erk/s1600/SafepointOverheads.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPUB1OvwpKSyn-263emaV1gCMIYoKTpu2mzkXGUxnMRkjyEYyEeHLSnRWqFIoJ1I1KkGo_hs09HycuC2gHNyFNbdBZamQtU41YEsDqa89x4I5x1yznWQ-4ktQgP7-BAqm7ejuxGYw6erk/s640/SafepointOverheads.png" width="640" /></a></div>
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><b><span style="background-color: #6aa84f;">GREEN </span>: Executing Java</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b><span style="background-color: #ffd966;">YELLOW </span>: Descheduled Java</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b><span style="background-color: #cc0000;">RED </span>: Native</b></span><br />
This diagram highlights several components of safepoint operation overheads:<br />
<ol>
<li>Time To Safe Point(TTSP): Each thread enters a safepoint when it hits a safepoint poll. But arriving at a safepoint poll requires executing an unknown number of instructions. We can see J1 hits a safepoint poll straight away and is suspended. J2 and J3 are contending on the availability of CPU time. J3 grabs some CPU time pushing J2 into the run queue, but J2 is not in a safepoint. J3 arrives <b>at a safepoint</b> and suspends, freeing up the core for J2 to make enough progress to get to a safepoint poll. J4 and J5 are already at a safepoint while executing JNI code, they are not affected. Note that J5 is trying to leave JNI halfway through the safepoint and is suspended before resuming Java code. Importantly we observe that the time to safepoint varies from thread to thread and some threads are paused for longer than others, Java threads which take a long time to get to a safepoint can delay many other threads.</li>
<li>Cost of the <b>safepoint operation</b>: This will depend on the operation. For GetStackTrace the cost of the operation itself will depend on the depth of the stack for instance. If your profiler is sampling many threads (or all the threads) in one go (e.g. via JVMTI::GetAllStackTraces) the duration will grow as more threads are involved. A secondary effect may be that the JVM will take the opportunity to execute some other safepoint operations while the going is good. </li>
<li>Cost of resuming the threads.</li>
</ol>
Some take aways from this analysis:<br />
<ol>
<li>Long TTSP can dominate pause times: some causes for long TTSP include page faults, CPU over-subscription, long running counted loops.</li>
<li>The cost of stopping all threads and starting all threads scales with the number of threads. The more threads the higher the cost. Assuming some non-zero cost to suspend/resume and TTSP, multiply by number of threads...</li>
<li>Add -XX:+PrintGCApplicationStoppedTime to your list of favourite flags! It prints the pause time and the TTSP.</li>
</ol>
But why should you trust me? Lets break some shit!!! Please note in the following examples I will be switching between Oracle 8u60 and Zing (latest dev JDK8 build). I work for <a href="https://www.azul.com/products/zing/" target="_blank">Azul Systems on the Zing JVM</a>, so feel free to assume bias on my part. My main motivation in including both JVMs is so that I can demonstrate the effect of -XX:+KeepSafepointsInCountedLoops on Zing.<br />
NOTE: The benchmarks are not intended to spark a flame war and though Zing happens to compile this particular piece of code 'better' I would not get over excited about it. What is exciting is the extra option Zing makes available here to control and minimize TTSP, and the additional perspective it offers on how a JVM can deal with these challenges.<br />
<br />
<h3>
Example 0: Long TTSP Hangs Application</h3>
<div>
<script src="https://gist.github.com/nitsanw/69b0e08be90422248e52.js"></script></div>
<div>
The above code should exit in 5 seconds (or <a href="https://github.com/nitsanw/safepoint-experiments/blob/master/src/main/java/safepoint/hang/WhenWillItExitInt.java" target="_blank">see it on GitHub</a>). The silly computation running in the daemon thread should not stop it from exiting (the JVM should exit if all remaining running Java threads are daemon threads). But if we run this sample we will find that the JVM does not exit after 5 seconds. It doesn't exit at all<b>[UPDATE: this example turned out to be a bit flaky as per discussion in comments and following <a href="http://psy-lob-saw.blogspot.co.za/2016/02/wait-for-it-counteduncounted-loops.html" target="_blank">blog post</a>]</b>. Not only does it not exit, it will also refuse incoming JMX connections, it won't respond the jmap/jstack. It won't naturally die, you'll have to 'kill -9' this fucker.<br />
The reason it's so stuck is because the computation is running in a 'counted' loop, and counted loops are considered short by the JVM. Short enough to not have safepoints. This thread refusing to come to a safepoint will lead to all other threads suspending indefinitely at the next safepoint operation.<br />
We can fix this (somewhat contrived) problem in several ways:<br />
<ol>
<li>Prevent C1/C2 compilation: Use -Xint or a compile command to prevent the method compilation. This will force safepoint polls everywhere and the thread will stop hurting the JVM.</li>
<li>Replace the index type with long: this will make the JVM consider this loop as 'uncounted' and include a safepoint poll in each iteration.</li>
<li>On Zing you can apply the -XX:+KeepSafepointsInCountedLoops option on either the JVM or method level (via a -XX:CompileCommand=option,... or via a compiler oracle file). This option is <a href="https://bugs.openjdk.java.net/browse/JDK-6869327" target="_blank">coming shortly to an OpenJDK near you</a>. See also related bug where <a href="https://bugs.openjdk.java.net/browse/JDK-5014723" target="_blank">breaking the loop into an inner/outer loop with safepoints in the outer loop</a> is discussed.</li>
</ol>
This example is handy when considering shared JVM solutions. This is perfectly 'safe' Java code that will bring most application containers to their knees.<br />
<b>Summary: </b>Delaying a <b>safepoint poll </b>indefinitely can be harmful...</div>
<br />
<h3>
Example 1: More Running Threads -> Longer TTSP, Higher Pause Times</h3>
<div>
<script src="https://gist.github.com/nitsanw/4cdb09cb20dba1f8ed35.js"></script></div>
The benchmark has 1 thread triggering a <b>safepoint operation</b> by calling <i>safepointMethod()</i> (in this case the <i>System.gc() </i>will be called as a result) and many threads scanning a byte array. The byte array scan loop will not have a <b>safepoint poll </b>(it's a counted loop), so the size of the array will act as our intended <b>safepoint poll interval</b>. With this setup we can control the frequency of safepoint operations (with some overhead), avg. distance from safepoint poll and with JMH the number of threads. We can use -XX:+PrintGCApplicationStoppedTime to log the stopped time and TTSP. Armed with this experiment we can now find a large machine to play on and see what happens. I use the command line '-tg' parameter to control the number of threads executing the <i>contains1() </i>method.<br />
So, here goes on a 48 core Haswell Xeon machine, Cent OS, I'll be using Oracle 8u60. The benchmark class I'm running is <a href="https://github.com/nitsanw/safepoint-experiments/blob/master/src/main/java/safepoint/fair/SafepointUsingGc.java" target="_blank">here</a>.<br />
Lets start with no <i>contains1() </i>threads and explain the method and data collection as we go along:<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ java -jar safepoints.jar -f 1 -i 10 -wi 10 -jvmArgs="-XX:+PrintGCApplicationStoppedTime -Xms1g -Xmx1g" -tg 0,0,0,1 SafepointUsingGc</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">There's 4 thread groups because I add another benchmark to run with this crowd in the next sample.</span><br />
<span style="font-family: inherit;">This spouts allot of </span>PrintGCApplicationStoppedTime printouts similar to:<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Total time for which application threads were stopped: 0.0180172 seconds, Stopping threads took: 0.0000513 seconds</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">I'm no Unix shell guru, but with some googling even I can AWK something to summarize this shit:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">grep 'Total time' output.file | awk '{ stopped += $9;ttsp += $14; n++ } END { if (n > 0) print "stopped.avg=" stopped / n ", ttsp.avg=" ttsp / n; }'</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">It's summing up the warmup period measurements as well, but I'm willing to let it ride. So with that in mind, lets compare the benchmark output with the reported pause times:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">SafepointUsingGc.run_together:safepoint 100 10 32768 true avgt 10 118611444.689 ± 1288530.618 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">stopped.avg=0.015759, ttsp.avg=6.98995e-05</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Or formatted for humans:</span><br />
<span style="background-color: #eeeeee; font-family: "courier new" , "courier" , monospace; font-size: small;"><b><u>park+gc(ms) |Stp.ms|TTSP.ms|</u></b><br /> 119 ± 1.0 | 16.1 | 0.07 |</span><br />
<div>
<br />
We went with the default intervalMs=100, so we'd expect the <i>safepoint()</i> cost to be around 100 + stopped.avg, it's actually 2ms more than that. This is why we have the <i>park()</i> benchmark to baseline against. Running again with the park group enabled ('-tg 0,0,1,1'):</div>
<div>
<span style="background-color: #eeeeee; font-family: "courier new" , "courier" , monospace; font-size: small;"><b><u>park(ms) | park+gc(ms) |Stp.ms|TTSP.ms|</u></b><br />
101 ± 0.8 | 118 ± 1.4 | 15.9 | 0.07 |
</span></div>
<div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">We can see that there's some variance to the GC call and some scheduling variance which can account for delta between our expected value and actual. Small potatoes, lets move on. Start with measuring the cost of the <i>contains1() </i>call when we're not slapping it with a GC every 100ms ('-tg 1,0,1,0' we're running the park baseline):</span></div>
<div>
<span style="background-color: #eeeeee; font-family: "courier new" , "courier" , monospace; font-size: small;">
<b><u>contains1(us) | park(ms) |Stp.ms|TTSP.ms|</u></b><br />
32.2 ± 0.02 | 100.1 ± 0.0 | 0.0 | 0.0 |</span></div>
<div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Note the </span> <i>contains1()</i><span style="font-family: inherit;"> method cost is in microseconds. Awesome, now lets ramp up the number of </span><i>contains1() </i>thread with the GC thread running every 100ms:</div>
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;">
<b><u>T |contains1(us) | park(ms) | park+gc(ms) |Stp.ms|TTSP.ms|</u></b><br />
1 | 51.4 ± 3.1 | 100.7 ± 0.7 | 117.5 ± 0.7 | 15.1 | 0.1 |<br />
2 | 49.5 ± 17.1 | 101.6 ± 1.9 | 116.9 ± 0.5 | 14.4 | 0.1 |<br />
4 | 45.5 ± 2.6 | 102.8 ± 4.0 | 116.9 ± 0.7 | 14.4 | 0.1 |<br />
8 | 45.6 ± 2.1 | 103.6 ± 5.6 | 114.0 ± 0.7 | 13.3 | 0.1 |<br />
16 | 46.8 ± 1.3 | 103.4 ± 6.8 | 115.0 ± 1.5 | 12.8 | 0.2 |<br />
32 | 65.7 ± 2.0 | 101.8 ± 0.9 | 113.0 ± 1.3 | 11.4 | 0.7 |<br />
64 | 121.9 ± 1.9 | 106.0 ± 8.1 | 116.1 ± 1.4 | 14.5 | 5.4 |<br />
128 | 251.9 ± 6.6 | 103.6 ± 5.3 | 119.6 ± 2.3 | 16.9 | 6.4 |<br />
256 | 513.2 ± 13.4 | 109.2 ± 9.7 | 123.4 ± 1.7 | 20.7 | 8.0 |<br />
512 |1111.3 ± 69.7 | 143.9 ±24.8 | 134.8 ±15.3 | 27.8 | 10.0 |</span><br />
<div>
<br /></div>
<div>
OK, that escalated not so quickly. Keep in mind we only actually have 48 cores (hyper-threaded), and so when we cross the line between 32 and 64 we are now looking at the effect of contention and scheduling on the TTSP. It is evident that if you have more CPU bound threads than CPUs performance will degrade, but that is not really what we're after.<br />
At 1 thread we already see a big drop in performance for <i>contains1()</i>. This is somewhat understandable in the presence of 15ms stolen from the application every 100ms. But where you'd expect a 15% difference we're actually looking at a far more significant drop in performance. Going from 32us to 51us kinda sucks.<br />
We can see a steady growth in TTSP while within the core count, going from 0.1ms to 0.7ms, this is bad news to anyone looking to keep their latencies in the low microseconds count as touched upon in <a href="https://www.lmax.com/blog/staff-blogs/2015/08/05/jvm-guaranteed-safepoints/" target="_blank">this post from the brilliant Mark Price</a>. Mark highlights the fact that the JVM regularly calls for a global safepoint (<a href="http://muslimvilla.smfforfree.com/index.php?topic=3219.0;wap2" target="_blank">“whether she needed it or not”</a>), so you may find to your delight that these little hiccups come round even when you've been allocating nothing.<br />
Once we have more threads than cores the TTSP cost goes into the milliseconds, but escalates rather slowly once there. Going from 5ms to 10ms while we go from 64 to 512 threads is better than I expected.<br />
<b>Summary:</b> By increasing the thread count we increase the TTSP. CPU over-subscription will lead to a further increase in TTSP. Monitor machines load avg to detect this kind of misconfiguration.</div>
</div>
<h3>
Example 2: Long TTSP has Unfair Impact</h3>
<div>
<script src="https://gist.github.com/nitsanw/f677c6162e662c4525ed.js"></script></div>
<div>
Building on previous benchmark, but now we have 2 methods: <i>contains1() </i>and<i> </i><i>contains1toK()</i>. <i>contains1toK() </i>is K times further away from a <b>safepoint poll</b> (outer loop is counted, the contains needle methods gets inlined removing the method exit safepoint and the loop inside it is counted). By increasing K we can see the impact of one thread on the other, in particular we expect the cost of <i>contains1()</i> to increase as the cost of <i>contains1toK()</i> increases, in the presence of <b>safepoint operations</b>. The unfairness here being that while <i>contains1toK()</i> is making progress it is making <i>contains1(<span style="font-size: xx-small;">legen...</span>)</i> wait for it (<span style="font-size: xx-small;">...DARY!</span>). Because using GC as the safepoint operation is rather heavy handed (even though it's a small heap and there's no garbage) we'll switch to using another <b>safepoint operation</b> for this case: <i>ManagementFactory.getThreadMXBean().findDeadlockedThreads()</i>. This is allot more lightweight than GC but becomes more expensive as we pile on threads (not by that much, but still) so I did not use it for prior example (though come to think of it the number of threads should also impact GC length a bit, oh well). Anyway, variety is the spice of life. Here's the results with both methods and <i>park()</i>, but no <b>safepoint operation </b>(benchmark class is <a href="https://github.com/nitsanw/safepoint-experiments/blob/master/src/main/java/safepoint/fair/SafepointUsingFindDeadlocks.java" target="_blank">here</a>):<br />
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;">
<b><u>K |contain1(us)| containsK(us) | park(ms) |Stp.ms|TTSP.ms|</u></b><br />
1 | 32.3 ± 0.1 | 34.8 ± 1.7 | 100.0 ± 0.0 | 0.0 | 0.0 |<br />
10 | 32.4 ± 0.5 | 349.7 ± 18.0 | 100.0 ± 0.0 | 0.0 | 0.0 |<br />
100 | 38.4 ± 2.8 | 3452.7 ± 130.4 | 100.0 ± 0.0 | 0.2 | 0.1 |<br />
1000 | 39.8 ± 1.1 | 33419.9 ± 421.6 | 100.0 ± 0.0 | 0.0 | 0.0 |<br />
10000 | 41.7 ± 5.0 | 336112.5 ± 10306.5 | 100.5 ± 1.7 | 0.0 | 0.0 |</span></div>
<br />
Indeed the cost of <i>contains1toK()</i> grows linearly with K and the 2 methods don't seem to impact each other that significantly.<br />
Here's what happens as we throw in the safepoint ops:</div>
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;">
<b><u>K |contains1(us)| containsK(us) | park(ms) | park+FDT(ms) |Stp.ms |TTSP.ms|</u></b><br />
1 | 39.7 ± 0.9 | 39.6 ± 1.4 | 100.0 ± 0.0 | 100.3 ± 0.0 | 0.1 | 0.1 |<br />
10 | 40.0 ± 1.0 | 400.1 ± 14.4 | 100.0 ± 0.0 | 100.5 ± 0.0 | 0.2 | 0.2 |<br />
100 | 41.8 ± 1.1 | 3835.4 ± 141.2 | 100.1 ± 0.0 | 102.7 ± 0.1 | 2.1 | 2.0 |<br />
1000 | 67.4 ± 3.5 | 37535.7 ± 2237.2 | 102.5 ± 1.9 | 127.1 ± 1.1 | 20.7 | 20.7 |<br />
10000 | 183.4 ± 8.5 | 346048.6 ± 9818.2 | 194.9 ± 11.5 | 348.4 ± 11.4 | 152.8 | 152.8 |</span></div>
<br />
A terrible thing, right? The annoying/unfair thing is that being a shit has no effect on <i>contains1toK()</i> performance. It scales linearly and seems oblivious to the havoc it's wrecking for everyone else. Note that the problem we see here was present in the run where no safepoint operations were invoked, there were just no safepoint operations to bring it out. On a long enough timeline a safepoint operation will come along and around it we'll have seen the same issue.<br />
How can we fix it, and at what cost? Lets explore some options.<br />
<h4>
Solution 1: I use Zing -> use -XX:+KeepSafepointsInCountedLoops</h4>
<div>
So Zing is super awesome, and has this option to keep safepoint polls in counted loops which can come in handy in cases where one particular method is killing you softly with TTSP. Here's the results we get with Zing when running the benchmark above (same jvmArgs, same machine etc):</div>
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;">
<b><u>K |contains1(us) | containsK(us) | park(ms) | park+FDT(ms) | Stp.ms|</u></b><br />
1 | 31.9 ± 0.9 | 30.5 ± 1.0 | 100.0 ± 0.0 | 100.2 ± 0.0 | 0.0 |<br />
10 | 32.9 ± 0.2 | 317.8 ± 8.8 | 100.0 ± 0.0 | 100.2 ± 0.0 | 0.0 |<br />
100 | 34.1 ± 2.4 | 3337.8 ± 265.4 | 100.1 ± 0.0 | 100.8 ± 0.2 | 0.4 |<br />
1000 | 44.4 ± 1.9 | 33991.6 ± 1199.0 | 100.9 ± 0.2 | 117.0 ± 0.6 | 10.2 |<br />
10000 | 159.0 ± 12.1 |299839.3 ± 6297.2 | 156.3 ± 8.0 | 319.8 ± 30.5 | 113.4 |</span></div>
<br />
Still bad, but Zing happens to squeeze better performance from the <i>containsNeedle()</i> method and while <i>contains1()</i> performance degrades it's opening a wide gap with observed behaviour above probably because less time is lost to TTSP (because <i>contains1toK()</i> finishes quicker). Also note that -XX:+PrintGCApplicationStoppedTime only prints out the stopped time for Zing. A separate command line argument is available which prints out safepoint statistics, but as this post grows longer I'm losing my will to rerun and parse stuff...<br />
We add the JVM argument: "-XX:CompileCommand=option,*.contains1ToK,+KeepSafepointsInCountedLoops" and watch the world burn:<br />
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;">
<b><u>K |contain1(us)| containsK(us) | park(ms) | park+FDT(ms)| Stp.ms|</u></b>
<br />
1 | 33.0 ± 0.6 | 40.1 ± 1.2 | 100.0 ± 0.0 | 100.2 ± 0.0 | 0.0 |<br />
10 | 32.8 ± 0.7 | 405.1 ± 13.1 | 100.0 ± 0.0 | 100.2 ± 0.0 | 0.0 |<br />
100 | 33.3 ± 0.7 | 4124.8 ± 104.4 | 100.0 ± 0.0 | 100.3 ± 0.0 | 0.1 |<br />
1000 | 32.9 ± 0.2 | 40839.6 ± 854.0 | 100.0 ± 0.0 | 101.0 ± 0.3 | 0.6 |<br />
10000 | 40.2 ± 2.9 | 374935.6 ± 7145.3 | 111.3 ± 3.7 | 100.2 ± 0.0 | 0.0 |</span></div>
<br />
SHAZZZING!... This is pretty fucking awesome<i>. </i>The impact on <i>contains1toK() </i>is visible, but I didn't have to change the code and the TTSP issue is <a href="https://www.youtube.com/watch?v=CEEPaYD5KZE&feature=youtu.be&t=147" target="_blank">gone man, solid GONE</a>. A look at the assembly verifies that the inner loop is still unrolled within the outer loop, with the addition of the requested <b>safepoint polls</b>. There are more safepoint polls than I'd like here, since the option is applied to both the outer method and the inlined method (to be improved upon sometime down the road...).<br />
<br />
<h4>
Solution 2: I use OpenJDK/Oracle, disable inner loop inlining</h4>
If the inner loop is in it's own method as our example, or if we can perhaps modify the code such that it is, we can ask the compiler to not inline that method. This should work out nicely here. To force the JVM to not inline the <i>containsNeedle()</i> method only in <i>contains1toK()</i> I made a copy of <i>containsNeedle()</i> called <i>containsNeedle2()</i> and used this argument: "-XX:CompilerCommand=dontinline,*.containsNeedle2". The results:<br />
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;">
<b><u>K |contain1(us)| containsK(us) | park(ms) | park+FDT(ms)|Stp.ms|TTSP.ms|</u></b><br />
1 | 39.9 ± 1.0 | 40.0 ± 2.6 | 100.0 ± 0.0 | 100.3 ± 0.0 | 0.1 | 0.0 |<br />
10 | 39.9 ± .7 | 395.6 ± 10.0 | 100.0 ± 0.0 | 100.3 ± 0.0 | 0.1 | 0.0 |<br />
100 | 41.3 ± 3.9 | 4030.6 ± 405.0 | 100.0 ± 0.0 | 100.3 ± 0.0 | 0.1 | 0.0 |<br />
1000 | 39.6 ± 2.1 | 34543.8 ± 1409.9 | 100.0 ± 0.0 | 100.3 ± 0.0 | 0.1 | 0.0 |<br />
10000 | 39.1 ± 1.2 | 328380.6 ± 8478.1 | 100.0 ± 0.0 | 100.3 ± 0.0 | 0.1 | 0.0 |</span></div>
<br />
<div>
That worked well! No harm to <i>contains1()</i>, no TTSP issue, and <i>contains1toK() </i>performance didn't suffer for it. Inlining is usually a key optimization you want to make sure happens, but in this case the right thing to do is stop it from happening. The work inside <i>containsNeedle()</i> is chunky enough to make this a good tradeoff. If the external loop was at a very high count, and the array size rather small this would have been a very different story.</div>
<h4>
Solution 3: Last resort. stop compiling the offending method</h4>
<div>
This is a bit terrible, but if things get bad AND you can't change the code, you can use a CompileCommand to exclude the method from compilation. The method will be force to stay in the interpreter, so will effectively have a <b>safepoint poll</b> between every 2 bytecodes. Performance will suck, but at least the application will be responsive. I am not gonna run this option, but if you want to add the following argument to the command line and watch it roll: "-XX:CompilerCommand=exclude,*.contains1toK"</div>
<div>
<br />
<b>Summary: </b>Threads which suffer from large TTSP will cause other threads to wait for them in the presence of a global safepoint attempt.</div>
<h3>
</h3>
<h3>
Example 3: Safepoint Operation Cost Scale</h3>
<div>
<script src="https://gist.github.com/nitsanw/3b39a4abc44c7475c90c.js"></script><br />
For this example we replace the <i>safepoint()</i> method with gathering thread stacks using <i>ManagementFactory.getThreadMXBean().dumpAllThreads(false, false)</i>. Gathering thread stacks also occurs at a safepoint and the safepoint operation cost will scale with the number of threads. We construct a number of idle threads in the setup method to control the <b>safepoint operation</b> cost. To make sampling the threads stack expensive we park the fuckers 256 frames deep.<br />
Different <b>safepoint operations</b> scale with different aspects of your process. Certain GC algorithms stop the world phases will scale with heap size. Profiling by collecting thread dumps (as done by JVisualVM sampler for instance) will scale with the number of threads (idle or not) and the stack depth. Deflating monitors will scale with the number of monitors in your application etc.<br />
Here's what happens as we pile on the idle threads (running only <i>contains1()</i>):</div>
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;"><u><b>IT | contain1(us)| park(ms) | park+DTS(ms) | Stp.ms|TTSP.ms|</b></u>
<br />
0 | 38.0 ± 1.6 | 100.0 ± 0.0 | 100.5 ± 0.0 | 0.2 | 0.0 |<br />
1 | 40.7 ± 1.4 | 100.1 ± 0.0 | 102.2 ± 0.0 | 0.6 | 0.0 |<br />
10 | 44.7 ± 0.8 | 100.2 ± 0.1 | 117.2 ± 0.1 | 5.5 | 0.0 |<br />
100 | 75.5 ± 0.8 | 122.6 ± 3.5 | 261.9 ± 2.3 | 61.3 | 0.1 |<br />
1000 | 204.5 ± 38.5 | 257.5 ± 28.8 | 1414.2 ± 47.0 | 313.0 | 1.0 |<br />
</span></div>
<br />
We get long stops and short TTSPs! hoorah! You can play with the number of threads and the depth of the stack to get the effect you want. It is interesting that most profilers using JVMTI::Get*StackTrace opt for sampling all threads. This is risky to my mind as we can see above the profiling overhead is open ended. It would seem reasonable to sample 1 to X threads (randomly picking the threads we sample) to keep the overhead in check on the one hand and pick a reasonable balance of TTSP vs operation cost on the other.<br />
As mention previously, JVMTI::GetStackTrace does not cause a global safepoint on Zing. Sadly AFAIK no commercial profilers use this method, but if they did it would help. The JMX thread bean does expose a single thread profiling method, but this only delegates to the multi-thread stack trace method underneath, which does cause a global safepoint...<br />
<br />
<b>Summary: </b>This final step adds another interesting control for us to play with completing our set:<br />
<ul>
<li><b>Safepoint poll</b> interval -> Will drive time to safepoint. CPU scarcity, swapping and page faults are other common reasons for large TTSP.</li>
<li><b>Safepoint operation</b> interval -> Sometimes in your control (be careful how you profile/monitor your application), sometimes not.</li>
<li><b>Safepoint operation cost</b> -> Will drive time in safepoint. Will depend on the operation and your application specifics. Study your GC logs.</li>
</ul>
<br />
<ul>
</ul>
<h3>
Example 4: Adding Safepoint Polls Stops Optimization</h3>
<div>
This is a repeat of a benchmark I ran to demonstrate the side effect of safepoint polls in loops on loop optimizations in <a href="https://github.com/netty/netty/pull/3969#issuecomment-132559757" target="_blank">the context of changing one of the Netty APIs</a>. This post is so long by now that if you had the strength to read this far you can probably click through to the ticket. I'll quote the basic results here for completeness.<br />
The benchmark:<br />
<script src="https://gist.github.com/nitsanw/49ce14d65b4759a1af1d.js"></script>
We are debating replacing the length and DataSet index types from int to longs. The impact on usage inside loops is substantial as demonstrated by the following results:<br />
<div>
<span style="background-color: #f3f3f3; font-family: "courier new" , "courier" , monospace; font-size: small;"><u><b>Benchmark (size) Mode Cnt Score Error Units</b></u><br />
copyInt 1000 avgt 5 76.943 ± 14.256 ns/op<br />
copyLong 1000 avgt 5 796.583 ± 55.583 ns/op<br />
equalsInt 1000 avgt 5 367.192 ± 16.398 ns/op<br />
equalsLong 1000 avgt 5 806.421 ± 196.106 ns/op<br />
fillInt 1000 avgt 5 84.075 ± 19.033 ns/op<br />
fillLong 1000 avgt 5 567.866 ± 10.154 ns/op<br />
sumInt 1000 avgt 5 338.204 ± 44.529 ns/op<br />
sumLong 1000 avgt 5 585.657 ± 105.808 ns/op</span></div>
There are several optimizations negated by the transition from counted to un-counted loop (uncounted loops currently include loops limited by a long, and therefore include a safepoint):<br />
<ul>
<li>Loop unrolling : This is not really a safepoint issue, can be fixed as per the KeepSafepointsInCountedLoops option</li>
<li>Fill pattern replacement (-XX:+OptimizeFill): the current fill implementation has no safepoints and the loop pattern it recognizes does not allow it.</li>
<li>Superword optimizations (-XX:+UseSuperword): current Superword implementation doesn't allow for long loops. This will probably require the compiler to construct an outer loop with an inner superword loop </li>
</ul>
</div>
The latter 2 have been <a href="http://psy-lob-saw.blogspot.co.za/2015/04/on-arraysfill-intrinsics-superword-and.html" target="_blank">discussed on this blog before in some detail</a>.<br />
<br />
<h3>
Final Summary And Testament</h3>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkvriHNHxT7Q_9tJ2rzTunNLxjR9rQRuDy596Af1U52fKCzI-9JUGGC5_rFKCUHg4Xn35tKOTipGiuJFwkRhW76T_Zq1NHwH4fdbMP6lkmWQxuATfeNP2Of3i21Q5_2pBfqfFzLgFl1EY/s1600/fmercury-thumbsup.jpeg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkvriHNHxT7Q_9tJ2rzTunNLxjR9rQRuDy596Af1U52fKCzI-9JUGGC5_rFKCUHg4Xn35tKOTipGiuJFwkRhW76T_Zq1NHwH4fdbMP6lkmWQxuATfeNP2Of3i21Q5_2pBfqfFzLgFl1EY/s1600/fmercury-thumbsup.jpeg" /></a>If you've read this far, you deserve a summary, or a beer, or something. WELL DONE YOU!!! Here's what I consider the main takeaways from this discussion/topic:</div>
<div>
<ul>
<li>Safepoint/Safepoint poll/Safepoint operation are 3 distinct terms, often mingled. Separating them allows us to consider their costs and impacts individually.</li>
<li>In most discussions the term Safepoint is used to describe a global safepoint. Some JVMs can control safepoint transition on the thread level. By considering the individual thread state we can more easily reason about the transition costs on a thread by thread level. </li>
<li>The failure of any thread to reach a safepoint (by hitting a safepoint poll) can cripple the whole JVM. In this sense most any bit of Java code can be considered blocking (as in not lock free). Consider for example a CAS loop, it's not counted, it will have a safepoint poll in it, the thread executing can be blocked indefinitely if another Java thread is suspended while not at a safepoint.</li>
<li>JVMs have many safepoint operations. If you are worried about their impact you should enable -XX:+PrintGCApplicationStoppedTime. Further detail on what types of safepoint operations were involved can be made available using -XX:+PrintSafepointStatistics –XX:PrintSafepointStatisticsCount=X. Try it out, it's educational.</li>
</ul>
</div>
<div>
<h3>
References</h3>
</div>
<div>
<ul>
<li>From the <a href="http://openjdk.java.net/groups/hotspot/docs/RuntimeOverview.html#Thread%20Management|outline" target="_blank">OpenJDK Runtime Overview</a> look for the "VM Operations and Safepoints" section. Note that the term safepoint is used here to mean a global safepoint.</li>
<li><a href="http://jpbempel.blogspot.co.za/2013/03/safety-first-safepoints.html" target="_blank">J.Bempel's post</a> covering many safepoint operations and observing the impact of safepoints.</li>
<li><a href="http://blog.ragozin.info/2012/10/safepoints-in-hotspot-jvm.html" target="_blank">Alexey Ragozin's post</a> also covers similar definition to the Runtime Overview and explores the safepoint poll mechanism and logging options.</li>
<li><a href="https://groups.google.com/forum/#!msg/mechanical-sympathy/GGByLdAzlPw/cF1_XW1AbpEJ" target="_blank">Gil Tene post on the Mechanical Sympathy mailing list</a> which closely matches the "Thread State" approach above. Gil goes into the topic in the context of Unsafe operations and assumptions surrounding safepoints. </li>
</ul>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com14tag:blogger.com,1999:blog-5171098727364395242.post-10020461543282148912015-10-19T09:46:00.002+01:002015-10-19T09:46:34.495+01:00Expanding The Queue interface: Relaxed Queue Access<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaS3SPxEOO41h872W4NvLghpRO4Cwj0bSOtMreF9ucWzckPdYL_04QM8bEpdfVlcat4lodVHePWsukwfCFZkqKrhWjQ0McqUb0OzQZZ5msGTvOoOHKc4hOuqc8Ap-3Fb_UBUiVDxmv_UE/s1600/Relax_single.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaS3SPxEOO41h872W4NvLghpRO4Cwj0bSOtMreF9ucWzckPdYL_04QM8bEpdfVlcat4lodVHePWsukwfCFZkqKrhWjQ0McqUb0OzQZZ5msGTvOoOHKc4hOuqc8Ap-3Fb_UBUiVDxmv_UE/s200/Relax_single.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
Continuing from <a href="http://psy-lob-saw.blogspot.co.za/2015/08/an-extended-queue-interface.html" target="_blank">previous post</a> on the expansion of the Queue interface to support new ways of interacting with queues I have gone ahead and implemented relaxedOffer/Poll/Peek for the JCTools queues. This was pretty easy as the original algorithms all required a bit of mangling to support the strong semantic and relaxing it made life easier. Implementing batch/continuous interactions was a bit more involved, but the results were interestingly rewarding. I will save discussing the drain/fill results for a follow up post.<br />
The new interface is <a href="https://github.com/JCTools/JCTools/blob/master/jctools-core/src/main/java/org/jctools/queues/MessagePassingQueue.java" target="_blank">here</a>, feedback is welcome.<br />
TL;DR:<br />
<br />
<ul>
<li>Throughput is improved in all cases (~5-15% increase)</li>
<li>Cost of failure (when offer/poll fail to return/add a value) is dramatically reduced</li>
<li>Latency for bursts is improved (as much as 40% reduction)</li>
</ul>
<br />
<h3>
Relaxed Access API</h3>
<div>
This ended up a very small API change, most of the difficulty here being choice of name. I went with the 'relaxed' prefix as suggested by J.P.Bempel (as opposed to the 'weak' prefix considered earlier):</div>
<div>
<script src="https://gist.github.com/nitsanw/d2efff5b209a6429965f.js"></script></div>
<div>
<br /></div>
<h3>
Relaxed vs. Non-Relaxed Example</h3>
<div>
<div>
The 'relaxation' of the queue full/empty reporting in these methods allows us to remove an extra load of the 'opposite' index and avoiding waiting around for delayed visibility. Here's how <a href="https://github.com/JCTools/JCTools/blob/edaa8503b040b48e0f5d92eb4129813993e3ca13/jctools-core/src/main/java/org/jctools/queues/MpscArrayQueue.java#L234" target="_blank">MPSCArrayQueue::poll</a> was changed to become 'relaxed':<br />
<script src="https://gist.github.com/nitsanw/bb60c6701a989c7b6173.js"></script><br />
So now we can drop lines 11-15, it doesn't look like much. We wouldn't really expect this gap in offer (see <a href="https://github.com/JCTools/JCTools/blob/edaa8503b040b48e0f5d92eb4129813993e3ca13/jctools-core/src/main/java/org/jctools/queues/MpscArrayQueue.java#L153" target="_blank">offer</a> for details) to happen very often, so the overhead can typically amount to an extra load of the producerIndex and a predictable branch.<br />
Also, since poll on an MPSC is actually cheaper than the offer anyhow, aren't we optimizing a non-issue? Could we not be actually making throughput worse by making the consumer more aggressive?</div>
</div>
<div>
<br /></div>
<h3>
Throughput Impact Of Relaxation</h3>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpSdg1GVVlHSTytpF_qEeqFDhtsQgs8dtNmVyLO6ViBIj1ai_N1_4_3uWT7P1ZuLDl7Z6zuowvBfzVydkGUYkKU-eBogtf4MjrvXb7CtEEflqkJSPGllIEGqnL_kiRTHXp4oxb2hULf7w/s1600/frabz-How-do-I-relax-Chamomile-motherfers-Thats-how-I-fing-relax-21e4cb.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="175" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpSdg1GVVlHSTytpF_qEeqFDhtsQgs8dtNmVyLO6ViBIj1ai_N1_4_3uWT7P1ZuLDl7Z6zuowvBfzVydkGUYkKU-eBogtf4MjrvXb7CtEEflqkJSPGllIEGqnL_kiRTHXp4oxb2hULf7w/s200/frabz-How-do-I-relax-Chamomile-motherfers-Thats-how-I-fing-relax-21e4cb.jpg" width="200" /></a></div>
To test the impact of this change I constructed a <a href="http://psy-lob-saw.blogspot.co.za/p/jmh-related-posts.html" target="_blank">JMH</a> benchmark that measures throughput when both consumer and producer spin to offer/poll as fast as they can (see the code <a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/jmh/throughput/MpqThroughputBackoffNone.java" target="_blank">here</a>). I ran the benchmark on a Haswell machine(i7-4770 CPU@3.40GHz/Ubuntu/OracleJVM8u60). The benchmarks were pinned to run cross core (no 2 threads share a physical core, i.e. no hyperthreading). Benchmarks were run with 5 forks, 10 warmup iterations (1 second), 10 measured iterations (1 second). The score reports operations per microsecond, or millions of operation per second if you prefer. The throughput is the number of pollsMade, but the other figures are interesting as well.</div>
<div>
These results are with a single producer and consumer:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>Type | 1P1C | Normal± Err | Relax ± Err |</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spsc | offersFailed | 0.0 ± 0.1 | 0.1 ± 0.1 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spsc | offersMade | 439.1 ± 4.1 | 432.1 ± 10.0 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spsc | pollsFailed | 0.2 ± 0.5 | 0.1 ± 0.2 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spsc | pollsMade | 438.9 ± 4.1 | 431.9 ± 10.0 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | offersFailed | 0.0 ± 0.0 | 0.0 ± 0.0 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | offersMade | 22.7 ± 0.6 | 31.9 ± 0.8 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | pollsFailed | 0.1 ± 0.1 | 6.6 ± 0.9 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | pollsMade | 22.8 ± 0.6 | 31.9 ± 0.8 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spmc | offersFailed | 0.2 ± 0.2 | 81.0 ± 5.4 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spmc | offersMade | 23.5 ± 0.6 | 26.8 ± 0.6 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spmc | pollsFailed | 0.1 ± 0.1 | 0.2 ± 0.3 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Spmc | pollsMade | 23.2 ± 0.6 | 26.6 ± 0.6 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersFailed | 0.0 ± 0.0 | 0.0 ± 0.1 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersMade | 71.4 ± 5.1 | 71.7 ± 4.8 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsFailed | 0.0 ± 0.1 | 0.3 ± 0.9 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsMade | 71.4 ± 5.1 | 71.7 ± 4.8 |</span></div>
</div>
<br />
<ul>
<li>The SPSC case serves as a baseline, and I expected results to be the same. There's a 2% difference which is close to the error reported. Maybe the inlining depth impacted the performance slightly, or perhaps the benchmark is not as stable as I'd like.</li>
<li>The MPSC case shows 50% improvement. This is pretty awesome. We also notice the consumer now fails more. This is indicative to the reduced cost of failure to find elements in the queue. The cache miss on the producerIndex was slowing down both the consumer and the producer. This phenomena has been observed in other post, but my best attempt at an explanation is <a href="http://psy-lob-saw.blogspot.co.za/2013/09/diving-deeper-into-cache-coherency.html" target="_blank">here</a>. The bottom line being that the mutator suffers from losing the exclusive state of the cache line in his own cache, while the reader gets hit with a read miss (going to LLC as the threads are on separate cores).</li>
<li>The SPMC case shows 15% improvement on throughput. The cost of failing to offer has gone down dramatically.</li>
<li>The MPMC case shows no change, but does show great throughput. This could be down to the very different algorithm in play or just because the MPMC queue is well balanced and thus behaves very well in a balanced use case as above. This may lead you to believe MPMC is a better choice than MPSC, but as the latency results and contended throughput results show this is an anomaly of the benchmark load. Because MPSC presents a much cheaper poll than offer the queue is always empty, making the contention on the queue buffer worse than it would be for MPMC where offer and poll are of similar costs.</li>
</ul>
Using the command line controls over thread groups sizes (-tg) I ran a few more cases. Did I mention I love JMH?<br />
Here's MPMC and MPSC with 3 producers and 1 consumer:<br />
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>Type | 3P1C | Normal± Err | Relax ± Err |</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | offersFailed | 0.0 ± 0.0 | 0.1 ± 0.6 | </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | offersMade | 12.7 ± 0.1 | 13.4 ± 0.1 | </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | pollsFailed | 4.1 ± 0.5 | 115.3 ± 4.9 | </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpsc | pollsMade | 12.7 ± 0.1 | 13.4 ± 0.1 | </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersFailed | 0.0 ± 0.0 | 0.0 ± 0.0 | </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersMade | 11.1 ± 0.0 | 11.7 ± 0.1 | </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsFailed | 0.3 ± 0.1 | 60.4 ± 10.0 | </span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsMade | 11.1 ± 0.0 | 11.7 ± 0.1 |</span></div>
<ul>
<li>MPSC case shows slight improvement to throughput of roughly 5%, but the improvement to cost of failure is very visible here.</li>
<li>MPMC case improves by 5% and the improved cost of failure is similarly visible.</li>
</ul>
<div style="margin: 0px;">
Here's MPMC and SPMC with 1 producers and 3 consumer:</div>
<span style="font-family: Courier New, Courier, monospace;"> <b>Type | 1P3C | Normal± Err | Relax ± Err |</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"> Spmc | offersFailed | 1.5 ± 0.2 | 125.9 ± 4.6 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Spmc | offersMade | 13.4 ± 0.1 | 14.0 ± 0.0 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Spmc | pollsFailed | 0.0 ± 0.0 | 0.0 ± 0.0 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Spmc | pollsMade | 13.1 ± 0.1 | 13.7 ± 0.0 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersFailed | 23.9 ± 0.8 | 124.1 ± 3.2 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersMade | 11.0 ± 0.0 | 12.1 ± 0.0 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsFailed | 0.0 ± 0.2 | 1.0 ± 1.6 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsMade | 10.7 ± 0.0 | 11.9 ± 0.0 |</span><br />
<ul>
<li>SPMC case shows 5% improvement and great improvement to failure performance.</li>
<li>MPMC case improves by 11% and the improved cost of failure is similarly visible.</li>
</ul>
Here's MPMC only, with 2 producers and 2 consumers:<br />
<span style="font-family: Courier New, Courier, monospace;"> <b>Type | 2P2C | Normal± Err | Relax ± Err |</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersFailed | 1.4 ± 0.8 | 0.7 ± 0.6 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | offersMade | 14.8 ± 0.8 | 16.0 ± 0.6 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsFailed | 0.0 ± 0.0 | 0.4 ± 0.5 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> Mpmc | pollsMade | 14.7 ± 0.8 | 15.9 ± 0.6 |</span><br />
<ul>
<li>The improvement to throughput is a nice 8%, but the error is not that far from it so I'm not too confident without lots more runs. This is a similarly symmetric load to the 1P1C and the balance is showing through.</li>
</ul>
I could go on to higher core counts etc, but I'm actually feeling the win here is pretty conclusive:<br />
<div>
<ul>
<li>Throughput is generally improved</li>
<li>Failure to make progress is significantly cheaper</li>
</ul>
Why is this failure mode important? This is particularly important when you have a consumer/producer which is able to make progress in the face of such a failure. A common example of this pattern is a Selector thread which also has an incoming queue of commands to execute, when there's nothing in the queue we would like to go back to selecting on the selector.</div>
<div>
<br /></div>
<div>
<br />
<h3>
Latency impact of Relaxation</h3>
</div>
<div>
Is this going to help us in the case of burst handling latency? There's a benchmark for that too! Or rather there's an estimate of burst processing cost. Code is <a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/jmh/latency/spsc/MpqRelaxedBurstCost.java" target="_blank">here</a>. The general idea being to send a burst of messages from the producer, then wait for completion notice from the consumer.<br />
Here goes:<br />
<span style="font-family: Courier New, Courier, monospace;"> Size | Type | Normal±Err | Relax± Err |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 1 | Spsc | 126 ± 2 | 126 ± 1 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 1 | Mpsc | 144 ± 2 | 140 ± 1 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 1 | Spmc | 187 ± 2 | 189 ± 2 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 1 | Mpmc | 190 ± 8 | 203 ± 2 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 10 | Spsc | 193 ± 2 | 192 ± 3 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 10 | Mpsc | 611 ± 22 | 429 ± 7 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 10 | Spmc | 309 ± 3 | 307 ± 1 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 10 | Mpmc | 807 ± 19 | 677 ± 15 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 100 | Spsc | 530 ± 7 | 528 ± 4 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 100 | Mpsc | 5243 ± 158 | 3023 ± 107 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 100 | Spmc | 1240 ± 7 | 1250 ± 6 |</span><br />
<span style="font-family: Courier New, Courier, monospace;"> 100 | Mpmc | 4684 ± 382 | 3665 ± 111 |</span><br />
<br />
<ul>
<li><span style="font-family: inherit;">Same as before, SPSC serves as a sanity check/baseline. No change is good.</span></li>
<li><span style="font-family: inherit;">SPMC/MPSC results are pretty similar when sending a single message. MPMC seems slightly regressed.</span></li>
<li><span style="font-family: inherit;">When sending 10/100 messages in a burst we see significant benefits to using the relaxed methods, in </span>particular<span style="font-family: inherit;"> for the MPSC and MPMC cases.</span></li>
</ul>
</div>
<div>
<h3>
<br /></h3>
<h3>
Summary</h3>
</div>
<div>
Relaxing is good for you, deep down you always knew it. Turns out it ain't half bad for your queues either :-). The new API will be part of the next version of JCTools, coming out as soon as I can work through my Maven mojo phobia.</div>
<div>
<br /></div>
<div>
Thanks <a href="http://jpbempel.blogspot.com/" target="_blank">J.P. Bempel</a> and <a href="https://twitter.com/philip_aston" target="_blank">Phil Aston</a> for their kind review, remaining errors are a result of me refusing to listen to them persnickety bastards. </div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com0tag:blogger.com,1999:blog-5171098727364395242.post-55558741463238864022015-08-19T16:39:00.000+01:002015-08-19T16:39:27.651+01:00An extended Queue interface<div class="separator" style="clear: both; text-align: left;">
In my work on <a href="https://github.com/JCTools/JCTools/issues" target="_blank">JCTools</a> I have implemented a fair number of concurrent access queues. The Queue interface is part of the java.util package and offers a larger API surface area than I found core to concurrent message passing on the one hand, and still missing others. I'm hoping to solicit some discussion on some new methods, and see if I can be convinced to implement those I decided to avoid thus far.</div>
<br />
<h3>
Inter-thread Message Passing vs. Generic Collection</h3>
<div>
The queue data structure traditionally has an offer/push/enqueue method for writing into the queue and a poll/pop/dequeue method for removing elements. This should be enough for anyone, right?</div>
<div>
Turns out some other useful methods come to mind when using a queue:</div>
<div>
<ul>
<li>peek: looking, not touching</li>
<li>size: for monitoring and heuristics and so on</li>
<li>isEmpty: a potentially more efficient assertion than size == 0</li>
</ul>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyNsQiJhbgQ373WF85LBUqLNH5mNzplAlR-H90_yrvc1XF832zsCmPhSaPclNuE2RVGFr9QtTXVQ7d1S0hNYFAjNWvdOTxM6Ejh4Rwfimc-FjESxKOl6x5XHzskHVfv9Wsu_97p3ikOOc/s1600/funny-weird-facts-8-queue.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="156" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyNsQiJhbgQ373WF85LBUqLNH5mNzplAlR-H90_yrvc1XF832zsCmPhSaPclNuE2RVGFr9QtTXVQ7d1S0hNYFAjNWvdOTxM6Ejh4Rwfimc-FjESxKOl6x5XHzskHVfv9Wsu_97p3ikOOc/s320/funny-weird-facts-8-queue.jpg" width="320" /></a>But I would argue that implementing the full scope of the java.util.Collection methods is slightly redundant. Why is it good to have offer/poll also exposed as add/remove/element?</div>
<div>
I have chosen to avoid implementing iterators for queues in JCTools, they seem to introduce a large complexity and I'm not convinced users out there need an iterator for their concurrent queue. The duplicate collection methods I get from extending AbstractQueue for free are alright though I don't see why anyone would actually use them.<br />
So lots of methods I don't want, but also some methods missing, in particular for concurrent queue semantics. In particular I have come to the conclusion the following are both practical to implement and helpful to users:<br />
<ul>
<li>weakOffer/Poll/Peek: the same as the 'strong' methods but without the requirement that allowing for spurious negative return value. This is particularly useful for queues where we are able to detect the inability to claim the next slot independently of the queue being full or empty. The 'weak' prefix is different from the Atomic* classes definition of 'weak' as it has no ordering implications, but I've been unable to come up with a better name(suggestions are most welcome).</li>
<li>drain(receiver)/fill(provider): this is attractive for cases where a receiver/provider function exists such that calls to it are guaranteed to succeed (e.g. receiver could just throw elements away, producer might return a constant). There are some small savings to be made here around the function calling loop, but for some these small savings might prove significant. A perpetual version of these methods could also take some sort of backoff strategy and spin when hitting an empty/full condition (similar to the Disruptor's BatchProcessor workflow).</li>
<li>batchPoll/Offer: this is attractive for processing known number of elements out of/into a queue. This is particularly profitable when multiple consumer/producers contend as the granularity of contention is reduced. It is also potentially damaging as batches can introduce large 'bubbles' into the queue and increase the unfairness. These methods could support a receiver/provider interface or use an array (enables batch array copies, guarantees quick release of a 'bubble').</li>
<li>offeredElementsCount/polledElementsCount: this is purely for reporting and would tend to be a long reflecting the producer/consumer counter. For linked queues this adds a new overhead, so while this is a clearly useful feature I'm reluctant to make it a compulsory part of the new interface.</li>
</ul>
</div>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyNsQiJhbgQ373WF85LBUqLNH5mNzplAlR-H90_yrvc1XF832zsCmPhSaPclNuE2RVGFr9QtTXVQ7d1S0hNYFAjNWvdOTxM6Ejh4Rwfimc-FjESxKOl6x5XHzskHVfv9Wsu_97p3ikOOc/s1600/funny-weird-facts-8-queue.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a>I plan to implement the above for <a href="https://github.com/JCTools/JCTools" target="_blank">JCTools</a> in the near future. What do you think? Please opine in the comments or via the <a href="https://github.com/JCTools/JCTools/issues" target="_blank">projects suggestion box</a>.</div>
<div>
Thanks :)</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com3tag:blogger.com,1999:blog-5171098727364395242.post-80604926366813973342015-07-27T08:26:00.000+01:002017-03-31T07:59:32.159+01:00JMH perfasm explained: Looking at False Sharing on Conditional Inlining<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim9IOFHQWfzR5I4kV79NEy68vdJSA9YD8dDplVV93KJAzo_uAihL4iSAstITYX-VzVc7spphUfr1yqY9oH60_Xxn-87DTMcHmRTlmlSkoqdL8NI3QhOPaXNNde75eZphLyzaRqtuLcQcw/s1600/KATANA_FINAL_3K_02.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim9IOFHQWfzR5I4kV79NEy68vdJSA9YD8dDplVV93KJAzo_uAihL4iSAstITYX-VzVc7spphUfr1yqY9oH60_Xxn-87DTMcHmRTlmlSkoqdL8NI3QhOPaXNNde75eZphLyzaRqtuLcQcw/s200/KATANA_FINAL_3K_02.jpg" width="200" /></a></div>
There is an edge that <a href="http://openjdk.java.net/projects/code-tools/jmh/" target="_blank">JMH</a> (read the<a href="http://psy-lob-saw.blogspot.com/p/jmh-related-posts.html" target="_blank"> jmh resources page</a> for other posts and related nuggets) has over other frameworks. That edge is so sharp you may well cut yourself using it, but given an infinite supply of bandages you should definitely use it :-) This edge is the ultimate profiler, the perfasm (pronounced PERF-AWESOME!, the exclamation mark is silent). I've been meaning to write about it for a while and as it just saved my ass recently...<br />
<br />
<h3>
SpscGrowableArrayQueue's False Sharing Issue</h3>
<a href="https://github.com/JCTools/JCTools" target="_blank">JCTools</a> includes a specialized SPSC (single-producer/consumer) bounded queue aimed at actor systems, the <a href="https://github.com/JCTools/JCTools/blob/42fa9fab84026952fc31b06b69ba156b92c2d11b/jctools-core/src/main/java/org/jctools/queues/SpscGrowableArrayQueue.java" target="_blank">SpscGrowableArrayQueue</a>. This queue is quite neat because it is combining the compactness of a linked queue with the awesome throughput of an array backed queue. The idea is quite simple, and similar in spirit to an ArrayList:<br />
<ol>
<li>Start with a small buffer, default to 16 elements</li>
<li>As long as the queue doesn't grow beyond that just stay small</li>
<li>If on offer you find that the queue is full double the size of the underlying buffer</li>
</ol>
The mechanics of the queue resizing, handling intermediate queue states, and detecting a new buffer from the consumer are a bit involved and will perhaps be expanded on some other day.<br />
Because the queue is geared at actor systems per queue footprint is important, so I reworked the memory layout of SpscArrayQueue and skimped on the padding where possible. If you have no idea what I mean by padding, and why it is done you can read the following posts:<br />
<ol>
<li><a href="http://psy-lob-saw.blogspot.com/2013/09/diving-deeper-into-cache-coherency.html" target="_blank">False sharing and the MESI protocol details related to cached counters</a>: This explores the motivation of padding fields in the first place</li>
<li><a href="http://psy-lob-saw.blogspot.com/2013/05/know-thy-java-object-memory-layout.html" target="_blank">Discovering and controlling object layout</a>(Using an early version of <a href="http://openjdk.java.net/projects/code-tools/jol/" target="_blank">JOL</a>): This show cases the JOL tool and how it should be used to discover object layout</li>
<li><a href="http://psy-lob-saw.blogspot.com/2013/07/single-producer-single-consumer-queue.html" target="_blank">SPSC Revisited - part I: An empiricist tale</a>: This post discusses the inlining of counters into a queue use inheritance to control field order and introduce padding. In this post I show how to pad the counter from each other and also how to pre/post pad the class and buffer.</li>
<li><a href="http://psy-lob-saw.blogspot.com/2014/06/notes-on-false-sharing.html" target="_blank">A more high level summary of false sharing is given in this post</a></li>
</ol>
With an overhead of roughly 128b per padding this was instrumental to reducing the high memory cost per queue. The hot counters were still padded from each other, but the class pre/post padding were removed as well as the padding of the elements array.<br />
So, I gave up on some of the padding, but reasoned that in most cases this should not make a difference because the hottest fields were still padded from each other.<br />
<br />
<h3>
Why so slow?</h3>
<div>
Now, I expect a queue that is very similar to <a href="https://github.com/JCTools/JCTools/blob/42fa9fab84026952fc31b06b69ba156b92c2d11b/jctools-core/src/main/java/org/jctools/queues/SpscArrayQueue.java" target="_blank">SpscArrayQueue</a>, but adds some features to be slower. There's just very little you can do about this, doing more will cost you something. But given that resizing is a rather exceptional event for this queue I thought this will be a minor hit, maybe 10-20% reduction in performance (for a certain definition of performance). <a href="https://github.com/JCTools/JCTools" target="_blank">JCTools</a> has some <a href="https://github.com/JCTools/JCTools/tree/master/jctools-benchmarks" target="_blank">benchmarks</a> included and I ran the QueueThroughputBackoffNone which will have a producer and consumer threads chewing on the queue as hard as they can.<br />
Since the numbers are not very important here, I'll stick to running benchmarks on my laptop (an MBP 11,1/ Ubuntu 15.04/<a href="http://www.azulsystems.com/products/zulu/downloads" target="_blank">Zulu</a> JDK8u45 - Zulu is a supported and tested OpenJDK build) in this post. Rest assured that I have confirmed on real hardware the same results. The results I quote below are the <i>pollsMade</i> figure which reflects the actual delivered throughput.</div>
<div>
To my puzzlement I found:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SpscArrayQueue 361.223 ± 7.156 ops/us</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SpscGrowableArrayQueue 64.277 ± 31.803 ops/us</span></div>
<br />
Crap performance and terrible variance, bugger me sideways.<br />
Looking at the code, I thought there must be something I was doing to upset the mighty JIT spirits. Maybe my methods were too big? My branches too unpredictable? My variable names too offensive? So I tweaked the code this way and that, looked at the inlining log (-XX:+PrintInlining) and the assembly (-XX:+PrintAssembly/-XX:CompileCommand=print,*) I got some minor improvements, but it mostly still sucked. What's wrong? A quick look at "-prof perfasm" and a fair amount of head scratching lead to the answer. The code (<a href="https://github.com/JCTools/JCTools/blob/7f78a25f40740df12def944faa67b318e6f5e8f2/jctools-core/src/main/java/org/jctools/queues/SpscGrowableArrayQueue.java#L26" target="_blank">before</a> and <a href="https://github.com/JCTools/JCTools/blob/42fa9fab84026952fc31b06b69ba156b92c2d11b/jctools-core/src/main/java/org/jctools/queues/SpscGrowableArrayQueue.java#L29" target="_blank">after</a>) is on github, the main focus for this post is perfasm and its usage so I won't dive into it.<br />
<br />
<h3>
Before we start: How does perfasm work?</h3>
<div>
To use perfasm you'll need a Linux(or Windows) OS, running on real hardware, the relevant perf tool installed, and a <a href="http://psy-lob-saw.blogspot.com/2013/01/java-print-assembly.html" target="_blank">JVM setup to print assembly</a>.</div>
<div>
Perf is (amongst other things) an instruction level profiler, of the kind that don't usually work with Java traditionally (though <a href="http://thread.gmane.org/gmane.linux.kernel/1890431" target="_blank">things are slowly changing</a>). One of the features offered by perf is the "perf record/annotate" workflow. Perf interrupts your process repeatedly and records the current PC (program counter) at each sample. This sampling of the program counter is recorded over a period of time to be post processed by the annotate feature which correlates samples to methods, lines of code and assembly instructions. The challenge for perf when dealing with Java code is that the binary form of each method only exists for the duration of that process lifetime. This means the PC is mostly referring to methods that are nowhere to be found when the annotation stage comes along.</div>
<div>
To summarize: perf record works, but perf annotate is broken for Java.</div>
<div>
To make perfasm work JMH captures the JVM compiler outputs by enabling the following flags:</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+UnlockDiagnosticVMOptions</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+LogCompilation</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:LogFile=...</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintAssembly</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintInterpreter</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintNMethods</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintNativeNMethods</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintSignatureHandlers</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintAdapterHandlers</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintStubCode</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintCompilation</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+PrintInlining</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:+TraceClassLoading</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> -XX:PrintAssemblyOptions=syntax</span></div>
</div>
The data collected here allows perfasm to do the annotation process by itself. I'll not bore you with the details of the output processing, but hat off to Mr. Shipilev who ploughed through the gory mess and made this work. The end result is a detailed output offering you the assembly of the hottest regions in your program, along with a list of the top hottest methods including both native and Java parts of the stack.<br />
Because perf allows the recording of any number of events the definition of hottest depends on the events you choose to profile. The default events are cycles and instructions, but you can specify any number of events (E.g. -Djmh.perfasm.events=cycles,cache-misses). The first event specified will be the 'hotness' qualifier.<br />
<br />
<h3>
What do you get?</h3>
<div>
The perfasm output is split into 4 sections:</div>
<div>
<ol>
<li>Annotated assembly output for the top HOT regions(titled "Hottest code regions (>10.00% "cycles" events):")</li>
<li>A list of the hot regions in your benchmark (titled "[Hottest Regions]"). This is the list of compiled methods inclusive of native methods, which is why we have the next section.</li>
<li>A list of the hottest methods after inlining, so only java methods in this section (titled "[Hottest Methods (after inlining)]").</li>
<li>A distribution of the cycles between types of regions (titled "[Distribution by Area]"). This will inform you of how the split goes between compiled code, kernel, JVM etc.</li>
</ol>
<div>
The most interesting of the 4 sections (and the only one I'm going to explain here) is the annotated hot regions section. I tend to edit the output of assembly spouting tools to make it more readable, but for the sake of explaining what's what here's the original output with footnotes:</div>
<div>
<script src="https://gist.github.com/nitsanw/97bf0b5a15fa1cd89727.js"></script></div>
<div>
To focus on the main features I trimmed out allot of the assembly code, which is where you see the "<MORE OF SAME/>" comments. Note the legend at the start of the output describing the 2 left most columns: cycles and instructions sample percentages.</div>
<div>
Here's the footnotes explained:</div>
<div>
<ol>
<li>(Line 7) This column is the instruction address. This is important as this column is both how perfasm matches the perf record output with the PrintAssembly output and how poor sods like yours truely have to look for jump destinations in the code.</li>
<li>(Line 7) This column has the actual instructions, in AT&T syntax (src -> dst).</li>
<li>(Line 7) This column is the instruction annotation generated by PrintAssembly. It is not always correct, but it's helpful in connecting the dots. Multi-level inlining of methods can be seen in action everywhere.</li>
<li>(Line 12) The condition/loop edge annotation is a wonderful recent addition to perfasm. It makes life marginally more bearable for the assembly consumer by connecting the jumps with their destinations. This is the start point annotation "<span style="background-color: white; color: #333333; font-family: "consolas" , "liberation mono" , "menlo" , "courier" , monospace; font-size: 12px; line-height: 16.7999992370605px; white-space: pre;">╭</span>"</li>
<li>(Line 16) This is the end point annotation "<span style="background-color: white; color: #333333; font-family: "consolas" , "liberation mono" , "menlo" , "courier" , monospace; font-size: 12px; line-height: 16.7999992370605px; white-space: pre;">↗</span>"</li>
</ol>
<div>
When looking at this you will be searching for suspect hot spots in the code. Some instruction which is chewing away your cycles. If you're in luck you'll be able to tweak your code just right to side step it and then you are off and away :-)</div>
</div>
</div>
<div>
<br /></div>
<h3>
Back on Track: Show me the cycles!</h3>
<div>
Right, so I point Captain Awesome at my benchmark and what do I see (trimmed addresses and edges as there's no important conditions here, also trimmed package names and moved code annotation left):</div>
<div>
<script src="https://gist.github.com/nitsanw/12cf2e56fae1fbda1e8b.js"></script></div>
<div>
This instruction, eating 30% of my cycles (line 7, needs <a href="https://www.youtube.com/watch?v=rjlSiASsUIs" target="_blank">HEALING!</a>), is part of a guard generated to ensure that inlining the poll method call is still a valid decision, that the instance observed at the callsite is of the class for which we inlined the poll method. This is not really a Java code line, this is the aftermath of inlining a virtual call (through an interface, see <a href="http://shipilev.net/blog/2015/black-magic-method-dispatch/" target="_blank">Aleksey's method dispatch inlining article</a> to learn more on inlining).<br />
To be fair, the above is kind of hard to understand. Why is a comparison between a constant and a register eating 30% of the cycles? according to every instruction 'cost' manual this should take 1 cycle (and the CPU can do 3-4 of them in parallel too), this is obviously not the instruction I'm looking for.<br />
This is a phenomena known as 'skid' where the reported instruction at a given sample is inaccurate because modern CPUs are <i>complicated.</i> See the <a href="http://www.spinics.net/lists/linux-perf-users/msg02157.html" target="_blank">following dialogue on the linux-perf-user mailing list</a>:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG3A5Ce76pYOLcn2rm-gboblXa8wwWFQhZ7u79aJgPt_i_gAALq5OUtpBns41MZCMZ5yxJTZlvLVD1ZJA-FNYx8qRZlY_bOLFlWJZVFgmM03iSr3GAJtqAghj8HnlbkeP0r8ZDdeUOlRM/s1600/marvin-gaye.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG3A5Ce76pYOLcn2rm-gboblXa8wwWFQhZ7u79aJgPt_i_gAALq5OUtpBns41MZCMZ5yxJTZlvLVD1ZJA-FNYx8qRZlY_bOLFlWJZVFgmM03iSr3GAJtqAghj8HnlbkeP0r8ZDdeUOlRM/s320/marvin-gaye.jpg" width="320" /></a></div>
<blockquote class="tr_bq">
<i>> I think Andi mentioned this to me last year -- that instruction profiling was no longer reliable.</i> </blockquote>
<blockquote class="tr_bq">
<b>It never was.</b> </blockquote>
<blockquote class="tr_bq">
<i>> Is this due to parallel and out-of-order execution? (ie, we're sampling the instruction pointer, but that's set to the resumption instruction, not the instructions being processed in the backend?).</i> </blockquote>
<blockquote class="tr_bq">
<b>Most problems are due to '<u>skid</u>': It takes some time to trigger the </b><b>profiling interrupt after the event fired. </b><b>[...] </b><b>There are also other problems, for example an event may not be tied </b><b>to an instruction. Some events have inherently large skid.</b></blockquote>
</div>
This sounds grim, but at the end of the day this is as accurate as profiling can get. It's not perfect, but it's still very valuable and you get used to it (there are ways to minimize skid discussed in the link, I've not tried those).<br />
What you end up doing is looking for instructions just before the blamed instruction, or instructions on which the blamed instruction is dependant, which may be more reasonably blamed for the bottleneck. Given the CMP is not the problem, we must ask why would the CPU spend so much time at it? A CMP or a TEST will often get blamed for the price of the load into the registers they use, in this case the CMP is most probably being blamed for the load of the queue type from the object header one instruction back:<br />
"<span style="font-family: "courier new" , "courier" , monospace;">0x8(%r12,%r11,8),%r8d ; implicit exception: dispatches to 0x00007f83c93f929d</span>"<br />
It doesn't help that the comment talks about some implicit exception (this is an implicit null check), where it could say "<span style="font-family: "courier new" , "courier" , monospace;">getkid(q) + implicit_nullchk(q)</span>" or something similar to indicate we are loading the kid (klass id) from the object header (see object header layout details <a href="http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/tip/src/share/vm/oops/oop.hpp" target="_blank">here</a>).<br />
Now that I had pointed a finger at this operation, I was still quite confused. This was not an issue for any of the other queues, why would loading the kid be such a bottleneck in this case? Maybe I'm wrong about this (always a good assumption)? To prove this is the issue I created a duplicate of the original benchmark where I use the SpscGrowableArrayQueue directly instead of going via the interface, for comparison I also benchmarked the SpscArrayQueue in the same fashion:<br />
<span style="font-family: "courier new" , "courier" , monospace;">SpscArrayQueue 369.544 ± 1.535 ops/us<br />
SpscGrowableArrayQueue 272.021 ± 12.133 ops/us</span><br />
<span style="font-family: inherit;">Now that's more like it! the expected 20% difference is more like 30%, but this is much closer. This still begs the question, why is the type check for </span>SpscGrowableArrayQueue so expensive? We can see that for SpscArrayQueue this makes very little difference, how is SpscGrowableArrayQueue different?<br />
<br />
<h3>
<a href="http://www.youtube.com/watch?v=DPYKfawYfVQ" target="_blank">Messing with the KID</a></h3>
I had a long hard look at this issue, which didn't help, then slept on it, which did help, and realized the problem here is that the object header is false-sharing with the producer fields. When I trimmed down the padding on this class in an effort to minimize allocation, I removed the array post and pre-padding as well as the class pre/post padding and reasoned that for the most part I need not worry about the object's neighbours false sharing. What I failed to realize was that the consumer and producer threads might be frequently hitting the object header, in this benchmark on every call. Once I realized this was the issue I reinstated the pre-padding such that the producer index is far enough from the object header to stop interfering with it and the problem went away (see <a href="https://github.com/JCTools/JCTools/blob/7f78a25f40740df12def944faa67b318e6f5e8f2/jctools-core/src/main/java/org/jctools/queues/SpscGrowableArrayQueue.java#L26" target="_blank">before</a> and <a href="https://github.com/JCTools/JCTools/blob/42fa9fab84026952fc31b06b69ba156b92c2d11b/jctools-core/src/main/java/org/jctools/queues/SpscGrowableArrayQueue.java#L29" target="_blank">after</a>, it's a messy diff as I fixed some other niggles while I was there, you can take the last version and play with adding/removing the padding and reordering the index and cold fields to verify my claims).<br />
Here's the (slightly trimmed for readability) perfasm output for the same conditional inlining check in the padded version:<br />
<script src="https://gist.github.com/nitsanw/26fdebafabc59d80ab18.js"></script><br />
As an interesting side note, this issue was not visible in the hand rolled throughput benchmarks. This is because in those benchmarks the queue is hoisted into a variable before the consume/produce loop which means the conditional inlining check can be done out of loop as well. This is great for getting nice numbers but hides an issue which users are likely to hit. Credit goes to JMH for herding benchmarks down a path which forces these issues into the measured scope.<br />
<br />
<h3>
Summary</h3>
<div>
The main focus of this post is perfasm, I hope it helps get you started on using it. The broader context in which you would use this tool is a less explicit background. I use perfasm regularly, but I'm also reasonably happy to look at assembly and compiler annotation which I know most people are not. I find it to be invaluable in issues, like the one described in this post, where the cost is between the Java lines rather than the lines themselves. Any profiler can give you a broad feel of where performance bottlenecks are, and a non-safepoint-biased profiler can show you the hottest line of code. What a Java profiler will not tell you is about the generated JVM code between your lines. It will attribute the cost of those instructions to the nearest Java line, and that can become a very confusing chase.<br />
It also worth pointing out that any nano-benchmark (measuring operations in the 0-500 nanoseconds range) practically requires you to look at the assembly for it's analysis. But 500ns can still be an aweful lot of code and an assembly level profiler is very handy. At this point it worth mentioning the Oracle Solaris Studio (first prize for least known profiler every year since release). It is a great assembly level profiler, and just generally a great profiler. If your measurement needs to take place outside the cosy comforts of a JMH benchmark I would recommend you give it a spin.<br />
Finally, this investigation came in the context of a workflow that is followed in the development of JCTools. I would loosely describe it as follows:
<br />
<ol>
<li>Implement new feature/data structure</li>
<li>Reason about expected impact/performance as part of design and implementation</li>
<li>Test expectations using the existing set of benchmarks</li>
<li>Expectations are far off the mark (if not... not sure, will figure it out when it happens)</li>
<li>Dig in until either expectations or code are cured.</li>
</ol>
This has a whiff of scientific exploration to it, but I assure you it is not done quite so seriously and I often fail to follow my own advice (or worse other people's advice). The habit of testing performance assumptions/expectation has offered me many an afternoon spent banging my head on a variety of surfaces. Perfasm has been instrumental in reducing the amount of head banging, but I fear nothing short of permanent brain damage will actually solve the problem.<br />
This post has been kindly reviewed by <a href="http://shipilev.net/" target="_blank">Aleksey Shipilev</a>, <a href="https://twitter.com/darachennis" target="_blank">Darach Ennis</a> and <a href="http://insightfullogic.com/" target="_blank">Richard Warburton</a>. Any remaining errors are entirely their fault ;-)</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com10tag:blogger.com,1999:blog-5171098727364395242.post-87586895475273803712015-05-22T06:35:00.000+01:002015-05-22T06:35:08.554+01:00Object.equals, primitive '==', and Arrays.equals ain't equal<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB-_FWpNiVJLZvZzNTd0S1DLTlExpe_fV6wMXAkVaBoVOg29Ab9Ss_91qP0I8R0IYnmtLH_59fCav-2VmkWxc1v6-lXHDrFUqWI4Lt7ZH9114n0GjP7c2RImKYCdNwSBGsK1NzNFfAFog/s1600/jabberwocky.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB-_FWpNiVJLZvZzNTd0S1DLTlExpe_fV6wMXAkVaBoVOg29Ab9Ss_91qP0I8R0IYnmtLH_59fCav-2VmkWxc1v6-lXHDrFUqWI4Lt7ZH9114n0GjP7c2RImKYCdNwSBGsK1NzNFfAFog/s200/jabberwocky.jpg" width="134" /></a></div>
It is a fact well known to those who know it well that "==" != "equals()" the example usually going something like:<br />
<span style="font-family: Courier New, Courier, monospace;"> String a = "Tom";</span><br />
<span style="font-family: Courier New, Courier, monospace;"> String b = new String(a);</span><br />
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> a != b but a.equals(b)</span><br />
<br />
It also seems reasonable therefore that:<br />
<span style="font-family: Courier New, Courier, monospace;"> String[] arr1 = {a};</span><br />
<span style="font-family: Courier New, Courier, monospace;"> String[] arr2 = {b};</span><br />
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> arr1 != arr2 but Arrays.equals(arr1, arr2)</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">So far, so happy... That's examples for you...</span><br />
<span style="font-family: inherit;">For primitives we don't have the equals method, but we can try boxing:</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int i = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> int j = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> i == j and also ((Integer)i).equals((Integer)j), and Arrays.equals({i}, {j})</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJlvYcSvwQClmpK-aCtoGYEfQ1YWUJLuUnzrT4flqnRbBQi0opjIhy0Y-cQRhuciLxOlFNkt0UDlULZHqjSneF5qlKUtc4WIoCX6jth-G0CL9J6-wEVDjDRjn_90iWrFaDvdgFado8_K4/s1600/laughter.jpeg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJlvYcSvwQClmpK-aCtoGYEfQ1YWUJLuUnzrT4flqnRbBQi0opjIhy0Y-cQRhuciLxOlFNkt0UDlULZHqjSneF5qlKUtc4WIoCX6jth-G0CL9J6-wEVDjDRjn_90iWrFaDvdgFado8_K4/s200/laughter.jpeg" width="200" /></a></div>
<h3>
<span style="font-family: inherit;">Floats ruin everything</span></h3>
<span style="font-family: inherit;">But... some primitives are more equal then others. Consider for instance the following example:</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: Courier New, Courier, monospace;">float f1 = Float.Nan;</span><br />
<div>
<span style="font-family: Courier New, Courier, monospace;"> float f2 = Float.Nan;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> f1 != f2, and also f1 != f1, Nan's are nasty</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">This is what floats are like children, they be <a href="https://www.youtube.com/watch?v=O_aziIIp8U8" target="_blank">treacherous little hobbitses</a> and no mistake. The fun starts when you box them fuckers:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> </span><span style="font-family: 'Courier New', Courier, monospace;">((Float)f1).equals((</span><span style="font-family: 'Courier New', Courier, monospace;">Float</span><span style="font-family: 'Courier New', Courier, monospace;">)f2), because Java likes to fix shit up</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> </span><span style="font-family: 'Courier New', Courier, monospace;">Arrays.equals({</span><span style="font-family: 'Courier New', Courier, monospace;">((Float)f1)},{((</span><span style="font-family: 'Courier New', Courier, monospace;">Float</span><span style="font-family: 'Courier New', Courier, monospace;">)f2)}), hmm consistent</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> but also: </span><span style="font-family: 'Courier New', Courier, monospace;">Arrays.equals({</span><span style="font-family: 'Courier New', Courier, monospace;">f1},{</span><span style="font-family: 'Courier New', Courier, monospace;">f2})...</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">This is counter to how one would normally think arrays are compared. You would think that for primitives (skipping arrays null and length checks):</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> boolean feq = true;</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> for(int i=0;i<farr1.length;i++){</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> if(</span><span style="font-family: 'Courier New', Courier, monospace;">farr1[i] != farr2[i]</span><span style="font-family: 'Courier New', Courier, monospace;">) {</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> feq = false; break;</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: inherit;">Is the same as:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> boolean feq = Arrays.equals(farr1,farr2);</span></div>
<div>
<span style="font-family: inherit;">But for <i>double[]</i> and <i>float[]</i> the contract has been changed to accommodate <i>Nan</i>. This is perhaps a good decision on the JDK authors side, but it is somewhat surprising.</span></div>
<div>
<span style="font-family: inherit;"><u>Conclusion: 2 arrays can be equal, even if some elements are not equal.</u></span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<h3>
<span style="font-family: inherit;">Objects are weird<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwdyc6zQY04Eq14p-Pd2ihFTWlF9-mFXdX22lmvZ4UCWDdeK3TCOhVhk1KjFf432dOFCIpaFPnXn4YMDhT0IGNe4bW2S72fwGgAwjdHC7qyJrJXaFRyEzUZlZVigXNdbmdIgPIuJnqBLc/s1600/sad-panda.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwdyc6zQY04Eq14p-Pd2ihFTWlF9-mFXdX22lmvZ4UCWDdeK3TCOhVhk1KjFf432dOFCIpaFPnXn4YMDhT0IGNe4bW2S72fwGgAwjdHC7qyJrJXaFRyEzUZlZVigXNdbmdIgPIuJnqBLc/s200/sad-panda.jpg" width="200" /></a></div>
</span></h3>
<div>
<span style="font-family: inherit;">Let's consider objects for a second. We started with <b>a != b</b> does not predicate that <b>!a.equals(b)</b>, but what about <b>a == b</b>?</span></div>
<span style="font-family: inherit;">The Object.equals() javadoc offers the following wisdom:</span><br />
<span style="font-family: Courier New, Courier, monospace;">The <b>equals</b> method implements an equivalence relation on non-null object references:</span><br />
<br />
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">It is <i><u>reflexive</u></i>: for any non-null reference value <b>x</b>, <b>x.equals(x)</b> should return <b>true</b>.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">It is <i>symmetric</i>: for any non-null reference values <b>x</b> and <b>y</b>, <b>x.equals(y)</b> should return <b>true</b> if and only if <b>y.equals(x)</b> returns <b>true</b>.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">It is <i><u>transitive</u></i>: for any non-null reference values <b>x</b>, <b>y</b>, and <b>z</b>, if <b>x.equals(y)</b> returns <b>true</b> and <b>y.equals(z)</b> returns <b>true</b>, then <b>x.equals(z)</b> should return <b>true</b>.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">It is <i><u>consistent</u></i>: for any non-null reference values <b>x</b> and <b>y</b>, multiple invocations of <b>x.equals(y)</b> consistently return <b>true</b> or consistently return <b>false</b>, provided no information used in equals comparisons on the objects is modified.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">For any non-null reference value <b>x</b>, <b>x.equals(null)</b> should return <b>false</b>.</span></li>
</ul>
You know what the most important word in the above lines is? SHOULD<br />
<div>
In the <a href="https://www.youtube.com/watch?v=Mm3ypbAbLJ8" target="_blank">wonderful land of Should</a>, all classes behave and if they bother implementing an equals method they follow the above rules (and also override the <i>hashcode</i> method while they are there).</div>
<div>
But what happens when an object is implement perversely, like this:</div>
<div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"> public<span class="s1"> </span>class<span class="s1"> Null {</span></span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace;"><span class="s1"> </span>@Override</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> <span class="s2">public</span> <span class="s2">boolean</span> equals(Object <span class="s3">obj</span>) {</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> <span class="s2">return</span> <span class="s3">obj</span> == <span class="s2">null</span>;</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div class="p3">
<span style="font-family: inherit;"><br /></span></div>
<div class="p3">
<span style="font-family: inherit;">This has some nice implications:</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> Null n = new Null();</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> n != null, but n.equals(null)</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> Object[] a = {n};</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> Object[] b = {null};</span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace;"> Object[] c = {n};</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> Arrays.equals(a, b) is true, because n.equals(null)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> Arrays.equals(b, a) is false, because null != n</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> </span><span style="font-family: 'Courier New', Courier, monospace;">Arrays.equals(a, a) is true, because a == a</span></div>
</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> <b>-></b> </span><span style="font-family: 'Courier New', Courier, monospace;">Arrays.equals(a, c) is false, because !n.equals(n)</span></div>
</div>
<div>
<br /></div>
<div>
Or to quote The Dude: "<a href="https://www.youtube.com/watch?v=1jRhgNp-fNc" target="_blank">Fuck it man, let's go bowling</a>"</div>
<div>
<br /></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span></div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com1tag:blogger.com,1999:blog-5171098727364395242.post-35158308747418121162015-05-19T20:33:00.000+01:002015-05-19T20:33:40.720+01:00Degrees Of (Lock/Wait) Freedom<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-WjJTep1DS6AIFLSYEyJxlpi9mzr4ltCfq0yfzDY-0WQ2XMDExxiNd-87QA3bipJyENeA_qboarNvB7xauzYQheu3-MawDDyGeG9PkPeChqAgEneaKMWvUbiPf4m94j3nMHgR-Q9qPbE/s1600/Y+do+we+care.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-WjJTep1DS6AIFLSYEyJxlpi9mzr4ltCfq0yfzDY-0WQ2XMDExxiNd-87QA3bipJyENeA_qboarNvB7xauzYQheu3-MawDDyGeG9PkPeChqAgEneaKMWvUbiPf4m94j3nMHgR-Q9qPbE/s1600/Y+do+we+care.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Yo, Check the diagonal, three brothers gone...</td></tr>
</tbody></table>
I've been throwing around the terms lock-free and wait-free in the context of the queues I've been writing, perhaps too casually. The definition I was using was the one from D. Vyukov's <a href="http://www.1024cores.net/home/lock-free-algorithms/introduction" target="_blank">website</a> (direct quote below):<br />
<ul>
<li><span style="font-family: Courier New, Courier, monospace;"><u><b>Wait-freedom:</b></u> Wait-freedom means that each thread moves forward regardless of external factors like contention from other threads, other thread blocking. Each operations is executed in a bounded number of steps. </span></li>
<li><span style="font-family: Courier New, Courier, monospace;"><b><u>Lock-freedom:</u></b> Lock-freedom means that a system as a whole moves forward regardless of anything. Forward progress for each individual thread is not guaranteed (that is, individual threads can starve). It's a weaker guarantee than wait-freedom. [...] A blocked/interrupted/terminated thread can not prevent forward progress of other threads. Consequently, the system as a whole undoubtedly makes forward progress.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;"><b><u>Obstruction-freedom:</u></b> Obstruction-freedom guarantee means that a thread makes forward progress only if it does not encounter contention from other threads. That is, two threads can prevent each other's progress and lead to a livelock. </span></li>
<li><span style="font-family: Courier New, Courier, monospace;"><b><u>Blocking Algorithms:</u></b> It's the weakest guarantee - basically all bets are off, the system as a whole may not make any forward progress. A blocked/interrupted/terminated thread may prevent system-wide forward progress infinitely. </span></li>
</ul>
The above definitions refer to "forward progress/moving forward" in the context of "system" and "thread". In particular they translate "X-freedom" to "a guarantee that T/S makes forward progress".<br />
Now "thread" is a well defined term, but what counts as forward progress of a given thread?<br />
To my mind a thread which is free to return from a method is free to 'make progress'. For example: if the next element in a queue is not visible to a consumer thread I can return control to the consumer thread which is then free to make progress on anything it feels like and try getting an element out of the queue later. Similarly a producer thread which is unable to add elements to a queue is 'free' to 'make progress' even if the queue is full. In short, I interpreted 'freedom' as regaining control of execution.<br />
<br />
<h3>
<a href="https://youtu.be/Xk2uObQDKtw?t=277" target="_blank">Freedom</a>? yeah, right...</h3>
My interpretation however is limited to the scope of the thread and assumes it has other things to do (so placing the thread within the context of a given 'system'). So the freedom to make progress is really a freedom to make progress on 'other things'. The definitions above when applied to a given data structure have no knowledge of the 'system' and it is therefore fair to assume nothing. And so if the 'system' is viewed as being concerned only with using the data structure, it seems my view of 'regained control freedom' is not inline with the 'progress making freedom'.<br />
Let's consider for example the linked Multi-Producer-Single-Consumer queue discussed in the <a href="http://psy-lob-saw.blogspot.com/2015/04/porting-dvyukov-mpsc.html" target="_blank">previous post</a> (this is the not the j.u.Queue compliant version):<br />
<script src="https://gist.github.com/nitsanw/365ab84d4b8712ec88f1.js"></script><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3E-Sb8WOwiDyV5qx_ASMhSViQppwadZB_cxDPUYsxVw0tF6Oi9DkIeJ4NvVanzrMMMhAutgkGkpwAkujY9xJdDCYHTDJCM-xBqzgYzu0i5pieo3QGK9rEqrAx9EQnIA-Sdf-ce-8-i4E/s1600/220px-RageAgainsttheMachineRageAgainsttheMachine.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3E-Sb8WOwiDyV5qx_ASMhSViQppwadZB_cxDPUYsxVw0tF6Oi9DkIeJ4NvVanzrMMMhAutgkGkpwAkujY9xJdDCYHTDJCM-xBqzgYzu0i5pieo3QGK9rEqrAx9EQnIA-Sdf-ce-8-i4E/s1600/220px-RageAgainsttheMachineRageAgainsttheMachine.jpg" style="cursor: move;" /></a>Now let us assume a producer has stalled in that unpleasant gap between setting the <i>head</i> and pointing <i>prev</i> to the new head(at line 34), this is what I've come to think of as a 'bubble' in the queue. The next producer will see the new head, but the consumer will not see it (or any nodes after it) until <i>prev.next</i> is set.<br />
Others producers can 'make progress' in the sense that elements will get added to the queue. Those elements visibility to the consumer however is blocked by the 'bubble'. What is the correct degree of freedom for these producers? what about the consumer?<br />
When <a href="http://www.1024cores.net/home/lock-free-algorithms/queues/non-intrusive-mpsc-node-based-queue" target="_blank">describing the queue</a> Vyukov makes the following statements:<br />
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">Wait-free and fast producers. One XCHG is maximum what one can get with multi-producer non-distributed queue.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">Push [offer in the Java version] function is <b>blocking</b> wrt consumer. I.e. if producer blocked in (* [line 34]), then consumer is blocked too. Fortunately 'window of inconsistency' is extremely small - producer must be blocked exactly in (* [line 34]).</span></li>
</ul>
So producers are wait-free, but consumer is blocking. The consumer is truly blocked by the 'bubble' and can make no progress in the terms of the data structure. Despite other producers adding nodes the consumer cannot see those nodes and is prevented from consuming the data. The fact that control is returned to the caller is considered irrelevant. And if we are being precise we should call this a blocking queue.<br />
<br />
<h3>
Life imitates Art</h3>
<div>
The "<a href="http://www.amazon.com/The-Multiprocessor-Programming-Revised-Reprint/dp/0123973376" target="_blank">Art of Multiprocessor Programming</a>" offers a different definition:</div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">"A concurrent object implementation is wait free if each method call completes in a finite number of steps. A method is lock-free if it guarantees that infinitely often some method call finishes in a finite number of steps" - page 99</span></blockquote>
This sentiment is similarly echoed in <a href="https://offblast.org/stuff/books/lockfreequeues_ppopp11.pdf" target="_blank">"Wait-Free Queues With Multiple Enqueuers and Dequeuers"</a> a paper on lock free queues:<br />
<br />
<ul>
<li>[...] to ensure that a process (or a thread) completes its operations in a bounded number of steps, regardless of what other processes (or threads) are doing. This property is known in the literature as (bounded) <b>wait-freedom</b>.</li>
<li><b>lock-freedom</b> ensures that among all processes accessing a queue, at least one will succeed to finish its operation.</li>
</ul>
<br />
Hmmm... no system, no progress. The boundaries are now an object and a method, which are well defined. I think this definition matches my understanding of 'regained control freedom' (open to feedback). If a method returns, progress or no progress, the condition is fulfilled. Under this definition the queue above is wait-free.<br />
The <a href="http://en.wikipedia.org/wiki/Non-blocking_algorithm" target="_blank">wiki definition</a> is again closer to the one outlined by Vyukov:<br />
<div>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">An algorithm is called non-blocking if <u>failure or suspension of any thread cannot cause failure or suspension of another thread for some operations</u>. A non-blocking algorithm is lock-free if there is guaranteed system-wide progress, and wait-free if there is also guaranteed per-thread progress.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">Wait-freedom is the strongest non-blocking guarantee of progress, combining guaranteed system-wide throughput with starvation-freedom. An algorithm is wait-free if every operation has a bound on the number of steps the algorithm will take before the operation completes.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">Lock-freedom allows individual threads to starve but guarantees system-wide throughput. An algorithm is lock-free if it satisfies that when the program threads are run sufficiently long at least one of the threads makes progress (for some sensible definition of progress). All wait-free algorithms are lock-free.</span></li>
</ul>
</div>
We are back to a fuzzy definition of system, progress, but the underlined sentence above is interesting in highlighting suspension. In particular I'm not sure how this is to be interpreted in the the context of the lock-free definition where suspension is not an issue if some threads can keep going.<br />
Note that once we have fixed poll to behave in a j.u.Queue compliant way:
<script src="https://gist.github.com/nitsanw/090a6682bf9c7202f1ea.js"></script>
The poll method is no longer wait-free by any definition.<br />
<div>
<br /></div>
<h3>
What about the JCTools queues?</h3>
<div>
<a href="http://www.github.com/JCTools/JCtools" target="_blank">JCTools</a> covers the full range of MPMC/MPSC/SPMC/SPSC range of queues, and aims for high performance rather than strict adherence to any of the definitions above. To be more precise in the definitions I would say:<br />
<ul>
<li>SpscArrayQueue/SpscLinkedQueue are Wait Free (on both control and progress senses)</li>
<li>MpscArrayQueue is lock free on the producer side and blocking on the consumer side. I'm planning to add a weakPoll method which will be wait free in the control sense.</li>
<li>MpscLinkedQueue is wait free on the producer side and blocking on the consumer side. I'm planning to add a weakPoll method which will be wait free in the control sense.</li>
<li>SpmcArrayQueue is lock free on the consumer side and blocking on the producer side. I'm planning to add a weakOffer method which will be wait free in the control sense.</li>
<li>MpmcArrayQueue is blocking on the both producer and consumer side. I'm planning to add a weakOffer/Poll method which will be lock free in the control sense.</li>
</ul>
Does it matter that the queues do not meet the exact definitions set out below? As always it depends on your needs...<br />
<ul>
</ul>
</div>
<div>
<h3>
Non-Blocking Algorithms On The JVM?</h3>
</div>
<div>
<span style="font-size: x-small;">{<b><u>Full disclosure</u></b>: please note when reading the next section that I work with <a href="http://www.azulsystems.com/" target="_blank">Azul Systems</a> on the <a href="http://www.azulsystems.com/products/zing/whatisit" target="_blank">Zing JVM</a>, I'm not trying to sell anything, but I am naturally more familiar with it and consider it awesome :-)}</span></div>
<div>
What happens when we include the JVM in our definition of a system? Can you build a non-blocking algorithm on the JVM? All the JVMs I know of are blocking in some particular cases, important to the discussion above are:</div>
<div>
<ol>
<li><b>Allocation</b>: The common case for allocation is fast and involves no locking, but once the young generation is exhausted a collection is required. Young generation collections are stop the world events for Oracle/OpenJDK. Zing has a concurrent young generation collector, but under extreme conditions or bad configuration it may degrade to blocking the allocator. <b>The bottom line is that allocation on the JVM is blocking and that means you cannot consider an algorithm which allocates non-blocking.</b> To be non-blocking a system would have to provably remove the risk of a blocking collection event on allocation.</li>
<li><b>Deoptimization</b>: Imagine the JIT compiler in it's eager and aggressive compilation strategy has decided your <i>offer</i> method ends up getting compiled a particular way which ends up not working out (assumptions on inheritance, class loading, passed in values play a part). When the assumption breaks a deoptimization may take place, and that is a blocking event. It is hard to prove any piece of code is deoptimization risk free, and therefore it is hard to prove any Java code is non-blocking. Deoptimization is in many cases a warmup issue. Zing is actively battling this issue with <a href="http://www.azulsystems.com/solutions/zing/readynow" target="_blank">ReadyNow</a> which reloads previously recorded compilation profiles for the JVM, greatly reducing the risk of deoptimization. Oracle/OpenJDK users can do a warmup run of their application before actually using it to reduce the risk of deoptimization. My colleague <a href="https://twitter.com/dougqh" target="_blank">Doug Hawkins</a> gave a <a href="https://vimeo.com/120533011" target="_blank">great talk on this topic</a> which will give you far more detail then I want to go into here :-). </li>
<li><b>Safepoints</b>: If your algorithm includes a <a href="http://psy-lob-saw.blogspot.com/2014/03/where-is-my-safepoint.html" target="_blank">safepoint poll</a> it can be brought to a halt by the JVM. The JVM may bring all threads to a safepoint for any number of reasons. Your method, unless inlined, already has a safepoint on method exit(OpenJDK) or entry(Zing). Any non-counted loop will have a safepoint-poll, and this includes your typical CAS loop. Can you prevent safepoints from ever happening in your system? It might be possible for a JVM to provide users with compiler directives which will prevent safepoint polls from being inserted in certain code paths, but I don't know of any JVM offering this feature at the moment.</li>
</ol>
In short, if you are on the JVM you are probably already blocking. Any discussion of non-blocking algorithms on the JVM is probably ignoring the JVM as part of the 'system'. This needs to be considered if you are going to go all religious about wait-free vs. lock-free vs. blocking. If a system MUST be non-blocking, it should probably not be on the JVM.</div>
<div>
<br /></div>
<h3>
Summary</h3>
<div>
Any term is only as good as our ability to communicate with others using it. In that sense, splitting hairs about semantics is not very useful in my opinion. It is important to realize people may mean any number of things when talking about lock-free/wait-free algorithms and it is probably a good idea to check if you are all on the same page.</div>
<div>
I personally find the distinction between control/progress freedom useful in thinking about algorithms, and I find most people mean lock/wait-free excluding the effects of the JVM...</div>
<div>
Thanks <a href="https://twitter.com/mjpt777" target="_blank">Martin</a> & <a href="https://twitter.com/dougqh" target="_blank">Doug</a> for the review, feedback and discussion, any remaining errors are my own (but their fault ;-) )</div>
<div>
<br /></div>
<blockquote class="tr_bq">
</blockquote>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com9tag:blogger.com,1999:blog-5171098727364395242.post-22932891605425256752015-04-22T10:51:00.002+01:002015-04-24T10:07:07.627+01:00Porting Pitfalls: Turning D.Vyukov MPSC Wait-free queue into a j.u.Queue<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLo0JThP3uP2-JzvWFtUzD3qexXim1xeN9IXqrVE1Q3SuDJfVOEXdhjBXuFiWlbgJGkp2y8Vxwe2cLw4NoZWK-EOonu2sIGakfgqEHC5R2uorsD6DtdERwxnXdhAqsZlNLKkBHDHlPMVM/s1600/domestic-sluttery-sluttishly-silly-ninja-russian-dolls.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLo0JThP3uP2-JzvWFtUzD3qexXim1xeN9IXqrVE1Q3SuDJfVOEXdhjBXuFiWlbgJGkp2y8Vxwe2cLw4NoZWK-EOonu2sIGakfgqEHC5R2uorsD6DtdERwxnXdhAqsZlNLKkBHDHlPMVM/s1600/domestic-sluttery-sluttishly-silly-ninja-russian-dolls.png" height="320" width="320" /></a></div>
<b><span style="font-size: x-small;">{This post is part of a long running series on lock free queues, <a href="http://psy-lob-saw.blogspot.com/p/lock-free-queues.html" target="_blank">checkout the full index to get more context here</a>}</span></b><br />
<a href="https://twitter.com/dvyukov" target="_blank">D. Vyukov</a> is an awesome lock-free dude, and I often refer to his instructive and invaluable
website <a href="http://www.1024cores.net/" target="_blank">1024cores.net</a> in my posts. On his site he covers lock free queue
implementations and in particular a wait-free <a href="http://www.1024cores.net/home/lock-free-algorithms/queues/non-intrusive-mpsc-node-based-queue" target="_blank">MPSC linked node queue</a>. This is really rather
special when you consider that normally MP would imply lock-free rather than wait-free
guarantees. I've ported his algorithm to Java (and so have many others: Netty/Akka/RxJava etc.), and had to tweak
it to match the Queue interface. In this post I'd like to explain the algorithm, it's
translation to Java, and the implications of making it a j.u.Queue.
<br />
<br />
<h3>
Lock free vs. Wait free</h3>
Let's review the <a href="http://www.1024cores.net/home/lock-free-algorithms/introduction" target="_blank">definitions</a>:<br />
<ul>
<li>Wait-free: thread progress is guaranteed, all operations finish in a determined number of steps.</li>
<li>Lock-free: global progress is guaranteed, though a particular thread may be blocked.</li>
</ul>
<br />
An example of a transition from lock free to wait free
is available with JDK8 changes to <span style="font-family: Courier New, Courier, monospace;">AtomicReference::getAndSet()</span>. The change was made by utilizing
the newly available <span style="font-family: Courier New, Courier, monospace;">Unsafe::getAndSetObject</span> intrinsic which translates directly to <span style="font-family: Courier New, Courier, monospace;">XCHG</span> (on x86). So
where we used to have for AtomicReference:<br />
<span style="font-family: Courier New, Courier, monospace;">T getAndSet(T newVal) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> T currentValue;</span><br />
<span style="font-family: Courier New, Courier, monospace;"> do {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> currentValue = val; // val is a volatile field</span><br />
<span style="font-family: Courier New, Courier, monospace;"> } while (!Unsafe.compareAndSwapObject(this, VAL_FIELD_OFFSET, currentValue, newValue));</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return currentValue;</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
Now we have:<br />
<span style="font-family: Courier New, Courier, monospace;">T getAndSet(</span><span style="font-family: 'Courier New', Courier, monospace;">T newVal</span><span style="font-family: 'Courier New', Courier, monospace;">) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"> return Unsafe.getAndSetObject(this, VAL_FIELD_OFFSET, newValue);</span><br />
<span style="font-family: Courier New, Courier, monospace;">}</span><br />
I discussed a similar change to AtomicLong.getAndAdd in <a href="http://psy-lob-saw.blogspot.com/2014/06/jdk8-update-on-scalable-counters.html" target="_blank">a previous post</a>, replacing the CAS loop with
<span style="font-family: Courier New, Courier, monospace;">LOCK XADD</span>.<br />
<br />
<h3>
The Vyukov Wait Free MPSC queue</h3>
This is a LinkedList type structure and the interesting methods are offer and poll, here's the
original (I did some formatting):<br />
<script src="https://gist.github.com/nitsanw/b7b3a61d4fc022c9ee69.js"></script>
Awesome in it's simplicity, deceptive in it's genius. Be aware that head/tail meaning is the other way around than what most people are used to. I personally go for producer/consumerNode (head = producer side, tail = consumer side in the above snippet) in my code, but for consistency I'll stick with Mr. Vs notation for the porting exercise.<br />
But how do we manage the same memory barriers
in Java? We can be nice about it and use the AtomicFieldUpdater or more brutal and use Unsafe. I
find you get better performance with Unsafe, but you should consider how appropriate this is for you.
In any case, here's what we end up with:<br />
<script src="https://gist.github.com/nitsanw/365ab84d4b8712ec88f1.js"></script>
The code is pretty similar. Now if we wanted to complete the j.u.Queue we could extend AbstractQueue and
implement only size()/peek() and we're done (I'm not going to bother with the iterator()):<br />
<script src="https://gist.github.com/nitsanw/51cd025838c494cd0c11.js"></script>
There we go, seems reasonable don't it? off to the pub to celebrate some kick ass lock free programming!<br />
<br />
<h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijjhezRHeMM5DoFw9cPN7TrGSZ3nQCA_bZPuSelbjjATpazorpvuQiFa0hezlj9HiYCk9BSxEmmD9X-jYPAMb6zLvNXjf6H4hfjDkLdSqXshw5q68nKtr2OTIH7aFvNAsN0vaZa1HWyNk/s1600/its+ok+im+a+ninja.jpeg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijjhezRHeMM5DoFw9cPN7TrGSZ3nQCA_bZPuSelbjjATpazorpvuQiFa0hezlj9HiYCk9BSxEmmD9X-jYPAMb6zLvNXjf6H4hfjDkLdSqXshw5q68nKtr2OTIH7aFvNAsN0vaZa1HWyNk/s1600/its+ok+im+a+ninja.jpeg" height="200" width="200" /></a>
Vyukov::poll() ain't Queue::poll!</h3>
I've discussed this particular annoyance in a <a href="http://psy-lob-saw.blogspot.com/2014/07/poll-me-maybe.html" target="_blank">previous post which has some overlap with this one</a>. The names are the same, but as it turns out the guarantees are quite different. While Vyukov is doing a
fine job implementing a queue, not any queue is a j.u.Queue. In particular for this case, the poll()
method has a different contract:<br />
<ul>
<li>j.u.Queue: Retrieves and removes the head of this queue, or returns null if this queue is empty.</li>
<li>Vyukov: Retrieves and removes the head of this queue, or returns null if next element is not available.</li>
</ul>
Why wouldn't the next element be available? Doesn't that mean the queue is empty? Sadly this ain't the case. Imagine for instance there are 2 producers and we run both threads step by step in a debugger. We break at line 33 (4th line of offer, the Java version):<br />
<ol>
<li><i>Producer 1</i>: we step 1 line, we just replaced head with node n. We are suspended before executing line 35.</li>
<li><i>Producer 2</i>: we let the program continue. We've replaced head and linked it to the previous head.</li>
</ol>
What state are we in? Let's look at head/tail nodes and where they lead (I number the nodes in order of assignment to head):<br />
<ul>
<li><i>head = Node[2]</i>, this is the node created by <i>Producer 2</i>. We also know that <i>Node[1].next = Node[2]</i>, because we let offer run it's course on <i>Producer 2</i>.</li>
<li><i>tail = Node[0]</i> the node we allocated in the constructor. This is the node head was before the first producer came along. This is what <i>prev</i> is equal to for <i>Producer 1</i>, but because we suspended that thread it never set it's <i>next</i> value. <i>Node[0].next</i> is still null!</li>
</ul>
<br />
If a consumer came along now they would get a null from poll(), indicating the queue is empty. But the queue is obviously not empty!<br />
So it seems we cannot deduce the queue is empty from looking at <i>tail.next </i>here's 2 valid indicators that the queue is empty:<br />
<ul>
<li><i>head == tail</i> : this is the starting point set in the constructor and where the consumer ends up after consuming the last element</li>
<li><i>head.val == null</i> : head can only have a value of null if it is tail</li>
</ul>
Here's a solution to a correct poll() and the knock on effect on peek():<br />
<script src="https://gist.github.com/nitsanw/a9b0b061bc212551383f.js"></script>
This is a bit annoying because now we have a wait-free offer(), but poll() and peek() are lock-free(only block one thread, producers can make progress).<br />
This pitfall is tricky enough that not only did <a href="https://github.com/JCTools/JCTools/issues/21" target="_blank">I fall for it </a>(on another queue algorithm, but same mistake), it also took by surprise Mr. Manes writing an interesting variant on this queue for his <a href="https://github.com/ben-manes/caffeine" target="_blank">high performance cache implementation</a> (I filed an <a href="https://github.com/ben-manes/caffeine/issues/9" target="_blank">issue which he promptly fixed</a>), and struck the notorious <a href="https://blogs.oracle.com/dave/entry/ptlqueue_a_scalable_bounded_capacity" target="_blank">Dave Dice when considering Vyukov's MPMC queue</a> (see comments for discussion).<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-8FT_pJkD5GDGIOmM8S_bP-vo7kxgTkPDvHjilUrZui_dwwflfoA3HBT3o4EnBnqNEDdbXWjdQIiaTDCUlwB2pfb2bo66xHwkOM3uPmRMuLUD0AhRRJgn-YAo-n4Q6jz9nEUxoyiKpfc/s1600/fat+ninja.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-8FT_pJkD5GDGIOmM8S_bP-vo7kxgTkPDvHjilUrZui_dwwflfoA3HBT3o4EnBnqNEDdbXWjdQIiaTDCUlwB2pfb2bo66xHwkOM3uPmRMuLUD0AhRRJgn-YAo-n4Q6jz9nEUxoyiKpfc/s1600/fat+ninja.png" height="224" width="320" /></a>So is this it? Are we done?<br />
Almost... <i>size()</i> is still broken.<br />
<br />
<h3>
A Good Size</h3>
<div>
It's not really surprising that size is broken given the terminating condition for the size loop was relying on next == null to terminate the count. Size was also broken in 2 other subtle ways:</div>
<div>
<ul>
<li>The interface for size dictates that it returns a positive int. But given that the queue is unbounded it is possible (though very unlikely) for it to have more than 2^31 elements. This would require that the linked list consume over 64GB (16b + 8b + 8b=32b per element, refs are 8b since this requires more than 32GB heap so no compressed oops). Unlikely, but not impossible. This edge condition is handled for ConcurrentLinkedQueue (same as here), and for LinkedBlockingQueue (by bounding it's size to MAX_INT, so it's not really unbounded after all) but not for LinkedList (size is maintained in an int and is blindly incremented).</li>
<li>Size is chasing a moving target and as such can in theory never terminate as producers keep adding nodes. We should limit the scope of the measurement to the size of the queue at the time of the call. This is not done for CLQ and should perhaps be considered.</li>
</ul>
Here's the end result:</div>
<script src="https://gist.github.com/nitsanw/a12330da236e8ba7c931.js"></script>
If you need a full SPSC/MPSC linked queue implementation they are available in <a href="https://github.com/JCTools/JCTools" target="_blank">JCTools</a> for your pleasure, enjoy!<br />
<br />
Special thanks to <a href="http://twitter.com/switchology" target="_blank">Doug</a> and <a href="https://github.com/ben-manes" target="_blank">Ben</a> for reviewing!<br />
<div>
<br /></div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com8tag:blogger.com,1999:blog-5171098727364395242.post-29222486760951464592015-04-13T10:45:00.002+01:002015-04-14T11:55:14.595+01:00On Arrays.fill, Intrinsics, SuperWord and SIMD instructions<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSdxeVhJ_zUjzvpqXb-sxXzwFZn8IhIKqblwqxa7Fm1nGl3ZUG914g64C0CwPv_94YVMje-zeAQ_yzHnkkSD2uLO7iyX3vuiwweAFBWcszJDY8KeWfjo9sHmD-ZD3TdPaW8mbAgxj8oX0/s1600/empire-strikes-back-chewbacca.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSdxeVhJ_zUjzvpqXb-sxXzwFZn8IhIKqblwqxa7Fm1nGl3ZUG914g64C0CwPv_94YVMje-zeAQ_yzHnkkSD2uLO7iyX3vuiwweAFBWcszJDY8KeWfjo9sHmD-ZD3TdPaW8mbAgxj8oX0/s1600/empire-strikes-back-chewbacca.jpg" height="197" width="320" /></a></div>
<b>{This post turned rather long, if you get lazy feel free to skip to the summary}</b><br />
Let's <a href="https://www.youtube.com/watch?v=1RW3nDRmu6k" target="_blank">start at the very beginning, a very good place to start</a>... My very first post on this blog was <a href="http://psy-lob-saw.blogspot.co.il/2012/10/java-intrinsics-are-not-jni-calls.html" target="_blank">a short rant on intrinsics</a>, and how they ain't what they seem. In that post I made the following statement:<br />
<blockquote class="tr_bq">
<span style="font-family: inherit;">"<span style="background-color: white; color: #222222; line-height: 18.4799995422363px;">intrinsic functions show up as normal methods or native methods</span>"</span></blockquote>
Which is correct. An intrinsic function is applied as a method substitution. A method call will appear in the code and the compiler will replace it's apparent source-level implementation with a pre-cooked implementation. In some cases intrinsics are sort of compilation cheats, the idea being that some bits of functionality are both very important (i.e. worth while optimizing) and can benefit from a hand crafted solution that will be better than what the compiler can achieve. The end result can be in one of a few flavours:<br />
<ol>
<li>Method call replaced with a call to a JVM runtime method: E.g. System.arrayCopy is replaced with a call to a method stub generated by the runtime for all array types. This method call is not a JNI call, but it is a static method call that is not inlined.</li>
<li>Method call replaced with one or more instructions inlined: E.g. Unsafe.getByte/compareAndSet/Math.max</li>
<li>Method call replaced with compiler IR implementation: E.g. java.lang.reflect.Array.getLength</li>
<li>A mix of the above: E.g. String.equals is partially implemented in IR, but the array comparison is a call to a method stub.</li>
</ol>
The intrinsics are all set up in <a href="http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/tip/src/share/vm/classfile/vmSymbols.hpp">vmSymbols.hpp</a> and if you look, you'll see Arrays.fill is NOT on the list. <a href="https://www.youtube.com/watch?v=clKi92j6eLE" target="_blank">So why am I talking about Chewbacca?</a> Because it is something like an intrinsic...<br />
<div>
<br /></div>
<div>
<h3>
The Arrays.fill SIMD Opportunity</h3>
<div>
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#fill-byte:A-byte-" target="_blank">Arrays.fill</a> is the Java <a href="http://www.cplusplus.com/reference/cstring/memset/" target="_blank">memset</a> (fills an array with a given value), and just like System.arrayCopy (memcpy in C lingo) is worth the effort to optimize and offers the same kind of opportunity. What opportunity might that be, you ask? the opportunity to use SIMD (Single Instruction Multiple Data) instructions when the underlying CPU offers them (I assume for the sake of discussion <a href="http://en.wikipedia.org/wiki/Advanced_Vector_Extensions" target="_blank">AVX enabled CPUs i.e. since Sandy Bridge</a>, I find this <a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/" target="_blank">listing of intel intrinsics</a> useful to explain and sort through the available instructions). These instructions allow the CPU to operate on up to 256 bit (<a href="http://en.wikipedia.org/wiki/AVX-512" target="_blank">512 bit soon</a>) chunks of data, thus transforming 32 byte sized MOV instructions into a single wide MOV instruction (E.g. the intel C instrinsic <span style="background-color: white; font-size: 16px;"><span style="font-family: Courier New, Courier, monospace;">_mm256_storeu_si256</span></span> <span style="font-family: inherit;">or the corresponding instruction </span><span style="background-color: white;"><span style="font-family: Courier New, Courier, monospace;">vmovdqu</span></span>). SIMD instructions are good for all sorts of operations on vectors of data, or arrays, which is why the process of transforming element by element operations into SIMD instructions is also referred to as vectorization.<br />
The actual assembly stub is generated dependent on CPU and available instruction set. For x86 the code is generated by the <a href="http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/61746b5f0ed3/src/cpu/x86/vm/macroAssembler_x86.cpp#l6085" target="_blank">macroAssembler_x86.cpp</a>, and the observant digger into the code will find it makes use of the widest memory instructions it can identify the processor is capable of. Wider is better baby! If you are not morbidly curious about what the implementation looks like, skip the next wall of assembly and you'll be back in Java land shortly.<br />
Here's what the assembly boils down to when UseAVX>=2/UseSSE>=2/UseUnalignedLoadStores=true:</div>
</div>
<div>
<script src="https://gist.github.com/nitsanw/413f64350e73cb9bda17.js"></script><br />
Roughly speaking the algorithm above is:<br />
<ol>
<li>Fill up an XMM register with the intended value</li>
<li>Use the XMM register to write 64 byte chunks (2 <span style="background-color: white; font-family: 'Courier New', Courier, monospace;">vmovdqu</span>) until no more are available</li>
<li>Write leftover 32 byte chunk (skipped if no matching leftovers)</li>
<li>Write leftover 8 byte chunks (skipped if no matching leftovers)</li>
<li>Write leftover 4 bytes (skipped if no matching leftovers)</li>
<li>Write leftover 2 bytes (skipped if no matching leftovers)</li>
<li>Write leftover 1 bytes (skipped if no matching leftovers)</li>
</ol>
It ain't nice, but we do what we gotta for performance! There are variations of the above described across the internets as the done thing for a memset implementation, this might seem complex but is pretty standard... anyway, moving right along.<br />
<br />
<h3>
The Arrays.fill 'intrinsic'</h3>
<div>
Arrays.fill is different from System.arrayCopy because, as it's absence from vmSymbols suggests, it's not a method substitution kind of intrinsic (so technically <b>not</b> an intrinsic). What is it then? Arrays.fill is a code pattern substitution kind of compiler shortcut, basically looking for this kind of loop:</div>
<div>
<script src="https://gist.github.com/nitsanw/298c5daa0e3650f3fd4b.js"></script>
</div>
And replacing it with a call into the JVM memset implementation (I recently learnt the <a href="https://github.com/RichardWarburton/honest-profiler/issues/77" target="_blank">same thing is done by GCC as well</a>, see code to assembly <a href="http://gcc.godbolt.org/#%7B%22version%22%3A3%2C%22filterAsm%22%3A%7B%22labels%22%3Atrue%2C%22directives%22%3Atrue%2C%22commentOnly%22%3Atrue%2C%22intel%22%3Atrue%2C%22colouriseAsm%22%3Atrue%7D%2C%22compilers%22%3A%5B%7B%22sourcez%22%3A%22FAYglgdgxgNgrgEwKYB4oGcAuDkDMB8wwAbgPZgIAE6AhrkgPoBOS6SmAFGRZQFRY0mmADTUwAL0aYxkgJSUA3sEorKUABaC%2BAIxptKAXkotImJEwAOLTAyh7MaTUz74OAobIDcy1Rq28kCCojXX0Aahkkb1VKXFJnDj9nXgtDSlCoylSUSkCETyywsPklGJiUtIAGaNUAX2BaoA%22%2C%22compiler%22%3A%22%2Fopt%2Fgcc-4.9.0%2Fbin%2Fg%2B%2B%22%2C%22options%22%3A%22-O3%20-std%3Dc%2B%2B0x%22%7D%5D%7D" target="_blank">here</a>). The pattern matching bit is done in <a href="http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/55fb97c4c58d/src/share/vm/opto/loopTransform.cpp#l2629" target="_blank">loopTransform.cpp</a>. This feels enough like an intrinsic grey area that the method doing the pattern match and replace is called <span style="font-family: Courier New, Courier, monospace;">intrinsify_fill.</span><br />
Pattern matching makes this optimization potentially far more powerful than method substitution as the programmer doesn't have to use a special JDK method to convey meaning, they can just express their meaning in code and the compiler 'knows' that this simple loop means 'fill'. Compare that with System.arrayCopy where rolling your own leads to performance that is much worse than that offered by the intrinsic.<br />
Let's prove me right (my favourite thing, beats <a href="https://youtu.be/0IagRZBvLtw?t=15" target="_blank">kittens and all that crap</a>), here's a JMH (see the <a href="http://psy-lob-saw.blogspot.com/p/jmh-related-posts.html" target="_blank">JMH reference page</a> for more JMH info/examples) benchmark comparing Arrays.fill to a hand rolled fill, and System.arrayCopy to handrolled array copy:
<script src="https://gist.github.com/nitsanw/5f1b538d80f6be2c5546.js"></script>
<br />
And the results are (Oracle JDK8u40/i7-4770@3.40GHz/Ubuntu, array is 32K in size)?<br />
<span style="font-family: Courier New, Courier, monospace;">
ArrayFill.fillBytes 561.540 ± 10.814 ns/op<br />
ArrayFill.manualFillBytes 557.901 ± 5.255 ns/op<br />
ArrayFill.manualReversedFillBytes 1017.856 ± 0.425 ns/op<br />
ArrayFill.copyBytes 1300.313 ± 13.482 ns/op<br />
ArrayFill.manualCopyBytes 1477.442 ± 13.030 ns/op</span></div>
<br />
We can verify that the call out to the JVM fill method happens for fillBytes/manualFillBytes by <a href="http://psy-lob-saw.blogspot.com/2013/01/java-print-assembly.html" target="_blank">printing out the assembly</a>:<br />
<br />
<script src="https://gist.github.com/nitsanw/099b1de013185aba8863.js"></script>
So what have we learnt so far:<br />
<ul>
<li>Use System.arrayCopy, it is better than your handrolled loop. But surprisingly not hugely better, hmmm.</li>
<li>You don't have to use Arrays.fill, you can roll your own and it works the same. Notice the call out to the fill method. But...</li>
<li>Don't get too creative rolling your own. If you get too funky (like filling the array backwards) it'll fall apart and the 'intrinsic' won't happen. But do note that the reverse fill still has some of that good SIMD stuff going, we'll get to that in a sec.</li>
</ul>
<h3>
Are The Other Types Filling The Love?</h3>
It all sounds great don't it? Let's see how this pans out for other types. We'll be filling an array of 32KB. To be uniform across data types that means a 16K chars/shorts array, an 8K ints/floats array and a 4K array of longs. I added an 8K array of objects, which is the same size for compressed oops on the Oracle JVM (reference size is 4 bytes, same as an int).<br />
The JMH benchmark code is as you'd expect:<br />
<script src="https://gist.github.com/nitsanw/156fc152478081ea98b1.js"></script>
Here's some reasonable expectations:
<br />
<ul>
<li>If no optimizations are present, wider writes are more efficient. It follows that the longFill would be the fastest. But...</li>
<li>Given a clever compiler the fill loop is replaced with the widest writes possible, so there should be no significant difference. But the fill optimization does not cover double/long/object arrays, so we might expect longFill to be the worst performer.</li>
<li>An objects array is not that different from an int array, so performance should be similar. Sure there's a write barrier, but it need only be done once per card (not once for the whole array as I thought initially, god bless Shipilev and PrintAssembly), so that's an extra byte write per card of elements filled. A card is per 512 bytes, each element is 4 bytes, so that's one card per 128 elements. Given there is no fill method implemented for it we may expect it to be slightly worse than the longFill.</li>
<li>We should not rely on expectations, because performance is best measured.</li>
</ul>
As you'd expect the results are somewhat different than the expectations (Oracle JDK8u40/i7-4770@3.40GHz/Ubuntu):<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillBytes 561.540 ± 10.814 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillChars 541.901 ± 4.833 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillShorts 532.936 ± 4.508 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillInts 543.165 ± 3.788 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillFloats 537.599 ± 2.323 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillLongs 326.770 ± 3.267 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillDoubles 346.840 ± 5.786 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillObjects 4388.242 ± 11.945 ns/op</span></div>
<br />
Say WOT?<br />
For bytes/chars/shorts/ints/floats Arrays.fill performs very similarly. This much is as expected from the second point above. But filling an array of longs/doubles is better than the others. The funny thing is, there's no fill function implemented for the long array, how come it is so darn quick? Also, why does the objects fill suck quite so badly when compared with the rest (I will not be addressing this last question! I refuse! this post is too fucking long as it is!)?<br />
This is what happens when we turn off the <span style="font-family: Courier New, Courier, monospace;">OptimizeFill</span> flag:<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillBytes 1013.803 ± 0.227 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillChars 323.806 ± 3.879 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillShorts 323.689 ± 4.499 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillInts 326.336 ± 1.559 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillFloats 319.449 ± 2.048 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillLongs 328.692 ± 3.282 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillDoubles 345.035 ± 6.362 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.fillObjects 4397.130 ± 7.161 ns/op</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div class="p1">
<span style="font-family: inherit;">Strange innit? now we got char/int/long arrays all performing similarly. In fact, with the exception of the byte array, everything is better than it was with the optimization.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h3>
<span style="font-family: inherit;"><a href="https://www.youtube.com/watch?v=t-wUe5aEwHM" target="_blank">Superword to the rescue!</a> </span></h3>
Turns out the JIT compiler is clued up on the topic of SIMD parallelisation by way of Superword Level Parallelism (see the <a href="http://groups.csail.mit.edu/cag/slp/SLP-PLDI-2000.pdf" target="_blank"><span id="goog_352674027"></span>original paper here<span id="goog_352674028"></span></a>):<br />
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">In some respects, superword level parallelism is a restricted form of ILP (Instruction Level Parallelism). ILP techniques have been very successful in the general purpose computing arena, partly because of their ability to find parallelism within basic blocks. In the same way that loop unrolling translates loop level parallelism into ILP, vector parallelism can be transformed into SLP. This realization allows for the parallelization of vectorizable loops using the same basic block analysis. As a result, our algorithm does not require any of the complicated loop transformations typically associated with vectorization. In fact, vector parallelism alone can be uncovered using a simplified version of the SLP compiler algorithm.<br />...<br />Superword level parallelism is defined as short SIMD parallelism in which the source and result operands of a SIMD operation are packed in a storage location.<br />...<br />Vector parallelism is a subset of superword level parallelism.</span></blockquote>
The Hotspot compiler implements SLP optimizations in <a href="http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/55fb97c4c58d/src/share/vm/opto/superword.cpp" target="_blank">superword.cpp</a> and you are invited to dive into the implementation if you like. I'm going to focus on it's impact here, and to do that I only need to know how to turn it on and off (core competency for any software person). It's on by default, so above results are what happens when it is on, here's what life looks like when it is off too (so -XX:-OptimizeFill -XX:-UseSuperWord):<br />
<span style="font-family: Courier New, Courier, monospace;">
ArrayFill.fillBytes 8501.270 ± 2.896 ns/op<br />
ArrayFill.fillChars 4286.161 ± 4.935 ns/op<br />
ArrayFill.fillShorts 4286.168 ± 3.146 ns/op<br />
ArrayFill.fillInts 2152.489 ± 2.653 ns/op<br />
ArrayFill.fillFloats 2140.649 ± 2.587 ns/op<br />
ArrayFill.fillLongs 1105.926 ± 2.228 ns/op<br />
ArrayFill.fillDoubles 1105.820 ± 2.393 ns/op<br />
ArrayFill.fillObjects 4392.506 ± 11.678 ns/op</span><br />
<br />
Life is revealed in all it's sucky splendour! This is what happens when the compiler shows you no love... did I say no love? hang on, things can get a bit worse.<br />
<h3>
Detour: Unsafe? We don't serve you kind here</h3>
To all the Unsafe fans, I got some sad news for y'all. Unsafe 'fill' loops are not well loved by the compiler. This is the price of stepping off the beaten path I guess. Consider the following benchmark:
<script src="https://gist.github.com/nitsanw/c3a8c7b99d5deb7e1d45.js"></script>
<br />
The results are:<br />
<span style="font-family: Courier New, Courier, monospace;">
ArrayFill.unsafeFillOffheapBytes 9742.621 ± 2.270 ns/op<br />
ArrayFill.unsafeFillOnHeapBytes 12640.019 ± 1.977 ns/op<br />
ArrayFill.fillBytes(for reference) 561.540 ± 10.814 ns/op</span>
<br />
The Unsafe variant do not enjoy the 'fill' pattern matching magic, nor do they get the SuperWord optimizations. What can you do? For this kind of thing you should use the Unsafe.setMemory method instead:<br />
<script src="https://gist.github.com/nitsanw/d288061d1070bde7226b.js"></script>
With the result:<br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.unsafeSetOffheapBytes 1259.281 ± 21.294 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace;">ArrayFill.unsafeSetOnHeapBytes 1275.158 ± 27.950 ns/op</span><br />
Not quite there, still ~2x worse (why? how come it doesn't just call the bytes fill method? a bit of digging shows it ends up calling the underlying platform's memset...) but beats being 20-25x worse like the handrolled method is.
<br />
<br />
<h3>
Summary and Musings</h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsd37Lgw2dWok484Z4u7DGrr6IxN_AwzaeagK9uMWCjqWKzjrK9CslTA_HZF2OnfmTTtYShenq8YortJuLOyUZvcXAE9aP1ezkRXmbr52Qv2bq9crLex68Xgfgj_66SowjsBkCyDMXzCg/s1600/TheRealCircleOfLife-52687.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsd37Lgw2dWok484Z4u7DGrr6IxN_AwzaeagK9uMWCjqWKzjrK9CslTA_HZF2OnfmTTtYShenq8YortJuLOyUZvcXAE9aP1ezkRXmbr52Qv2bq9crLex68Xgfgj_66SowjsBkCyDMXzCg/s1600/TheRealCircleOfLife-52687.jpg" height="320" width="285" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">It's the circle of life!</td></tr>
</tbody></table>
<div>
So what did we learn:</div>
<div>
<ul>
<li>There's another kind of 'intrinsic' like optimization, which uses pattern matching to swap a block of code rather than a method. This is employed for memset like memory fill loops (in particular Arrays.fill) intrinsicfication. It's not an intrinsic technically, but you know what I fucking mean. </li>
<li>System.arrayCopy/Arrays.fill implementations utilize SIMD instructions to improve their efficiency. These instructions are not available in plain Java, so some compiler intervention is required.</li>
<li>The JIT compiler is also able to use SuperWord Level Parallelism to derive SIMD code from 'normal' sequential code.</li>
<li>In the case of Arrays.fill, it looks like the SuperWord optimized code is faster than the fill specialized implementation for all types except bytes (on the system under test)</li>
<li>If you use Unsafe you will be excluded from these optimizations.</li>
</ul>
So I look at this process and I imagine history went something like this:</div>
<div>
We want to use SIMD instructions, but the JIT compiler isn't really clever enough to generate them by itself. Memset implementations are rather specialized after all. Let's make life a bit easier for the compiler by creating an intrinsic. We'll even go the extra mile and make an effort to automatically identify opportunities to use this intrinsic, so now it's not really an intrinsic any more. The Arrays.fill optimization is available on Oracle JDK6u45 (the oldest I keep around, maybe it was there a while before that) and on that JVM it is twice as fast as the SLP generated code.</div>
<div>
Over time, SLP gets better and eventually the compiler is now good enough to optimize the fill loop by itself and beat the specialized method. That is an awesome thing. We just need to remove the training wheels now.</div>
<div>
And there's a final punch line to this story. Memset/Memcpy are such common and important opportunities for optimization, so Intel has decided to offer an assembly 'recipe' for them and save everyone the effort in writing them:</div>
<div>
<blockquote class="tr_bq">
<b>3.7.6 Enhanced REP MOVSB and STOSB operation (ERMSB)</b><br />
Beginning with processors based on Intel microarchitecture code name Ivy Bridge, REP string operation using MOVSB and STOSB can provide both flexible and high-performance REP string operations for soft- ware in common situations like memory copy and set operations. Processors that provide enhanced MOVSB/STOSB operations are enumerated by the CPUID feature flag: CPUID:(EAX=7H, ECX=0H):EBX.ERMSB[bit 9] = 1. - [From the Intel Optimization Manual(September 2014)]</blockquote>
From the manual it seems that this method of implementing memcpy/memset can perform well, but like anything else, YMMV (the intel manual discussion of the performance differences is in itself interesting both on the results and the methodology level). One obvious advantage of this method is that it results in much much smaller code that should be trivial to inline into callers. This will however put the SuperWord method at a slight disadvantage, and the tide will change again.<br />
[UPDATE 14/03/2015: It seems the good folks of Oracle have <a href="http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8005522" target="_blank">considered and rejected the use of REP MOVSB for array copy.</a>]<br />
Thanks go to the kind reviewers <a href="https://twitter.com/peterhughesdev" target="_blank">Peter 'Massive' Hughes</a>, <a href="https://twitter.com/darachennis" target="_blank">Darrach</a> and the <a href="https://twitter.com/shipilev" target="_blank">Shipster</a></div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com12tag:blogger.com,1999:blog-5171098727364395242.post-50019176851648611832015-03-11T10:45:00.001+00:002015-03-11T10:45:48.267+00:00Correcting YCSB's Coordinated Omission problemYCSB is the <a href="http://labs.yahoo.com/news/yahoo-cloud-serving-benchmark/" target="_blank">Yahoo Cloud Serving Benchmark</a>(also on <a href="http://en.wikipedia.org/wiki/YCSB" target="_blank">wiki</a>): a generic set of benchmarks setting out <br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyYm4y_WmTX2vz0WjMBVQ7djSdPYyBERUBqCHFt06L5ZQnTFzQiZF8qUEIlvRG9VzOUs7iSOQ3V4EvAcc137cf6luM7fvc19kmMUG70Hg8YmyTCto7d06r1ka0ERTZIyX8dITZVv9IEnE/s1600/NIMBUS-CLOUD-SERVING-BOARD_zpsd68a9cf2.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyYm4y_WmTX2vz0WjMBVQ7djSdPYyBERUBqCHFt06L5ZQnTFzQiZF8qUEIlvRG9VzOUs7iSOQ3V4EvAcc137cf6luM7fvc19kmMUG70Hg8YmyTCto7d06r1ka0ERTZIyX8dITZVv9IEnE/s1600/NIMBUS-CLOUD-SERVING-BOARD_zpsd68a9cf2.jpeg" height="213" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Nimbus Cloud Serving Board</td></tr>
</tbody></table>
to compare different key-value store providers under a set of loads:<br />
<blockquote class="tr_bq">
<span style="background-color: white; color: #181818; line-height: 16px;"><span style="font-family: inherit;">The goal of the Yahoo Cloud Serving Benchmark (YCSB) project is to develop a framework and common set of workloads for evaluating the performance of different "key-value" and "cloud" serving stores.</span></span></blockquote>
The code is open for extension and contribution and all that good stuff, you can get it <a href="https://github.com/brianfrankcooper/YCSB" target="_blank">here</a>. And it has become tool for comparing vendors in the NoSQL space. The benchmarks set out to measure latency and throughput. The terms are not directly defined in the <a href="https://www.cs.duke.edu/courses/fall13/compsci590.4/838-CloudPapers/ycsb.pdf" target="_blank">paper</a>, but the following statement is made:<br />
<blockquote class="tr_bq">
The Performance tier of the benchmark focuses on the
latency of requests when the database is under load. Latency
is very important in serving systems, since there is
usually an impatient human waiting for a web page to load. [...] Typically application designers must decide on an
acceptable latency, and provision enough servers to achieve
the desired throughput while preserving acceptable latency. [...] The YCSB
Client allows the user to define the offered throughput as
a command line parameter, and reports the resulting latency,
making it straightforward to produce latency versus
throughput curves.</blockquote>
What could possibly go wrong?™<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUpizleOFJVDCZryfxbHmZZbGpa_2NhXV9rnmNz5v9C4LsXVLrQ1L9Oe6dXRR9tm7t4nUl2FNbb_KSFPoptiiGlyFvInW2S5iXMYNj_tgjV1lXzIXtgiHQakymnb-gABxETHlEIKsyQ6g/s1600/Histogram+(2).png" imageanchor="1" style="font-size: medium; font-weight: normal; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUpizleOFJVDCZryfxbHmZZbGpa_2NhXV9rnmNz5v9C4LsXVLrQ1L9Oe6dXRR9tm7t4nUl2FNbb_KSFPoptiiGlyFvInW2S5iXMYNj_tgjV1lXzIXtgiHQakymnb-gABxETHlEIKsyQ6g/s1600/Histogram+(2).png" height="244" width="640" /></a>
<div>
It can go this wrong for instance, order of magnitude difference in results for different percentiles, leading to some poor decision making on how much hardware you'll need, leading to getting fired from your job and growing old bitter and twisted mumbling to yourself as you get drunk on the street corner until you freeze to death on a winter night. So potentially this is a risk to your future well being, listen up!</div>
<h3><br/>
It's broken? Coordinated WOT?</h3>
<div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7H1S-VVsqb6XrdSRtRBs1FKrNfx3V_tZGEQQR-TZOmuYVHaPHISG_c6atFsAIgKbgRzW2Zd1Q9VbNSmCkQKfYTWolR2JsKaQOmUds-8XG1AwGtxEVq4a3hOddjm7crs3YFw1z6p1MwZA/s1600/gil-UnderstandingLatency.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7H1S-VVsqb6XrdSRtRBs1FKrNfx3V_tZGEQQR-TZOmuYVHaPHISG_c6atFsAIgKbgRzW2Zd1Q9VbNSmCkQKfYTWolR2JsKaQOmUds-8XG1AwGtxEVq4a3hOddjm7crs3YFw1z6p1MwZA/s1600/gil-UnderstandingLatency.jpg" height="146" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: xx-small;">When you measure latency bad,</span><br />
<span style="font-size: xx-small;">Mr. Tene is sad</span><br />
<span style="font-size: xx-small;"> :(</span></td></tr>
</tbody></table>
My colleague at Azul, <a href="https://github.com/giltene" target="_blank">Gil Tene</a>, the magnificent, glorious, multi-dimensional, coding CTO, officer and gentleman (that's my bonus sorted) has been doing a lot of latency related preaching and teaching in the last few years. He has given the following talks at any number of conferences, but if you happened to have missed them, watch them NOW:</div>
<div>
<ul>
<li>"<a href="http://www.infoq.com/presentations/latency-pitfalls" target="_blank">How NOT to Measure Latency</a>" - At QCon London, 2013</li>
<li>"<a href="https://www.youtube.com/watch?v=9MKY4KypBzg" target="_blank">Understanding Latency</a>" - At React Conf SF, 2014</li>
</ul>
</div>
<div>
In particular he has coined the term "Coordinated Omission" (<a href="https://groups.google.com/forum/#!msg/mechanical-sympathy/icNZJejUHfE/BfDekfBEs_sJ" target="_blank">see raging discussion on Mechanical Sympathy</a>) to describe instances in which the measuring/monitoring system coordinates measurement with the system under test/measurement such that samples are biased. This issue manifests in many load generating frameworks where the call into the system under test is done synchronously and the measurement thread holds up the next sample while the call is ongoing. This enables the system under test to delay requests that would have been made during the synchronous call thus skewing the sample set. Consider for example a system where:</div>
<div>
<ul>
<li>We set out to measure a request every 1ms (from a single thread, synchronously)</li>
<li>The first 500 calls come back in 100µs each (so call K starts at Kms and returns at Kms + 100µs )</li>
<li>Call 501 takes <b>500 milliseconds </b>(starts at 500ms, returns at 1 second)</li>
<li>Call 502 takes 100µs</li>
</ul>
</div>
<div>
See the problem?</div>
<div>
The problem is that call 502 did NOT happen at it's designated time, and saying it took 100µs fails to capture this. It failed the assumptions laid out in the first sentence because we were blocked for 500ms. If we were to stick to our original <b>schedule </b>we would be making calls 502 to 1000 in the time it took for call 501 to execute. How should we treat this departure from plan?</div>
<div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdjDzJR6uL7IF-U8mtiC-UI9f-YoDUCRzLdnefIlwjt_wRnPS7mK92YpeRIlEJjIkAhY0fdQPRebJ7ORw6bz3NljdcSa9R35Ft93Ro_QLXOK9NTGnyN2cRrmrDuOZValxXqiI3VCc2vGk/s1600/heads+in+sand.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdjDzJR6uL7IF-U8mtiC-UI9f-YoDUCRzLdnefIlwjt_wRnPS7mK92YpeRIlEJjIkAhY0fdQPRebJ7ORw6bz3NljdcSa9R35Ft93Ro_QLXOK9NTGnyN2cRrmrDuOZValxXqiI3VCc2vGk/s1600/heads+in+sand.jpg" height="92" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Is it safe to come out yet?</td></tr>
</tbody></table>
<ol>
<li><b>Ignore it!</b> it will go away by itself! - This is the coordinated omission way. We are now reporting numbers that are no longer according to the test plan, which means that our "latency versus throughput curves" are off the mark. This is a very common solution to this issue.</li>
<li><b>Fail the test</b>, we wanted a call every ms and we didn't get that - This is an honest hardline answer, but it potentially throws the baby with the bath water. I think that if you set out to schedule 1000 calls per second you might want to see how often this falls apart and how. But this answer is The Truth™, can you handle it? If one is to start from scratch and write their own load generator I propose a read of the <a href="http://twitter.github.io/iago/philosophy.html" target="_blank">Iago load test framework philosophy</a> page: "<span style="background-color: white; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 18px;">Iago accurately replicates production traffic. It models open systems, systems which receive requests independent of their ability to service them. Typical load generators measure the time it takes for </span><var style="background-color: white; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 18px;">M</var><span style="background-color: white; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 18px;"> threads to make </span><var style="background-color: white; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 18px;">N</var><span style="background-color: white; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 18px;"> requests, waiting for a response to each request before sending the next; if your system slows down under load, these load testers thus mercifully slow down their pace to match. That's a fine thing to measure; many systems behave this way. But maybe your service isn't such a system; maybe it's exposed on the internet. Maybe you want to know how your system behaves when </span><var style="background-color: white; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 18px;">N</var><span style="background-color: white; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 18px;"> requests per second come in with no "mercy" if it slows down.</span>". This is a fine sentiment.</li>
<li><b>Coordinated Omission correction:</b> Adjust the results to reflect the expected call rate. This can be done in a straight forward manner if the 'missing' calls are added back with a latency which reflects the period for which they were delayed. This correction method is supported out of the box by HdrHistogram but the discussion regarding it's over or under estimation of the impact of the delay is outside the scope of this post.</li>
<li><b>Coordinated Omission avoidance:</b> Measure all calls according to original <b>schedule</b>. We are now saying: "If I can't make the call, the meter is still running!". This is particularly relevant for systems where you would typically be making the requests to the system under test from a thread pool. That thread pool would be there to help you support asynchronous interaction where the API failed in giving you that option. Like JDBC... Like many key-value pair provider APIs.</li>
</ol>
This last solution is the one we'll go for in this post, but I would urge you to consider the results critically. In particular if you are trying to simulate independent access to a web server (as opposed to a DB via a thread pool) then the adherence to schedule might be hugely optimistic of the results in your case. This is because failing to generate independent load may have all sorts of beneficial effects on the system under test.<br />
For the YCSB benchmark I'm assuming the harness/load generator is simulating a web serving layer accessing the key-value store in an effort to serve an unbounded, uncoordinated user request load via a predefined thread pool. So it's door number 3 for me. <a href="https://github.com/LatencyUtils/YCSB2" target="_blank">The corrected load generator is here</a>.</div>
<h3><br/>
Step 0: Some preliminary work (not strictly required)</h3>
<div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgem0BxGkkKUVIBTpyZS9CsV3V9s6Vg65ei_o2RIxIti55JpfxN_rpBZmoI6hoLZqxr-51eBhBBzohUiCEGzQpLMMf3r3wMhe1BDExHpPT4sV8hUUGjkS6vQF1TpGYKizLsNqJ28C-ud4k/s1600/fmercury-thumbsup.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgem0BxGkkKUVIBTpyZS9CsV3V9s6Vg65ei_o2RIxIti55JpfxN_rpBZmoI6hoLZqxr-51eBhBBzohUiCEGzQpLMMf3r3wMhe1BDExHpPT4sV8hUUGjkS6vQF1TpGYKizLsNqJ28C-ud4k/s1600/fmercury-thumbsup.jpeg" height="129" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">HdrHistogram, as approved by Freddie!</td></tr>
</tbody></table>
As described previously here, we should all <a href="http://psy-lob-saw.blogspot.com/2015/02/hdrhistogram-better-latency-capture.html" target="_blank">just get on with capturing latency using HdrHistograms.</a> So as a first step toward correcting YCSB I have gone in and added an HdrHistogram measurement container. This is pretty straight forward as all I needed to modify was the <a href="https://github.com/LatencyUtils/YCSB2/blob/master/core/src/main/java/com/yahoo/ycsb/measurements/Measurements.java" target="_blank">Measurements</a> class to allow a new measurement type. While I was there I tweaked this and that and the following list of changes to that class emerged:</div>
<div>
<ol>
<li>Add new measurement type and corresponding command-line option("<i>-p </i><i>measurementtype=</i><i>hdrhistogram</i>")</li>
<li>Add combined measurement option allowing old/new measurement side by side: "<i>hdrhistogram+histogram</i>"
</li>
<li><div class="p1">
Add support for capturing both corrected and uncorrected measurements for the same run.</div>
</li>
<li><div class="p1">
Use CHM instead of synchronizing around a HashMap.</div>
</li>
</ol>
The <a href="https://github.com/LatencyUtils/YCSB2/blob/master/core/src/main/java/com/yahoo/ycsb/measurements/OneMeasurementHdrHistogram.java" target="_blank">new measurement type</a> supports logging loss less HdrHistogram data to a file (controlled by the <i>hdrhistogram.fileoutput=<true|false> </i>option and the <i>hdrhistogram.output.path=<path></i> option) as well as better precision percentile data and lock free logging of latencies. This is not very interesting work but if you are interested in the "How would I plug in my own data structure to capture latency into YCSB?" topic have fun. It was not necessary for correction but it was good to do so that better quality results can be observed. You're welcome.</div>
<h3><br/>
Step 1: Demonstrate the issue</h3>
<div>
YCSB includes a very useful means of verifying the measurements in the form of a mock DB driver. This means we can test our assertions regarding coordinated omission without setting up a key value store of any kind. The mock DB is called BasicDB and is the default DB used. We can configure it to simulate a pause and see what happens (<i>-p basicdb.verbose=false -p basicdb.simulatedelay=4 </i> will make the mock DB stop logging every action and simulate a latency of 0-4ms for each action). I added a further option to the BasicDB which allows us to turn off the randomization of the delay (<i>-p basicdb.randomizedelay=false</i>).</div>
<div>
Let's consider our expectations in the case where a DB simply cannot handle request quickly enough. We can setup an experiment with the following settings: <i>-target 1000 -threads 1 -s </i><i>-p status.interval=1 </i><i>-p workload=com.yahoo.ycsb.workloads.CoreWorkload -p basicdb.verbose=false -p basicdb.simulatedelay=4 -p basicdb.randomizedelay=false -p measurementtype=hdrhistogram -p maxexecutiontime=60</i></div>
<div>
Here's what they all mean:</div>
<div>
<ul>
<li><i>-target 1000 -> </i>We aim to test 1000 requests per second</li>
<li><i>-threads 1 -> </i>We have a single client thread</li>
<li><i>-s </i><i>-p status.interval=1 -> </i>We will be printing out status every second (I made the status interval configurable)</li>
<li><i>-p basicdb.verbose=false -p basicdb.simulatedelay=4 -p basicdb.randomizedelay=false -> </i>The DB will sleep 4ms on each request, so the maximum we can hope for is 250, no noisy printing per operation please</li>
<li><i>-p measurementtype=hdrhistogram -> </i>Use HdrHistogram to capture the latencies</li>
<li><i>-p maxexecutiontime=60 -></i> Run for one minute, then exit and print summary</li>
</ul>
This DB is obviously failing, it can't keep up with the rate of incoming requests and according to our model they queue up. The time measured per call is reflected in the summary for the READ operations:</div>
<div>
<blockquote class="tr_bq">
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Operations, 12528.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], AverageLatency(us), 4477.102809706258</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MinLatency(us), 4018.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MaxLatency(us), 44703.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 95thPercentileLatency(ms), 4.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 99thPercentileLatency(ms), 4.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Return=0, 12528</span></div>
</blockquote>
But this completely ignores the time spent on the queue. If we were measuring according to schedule we'd get the following set of latencies:</div>
<blockquote class="tr_bq">
Latency[k] = 4 + 3*(k-1) ms</blockquote>
The max latency would be for the last request to get in. We ran for 60 seconds, at 250 requests/sec which means our last request was (k=15000) and had a latency of <b>45 seconds</b> when measured from the time we <u>intended to make it</u>. This number reflects the system's failure to handle load far more correctly than the numbers quoted above.<br />
<h3><br/>
Step 2: Working to Schedule</h3>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvQLsoodL6soLI8cbu4fwMY_A-01mty8RpKk1ZjjLViTKkT0fGyJoYWTy4qEpuS86gKtCTOAl_AswQgiC2YU5FjgI1fOF38BMZyP9JkjeQrUBsjNTA-TgJ3lV93CkbLXfMcpp4NvIPUTA/s1600/440px-Wittner_metronome.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvQLsoodL6soLI8cbu4fwMY_A-01mty8RpKk1ZjjLViTKkT0fGyJoYWTy4qEpuS86gKtCTOAl_AswQgiC2YU5FjgI1fOF38BMZyP9JkjeQrUBsjNTA-TgJ3lV93CkbLXfMcpp4NvIPUTA/s1600/440px-Wittner_metronome.jpg" height="150" width="200" /></a></div>
The YCSB load generator has a weak notion of schedule, in the sense that it opts for option number 1 above and will just execute the operations when it can. When faced with the task of correcting this kind of issue (in a pile of foreign code) we need to look for 2 things in the load generator:<br />
<ol>
<li>"Scheduling an action to run at time X" - This will involve some calls to one of the many scheduling facilities in the JDK:</li>
<ol>
<li>Thread.sleep is an old favourite, but <a href="http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/TimeUnit.html" target="_blank">TimeUnit</a> also supports a sleep method. A search for sleep in the code base will cover both. This is <a href="https://github.com/brianfrankcooper/YCSB/blob/5659fc582c8280e1431ebcfa0891979f806c70ed/core/src/main/java/com/yahoo/ycsb/Client.java#L259" target="_blank">what YCSB was using</a> to schedule next event to fire.</li>
<li>Code submitting tasks to <a href="http://docs.oracle.com/javase/8/docs/api/java/util/Timer.html" target="_blank">java.util.Timer</a>, or alternatively the <a href="http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ScheduledExecutorService.html" target="_blank">ScheduledExecutorService</a></li>
<li>Code using <a href="http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/LockSupport.html" target="_blank">LockSupport</a>.parkNanos</li>
<li>Object.wait(...)</li>
<li>others?</li>
</ol>
<li>"Measuring the length of an operation" - This will involve calls to System.nanoTime() or currentTimeMillis(). For YCSB this is found <a href="https://github.com/brianfrankcooper/YCSB/blob/5659fc582c8280e1431ebcfa0891979f806c70ed/core/src/main/java/com/yahoo/ycsb/DBWrapper.java#L89" target="_blank">to happen for example here</a>.</li>
</ol>
<br />
To correct this problem I had to introduce the concept of 'intended start time' to the operations measurement. Schedule for YCSB is specified by the <i>-target </i>command line option which sets the overall number of operations per second to be attempted by the load generator. This is optional, and the default is to go as fast as you can manage, i.e. with no schedule but the back pressure from the system under test to guide us. I'm not sure what a good assumed rate of requests is reasonable in this case, so I did not correct this case.<b> NOTE: If you don't specify <i>target</i> no correction will take place.</b></div>
<div>
The <i>target</i> parameter is translated to a per-thread operation rate (number of threads is set via the <i>threads</i> option, default is 1) so if we have 10 threads, and the target request rate is 1000 (<i>-target 1000 -threads 10</i>) we will have each thread hitting the store with 100 requests per second. The client threads randomize the first operation time to avoid all hitting the store on the same interval. I did some ground work here by setting the units across the board to nanoseconds and naming interval parameters appropriately, nothing too exciting.<br />
The actual correction at it's core involves:<br />
<ol>
<li>Record the operation's intended start time</li>
<li>Use the intended start time when computing latency</li>
</ol>
Sadly the way YCSB measures latency does not lend itself to a simple in place fix. The operations are scheduled by the ClientThread which calls into a workload, calling into a DB, which is actually the DBWrapper which measures the latency (for calling into an actual DB implementation) and reports it to the central Measurements singleton. This means that changing the Workload/DB API to include a startTime parameter to each call is quite a far reaching change which would require me to dig through all the DB drivers implementations and would result in a very unpleasant time for all.<br />
I settled on using a thread local on the Measurements object to transfer the start time to the DBWrapper, it is not a nice way to do things (and I'm happy to hear better suggestions) but it does the job without modifying the API.<br />
Once we have:<br />
<ol>
<li>ClientThread <a href="https://github.com/LatencyUtils/YCSB2/blob/master/core/src/main/java/com/yahoo/ycsb/Client.java#L305" target="_blank">setting up the start time for the operation via Measurements</a></li>
<li>DBWrapper <a href="https://github.com/LatencyUtils/YCSB2/blob/master/core/src/main/java/com/yahoo/ycsb/DBWrapper.java#L156" target="_blank">using the start time from Measurements</a> to compute the operation latency</li>
</ol>
That's pretty much it. For extra points I wanted to include some facilities to compare measurements before/after the change. These can be removed if we accept HdrHistogram as a replacement and if we accept we only want to measure the intended latency, which would result in a much smaller PR.<br />
<h3><br/>
Step 3: is the issue solved?</h3>
</div>
<div>
Running the setup from step 1 such that it produces the intended latency as well as the original measurement side by side(<i>-p measurement.interval=both</i>) yields the following result for the READ operations:<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Operations, 12414.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], AverageLatency(us), 4524.981069759949</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MinLatency(us), 4018.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MaxLatency(us), 24703.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 95thPercentileLatency(ms), 4.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 99thPercentileLatency(ms), 4.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Return=0, 12414</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], Operations, 12414.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], AverageLatency(us), 2.359010991606251E7</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MinLatency(us), 4256.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MaxLatency(us), 4.6989311E7</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 95thPercentileLatency(ms), 42369.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 99thPercentileLatency(ms), 46530.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
</div>
<div>
This reflects the effect a backed up system would have on latency as we express in Step 1 above. It's actually a bit worse because the average cost of calling the mock DB with a sleep of 4ms is 4.5ms. As we can see the maximum latency is 46.9 seconds, reflecting the fact that the last read to execute was scheduled to hit the system 13.1 seconds into the run.<br />
<h3><br/>Step 4: The limitations of the harness</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitePGFdnmg920ng2yVVd-TMdiMUp5mQGOXUZuY6TEZ3MrHWH-tqBBaQwBQakb80b_IX8ClG57lVqG-Z-mTEfAPOSf8FSCEvurcHQNvWKgaiDHQqxjeMVPcTOTjSTPgXqj8LLruaEAEGrE/s1600/mesh+non-pull+harness.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitePGFdnmg920ng2yVVd-TMdiMUp5mQGOXUZuY6TEZ3MrHWH-tqBBaQwBQakb80b_IX8ClG57lVqG-Z-mTEfAPOSf8FSCEvurcHQNvWKgaiDHQqxjeMVPcTOTjSTPgXqj8LLruaEAEGrE/s1600/mesh+non-pull+harness.jpg" height="200" width="200" /></a></div>
We can now also consider the perfect DB for the sake of observing the short comings of the test harness by setting the mock DB delay to 0(<i>-p basicdb.simulatedelay=0</i>):<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Operations, 56935.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], AverageLatency(us), 0.01796785808377975</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MinLatency(us), 0.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MaxLatency(us), 49.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 95thPercentileLatency(ms), 0.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 99thPercentileLatency(ms), 0.0</span></div>
<div class="p1">
</div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Return=0, 56935</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], Operations, 56935.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], AverageLatency(us), 232.37026433652412</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MinLatency(us), 0.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MaxLatency(us), 39007.0</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 95thPercentileLatency(ms), 0.0</span></div>
<div class="p1">
</div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 99thPercentileLatency(ms), 0.0</span></div>
<div class="p1">
<br /></div>
<div class="p1">
How come it take so long to measure a noop? why such large differences? Here's some generic theories and how they panned out:</div>
<div class="p1">
</div>
<ul>
<li><b>The JVM running the load generator is running with suboptimal settings(<i>-Xms64m -Xmx64m -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime, </i>Oracle JDK8u31) on a busy Mac laptop running on battery</b></li>
</ul>
This is no way to benchmark anything, but the interesting thing is that if we have no schedule to stick to the test harness is willing to just ignore the issue. If we run on a decent machine (with a decent OS) we get nicer results. <span style="font-family: inherit;">This is from a server class machine running CentOS6.3/OracleJDK8u25 with same settings:</span><br />
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Operations, 56930.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], AverageLatency(us), 0.44417705954681186</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MinLatency(us), 0.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MaxLatency(us), 20.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 95thPercentileLatency(ms), 0.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 99thPercentileLatency(ms), 0.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Return=0, 56930</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], Operations, 56930.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], AverageLatency(us), 146.31262954505533</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MinLatency(us), 15.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MaxLatency(us), 14255.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 95thPercentileLatency(ms), 0.0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 99thPercentileLatency(ms), 0.0</span></div>
<div>
<span style="font-family: inherit;">This is still significant.</span><br />
<span style="font-family: inherit;"><br /></span></div>
<ul>
<li><b>The JVM suffers from warmup related artefacts</b></li>
</ul>
This certainly correlated to the max values I'm seeing here. When looking at the status line for the first second I see:</div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[READ: Count=22, Max=14, Min=0, Avg=0.64, 90=0, 99=14, 99.9=14, 99.99=14]</span></div>
<div>
<span style="font-size: x-small;"><span style="font-family: 'Courier New', Courier, monospace;">[Intended-READ: Count=23, Max=</span><b style="font-family: 'Courier New', Courier, monospace;">14255</b><span style="font-family: 'Courier New', Courier, monospace;">, Min=15, Avg=5563.39, 90=13287, 99=14255, 99.9=14255, 99.99=14255]</span></span><br />
But after a few seconds the process settles and we see much better results, this is typical:<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ: Count=947, Max=14, Min=0, Avg=0.02, 90=0, 99=0, 99.9=2, 99.99=14]</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ: Count=946, Max=194, Min=61, Avg=151.29, 90=165, 99=175, 99.9=186, 99.99=194]</span><br />
<span style="font-family: inherit;">A good way to handle this issue is by relying on the HdrHistogram output files to grab relevant time periods for analysis. With the original YCSB output we have the percentile summary data, but this is not something we can combine for analysis. With the loss-less interval histogram logs we can look at any sub-period(which is longer than one interval, but shorter than the whole run) and get accurate full range histogram data. A common practice is to discard warmup period results, I'm no a fan of throwing away data, but since this is the load generator warmup I'd think it's quite legitimate. It's perhaps an interesting feature to add to such a framework that the framework can be warmed up separately from the system to examine cold system behaviour.</span><br />
<ul>
<li><b>Thread.sleep/LockSupport.parkNanos are not super accurate and may wakeup after the intended operation start time</b></li>
</ul>
I've added an option for spinning instead of sleeping (so burn a CPU). This has improved the average value dramatically from ~146µs to ~3.1µs. A typical status line now looks like:</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ: Count=947, Max=13, Min=0, Avg=0.02, 90=0, 99=0, 99.9=3, 99.99=13]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ: Count=948, Max=57, Min=0, Avg=0.47, 90=1, 99=12, 99.9=26, 99.99=57]</span></div>
<div>
It is obviously not desirable for the load generator to burn a CPU instead of using sleep, but it also introduces scheduling inaccuracies. This is an accuracy issue we didn't have to deal with when not measuring from a schedule. This didn't impact the measured outliers, but has dramatically reduced their number. The take away here is just that there are accuracy limitations to the load generators ability to stick to schedule.</div>
<div>
<br />
<ul>
<li><b>GC pauses that are large enough to derail the schedule on the load generator side are now captured. Unless the GC pauses happen inside the measurement gap we will have no idea we have gone off schedule if we don't track the intended start time.</b></li>
</ul>
We should capture GC logs on load generator side and make sure we correlate the GC events with recorded latencies. Here's a GC pause being captured by the corrected measurement:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ: Count=952, Max=0, Min=0, Avg=0, 90=0, 99=0, 99.9=0, 99.99=0] </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ: Count=952, Max=14, Min=0, Avg=0.03, 90=0, 99=0, 99.9=3, 99.99=14]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[GC (Allocation Failure) [PSYoungGen: 17895K->1824K(18944K)] 17903K->1840K(62976K), <b>0.0024340</b> secs] [Times: user=0.01 sys=0.01, real=0.01 secs]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Total time for which application threads were stopped: <b>0.0026392</b> seconds</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ: Count=957, Max=0, Min=0, Avg=0, 90=0, 99=0, 99.9=0, 99.99=0]</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">[Intended-READ: Count=957, Max=<b>2719</b>, Min=0, Avg=5.21, 90=0, 99=0, 99.9=2119, 99.99=<b>2719</b>]</span></div>
<div>
<span style="font-family: inherit;">This process is running with a 64M heap, you can expect longer pauses as the heap grows (in particular as the young generation grows).</span><br />
<span style="font-family: inherit;"><br /></span></div>
<ul>
<li><b>The operation setup time is now being measured as well as the operation itself.</b></li>
</ul>
When running with the spin option we can see the average operation cost is ~3.1µs, this is all test harness overhead and is really quite negligible in the context of network hopping operations. In other words, nothing to worry about for this harness but could well prove an issue for others.<br />
<h3><br/>Step 5: The Good, The Bad And The STW pausing DB</h3>
Many software processes have a latency profile that is far from normally distributed. To see what YCSB makes of this kind of profile now that we can compare corrected vs. uncorrected measurement I have built a mock DB that has 4 mods of latency (p is uniform random number [0,1]):<br />
<ol>
<li>Awesome (p < 0.9): we return in 200µs-1ms</li>
<li>Good (0.9 < p < 0.99): we return in 1-10ms</li>
<li>Minor Hiccup( 0.99 < p < 0.9999): we hit a bump, but only one thread is affected 10-50ms</li>
<li>Major Hiccup(0.9999 < p): we hit a STW pause(because GC/THP/LBJ/STD/others), all threads stop for 50-200ms</li>
</ol>
I implemented the above with a read write lock, where the STW pause grabs the write lock and all the others grab the read lock. This is far from a perfect representation of a system (everyone waits for STW as intended, but also STW waits for everyone to start), but it will do. If you feel strongly that a better simulation is in order, write one and let's discuss in the comments!<br />
What sort of profile will we see? How far off course will our measurements be if we don't stick to schedule? Here's this setup run at a rate of 10,000 requests per second, with 25 threads (so each thread is trying for 250 reqs/sec or 1 request per 4ms):<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Operations, 569516.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], AverageLatency(us), 1652.1852871561116</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MinLatency(us), 210.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], MaxLatency(us), 142463.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 95thPercentileLatency(ms), 1.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], 99thPercentileLatency(ms), 19.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ], Return=0, 569516</span><br />
<br />
According to these numbers, the max is quite high but the overall impact of hiccups is not too severe (all depends on your requirements of course). Even at this stage we can see that the effect of global pauses is skewing the other measurements (if you hit a short operation while a STW pause is in progress you still have to wait for the STW event to finish).<br />
The corrected measurements tell a different story:<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], Operations, 569516.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], AverageLatency(us), 24571.6025835973</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MinLatency(us), 268.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], MaxLatency(us), 459519.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 95thPercentileLatency(ms), 83.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ], 99thPercentileLatency(ms), 210.0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<br />
<div class="p1">
How can this be right? Can this be right?<br />
<ul>
<li>At a rate of 10000 request per second, the unlikely Major Hiccup is likely to happen every second. Consider this next time someone tells you of a 99.99%ile behaviour. Given an event rate of 10K per second, 99.99% is suddenly not very rare. Consider that at this rate there's likely to be a few events that are worse.</li>
<li>The average major hiccup is 125ms long, in this time 125/4 events are delayed on all 25 threads -> 125 * 25 / 4 = 781 events are delayed from starting, they will further delay each other as they execute. In roughly 12 seconds we can see how it is quite probable that one of these events is another major hiccup. What with all the queuing up behind the first one etc, the pileup becomes quite reasonable.</li>
<li>The probability of a 'mode' is not the probability of the per event latency once STW and queuing effects are in play.</li>
</ul>
I've made the mock DB print out 'OUCH' every time we get slapped with a STW event. It turns out that we got very unlucky in this run and hit three of these in a row:<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">56 sec:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ: Count=9192, Max=83903, Min=238, Avg=1745.13, 90=1531, 99=26959, 99.9=79551, 99.99=83775]</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ: Count=9208, Max=159999, Min=303, Avg=16496.92, 90=54271, 99=103807, 99.9=150527, 99.99=158335]</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">OUCH</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">OUCH</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">OUCH</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><span style="font-size: x-small;">57 sec: </span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[READ: Count=9642, Max=129727, Min=247, Avg=2318, 90=1799, 99=40607, 99.9=125631, 99.99=127359] </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[Intended-READ: Count=9635, Max=459519, Min=320, Avg=102971.39, 90=200319, 99=374271, 99.9=442367, 99.99=457983]</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">This is quite telling.</span><br />
<span style="font-family: inherit;">The view on what's the worst second in this run is wildly different here. Because the uncorrected measurement takes each event as it comes it will take the view that 75 events were delayed by these hiccups, and none by more than 130ms. But from the corrected measurement point of view all the queued up measurements were effected and were further delayed by each other.</span><br />
<span style="font-family: inherit;">I've re-run, this time logging interval histograms in their compressed form for every second in the run. Logging a 60 seconds run with 1 second interval data cost me 200k (we can tweak the construction in </span>OneMeasurementHdrHistogram to minimize the cost<span style="font-family: inherit;">). I can take the compressed logs and use the </span><a href="https://github.com/HdrHistogram/HdrHistogram/blob/master/HistogramLogProcessor" target="_blank">HistogramLogProcessor</a> script provided with HdrHistogram to process the logs (you need to copy the HdrHistogram.jar into the script folder first). Running:<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">./HistogramLogProcessor -i READ.hdr -o uncorrected -outputValueUnitRatio 1000</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">./HistogramLogProcessor -i Intended-READ.hdr -o corrected -outputValueUnitRatio 1000</span></div>
<div class="p1">
Will produce *.hgrm files for both. I then use the <a href="https://github.com/HdrHistogram/HdrHistogram/blob/master/GoogleChartsExample/plotFiles.html" target="_blank">plotFiles.html</a> to generate the following comparison:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUpizleOFJVDCZryfxbHmZZbGpa_2NhXV9rnmNz5v9C4LsXVLrQ1L9Oe6dXRR9tm7t4nUl2FNbb_KSFPoptiiGlyFvInW2S5iXMYNj_tgjV1lXzIXtgiHQakymnb-gABxETHlEIKsyQ6g/s1600/Histogram+(2).png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUpizleOFJVDCZryfxbHmZZbGpa_2NhXV9rnmNz5v9C4LsXVLrQ1L9Oe6dXRR9tm7t4nUl2FNbb_KSFPoptiiGlyFvInW2S5iXMYNj_tgjV1lXzIXtgiHQakymnb-gABxETHlEIKsyQ6g/s1600/Histogram+(2).png" height="244" width="640" /></a></div>
<div class="p1">
<br /></div>
<br />
They tell very different stories don't they.<br />
The red line will have you thinking your system copes gracefully up to the 99%ile slowly degrading to 20ms, when measuring correctly however the system is shown to degrade very quickly with the 20ms line crossed as early as the median, and the 99%ile being 10 times the original measurement. The difference is even more pronounced when we look at one of those terrible seconds where we had back to back STW hiccups. I can use the <a href="https://github.com/HdrHistogram/HdrHistogram/blob/master/HistogramLogProcessor" target="_blank">HistogramLogProcessor</a> script to produce partial summary histograms for the 3 seconds around that spike:<br />
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">./HistogramLogProcessor -i Intended-READ.hdr -o correctedOuch3 -outputValueUnitRatio 1000 -start 1425637666.488 -end 1425637668.492</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoPQmXJWO0Ye2xy5O_gWjEiadDQLtyvEZfxex4OCe0f8S0bwMkLt2No4E8KlZdXx6AGoYwjZ5fmZDHNRrBcck6iJ1bznZMtHbBtLLpz-yZXZYDz_hXm-E_QucGPAZwarScdkNDdTa-UzQ/s1600/Histogram+(3).png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoPQmXJWO0Ye2xy5O_gWjEiadDQLtyvEZfxex4OCe0f8S0bwMkLt2No4E8KlZdXx6AGoYwjZ5fmZDHNRrBcck6iJ1bznZMtHbBtLLpz-yZXZYDz_hXm-E_QucGPAZwarScdkNDdTa-UzQ/s1600/Histogram+(3).png" height="246" width="640" /></a></div>
<div class="p1">
<span style="font-family: inherit;">Similarly we can compare a good second with no STW pauses:</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKUs3rxSuFjcXD-GLsYSyc98K338vvTmZ9Bl5fiX_nwlzeme4kJE8zmHivENkqiPDnO4L9hZfPO6JPfw-_DLbx0CJR2wNd8ACI0YDYbelu-ljJ6qgFa8vAamsi2GESApsDePri9c0y72c/s1600/Histogram+(4).png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKUs3rxSuFjcXD-GLsYSyc98K338vvTmZ9Bl5fiX_nwlzeme4kJE8zmHivENkqiPDnO4L9hZfPO6JPfw-_DLbx0CJR2wNd8ACI0YDYbelu-ljJ6qgFa8vAamsi2GESApsDePri9c0y72c/s1600/Histogram+(4).png" height="246" width="640" /></a></div>
<div class="p1">
<br /></div>
<div class="p1">
<h3><br/>Summary</h3>
</div>
<div class="p1">
Coordinated Omission is a common problem in load generators (and other latency reporters), we had a look at fixing YCSB, an industry standard load generator:</div>
<div class="p1">
</div>
<ul>
<li>Replaced the data structure used to capture latency with HdrHistogram: that is just generally useful and gives us better data to work with when examining the corrected measurement</li>
<li>Found scheduling code and introduced notion of operation start time.</li>
<li>Found measuring code and captured both operation cost (uncorrected measurement) and scheduled time latency (corrected measurement).</li>
<li>Use a mock system under test to evaluate measurement of known scenario. This is a very handy thing to have and luckily YCSB had this facility in place. In other places you may have to implement this yourself but it's a valuable tool to have in order to better understand the measurement capabilities of your harness. This helped highlight the scale of scheduling inaccuracies and test harness overhead per operation, as well as the scale of test harness error during its own warmup period.</li>
<li>Use HdrHistogram facilities to visualise and analyse latency histogram data from the compressed histogram logs.</li>
</ul>
Thanks goes to this posts kind reviewers: <a href="https://twitter.com/peterhughesdev" target="_blank">Peter Huges</a>, <a href="http://twitter.com/darachennis" target="_blank">Darach</a>, and <a href="https://twitter.com/philip_aston" target="_blank">Philip Aston</a><br />
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
</div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com0tag:blogger.com,1999:blog-5171098727364395242.post-42962202337231968052015-02-16T10:28:00.000+00:002015-02-16T10:28:57.851+00:00HdrHistogram: A better latency capture methodSome years back I was working on a latency sensitive application, and since latency was sensitive it was a requirement that we somehow capture latency both on a transaction/event level and in summary form. The event level latency was post processed from the audit logs we had to produce in any case, but the summary form was used for live system monitoring. We ended up with a solution I've seen since in many places (mild variations on a theme), which is what I've come to think of as the linear buckets histogram:<br />
<script src="https://gist.github.com/nitsanw/3e6c59e085b659afb7c3.js"></script><br />
The above data structure was an easy solution to a problem we had little time to solve, but it left much to be desired.<br />
These days the histogram problem is gloriously solved by the <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> (High Dynamic Range), and though it's been around for a couple of years now I still find olden hand rolled histograms in many a code base. Cut that shit out boys and girls! Let me show you a better way :-)<br />
<br />
HdrHistogram highlights:<br />
<ul>
<li>Mature and battle tested, this data structure has been in production for many companies for a while now. Kinks have been unkinked and knickers untwisted.</li>
<li>Multi-lingual support: current implementations available in <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">Java</a>, <a href="https://github.com/HdrHistogram/HdrHistogram_c" target="_blank">C</a>, <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">C#</a>, <a href="https://github.com/HdrHistogram/hdr_histogram_erl" target="_blank">Erlang</a>, Go and more are on the way.</li>
<li>Auto-resizing histograms (if you exceed your initial miximum value estimate)</li>
<li>Compact memory footprint supporting high precision of values across a wide range.</li>
<li>Compressed lossless serialization/de-serialization</li>
<li>Plotting scripts for gnuplot, a webby charting tool and an excel sheet chart</li>
<li>Lock-free concurrency support for recording and logging from multiple threads</li>
<li>Zero allocation on recording path (unless resizing which is optional, and then only if value exceeds initially specified max recorded value)</li>
<li>Constant time measurement which is less than cost of calling System.nanoTime() (on the cost, scalability and trustworthiness of nanoTime <a href="http://shipilev.net/blog/2014/nanotrusting-nanotime/" target="_blank">read Shipilev's excellent report</a>)</li>
</ul>
It is truly as if the bees-knees and the dogs-bollocks had a baby, is it not?<br />
<br />
<h3>
Mama, what's a histogram?</h3>
<div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgsJnVXrdWMPO3XlYxw_ZG4CRQLzLKEGk8dyN9Q-ytejlFBHN2V337GqB7dBB96NZc6dItljd2TzwR6a2bQVl4ZPC0hE6seyXb_Lkaf5585r4rN6Rj52SCW8mMGnHYQwJFvvmeTWSTvdE/s1600/500px-Histogram_of_arrivals_per_minute.svg.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgsJnVXrdWMPO3XlYxw_ZG4CRQLzLKEGk8dyN9Q-ytejlFBHN2V337GqB7dBB96NZc6dItljd2TzwR6a2bQVl4ZPC0hE6seyXb_Lkaf5585r4rN6Rj52SCW8mMGnHYQwJFvvmeTWSTvdE/s1600/500px-Histogram_of_arrivals_per_minute.svg.png" height="168" title="" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: xx-small;">"Histogram of arrivals per minute" by DanielPenfield<br />Own work. Licensed under CC BY-SA 3.0 via Wikimedia Common</span></td></tr>
</tbody></table>
Well, you could look it up on <a href="http://en.wikipedia.org/wiki/Histogram" target="_blank">wikipedia</a>, but the short answer is that a histogram is a summary of data in terms of the frequency of ranges of values. So given the following data set [1,2,3,...,100000] captured for a second I could summarize in several ways:</div>
<div>
<ul>
<li>I could capture the whole range of values each value in a bucket of it's own, assigning each value a frequency of 1 per second.</li>
<li>I could have a single bucket for values between 0 and 100,000, this bucket will have the frequency of 100,000 times per second.</li>
</ul>
These are both a bit silly, the first is as bad as dealing with the full set of data, the second is telling us nothing about the way the 100,000 values break down within the range. Still these are the 2 extremes of histograms, the alternatives lie within that range in terms of the data they offer but there are many ways to skin a cat (apparently, :( poor cats):</div>
<div>
<ul>
<li>Capture the values in 1000 buckets, each bucket representing a range of 100 values: [1..100][101..200]...[99,901..100,000] that will result in 1,000 buckets each with a frequency of 100. This is the sort of histogram described above where all buckets capture the same fixed range.</li>
<li>Capture the values in 17 buckets, each bucket K representing a range [2^K..(2^(K+1)-1)] e.g. [1..1][2..3][4..7]...[65,536..131,071]. This would be a good solution if we thought most values are likely to be small and so wanted higher precision on the lower range, with lower and lower precision for larger values. Note that we don't have to use 2 as the base for exponential histogram, other values work as well.</li>
</ul>
Both of the above solutions trade precision across a large range with storage space. In both solutions I am required to choose up-front the histogram precision and we expect to pay for a large range with either space or precision. Now that we realize what the variables are we can describe these solutions:</div>
<div>
<ul>
<li>Linear buckets: For a range 0..R we will have to pay R/B space for buckets of range B. The higher R is the more space we require, we can compensate by picking a large B.</li>
<li>Exponential buckets: For a range 0..R we require space of log2 of R. The bucket size grows exponentially as we track higher values.</li>
</ul>
The problem we face with latency data points is that the range of values we want to capture is rather large. It is not unreasonable to expect the latency outliers to be several orders of magnitude larger than the typical observed measurement. For example, it may well be that we are timing a method whose cost is in the 100s of nanoseconds, or a high speed connection round trip on the order of 1-100µs but on occasion our latency is dominated by some stop the world GC pause, or network congestion, which is in the order of 10ms to a few seconds. How can we correctly size the range of our histogram? Given the possibility of multi-second GC delays we need to cover a typical range of 1000ns to 100,000,000,000ns. If we used a linear histogram with a 100µs bucket we'd need 1,000,000 buckets (assuming an int counter this will add up to a ~4MB data structure).</div>
<div>
The HdrHistogram follows a different path to the above solutions and manages to accommodate a large range with a high precision across the range in a limited space.</div>
<div>
<br /></div>
<h3>
How does it work?</h3>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsb5-a_7z1wHnuf4p33BHnHjYaIcEk2DrbM857rSyQ38W749WBxd-e2A2PDJ9idIBNfy7mgrl3EM6WgcHZ4q5rIM8azaTESibplbU5kWy-2tiZvEY8X-yPCgfTZYdYzyFBlbkzRIzXPFA/s1600/Snowman+evoloution.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsb5-a_7z1wHnuf4p33BHnHjYaIcEk2DrbM857rSyQ38W749WBxd-e2A2PDJ9idIBNfy7mgrl3EM6WgcHZ4q5rIM8azaTESibplbU5kWy-2tiZvEY8X-yPCgfTZYdYzyFBlbkzRIzXPFA/s1600/Snowman+evoloution.png" /></a></div>
Here's what the documentation has to say:</div>
<blockquote class="tr_bq">
"<span style="background-color: white; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6000003814697px;">Internally, data in HdrHistogram variants is maintained using a concept somewhat similar to that of floating point number representation: Using an exponent a (non-normalized) mantissa to support a wide dynamic range at a high but varying (by exponent value) resolution. AbstractHistogram uses exponentially increasing bucket value ranges (the parallel of the exponent portion of a floating point number) with each bucket containing a fixed number (per bucket) set of linear sub-buckets (the parallel of a non-normalized mantissa portion of a floating point number). Both dynamic range and resolution are configurable, with highestTrackableValue controlling dynamic range, and numberOfSignificantValueDigits controlling resolution."</span></blockquote>
<div>
Hmmm... I'll admit to having difficulty immediately understanding what's happening from the above text, precise though it may be. I had to step through the code to get my head around what works why, read the above again and let it simmer. I'm not going to dig into the implementation because, while interesting, it is not the point of this post. I leave it to the reader to pester <a href="https://twitter.com/giltene" target="_blank">Gil Tene</a>(the author of <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a>) with implementation related questions.</div>
<div>
The principal idea is a mix of the exponential and linear histograms to support a <a href="http://en.wikipedia.org/wiki/Dynamic_range" target="_blank">dynamic range</a> precision that is appropriate to the time unit scale. At the scale of seconds we have a precision of milliseconds, at the scale of milliseconds we have a precision of microseconds etc. This translates roughly into exponential scale buckets which have linear sub-buckets.</div>
<div>
<br /></div>
<h3>
Example: From raw recording to histogram</h3>
<div>
I have posted a while back a Java ping utility which measures the round trip between minimal client and server processes. Each round trip was recorded into a large array and every set number of round trips the measurements were summarized in percentiles:</div>
<div>
<script src="https://gist.github.com/nitsanw/48269a431cc0600c795f.js"></script></div>
<div>
Recording raw data is the simplest way of capturing latency, but it comes at a price. The long[] used to capture the latencies is ~8MB in size, this is for a million samples and in a real application can grow without bounds until some cutoff point where we decide to summarize or discard the data. When we want to report percentiles we have to sort it and pick the relevant data points. This is not usually an acceptable solution (because of the memory footprint), but it offers absolute accuracy and is trivial to implement (until you have to consider serialization, concurrency and visualization, but otherwise trivial).</div>
<div>
Replacing this measurement method with a histogram is straight forward:</div>
<div>
<script src="https://gist.github.com/nitsanw/b01d2cfca5b9ddf678c9.js"></script></div>
<div>
This histogram is 31KB when using 2 decimal places precision which is good enough in most case (according to <a href="http://openjdk.java.net/projects/code-tools/jol/" target="_blank">JOL</a>, add it to your utility belt if it ain't there already. Increasing the precision to 3 increases the size to 220KB), which is a large improvement over 8MB. We could reduce the memory consumption further if we were willing to limit the maximum data point count per bucket and use an integer/short histogram(ints seem like a reasonable choice).</div>
<div>
If we print both measurements methods for the same run we can see the difference between the raw data and the HDR representation which is naturally slightly less accurate (# lines are <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a>,@ is for raw data, each line represents 1M data points):</div>
<div>
<script src="https://gist.github.com/nitsanw/73755b6ff4a3c625452f.js"></script></div>
<div>
We can see that reported percentiles are pretty close to the raw data:<br />
<br />
<ul>
<li>Note that the nanoTime on Mac reports in µs granularity, which is why the real values(@ lines) all end with 3 zeros.</li>
<li>Note that the max/min reported are adjusted to the correct histogram resolution (not a big deal, but slightly surprising).</li>
</ul>
<br />
What would have happened with our hand rolled solution? To keep a range of 0 to 60,000,000,000ns in a linear histogram of the same memory footprint we would need to limit ourselves to roughly 220k/8=64k buckets. Each bucket would have a granularity of roughly 1ms which would have translated to a very limited visibility on the lower end of the spectrum as most data sets are actually all under 1ms. This would have also completely skewed our percentiles (i.e 99.99% results in 1ms bucket, no breakdown of behaviour in percentiles). We could try and tackle the issue by picking a lower range to cover (which if applied to <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> will minimize memory usage further) or by blowing the memory budget on finer grained buckets.<br />
<br />
<h3>
Example: Aeron samples, much percentiles! Such graphs! Wow!</h3>
</div>
<div>
<a href="http://en.wikipedia.org/wiki/Percentile" target="_blank">Percentiles</a> are commonly used to describe latency <a href="http://en.wikipedia.org/wiki/Service-level_agreement" target="_blank">SLA</a> of a given system and a typical application will report a range of percentiles to reflect the probability of a given response time. In this context we can say that a 99%ile latency of 1ms means that 99% of all requests were handled in =< 1ms.<br />
<a href="https://github.com/real-logic/Aeron" target="_blank">Aeron</a> is a low latency, reliable UDP messaging library. Since latency is an obvious concern, all the samples utilize the <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> lib to demonstrate measurement and report results, here are the relevant excerpts from the <a href="https://github.com/real-logic/Aeron/blob/master/aeron-samples/src/main/java/uk/co/real_logic/aeron/samples/Ping.java" target="_blank">Ping sample</a>:<br />
<script src="https://gist.github.com/nitsanw/72ec51b89e0939d4faa9.js"></script>
This results in a pile of text getting poured into the console, not that friendly:
<script src="https://gist.github.com/nitsanw/3d6a15cf7a1554e68109.js"></script>
<br />
Fear not, <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> comes packed with a handy charty thingy! Here's what the above histogram looks like when plotted:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieykWJotCR1THMECpCVrEmUleVyb3ilj_16kP4NFs5VN1wDl9-2R8cDiGaxE__nIcTkasj4hx1fKudOotbbdn4C9MeYA9lJVx3ogBj3G_EdbaR1nK_DW3wsMN-2DxJvh1o0kX9h1nm-Vc/s1600/Histogram+(1).png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieykWJotCR1THMECpCVrEmUleVyb3ilj_16kP4NFs5VN1wDl9-2R8cDiGaxE__nIcTkasj4hx1fKudOotbbdn4C9MeYA9lJVx3ogBj3G_EdbaR1nK_DW3wsMN-2DxJvh1o0kX9h1nm-Vc/s1600/Histogram+(1).png" height="244" width="640" /></a></div>
<br />
To get this graph:<br />
<ul>
<li>Save output above to a text file</li>
<li>Open a browser, and go <a href="http://hdrhistogram.github.io/HdrHistogram/plotFiles.html" target="_blank">here</a> (the same HTML is in the project <a href="https://github.com/HdrHistogram/HdrHistogram/blob/master/GoogleChartsExample/plotFiles.html" target="_blank">here</a>)</li>
<li>Choose your file, choose percentiles to report and unit to report in</li>
<li>Export the picture and stick it in your blog!</li>
</ul>
This histogram was provided to me by Martin Thompson (one of the Aeron developers) and is from a test run in a performance lab. We can see that Aeron is delivering a solid 7µs RTT up to the 90%ile where latency starts to gradually grow. In this particular data set the maximum observed latency was 38µs. This is a great latency profile. It is far more common for the max and 99.99%ile to be orders of magnitude more that the 90%ile.<br />
I could similarly plot this histogram using a gnuplot script to be found <a href="https://github.com/HdrHistogram/HdrHistogram/tree/master/gnuplotExample" target="_blank">here</a>. The gnuplot script is very handy for scripted reporting on large runs. It also allows for plotting several histograms on the same graph to allow visual comparison between benchmark runs for instance.<br />
<br />
<h3>
Example: Compressed histogram logging</h3>
Because the SLA is often specified in percentiles, it is common for applications to log only percentiles and not histograms. This leads to a reporting problem as <a href="http://latencytipoftheday.blogspot.com/2014/06/latencytipoftheday-you-cant-average.html" target="_blank">it turns out that percentiles output cannot be combined to produce meaningful average percentiles.</a> The solution would be to log the full histogram data, but who want a log that grows by 31KB every 5 seconds just to capture one histogram? Worry not, <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> comes with a compressed logging format and log writer and all that good stuff:</div>
<div>
<script src="https://gist.github.com/nitsanw/7c657672e126a937b466.js"></script></div>
<div>
How many bytes is that logged histogram costing us? The 31KB histogram compressed down to 1KB in my example, but the result of compression will depend on the histogram captured. It is fair to assume that histograms compress well as the array of buckets is full of zeros (on the byte level) as most buckets are empty or low count.<br />
If 1KB sounds like allot consider that a days worth of 10s interval histograms will result in an 8MB log file, which seems pretty acceptable even if you have a hundred such files. The benefit is that you will now have high precision interval latency data that you can reliably use to create longer interval latency data. You can use the <a href="https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/HistogramLogProcessor.java" target="_blank">HistogramLogProcessor</a> to produce a full or partial log summary histogram for plotting as above.<br />
I believe there's some truly exciting data visualizations one could build on top of this data, but sadly thats not where my skills lie. If you got skills to show of in this area I'm sure <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> would value your contribution.<br />
<br />
<h3>
Example: jHiccup and concurrent logging</h3>
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOtodJcWE-e1lA8AHaTHtqgJT_v8uxVYWeuGvYUYnLCQ8sIyLk7SoM8_8qE0OUmA2GBnHXlDDnOJgWont-gGa73qPNFKw8E-Ku2yma6m4PmUzgfWrHGJxZ43GlUCXtMsb373M2XmbJoUs/s1600/oh-damn-it_o_192211.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOtodJcWE-e1lA8AHaTHtqgJT_v8uxVYWeuGvYUYnLCQ8sIyLk7SoM8_8qE0OUmA2GBnHXlDDnOJgWont-gGa73qPNFKw8E-Ku2yma6m4PmUzgfWrHGJxZ43GlUCXtMsb373M2XmbJoUs/s1600/oh-damn-it_o_192211.jpg" height="155" width="200" /></a></div>
<a href="https://github.com/giltene/jHiccup" target="_blank">jHiccup</a> is a pause measurement tool used to capture OS or JVM 'hiccups'. It deserves it's own post but I'll try and summarize it in a few points:<br />
<ul>
<li>jHiccup runs a HiccupRecorder thread which sleeps for a period of time (configurable) and measures the delta between the wakeup time and actual time. The failure to be re-scheduled is taken as a potential OS/JVM hiccup. The size of the hiccup is recorded in a histogram.</li>
<li>jHiccup can be run as an agent in your own process, an external process, or both.</li>
<li>jHiccup has been ported to C as well.</li>
<li>People typically use jHiccup to help charecterize and diagnose disruptions to execution in their system. While not every hiccup is the result of a STW pause we can use the jHiccup agent evidence correlated with an external jHiccup process and the JVM gc logs to support root cause analysis. A significant hiccup is a serious sign of trouble meaning a thread was denied from scheduling for the length of the hiccup. We can safely assume in most cases that this is a sign that other threads were similarly disturbed.</li>
</ul>
Gil Tene originally wrote <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> as part of jHiccup, but as <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> turned out to be more generally useful the two were split. The reason I bring jHiccup up in this context is that it serves as a regularly maintained full blown real world example of using an <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a>.<br />
jHiccup has 2 interesting threads, with roles that parallel many real world applications out there:<br />
<ul>
<li>The measuring thread(<a href="https://github.com/giltene/jHiccup/blob/master/src/main/java/org/jhiccup/HiccupMeter.java#L408" target="_blank">HiccupRecorder</a>): This is the thread that sleeps and wakes up and so on. The rate at which it does that is potentially quite high and we don't want to skew to measurement by performing IO on this thread. Similarly many real world application will have critical threads where it is not desirable to introduce IO. Since this is the case actual persistence will be performed on another thread.</li>
<li>The monitoring/logging thread(<a href="https://github.com/giltene/jHiccup/blob/master/src/main/java/org/jhiccup/HiccupMeter.java#L133" target="_blank">HiccupMeter</a>): This thread will wake up at regular intervals and write the last interval's histogram to the log file. But since it is reading a histogram while another thread is writing to the histogram we now need to manage concurrency.</li>
</ul>
<a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> offers a synchronization facility to serve exactly this use case in the form of the <a href="https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/Recorder.java" target="_blank">Recorder</a>:<br />
<ul>
<li>Recording a value in the recorder is a wait free operation (on JDK8, can be lock free on older depending on the getAndAdd implementation for AtomicLongArray).</li>
<li>The Recorder also comes in a <a href="https://github.com/HdrHistogram/HdrHistogram/blob/master/src/main/java/org/HdrHistogram/SingleWriterRecorder.java" target="_blank">single-writer flavour</a>, which minimizes the concurrency related overheads.</li>
</ul>
<br />
Under the covers the recorder is using an active histogram and an inactive one, swapped seamlessly when an interval histogram is requested. Using a Recorder looks much like using a normal histogram(full code <a href="https://github.com/giltene/jHiccup/blob/master/src/main/java/org/jhiccup/HiccupMeter.java#L133" target="_blank">here</a>):<br />
<script src="https://gist.github.com/nitsanw/e1b87744b6371c8d357c.js"></script>
And that's concurrent logging sorted ;-).<br />
<br /></div>
<div>
<h3>
Summary</h3>
</div>
<div>
With <a href="https://github.com/HdrHistogram/HdrHistogram" target="_blank">HdrHistogram</a> now hitting version 2.1.4 and offering a wealth of tried and tested functionality along with cross platform implementations and a standardized compressed logging format it is definitely time you gave it a go! May your latencies always be low!<br />
If you are looking for a pet project and have a gift for UI thingies a latency explorer for the histogram interval logs would be an awesome contribution!</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com10tag:blogger.com,1999:blog-5171098727364395242.post-17794598050368091712015-01-19T12:38:00.000+00:002015-02-21T11:17:17.252+00:00MPMC: The Multi Multi Queue vs. CLQ<b><span style="font-size: x-small;">{This post is part of a long running series on lock free queues, <a href="http://psy-lob-saw.blogspot.com/p/lock-free-queues.html" target="_blank">checkout the full index to get more context here</a>}</span></b><br />
<a href="https://github.com/JCTools/JCTools" target="_blank">JCTools</a>, which is my spandex side project for lock-free queues and other animals, contains a lovely gem of a queue called the <a href="https://github.com/JCTools/JCTools/blob/2a30aa3ecd2bb50fc866b78668b633bb7a0d62ab/jctools-core/src/main/java/org/jctools/queues/MpmcArrayQueue.java" target="_blank">MpmcArrayQueue</a>. It is a port of <a href="http://www.1024cores.net/home/lock-free-algorithms/queues/bounded-mpmc-queue" target="_blank">an algorithm</a> put forward by <a href="http://www.1024cores.net/home/about-me" target="_blank">D. Vyukov</a> (the lock free ninja) which I briefly discussed in a <a href="http://psy-lob-saw.blogspot.com/2014/04/notes-on-concurrent-ring-buffer-queue.html" target="_blank">previous post</a> on ring-buffer queues.<br />
The <a href="https://github.com/JCTools/JCTools/blob/master/jctools-core/src/main/java/org/jctools/queues/MpmcArrayQueue.java" target="_blank">implementation</a> is covered to a large extent in that post, but no performance numbers are provided to evaluate this queue vs. the JDK's own MPMC queue the illustrious ConcurentLinkedQueue (CLQ for short). Let's fix that!<br />
<br />
<h3>
Welcome to the Machine</h3>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr1NXx2ItcWPQURiRXncdYntdL5FJtmwmx-8TP-1xisGxfBo23HtLVTnsFnFYeME3DNwuSwdPp9iHeRStlhlxp7ckLvP9RajFkoZmgT2vk27eCGCgr9SD0dCeJdtbB9qmfdmpPRlZJfsw/s1600/Bjork-Quiet_l.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr1NXx2ItcWPQURiRXncdYntdL5FJtmwmx-8TP-1xisGxfBo23HtLVTnsFnFYeME3DNwuSwdPp9iHeRStlhlxp7ckLvP9RajFkoZmgT2vk27eCGCgr9SD0dCeJdtbB9qmfdmpPRlZJfsw/s1600/Bjork-Quiet_l.jpg" height="150" width="200" /></a></div>
Before we start the party, we must establish the test bed for these experiments. The results will only be relevant for similar setups and will become less indicative the more we deviate from this environment. So here it is, the quiet performance testing machine (shitloads of power and nothing else running):</div>
<div>
<ul>
<li>OS: CentOS 6.4</li>
<li>CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz, dual socket, each CPU has 12 cores (24 logical core as HT is on). This is an Ivy Bridge, so decent hardware if not the fanciest/latest.</li>
<li>JVM: Oracle JDK 1.8u25</li>
<li>Load average (idle): 0.00, 0.02, 0.00 -> <a href="https://www.youtube.com/watch?v=TEC4nZ-yga8" target="_blank">it's oh so quiet</a>....</li>
</ul>
<div>
I'll be using <i><span style="font-family: Courier New, Courier, monospace;">taskset</span></i> to pin threads to a given topology of producers/consumers which is important because:<br />
<ul>
<li>Running 2 logical threads on same physical core will share queue in L1 cache.</li>
<li>Running on 2 physical cores will share queue in LLC.</li>
<li>Running on different socket (i.e 2 different CPUs) will force shared data via QPI.</li>
<li>Each topology has it's own profile, there's no point to averaging the lot together.</li>
</ul>
For the sake of these benchmarks I'll be pinning the producer/consumer threads to never share a physical core, but remain on the same CPU socket. In my particular hardware this means I'll use the 12 physical cores on one CPU using <i><span style="font-family: Courier New, Courier, monospace;">taskset -c 0-11.</span></i></div>
</div>
<div>
For the sake of keeping things simple I ran the benchmarks several times and averaged the results, no fancy graphs (still recovering from the new year). Throughput results are quoted in ops per microsecond, if you are more comfortable with ops per second just multiply by 1 million. Latency results are in nanoseconds per operation.<br />
<br /></div>
<h3>
Throughput Benchmarks: Hand rolled 1 to 1</h3>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYARwjddSfkuYpZmIlhJrv8UdHmAU49irkgOg7_dxzRUDw7xyBR43TDEOxH1S_LVi9OtNDyyu_0LYyh1NLBEBoj1EKG2PJvM0yM46YzQNzRm4tQw4rSmq1PtNEDwQ8EmCDOCr6CEAgDk8/s1600/handrolled.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYARwjddSfkuYpZmIlhJrv8UdHmAU49irkgOg7_dxzRUDw7xyBR43TDEOxH1S_LVi9OtNDyyu_0LYyh1NLBEBoj1EKG2PJvM0yM46YzQNzRm4tQw4rSmq1PtNEDwQ8EmCDOCr6CEAgDk8/s1600/handrolled.png" height="125" width="320" /></a></div>
Let's start with the simplest benchmark in JCTools. The <span style="font-family: Courier New, Courier, monospace;"><a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/handrolled/throughput/spsc/QueuePerfTest.java" target="_blank">QueuePerfTest</a></span><span style="font-family: inherit;"> is really a single producer/consumer test:</span></div>
<div>
<ul>
<li><span style="font-family: inherit;">A producer thread is spun up which feeds into the queue as fast as it can.</span></li>
<li><span style="font-family: inherit;">The main thread plays the consumer thread and empties the queue as fast as it can. </span></li>
</ul>
<span style="font-family: inherit;">The original benchmark was a sample used by Martin Thompson in one of his talks and I tweaked it further. It's nice and simple and has the added benefit of verifying that all the elements arrived at their destination. It also opens the door to loop unrolling which you may or may not consider a reasonable use case. This benchmark calls Thread.yield() if an offer/poll call fails which represents a rather forgiving backoff policy, which is why we also have the </span><span style="font-family: Courier New, Courier, monospace;"><a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/handrolled/throughput/spsc/BusyQueuePerfTest.java" target="_blank">BusyQueuePerfTest</a></span><span style="font-family: inherit;"> which busy spins on fail.</span></div>
<div>
<span style="font-family: inherit;">Here's what we get:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Queue Benchmark Throughput(ops/µs)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">CLQ <a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/handrolled/throughput/spsc/QueuePerfTest.java" target="_blank">QueuePerfTest</a> 10.7</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Mpmc </span><a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/handrolled/throughput/spsc/QueuePerfTest.java" style="font-family: 'Courier New', Courier, monospace;" target="_blank">QueuePerfTest</a><span style="font-family: Courier New, Courier, monospace;"> 65.1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">CLQ </span><a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/handrolled/throughput/spsc/BusyQueuePerfTest.java" style="font-family: 'Courier New', Courier, monospace;" target="_blank">BusyQueuePerfTest</a><span style="font-family: Courier New, Courier, monospace;"> 15.3</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">Mpmc </span><span style="font-family: Courier New, Courier, monospace;"> </span><a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/handrolled/throughput/spsc/BusyQueuePerfTest.java" style="font-family: 'Courier New', Courier, monospace;" target="_blank">BusyQueuePerfTest</a><span style="font-family: Courier New, Courier, monospace;"> 63.5</span></div>
<div>
</div>
<div>
The observed throughput for CLQ varies quite a bit between iterations, but the full run average is not too unstable. The performance also depends on the size of the heap as CLQ allocates a node per item passed and the GC overhead can become a significant part of management. In my runs I set the heap size to 1GB and the benchmark also calls System.gc() between measurement iterations. In any case, off to a good start, the JCTools queue is looking well.</div>
<div>
<br /></div>
<h3>
Throughput benchmarks: JMH Joy</h3>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc4uuMkOUcrljklRsVQrs2FoCcPXXcB-oECCz-o3NiXPRtdam7qzxGxEsT-tnuBIbyyUY8PvODUsdsOuogg_9h3tcwD13sddQpZlx-qMLkLyptAGM04g5LcmBNSoGrx5JzWcypDfKg8EY/s1600/lastingjoy2.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjc4uuMkOUcrljklRsVQrs2FoCcPXXcB-oECCz-o3NiXPRtdam7qzxGxEsT-tnuBIbyyUY8PvODUsdsOuogg_9h3tcwD13sddQpZlx-qMLkLyptAGM04g5LcmBNSoGrx5JzWcypDfKg8EY/s1600/lastingjoy2.jpg" height="200" width="200" /></a></div>
I'm a big JMH fan (see the <a href="http://psy-lob-saw.blogspot.com/p/jmh-related-posts.html" target="_blank">JMH reference page</a>), and <a href="https://github.com/JCTools/JCTools/tree/master/jctools-benchmarks/src/main/java/org/jctools/jmh/throughput" target="_blank">these benchmarks</a> show some of the power JMH has. The benchmark code is very simple and it's clear what's being measured:</div>
<div>
<ul>
<li>We have 2 methods, offer and poll</li>
<li>Thread groups hammer each method</li>
<li>We use <span class="s1">@</span>AuxCounters to report successful/failed interactions with the queue</li>
<li>The throughput is the number of pollsMade, i.e. the number of items delivered</li>
</ul>
</div>
<div>
The nice thing is that once I got this setup I can now play with the number of threads in each group with no further investment (see an <a href="http://psy-lob-saw.blogspot.com/2013/05/using-jmh-to-benchmark-multi-threaded.html" target="_blank">introduction to multi-threaded benchmarks with JMH</a>). I can run several iterations/forks and go make some coffee while JMH crunches through the lot and gives me a nice report (JMH options used for these benchmarks: "<span style="font-family: Courier New, Courier, monospace;">-gc -f 5 -i 5 -wi 5 -p qType=MpmcArrayQueue,ConcurrentLinkedQueue -p qCapacity=1000000</span>" the -gc option forces a GC cycle between iterations as before, I use the -tg option to control number of producer/consumer threads). Here's the results for <a href="https://github.com/JCTools/JCTools/blob/master/jctools-benchmarks/src/main/java/org/jctools/jmh/throughput/QueueThroughputBackoffNone.java" style="font-family: 'Courier New', Courier, monospace;" target="_blank">QueueThroughputBackoffNone</a> (again results are in ops/µs, so higher is better):<br />
<div>
<span style="font-family: Courier New, Courier, monospace;">Queue 1P1C </span><span style="font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;">2P2C </span><span style="font-family: 'Courier New', Courier, monospace;">3P3C </span><span style="font-family: 'Courier New', Courier, monospace;">4</span><span style="font-family: 'Courier New', Courier, monospace;">P4C 5P5C</span><span style="font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;"> 6P6C</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">CLQ </span><span style="font-family: 'Courier New', Courier, monospace;"> 7.9 ±1.7 , </span><span style="font-family: Courier New, Courier, monospace;">4.7 ±0.2, 3.8 ±0.1, 3.7 ±0.4, 3.1 ±0.1, 2.9 ±0.1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Mpmc </span><span style="font-family: Courier New, Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;">68.4 ±11.1, </span><span style="font-family: Courier New, Courier, monospace;">9.2 ±1.1, 7.3 ±0.5, 5.8 ±0.4, 5.2 ±0.4, 5.3 ±0.2</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
Note the columns stand for number of producers/consumers so 1P1C will be running with -tg 1,1 and will have one thread hitting the offer and one the poll. 2P2C will have 2 consumers and 2 producers etc.<br />
So on the one hand, joy, MPMC still consistently ahead, on the other hand it is not some sort of magic cure for contention. If you have multiple threads hitting a shared resource you will suffer for it. The initial hit is the worst, but the following degraded performance curve isn't too bad and we seem to stay ahead.<br />
<br />
<h3>
Average Round Trip Time Benchmarks</h3>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyUhYheTt73VPtBhYq-4Zi6eE-ofAij-j1GrM3Pq13tT3XPzPI-sD2kxy9JNVouxgbcDQZOii63ivjjlWWKQ3rjgUk4STFAkQcUXIbb0ugMgKrEpo0aP-BEz7wz5sZ0XzQoBAxQXlsnHs/s1600/ping-pong.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyUhYheTt73VPtBhYq-4Zi6eE-ofAij-j1GrM3Pq13tT3XPzPI-sD2kxy9JNVouxgbcDQZOii63ivjjlWWKQ3rjgUk4STFAkQcUXIbb0ugMgKrEpo0aP-BEz7wz5sZ0XzQoBAxQXlsnHs/s1600/ping-pong.jpg" height="133" width="200" /></a>This benchmark was set up to model a bursty load on queue implementations where we measure the time it takes for a burst of messages to travel from an originating thread, to a chain of threads inter-linked by queues, back to the same thread. This benchmark is focusing on near empty queues and the notification/wakeup time from when an item is placed in a queue until it becomes visible to another thread. It also highlights any batching optimisations impact as bursts grow larger.<br />
The benchmark has been discussed in some detail in 2 previous posts(<a href="http://psy-lob-saw.blogspot.com/2013/12/jaq-spsc-latency-benchmarks1.html" target="_blank">1</a>, <a href="http://psy-lob-saw.blogspot.com/2014/01/jaq-spsc-latency-benchmarks2.html" target="_blank">2</a>)<br />
We have 2 axis we can explore here, burst size and ring size, I have not explore all the possibilities. The results seem quite straightforward and I'm short on time. I tested with burst size=1,10,100 in the default configuration, i.e. chain length 2 (so a ping-pong kind of setup), and just for the heck of it I ran the burst size=100 with a chain length of 8 (so 1->2->3...->7->8->1). There you go (results are now in nanoseconds per operation, lower is better):<br />
<div>
<span style="font-family: Courier New, Courier, monospace;">Queue b=1,c=2 b=10,c=2 b=100,c=2 b=100,c=8</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">CLQ 488 ±11, 2230 ±98, 15755 ±601, 24886 ±2565</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">Mpmc </span><span style="font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;">422 ±7, </span><span style="font-family: 'Courier New', Courier, monospace;">1718 ±51, 5144 ±287, </span><span style="font-family: 'Courier New', Courier, monospace;">8714 ± 200</span></div>
<br />
Note that the headers shorthand stands for burst size(b) and chain length(c) so b=1,c=2 which is the default config for the benchmark stands for burst size of 1 (so send 1 message and wait until 1 message is received back) and chain length of 2 (so 2 threads linked up in this ring: 1->2->1).<br />
The difference on single element exchange is not that great, but as burst size grows the gap widens significantly. This is perhaps down to the fact MPMC enjoys a much denser delivery of data, minimising the cache misses experienced as part of the message exchange. Note that extending the length of the chain seems to add the same percentage of overhead for each implementation, resulting in the same ratio for chain length 2 as we did for chain length 8 (MPMC is 3 times 'cheaper' than CLQ)<br />
<br />
<h3>
Summary</h3>
</div>
<div>
This was all rather dry, but I hope it helps people place the MPMC alternative in context. I would suggest you consider using MpmcArrayQueue in your application as a replacement to CLQ if:</div>
<div>
<ul>
<li>You need a bounded queue</li>
<li>You are concerned about latency</li>
<li>You don't need to use the full scope of methods offered by Queue and can make do with the limited set supported in JCTools queues</li>
</ul>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com4tag:blogger.com,1999:blog-5171098727364395242.post-66424401711191845342014-12-01T15:38:00.000+00:002014-12-03T16:05:19.806+00:00The Escape of ArrayList.iterator() <div class="separator" style="clear: both; text-align: center;">
<a href="http://www.gocomics.com/calvinandhobbes/1986/04/13/" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1SIFO70OTcB5i23XryRCfKhGWvNqtwUZ-P-DhvU7YMJSKli5RrtOYBRPwOjU1VcYKnSCzZZst3tz0IoAzLWH5jCBdEaVm-Fs1cdGxsEYRKYZdyiHAiBXnIiafD35axeOY7r9dKIqRGjM/s1600/Screen+Shot+2014-12-01+at+17.28.29.png" height="203" width="320" /></a></div>
<b>{This post assumes some familiarity with JMH. For more JMH related content start at the new and improved <a href="http://psy-lob-saw.blogspot.com/p/jmh-related-posts.html" target="_blank">JMH Resources Page</a> and branch out from there!}</b><br />
<a href="http://en.wikipedia.org/wiki/Escape_analysis" target="_blank">Escape Analysis</a> was a much celebrated optimisation added to the JVM in Java 6u23:<br />
<blockquote class="tr_bq" style="font-family: Arial, Helvetica, FreeSans, Luxi-sans, 'Nimbus Sans L', sans-serif; font-size: 12px; margin-bottom: 17px; margin-top: 3px;">
"Based on escape analysis, an object's escape state might be one of the following:<br />
<ul style="font-family: Arial, Helvetica, FreeSans, Luxi-sans, 'Nimbus Sans L', sans-serif; font-size: 12px; margin-left: 13px; padding-left: 0px;">
<li style="list-style-image: url(https://docs.oracle.com/javase/webdesign/pubs/im/ul_bullet.gif); margin-left: 13px; padding-left: 0px;"><code style="color: #444444; font-family: 'Courier New', Monaco, Courier, monospace;">GlobalEscape</code> – An object escapes the method and thread. For example, an object stored in a static field, or, stored in a field of an escaped object, or, returned as the result of the current method.</li>
</ul>
<ul style="font-family: Arial, Helvetica, FreeSans, Luxi-sans, 'Nimbus Sans L', sans-serif; font-size: 12px; margin-left: 13px; padding-left: 0px;">
<li style="list-style-image: url(https://docs.oracle.com/javase/webdesign/pubs/im/ul_bullet.gif); margin-left: 13px; padding-left: 0px;"><code style="color: #444444; font-family: 'Courier New', Monaco, Courier, monospace;">ArgEscape</code> – An object passed as an argument or referenced by an argument but does not globally escape during a call. This state is determined by analyzing the bytecode of called method.</li>
</ul>
<ul style="font-family: Arial, Helvetica, FreeSans, Luxi-sans, 'Nimbus Sans L', sans-serif; font-size: 12px; margin-left: 13px; padding-left: 0px;">
<li style="list-style-image: url(https://docs.oracle.com/javase/webdesign/pubs/im/ul_bullet.gif); margin-left: 13px; padding-left: 0px;"><code style="color: #444444; font-family: 'Courier New', Monaco, Courier, monospace;">NoEscape</code> – A scalar replaceable object, meaning its allocation could be removed from generated code.</li>
</ul>
After escape analysis, the server compiler eliminates scalar replaceable object allocations and associated locks from generated code. The server compiler also eliminates locks for all non-globally escaping objects. It does <em>not</em> replace a heap allocation with a stack allocation for non-globally escaping objects." - from <a href="https://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html#escapeAnalysis" target="_blank">Java 7 performance enhancements</a> </blockquote>
Alas, proving an object never escapes is a difficult problem, and many people feel they cannot rely on this optimisation to kick in and "do the right thing" for them. Part of the problem is that there is no easy way to discover if a particular allocation has been eliminated (on a debug OpenJDK build one can use <span style="background-color: white; color: #444444; font-size: 13px; line-height: 17.2800006866455px;"><span style="font-family: Courier New, Courier, monospace;">-XX:+UnlockDiagnosticVMOptions -XX:+PrintEscapeAnalysis -XX:+PrintEliminateAllocations</span></span>).<br />
The EscapeAnalysis skepticism leads some people to go as far as claim that the JIT compiler fails to eliminate the iterator allocation of collections, which are everywhere:<br />
<script src="https://gist.github.com/nitsanw/b1fca1553f095ff18ab3.js"></script><br />
Buy why skepticize what you can analyze?<br />
<br />
<h3>
Theory: even simple iterators do not benefit from Escape Analysis</h3>
<div>
So lets give this some thought. I generally think the best of the JVM developer bunch. Sure they may miss here and there, they're just human after all, but we are talking about a good strong bunch of engineers. I tend to assume that if there is a simple case for an optimization that is worth while than they have not failed to capitalize on it. I would therefore be quite surprised if indeed iterators do not benefit from escape analysis as they seem quite natural candidates, particularly in the syntactic sugar case, but even in the explicit case. Still, my trust in the JVM engineers is no evidence, how can I prove this works for a given setup?</div>
<div>
<ol>
<li>Use a debug build... I invite the readers to try this method out, didna do it.</li>
<li>Use a memory profiler, like the one packed with VisualVM or YourKit</li>
<li>Setup an experiment to observe before/after effect of desired optimization. Use -XX:-+DoEscapeAnalysis and examine gc logs to observe effect.</li>
<li>Look at the assembly...</li>
</ol>
<h3>
Experiment: Observe a loop over an array list</h3>
</div>
<div>
Reach for your tool belts and pick the swiss army knife that is <a href="http://psy-lob-saw.blogspot.com/p/jmh-related-posts.html" target="_blank">JMH</a>. Here's the benchmark I will use for this:</div>
<div>
<script src="https://gist.github.com/nitsanw/c90ce9fd713c12572258.js"></script></div>
<div>
This is one of them times when I'm not particularly concerned with the performance of this bit of code as such, but JMH makes a good crucible for java code. The same effort that went into correct measurement enables the examination of code in an isolated manner. This makes JMH a good tool for testing the JIT compiler.<br />
<br /></div>
<h3>
</h3>
<h3>
Measurement: Profile the experiment</h3>
<div>
I want to plug the experiment into a profiler, so I set the number of iterations to 1000 and get into it man, profiling, doing it you know, like a... like a sex machine, can I count it all? Here's <a href="http://www.yourkit.com/java/profiler/" target="_blank">YourKit</a> reporting:</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi21PQL4VaRoLi7rbl39bdRFr12GjL91VC0cCNUFPF5LvEaXHOk0oyv-pxwj4zATErqrTWzVVYl8NHzDBxh40YvPnRLmUilT23r6C4kcZnEPZlyD_o_gWtH6Lv_RHjE3oV6vBeXwFvV2RY/s1600/Screen+Shot+2014-11-28+at+13.11.53.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi21PQL4VaRoLi7rbl39bdRFr12GjL91VC0cCNUFPF5LvEaXHOk0oyv-pxwj4zATErqrTWzVVYl8NHzDBxh40YvPnRLmUilT23r6C4kcZnEPZlyD_o_gWtH6Lv_RHjE3oV6vBeXwFvV2RY/s1600/Screen+Shot+2014-11-28+at+13.11.53.png" height="83" width="640" /></a></div>
<br /></div>
<div>
Ouch! that array list iterator is right there at the top! all that trust I put in the JVM developers? GONE! Let's get a second opinion from <a href="http://visualvm.java.net/" target="_blank">JVisualVM</a>, just to be sure:</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5HRtBMFayIuYNbQX-yow46U6rLsqeMC704qL2bjyqQbAiTswpiOzsrCUO05OgNS7G0n92nJ5zFqNRT7l_KObagJQmd5orc7N4FujJK8nAPXEiHJNwYyBW3pDcKzxOoCb_6v_omFxh9WM/s1600/Screen+Shot+2014-11-28+at+13.18.38.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5HRtBMFayIuYNbQX-yow46U6rLsqeMC704qL2bjyqQbAiTswpiOzsrCUO05OgNS7G0n92nJ5zFqNRT7l_KObagJQmd5orc7N4FujJK8nAPXEiHJNwYyBW3pDcKzxOoCb_6v_omFxh9WM/s1600/Screen+Shot+2014-11-28+at+13.18.38.png" height="113" width="640" /></a></div>
<br /></div>
<div>
Oh me, oh my, this is bad...<br />
Finally, with tears in our eyes, let us try see what <a href="http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html" target="_blank">Java Mission Control</a> has to say:</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkjqFoHH2e81Xc5imQOWwStdpOx2ABScOb_Vnisny9gtrrZZIp49823ICW3xh9MjahEPKMCO_xtltVmDdh4imACOwFuC72elHYkDpb0Iyzk0AmDv-ovdMNG_Rw4d-zvZ_HLUvh9c2vyUw/s1600/Screen+Shot+2014-11-28+at+13.23.39.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkjqFoHH2e81Xc5imQOWwStdpOx2ABScOb_Vnisny9gtrrZZIp49823ICW3xh9MjahEPKMCO_xtltVmDdh4imACOwFuC72elHYkDpb0Iyzk0AmDv-ovdMNG_Rw4d-zvZ_HLUvh9c2vyUw/s1600/Screen+Shot+2014-11-28+at+13.23.39.png" height="114" width="640" /></a></div>
<br /></div>
<div>
How curious! the iterator allocation is gone! but JMC is outnumbered 2 to 1 here. Could it be that some profilers are pooping their pants?</div>
<div>
<br /></div>
<h3>
Measurement: Measure with -XX:+/-DoEscapeAnalysis</h3>
<div>
Sometimes we get lucky and the optimization we want to examine comes with a handy flag to turn it on and off. We expect escape analysis to remove the iterator allocation and thus leave us with a benchmark which generates no garbage. We are also fortunate that JMH comes with the handy GC profiler which simply examines the GC JMX bean to inform us if any collections happened. Running the benchmark with a short list size and a small heap should trigger plenty of young generation GC cycle in each measurement iteration. Lo and behold, with escape analysis on:</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$java -jar target/microbenchmarks.jar -f 1 -i 10 -wi 10 -p size=1000 \</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> -jvmArgs="-Xms64m -Xmx64m -XX:+DoEscapeAnalysis" -prof gc ".*.sumIteratorOverList"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Benchmark (size) Score Error Units</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList 1000 816.367 ± 17.768 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.count.profiled 1000 0.000 ± NaN counts</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.count.total 1000 0.000 ± NaN counts</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.time.profiled 1000 0.000 ± NaN ms</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.time.total 1000 0.000 ± NaN ms</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$java -jar target/microbenchmarks.jar -f 1 -i 10 -wi 10 -p size=1000 \</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> -jvmArgs="-Xms64m -Xmx64m -XX:-DoEscapeAnalysis" -prof gc ".*.sumIteratorOverList"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Benchmark (size) Score Error Units</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList 1000 940.567 ± 94.156 ns/op</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.count.profiled 1000 19.000 ± NaN counts</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.count.total 1000 42.000 ± NaN counts</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.time.profiled 1000 9.000 ± NaN ms</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sumIteratorOverList:@gc.time.total 1000 27.000 ± NaN ms</span></div>
<div>
Now that is what we hoped for, escape analysis saves the day! Please note above numbers are from running on my laptop using Oracle Java 8u20, I was not aiming for measurement accuracy, just wanted to verify the compilation/GC aspects. Laptops are good enough for that.</div>
<div>
<br /></div>
<h3>
WTF Profilers? Why you lie?</h3>
<div>
There's a big difference in how JVisualVM/YourKit work and how JMC work, in particular:</div>
<div>
<ul>
<li>JVisualVM/YourKit: treat the JVM as a black box and profile via the JVMTI interfaces and bytecode injection. Work on all JVMs.</li>
<li>Java Mission Control: use internal JVM counters/reporting APIs only available to the Oracle JVM (so can't profile OpenJDK/Zing/IBM-J9 etc)</li>
</ul>
So why should this get in the way of escape analysis? Searching through the OpenJDK source code we can spot the following:<br />
<blockquote class="tr_bq">
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s1">void</span> C2Compiler::compile_method(<span class="s2">ciEnv</span>* env, <span class="s2">ciMethod</span>* target, <span class="s1">int</span> entry_bci) {</span></div>
<div class="p2">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="s3"> assert(is_initialized(), </span>"Compiler thread must be initialized"<span class="s3">);</span></span></div>
<div class="p3">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <span class="s1">bool</span> subsume_loads = SubsumeLoads;</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <span class="s1">bool</span> do_escape_analysis = DoEscapeAnalysis &&!env->jvmti_can_access_local_variables();</span></div>
</blockquote>
This explains a similar issue, but not this one. The above code is relevant for debuggers and other tools relying on stepping through code and examining variable values.<br />
This issue is the result of some instrumentation trapping the reference, at least for JVisualVM. We can look at the assembly to be certain, this is the iterator non-allocation before we connect JVisualVM to the process:
<script src="https://gist.github.com/nitsanw/dabd9c4d6d02a17c7f4f.js"></script><br />
Note how the iterator is initialized with no allocation actually taking place (note also how line 4 annotation is a big fat lie, this is actually getting <i>modCount</i> from the list). Method is recompiled to same when we attach the agent. And here's what happens when we turn on memory profiling:
<script src="https://gist.github.com/nitsanw/02830114e93f1cf492d7.js"></script>
The iterator is now allocated (see 'new' keyword at line 18 above).We can see that the instrumenting agent added a <i>traceObjAlloc </i>call into the iterator constructor (line 52 above). JVisualVM is open source, so we can dig into the <a href="http://visualvm.sourcearchive.com/documentation/1.1.1/ProfilerRuntimeObjLiveness_8java-source.html" target="_blank">code of the above trace method</a> and the instrumentation code, I leave it as an exercise to the reader. If I was a better man I might have a go at fixing it, maybe another day.<br />
<br />
<h3>
Summary</h3>
<div>
<ul>
<li>EscapeAnalysis works, at least for some trivial cases. It is not as powerful as we'd like it, and code that is not hot enough will not enjoy it, but for hot code it will happen. I'd be happier if the flags for tracking when it happens were not debug only.</li>
<li>Profilers can kill EscapeAnalysis. In the example above we have very little code to look at, so it is easy to cross examine. In the real world you'd be profiling a much larger codebase, looking for allocation elimination opportunities, I suggest you have a healthy doubt in your profiler.</li>
<li>Java Mission Control is my current favourite profiler for the Oracle JVM. It's a shame the underlying APIs are not made public and standard so that tool makers can rely on them for all JVMs. Perhaps one day at least part of these APIs will be made public.</li>
</ul>
Thanks <a href="http://vanillajava.blogspot.com/" target="_blank">Peter Lawrey</a> for reviewing this post!<br />
<ul>
</ul>
</div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com12tag:blogger.com,1999:blog-5171098727364395242.post-1923954146750523222014-11-03T11:16:00.000+00:002016-02-15T07:09:18.030+00:00The Mythical Modulo Mask<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB-_FWpNiVJLZvZzNTd0S1DLTlExpe_fV6wMXAkVaBoVOg29Ab9Ss_91qP0I8R0IYnmtLH_59fCav-2VmkWxc1v6-lXHDrFUqWI4Lt7ZH9114n0GjP7c2RImKYCdNwSBGsK1NzNFfAFog/s1600/jabberwocky.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB-_FWpNiVJLZvZzNTd0S1DLTlExpe_fV6wMXAkVaBoVOg29Ab9Ss_91qP0I8R0IYnmtLH_59fCav-2VmkWxc1v6-lXHDrFUqWI4Lt7ZH9114n0GjP7c2RImKYCdNwSBGsK1NzNFfAFog/s1600/jabberwocky.jpg" width="134" /></a>It is an optimisation well known to those who know it well that % of power of 2 numbers can be replaced by a much cheaper AND operator to the same power of 2 - 1. E.g:<br />
x % 8 == x & (8 - 1)<br />
<span style="background-color: #f4cccc; font-size: x-small;">[4/11/2014 NOTE] This works because the binary representation for N which is a power of 2 will have a single bit set to 1 and (N-1) will have all the bits below that set to 1 (e.g 8 = 00001000, 8-1= 00000111). When we do x AND (N-1) only the remainder of x / N will match the N-1 mask.</span><br />
<span style="background-color: #f4cccc; font-size: x-small;"><u>[4/11/2014 NOTE + slight spoiler: this only works when x >= 0]</u></span><br />
<span style="background-color: #f4cccc; font-size: x-small;"><u>[15/02/2016 NOTE: It has been pointed out that % is not <b>modulo</b>, but <b>remainder</b>. Why everyone calls it modulo I'm not sure]</u></span><br />
The reason the & is so much cheaper is because while % is implemented using the DIV instruction, & is just AND and as it turns out DIV is expensive and AND is cheap on x86 CPUs (and other places too I think). The optimisation is used in the Disruptor as well as the JCTools circular array queues and in the ArrayDequeue and other JDK classes. Is it time to replace % with & everywhere in your code which has this opportunity?<br />
<span style="background-color: #f4cccc;"><span style="font-size: x-small;">[4/11/2014 NOTE] Technical term for this sort of optimization is </span><a href="http://en.wikipedia.org/wiki/Strength_reduction" style="font-size: small;" target="_blank">Strength Reduction</a><span style="font-size: x-small;">. </span></span><br />
<br />
<h3>
<span class="s1"><b>Starting Simple</b></span></h3>
Lets start with some basic benchmarks:<br />
<script src="https://gist.github.com/nitsanw/c88ffaa24f5235bc6455.js"></script>
And the results (on JDK8u5/E5-2697 v2 @ 2.70GHz/-XX:-UseCompressedOops for consistency between assembly and results):<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>Benchmark Score error Units</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> moduloLengthNoMask 3.448 ± 0.007 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> moduloConstantLengthNoMask 1.150 ± 0.002 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> moduloLengthMask 1.030 ± 0.006 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> moduloMask 0.862 ± 0.001 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> consume 0.719</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">±</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">0.001 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> noop 0.287</span><span style="font-family: "courier new" , "courier" , monospace;"> ± </span><span style="font-family: "courier new" , "courier" , monospace;">0.000 ns/op</span><br />
<br />
So pretty much as per expectation the modulo operation is far more expensive than the mask:<br />
<ul>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRT9SLxMoUiHE1g8yv0uVJJeVrYl7t0dgc1Iuv9tLwkqQlRsYf9l6SJsfm8xMV69ZyU6O2SPFyUwJavTLMy7Lft72H6dR7lWfRyXqmX5lQhpLCJZSBNbs-x1fO_JNjg7a07YbrlepsdaM/s1600/George_Peppard.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRT9SLxMoUiHE1g8yv0uVJJeVrYl7t0dgc1Iuv9tLwkqQlRsYf9l6SJsfm8xMV69ZyU6O2SPFyUwJavTLMy7Lft72H6dR7lWfRyXqmX5lQhpLCJZSBNbs-x1fO_JNjg7a07YbrlepsdaM/s1600/George_Peppard.jpg" width="320" /></a>
<li>The clever JIT is aware of the optimisation opportunity and will replace a constant % with the &. It is not a perfect replacement, but pretty close.</li>
<li>At this sort of low digit ns benchmark we can’t make a statement such as “modulo is 4 times more expensive” because the same machine produces a baseline of <span style="font-family: "courier new" , "courier" , monospace;">0.287ns/op</span> for the noop benchmark and <span style="font-family: "courier new" , "courier" , monospace;">0.719ns/op</span> for the consume benchmark. If we deduct the consume result from the other scores we see a 1 : 25 ratio between the costs. Is that a good way to model performance? not really either, performance is not additive so simply subtracting one cost from the other doesn't really work at this scale. The truth is somewhere fuzzy in between and if we really care we should look at the assembly.</li>
<li>It seems that using a pre-calculated mask field is more awesome than using the "array length - 1" as a mask. That is consistent with the expectation that the re-calculation of the mask on the fly, as well as loading the value to be used for that calculation, is more expensive than using the pre-calculated field.</li>
</ul>
<div>
<a href="http://en.wikipedia.org/wiki/John_%22Hannibal%22_Smith" target="_blank">I love it when a plan comes together...</a><br />
<br />
<h3>
Going Deeper</h3>
The reason we wanted the modulo in the first place was to read from the array, right? so let’s try that:
<script src="https://gist.github.com/nitsanw/1e7686741724f48e261f.js"></script>
<br />
<div class="p1">
<span class="s1">And the results:</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"> <b>Benchmark Score error Units</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readByLengthNoMask 3.736 ± 0.005 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readByConstantLengthNoMask 1.437 ± 0.001 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readByMask 1.347 ± 0.022 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readByLengthMask 1.181 ± 0.049 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readNoMask 1.175 ± 0.004 ns/op</span><br />
Well, what’s this I see? "length-1" mask is leading the chart! How’d that happen?</div>
<div class="p1">
<span class="s1">To quote from the famous “<a href="http://www.amazon.com/Jack-Flumflum-Tree-Julia-Donaldson/dp/0330504061" target="_blank">Jack and the FlumFlum Tree</a>”:</span></div>
<div class="p1">
<blockquote class="tr_bq">
<span class="s1">“Don’t get your knickers in a twist!” said Jack,<br />“Let’s have a look in the patchwork sack.”</span></blockquote>
</div>
<div class="p1">
<span class="s1">Lets start with the generated assembly for the constant modulo:</span></div>
<div class="p1">
<span class="s1"><script src="https://gist.github.com/nitsanw/aa1775cb5d2b20d9df0d.js"></script></span></div>
I didna see that one coming! the modulo on a constant is not your garden variety & mask affair since it turns out our original assertion about the mask/modulo equality is only true for positive numbers. The JIT in it’s wisdom is dealing with the negative case by doing (x = -x; x = x&15; x = -x;).<br />
I think the above case could be made a tiny bit faster by switching the branch around (so jump for negative value). It’s easy however to see what happens if we simplify the constant version further by using a constant mask:
<script src="https://gist.github.com/nitsanw/18a81ebf3cf336d5eba3.js"></script><br />
And results:<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>Benchmark Score error Units</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> moduloConstantLengthNoMask 1.150 ± 0.002 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <u>moduloConstantLengthMask 0.860 ± 0.001 ns/op</u></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readByConstantLengthNoMask 1.437 ± 0.001 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <u>readByConstantLengthMask 1.209 ± 0.017 ns/op</u></span><br />
So minor joy on the modulo, and reading is better than plain mask, nearly as good as the "length-1" mask. Oh well, let's move on.<br />
The big surprise was the mask calculated on the fly from the array length version. How can calculating the mask on the fly, which seemed to be slower, end up being faster when reading from the array? Who feels like more assembly?
<script src="https://gist.github.com/nitsanw/3d25510ba6aeb72d02dc.js"></script><br />
I was hoping the JVM was clever enough to remove the array bound checks, but that didn’t happen. What’s happening here is that the length load serves the purpose of both creating the mask and checking the bounds. This is not the case for the mask version where we load the mask for the index calculation and the length for the bounds check, thus paying for 2 loads instead of one:
<script src="https://gist.github.com/nitsanw/8191aeea81b94fe39730.js"></script><br />
So removing the computation did not make a difference because the bound check requires the extra load of the length anyhow, can we make the bounds check go away? Of course we can, but it’s Unsafe!!! Let’s do it anyways!
<script src="https://gist.github.com/nitsanw/04affeafca439526a400.js"></script><br />
<div>
The assembly:</div>
<script src="https://gist.github.com/nitsanw/58f201632eb943bbb3ec.js"></script>
<br />
<div>
Shazzam! no bounds check, but look at all the work that’s gone into the unsafe read of the array. It would have been so much better if the unsafe read enjoyed the same addressing mode as normal array reads like so “<span style="font-family: "courier new" , "courier" , monospace;">r8d,DWORD PTR [r9+r10*4+0x18]</span>”, but it seems the JIT compiler is not recognising the opportunity here. What’s the performance like?<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>Benchmark Score error Units</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readByMask 1.347 ± 0.022 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readByLengthMask 1.181 ± 0.049 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readNoMask 1.175 ± 0.004 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <u>unsafeReadByMask 1.152 ± 0.001 ns/op</u></span>
</div>
<div>
<br />
This is even better than no mask at all. Yay?</div>
<div>
Well… sort of. If you mean to have the fastest ‘get’ from the array that allows for an array size which is not an application constant, than this is a mini-win. In particular is saves you a load of the array length in this case and loads can cost anything really. In the case where index and mask are long we can get better code generated:</div>
</div>
<div>
<script src="https://gist.github.com/nitsanw/7333153e9d867ffa5719.js"></script></div>
<div>
<div>
But performance is much the same for this case. Seems like there’s not much left to win in this case.</div>
<div>
For completeness sake we can compare the no mask result with an Unsafe equivalent:</div>
<span style="font-family: "courier new" , "courier" , monospace;"> <b>Benchmark Score error Units</b></span><br />
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;"> </span></span><u><span style="font-family: "courier new" , "courier" , monospace;">unsafeReadByNoMask 1.038 ± 0.022</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;"> ns/op</span></u></div>
<div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace;"> </span></span><span style="font-family: "courier new" , "courier" , monospace;">readNoMask 1.175 ± 0.004 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: inherit;">So it seems slipping past the array boundary check is worth something, but is it generally worth it? what if we weren't dealing with just the one element?</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: inherit;"><br /></span></span></div>
<div class="p1">
<h3>
<span class="s1"><span style="font-family: inherit;">Bound Check Elimination</span></span></h3>
</div>
<div class="p1">
<span class="s1">Looking at the above optimisation we need to accept that it is probably only worth it if array bound checks happen on every access. If we now compare a sum over an array:</span></div>
<script src="https://gist.github.com/nitsanw/700185415420c034437e.js"></script>
<br />
<div class="p1">
<span class="s1"><span style="font-family: inherit;">We get the following results (length=100):</span></span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace;"> <b>Benchmark Score error Units</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> loopOverArrayStraight 26.855</span><span style="font-family: "courier new" , "courier" , monospace;"> ± </span><span style="font-family: "courier new" , "courier" , monospace;">0.060 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> loopOverArrayUnsafeInt 41.413</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">±</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">0.056 ns/op</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> loopOverArrayUnsafeLong 76.257</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">±</span><span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;">0.171 ns/op</span><br />
<span style="font-family: inherit;">Oh Unsafe, why you so sucky sucky? </span>How come the unsafe versions suck so significantly? isn’t Unsafe the cure to all performance problems?<br />
Once the bounds check is eliminated by the JIT we can see that for the UnsafeInt we have the same issue with addressing conversion, only now the cost is not compensated for by the bound check removal. The UnsafeLong version is even worse, how come?<br />
The generated loop for the int case is long and boring because it’s unrolled, the long case is pretty small:
<script src="https://gist.github.com/nitsanw/d89d938e68b4b06c0d41.js"></script><br />
2 'bad' things just happened:
<br />
<ol>
<li>Addressing didn’t workout the way we’d like. Instead of the desired “mov r11d,DWORD PTR [r9+rdi*4+0x18]” we get a two stage setup where we do:”lea r10,[r9+rdi*4]” and then “add r11d,DWORD PTR [r10+0x18]”. Bummer.</li>
<li>We got a safe point poll in the loop. This is happening because long indexed loops are considered potentially very long (as opposed to shorter int loops... heuristics for time to safe point) and so include a safe point poll.</li>
</ol>
So we want to fix the addressing mode and stick to having an int index. If we were to insist on using Unsafe (perhaps because we are trying to do this with off heap memory instead of an array) we’d have to do this:
<script src="https://gist.github.com/nitsanw/48a2a2e8d542b31ada63.js"></script><br />
<span style="background-color: #f4cccc; font-size: x-small;">[4/11/2014 NOTE] Note that what we really want here is more than just getting rid of the multiplication/widening, we want the JIT to identify the expression calculated for offset as relative array access and pick the correct addressing mode for MOV to use. There are clever people out there trying to make sure this will work better in the future.</span><br />
This removes the need for a safe point poll and simplifies addressing to the point where we nearly match the iteration over the array case (length=100):<br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> <b>Benchmark Score error Units</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> loopOverArrayStraight 26.855</span><span style="font-family: "courier new" , "courier" , monospace;"> ± </span><span style="font-family: "courier new" , "courier" , monospace;">0.060 ns/op</span></div>
<div>
<div style="font-family: 'Courier New', Courier, monospace;">
loopOverArrayUnsafePointer 27.377 ± 0.049 ns/op</div>
<div>
<div class="p1">
<span class="s1"><span style="font-family: inherit;">We can explore the relationship between the implementations by testing for different array sizes:</span></span></div>
<div class="p1" style="font-family: 'Courier New', Courier, monospace;">
<span class="s1"><b> 10 100 1000 10000</b></span></div>
<div class="p1" style="font-family: 'Courier New', Courier, monospace;">
<span class="s1"><b>straight</b> 4.3 26.8 289.2 2883.7</span></div>
<div class="p1" style="font-family: 'Courier New', Courier, monospace;">
<span class="s1"><b>unsafeP</b> 4.8 27.3 296.1 2886.4</span></div>
</div>
</div>
<br />
<div class="p1">
<span class="s1">So it seems that the smaller the array the more relative advantage the array iteration has when iterating in this fashion. This should not really be surprising, there's nothing here to confuse the JIT compiler and iterating over arrays is important enough to optimize. We have to work hard to get close to the JIT compiler when it does what it does best.</span><br />
<span class="s1"><br /></span>
<br />
<h3>
Summary</h3>
We had a simple optimisation in mind, replace a % with &:<br />
<ul>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghVW3Fs49B8GQ0op8JmbDu2VEP_QA0U532KhEvVC5IOgw7y6Ao0aEC6fskvrEJwDV_1riD7FSMm0oxwk6kFTEGkdOD0Zv5I3eRokwXEMqbDfi2ctavotZDqGd003TdKoEfclDY7xq0PJg/s1600/o_rlmente.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="199" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghVW3Fs49B8GQ0op8JmbDu2VEP_QA0U532KhEvVC5IOgw7y6Ao0aEC6fskvrEJwDV_1riD7FSMm0oxwk6kFTEGkdOD0Zv5I3eRokwXEMqbDfi2ctavotZDqGd003TdKoEfclDY7xq0PJg/s1600/o_rlmente.jpg" width="200" /></a>
<li>Observed that for the case where constants are used the JIT is able to perform that optimisation for us almost as well as we’d do ourselves (we have no way of specifying positive only modulo, i.e uint).</li>
<li>We proved the viability of the optimisation in 2 variations, using a pre-calculated mask field and using (array.length - 1)</li>
<li>Using the optimisation in the context of a circular array read showed an interesting reversal in performance. We observed the cause of this reversal to be the array.length load for the purpose of bound checks reused for the calculated mask as opposed to the re-calculated.</li>
<li>Using Unsafe we managed to bypass the array bound check and get the best result using the mask for a single read. </li>
<li>When we try the same method naively in a loop (over the whole array) array bound check is eliminated and plain old array access is the best performer.</li>
<li>To regain the performance for Unsafe access we have to tweak the code to avoid safe point polling as well as to get the addressing mode we want in the resulting assembly. Even then plain array access is better for smaller arrays.</li>
</ul>
</div>
<span style="font-family: inherit;">Simple innit?</span><br />
<span style="font-family: inherit;">Some notes on methodology:</span><br />
<ul>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAU5BKJtQkWS-D45CfjxcS4sgzZdVyKSZ_KVxLNtm3g7cLQNqLwOlyBghcPu2kGHHEVka-TWu5Z81QHf24zwqzC9ozHJrC8PUwPXmDRR1aXlhH73ajaVjQdh1lZ0fTL10e4WNDwTHS1iM/s1600/the+elephant.jpeg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAU5BKJtQkWS-D45CfjxcS4sgzZdVyKSZ_KVxLNtm3g7cLQNqLwOlyBghcPu2kGHHEVka-TWu5Z81QHf24zwqzC9ozHJrC8PUwPXmDRR1aXlhH73ajaVjQdh1lZ0fTL10e4WNDwTHS1iM/s1600/the+elephant.jpeg" /></a>
<li>I ran the same experiments on different intel generations, you get different results but the assembly remains the same. E.g. on older CPUs the maximum instructions per cycle would be less than on the Ivy Bridge CPU I've used here, this will lead to instruction spilling over to the next cycle. The L1 latency could be higher leading to loads dominating the costs etc. This ends up giving a slightly different balance to compute vs. memory load. Overall analysis holds.</li>
<li>Using -XX:-UseCompressedOops was done for the sake of consistent assembly and results. Using compressed oops makes loads look a bit clumsier and I wanted to have less to explain. But running with the flag on (as it is by default) also effects results on this scale. In particular because the compressed oops require a shift to be used and shifters are a limited resource the CPU (1 on Westmere, 2 on Ivy Bridge) it can end up adding a cycle to the results.</li>
<li>Running these same experiments on a laptop was good for getting the assembly out and a vague sense of scale for results, but measurements had far greater error in that environment. Also note that laptops and desktops tend to be a generation ahead of servers where processors are concerned.</li>
<li>An interesting experiment would be to look at same experiment with the JMH perfasm profiler. I did that but could not figure out how to get Intel syntax out of it and so for consistency sake stuck with what I had. Left as an exercise to the reader :P</li>
</ul>
Many thanks to <a href="http://jpbempel.blogspot.com/" target="_blank">J.P. Bempel</a> and <a href="https://twitter.com/peterhughesdev" target="_blank">Peter Hughes</a> for reviewing, any issues remaining were added by me after they reviewed the post.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /></div>
</div>
Nitsanhttp://www.blogger.com/profile/10496299147100350513noreply@blogger.com7