Comments on: Identifying bottlenecks with New Relic’s Java profiling tool http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/ All about browsers, performance testing, and load testing Tue, 07 Feb 2012 15:39:33 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: William Louth http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/comment-page-1/#comment-2016 William Louth Mon, 14 Jun 2010 02:52:59 +0000 http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/#comment-2016 Moving on from "Who is the Biggest Schmuck Here" competition I have posted recently some further information on the "best practices" used by vendor marketing departments in claiming "low overhead". http://williamlouth.wordpress.com/2010/05/25/the-java-application-performance-management-vendor-showdown/ Moving on from “Who is the Biggest Schmuck Here” competition I have posted recently some further information on the “best practices” used by vendor marketing departments in claiming “low overhead”.

http://williamlouth.wordpress.com/2010/05/25/the-java-application-performance-management-vendor-showdown/

]]> By: Bob Newhart http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/comment-page-1/#comment-2010 Bob Newhart Sun, 13 Jun 2010 09:46:44 +0000 http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/#comment-2010 I'm just wondering if your name is actually William Loudmouth-- I'm planning on launching a campaign to smear your product on the simple grounds that you're a simple @#$hole with a constant axe to grind. You're such a schmuck that it doesn't even matter whether you're right or wrong, but the tone of your messages has convinced me to ignore your software and anything to do with you at all costs. Do all of us a favor and please have your PR division prevent you from any more postings. I’m just wondering if your name is actually William Loudmouth– I’m planning on launching a campaign to smear your product on the simple grounds that you’re a simple @#$hole with a constant axe to grind. You’re such a schmuck that it doesn’t even matter whether you’re right or wrong, but the tone of your messages has convinced me to ignore your software and anything to do with you at all costs. Do all of us a favor and please have your PR division prevent you from any more postings.

]]> By: William Louth http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/comment-page-1/#comment-1773 William Louth Tue, 04 May 2010 06:31:37 +0000 http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/#comment-1773 What is it with you guys. I ask for real data you provide another meaningless chart. Did you actually read my comment above. It stated the cost factors (drivers) for call stack sampling overhead. You failed to even list one. You could not even manage to specify the cpu count on the machine. As I stated before you cannot beat the figures I listed. These are the baseline figures for the Java runtime. Your collection and processing costs will go on top of these. The only way you can reduce the impact is to pick a IO/DB bound app (which you have), one with very short call stacks (which you have), one with very little concurrent thread processing (which you have), and you have increased the sampling interval close to a second or more (which I suspect). Again if you dispute my findings (and I am THE expert in this field) you are more than welcome to use my test classes or benchmark against our technology. I know you unit costs are their horrendous so its no problem for me but for you well...... What is it with you guys. I ask for real data you provide another meaningless chart. Did you actually read my comment above. It stated the cost factors (drivers) for call stack sampling overhead. You failed to even list one. You could not even manage to specify the cpu count on the machine.

As I stated before you cannot beat the figures I listed. These are the baseline figures for the Java runtime. Your collection and processing costs will go on top of these. The only way you can reduce the impact is to pick a IO/DB bound app (which you have), one with very short call stacks (which you have), one with very little concurrent thread processing (which you have), and you have increased the sampling interval close to a second or more (which I suspect).

Again if you dispute my findings (and I am THE expert in this field) you are more than welcome to use my test classes or benchmark against our technology. I know you unit costs are their horrendous so its no problem for me but for you well……

]]> By: Jim Gochee http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/comment-page-1/#comment-1772 Jim Gochee Tue, 04 May 2010 03:34:27 +0000 http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/#comment-1772 Hi William, I claimed that our overhead was less than 10% cpu burn when sampling threads, and the data for Patrick's app confirms this. Here's a graph of CPU burn on the java process: http://skitch.com/jim-gochee/db36j/centcom-application-cpu-usage-broken-out-by-host-new-relic-rpm I've put arrows at the start/end of the sampling session. I'm not sure what ax you've got to grind, but our product is free to download and run, so knock yourself dead comparing us to your solution. Jim Hi William,
I claimed that our overhead was less than 10% cpu burn when sampling threads, and the data for Patrick’s app confirms this. Here’s a graph of CPU burn on the java process:

http://skitch.com/jim-gochee/db36j/centcom-application-cpu-usage-broken-out-by-host-new-relic-rpm

I’ve put arrows at the start/end of the sampling session.

I’m not sure what ax you’ve got to grind, but our product is free to download and run, so knock yourself dead comparing us to your solution.

Jim

]]> By: William Louth http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/comment-page-1/#comment-1763 William Louth Mon, 03 May 2010 07:25:20 +0000 http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/#comment-1763 Jim I have not figured out whether you are inexperienced enough to believe you what you and your company claims or experience enough to withhold enough information to prevent anyone showing how clearly wrong you are in your claims. Our testing of the impact of call stack sampling is independent of any profiling processing. You cannot do any better than the numbers we shown as this is the baseline from which you add you own overhead. So how can we both be right. Well there are a number of factors some of which reflect the type of web applications that your company generally monitors. 1. The web application is severely IO bound. Any overhead whether excessive compared to other solutions will pale in comparison to the amount of time spent executing database/file/messaging operations. 2. The degree of concurrency is relatively low compared to enterprise applications which can easily have more than 200 active threads at a time. 3. The maximum call stack depth is relatively low compared to enterprise enterprise applications (especially Spring based ones) which generally exceed 300 by the time they hit their exit point (messaging, database, rmi,...). 4. You have offloaded a large amount of (cpu) work onto other non-application specific threads assuming their is a lot of spare compute capacity. 5. The sampling time interval is set extremely high compared to other typical profilers. Now after reading my links I am surprised that you could not manage to provide any further information other than 10% which by the way I find pretty excessive and certainly not production ready but then again we operate at different ends of the spectrum in terms of performance whereas JXInsight Probes technology has a cost in the low nanoseconds yours is in the double digit microseconds for your trace technology - 1000x slower. If you dispute any of the above then we again extend our offer (which was previously reused) to have an official performance shootout - ours probes technology against your legacy trace technology. Then we can finally bring truth to the claims of zero overhead on and off both in terms of runtime and analysis. http://williamlouth.wordpress.com/2010/03/02/does-transaction-tracing-scale-analysis/ By the way Patrick we have also requested a similar shootout of dynaTrace (aka dynaCopy) on many occasions - each time they also refused. Our technology is hundreds (if not thousands) of times more efficient than both solutions and yet we do not go around making the unqualified "low overhead" claims both these cowboys make. William Jim I have not figured out whether you are inexperienced enough to believe you what you and your company claims or experience enough to withhold enough information to prevent anyone showing how clearly wrong you are in your claims.

Our testing of the impact of call stack sampling is independent of any profiling processing. You cannot do any better than the numbers we shown as this is the baseline from which you add you own overhead.

So how can we both be right. Well there are a number of factors some of which reflect the type of web applications that your company generally monitors.

1. The web application is severely IO bound. Any overhead whether excessive compared to other solutions will pale in comparison to the amount of time spent executing database/file/messaging operations.

2. The degree of concurrency is relatively low compared to enterprise applications which can easily have more than 200 active threads at a time.

3. The maximum call stack depth is relatively low compared to enterprise enterprise applications (especially Spring based ones) which generally exceed 300 by the time they hit their exit point (messaging, database, rmi,…).

4. You have offloaded a large amount of (cpu) work onto other non-application specific threads assuming their is a lot of spare compute capacity.

5. The sampling time interval is set extremely high compared to other typical profilers.

Now after reading my links I am surprised that you could not manage to provide any further information other than 10% which by the way I find pretty excessive and certainly not production ready but then again we operate at different ends of the spectrum in terms of performance whereas JXInsight Probes technology has a cost in the low nanoseconds yours is in the double digit microseconds for your trace technology – 1000x slower.

If you dispute any of the above then we again extend our offer (which was previously reused) to have an official performance shootout – ours probes technology against your legacy trace technology. Then we can finally bring truth to the claims of zero overhead on and off both in terms of runtime and analysis.

http://williamlouth.wordpress.com/2010/03/02/does-transaction-tracing-scale-analysis/

By the way Patrick we have also requested a similar shootout of dynaTrace (aka dynaCopy) on many occasions – each time they also refused.

Our technology is hundreds (if not thousands) of times more efficient than both solutions and yet we do not go around making the unqualified “low overhead” claims both these cowboys make.

William

]]> By: Jim Gochee http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/comment-page-1/#comment-1752 Jim Gochee Fri, 30 Apr 2010 23:52:55 +0000 http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/#comment-1752 Hi William, New Relic's sampling approach must be different from the one you benchmark in http://williamlouth.wordpress.com/2009/10/21/java-call-sampling-overhead-in-the-wild/. For our solution, overhead to the application is zero when you're not running a "collection session". And when you are running a collection session, it's only on a single JVM in your cluster and the impact to that single JVM is around 10% cpu burn. For a tool that gives complete thread state visibility that amount of overhead seems very reasonable. Jim Hi William,
New Relic’s sampling approach must be different from the one you benchmark in http://williamlouth.wordpress.com/2009/10/21/java-call-sampling-overhead-in-the-wild/. For our solution, overhead to the application is zero when you’re not running a “collection session”. And when you are running a collection session, it’s only on a single JVM in your cluster and the impact to that single JVM is around 10% cpu burn. For a tool that gives complete thread state visibility that amount of overhead seems very reasonable.

Jim

]]> By: William Louth http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/comment-page-1/#comment-1747 William Louth Fri, 30 Apr 2010 11:37:49 +0000 http://blog.browsermob.com/2010/04/identifying-bottlenecks-with-new-relics-java-profiling-tool/#comment-1747 But thread call stack sampling does not scale (certainly not in production) unless the concurrent workload is relatively small and you have low hanging fruit that could in all likelihood be found with very simplistic mechanism such as a thread stack dump like above. I wrote a number of articles on this. Sampling is cheap for the vendor development wise but its cost benefit analysis at runtime is very poor once you go beyond the obvious. http://williamlouth.wordpress.com/2009/10/21/java-call-sampling-overhead-in-the-wild/ http://williamlouth.wordpress.com/2009/01/16/profiling-sampling-versus-execution-part-2/ http://williamlouth.wordpress.com/2009/01/07/profiling-sampling-versus-execution-part-1/ William But thread call stack sampling does not scale (certainly not in production) unless the concurrent workload is relatively small and you have low hanging fruit that could in all likelihood be found with very simplistic mechanism such as a thread stack dump like above.

I wrote a number of articles on this. Sampling is cheap for the vendor development wise but its cost benefit analysis at runtime is very poor once you go beyond the obvious.

http://williamlouth.wordpress.com/2009/10/21/java-call-sampling-overhead-in-the-wild/

http://williamlouth.wordpress.com/2009/01/16/profiling-sampling-versus-execution-part-2/

http://williamlouth.wordpress.com/2009/01/07/profiling-sampling-versus-execution-part-1/

William

]]>