LoadRunner vs JMeter

LoadRunner vs. JMeter Tuesday, 26th October 2010

During some downtime on a recent project, I had a chance to look into open source performance testing software. Obviously, LoadRunner licenses are expensive, so there are a lot of situations where we, or clients, would prefer something cheaper.

The most sophisticated piece of software on the open source scene is JMeter. The question is, can it substitute for Loadrunner?

The short answer is that JMeter does about 60% of what LoadRunner does, but that is perhaps too short. Some projects need the other 40%, and some don't. I'll try to explain the key differences, but try to be objective (rather than thrifty) when evaluating your needs.

Firstly, what JMeter does well. For purely web-based scripts, JMeter is hard to beat. It's easier to learn than LoadRunner, and has no obvious deficiencies when used in this way. Also, JMeter is far better for anything involving string handling; it has well integrated support for regular expressions, as well as a handful of scripting languages (string handling in LoadRunner, via C is a nightmare).

It also supports protocols including SOAP, MQ (via JMS), SQL (via JDBC), TCP (cf. WinSock), as well as the usual suspects like SMTP and FTP. However, crucially JMeter can't record scripts in any of these protocols – scripts have to be either written from scratch, or recorded in something like Wireshark, then converted by hand.

Also, LoadRunner is hard to match for recording and monitoring. JMeter can only monitor Apache Tomcat servers (even then, the monitoring's pretty basic), and has nothing even approaching LoadRunner Analysis. In short, if your test fails, there's relatively little that JMeter can do to help figure out why.

So, what's JMeter useful for? If you're testing purely web-based systems, it'll likely do everything you need. If you're testing other protocols, you might have to work a bit harder, and if you're working to a deadline, you'll probably already be working pretty hard.

However, JMeter has good integration with Apache Ant, which makes it potentially useful for automated regression testing (where time would be less of an issue anyway). Also, if you already have some LoadRunner scripts, it's not too difficult to port them to JMeter (for example, because you want to use them on a system which doesn't have appropriate licenses).

JMeter is quite adequate for smaller or simpler projects, but LoadRunner is the market leader for a reason. LoadRunner is expensive, but it's hard to beat for big complicated projects. By all means, look at whether JMeter is a match for your particular needs, but remember: There's no surer way to look unprofessional, than using the wrong tool for the job.

This site was created with KompoZer and Screem. Contact me at james_pic@hotmail.com. Unless otherwise stated, all material on this website is released into the public domain.

For optimal viewing, Internet Explorer users should either download MathPlayer, or use a better browser.