Performance Blog
LoadRunner vs. JMeter Tuesday, 26th October 2010
During some downtime on a recent project, I had a chance to look into open source performance testing software. Obviously, LoadRunner licenses are expensive, so there are a lot of situations where we, or clients, would prefer something cheaper.
The most sophisticated piece of software on the open source scene is JMeter. The question is, can it substitute for Loadrunner?
The short answer is that JMeter does about 60% of what LoadRunner does, but that is perhaps too short. Some projects need the other 40%, and some don't. I'll try to explain the key differences, but try to be objective (rather than thrifty) when evaluating your needs.
Firstly, what JMeter does well. For purely web-based scripts, JMeter is hard to beat. It's easier to learn than LoadRunner, and has no obvious deficiencies when used in this way. Also, JMeter is far better for anything involving string handling; it has well integrated support for regular expressions, as well as a handful of scripting languages (string handling in LoadRunner, via C is a nightmare).
It also supports protocols including SOAP, MQ (via JMS), SQL (via JDBC), TCP (cf. WinSock), as well as the usual suspects like SMTP and FTP. However, crucially JMeter can't record scripts in any of these protocols – scripts have to be either written from scratch, or recorded in something like Wireshark, then converted by hand.
Also, LoadRunner is hard to match for recording and monitoring. JMeter can only monitor Apache Tomcat servers (even then, the monitoring's pretty basic), and has nothing even approaching LoadRunner Analysis. In short, if your test fails, there's relatively little that JMeter can do to help figure out why.
So, what's JMeter useful for? If you're testing purely web-based systems, it'll likely do everything you need. If you're testing other protocols, you might have to work a bit harder, and if you're working to a deadline, you'll probably already be working pretty hard.
However, JMeter has good integration with Apache Ant, which makes it potentially useful for automated regression testing (where time would be less of an issue anyway). Also, if you already have some LoadRunner scripts, it's not too difficult to port them to JMeter (for example, because you want to use them on a system which doesn't have appropriate licenses).
JMeter is quite adequate for smaller or simpler projects, but LoadRunner is the market leader for a reason. LoadRunner is expensive, but it's hard to beat for big complicated projects. By all means, look at whether JMeter is a match for your particular needs, but remember: There's no surer way to look unprofessional, than using the wrong tool for the job.
Wireshark: Tools and Weapons Tuesday, 26th October 2010
I was recently debugging a script. It didn't seem to like one of the requests the script made, so I ran through the procedure manually, and watched in Wireshark.
The firm I was working with at the time is very security conscious, and I got slapped on the wrist for using Wireshark.
The basic problem is that they saw Wireshark as a weapon for hackers. It's far more acceptable to re-record the script in a more business-like tool, like LoadRunner VUGen, and compare the new and the old script.
The difference between a tool and a weapon is mostly semantics. In the wrong hands, VUGen is just as dangerous as Wireshark – in fact it's more dangerous, in some ways, as it can hook into secure connections.
It's a problem I've encountered elsewhere, too. A port scanner's a really easy way of checking connectivity to a remote server, but terrifies system administrators. I don't know of any business-friendly tool that'll do quite the same thing (I had an MS SQL server I wanted to check connectivity to. In the end, I used telnet to open a connection on port 1433, which established connectivity, but it's obviously a poor solution).
By forcing IT people to use business-like tools, businesses are forcing them to use the wrong tool for the job. If their IT people are worth their salt, they can do just as much damage with these business "tools" as with hacker "weapons". Security is an illusion.
Businesses have to trust their IT people. Security measures don't make IT people any safer, they just make them less productive.
New Blog Tuesday, 26th October 2010
I've recently started work for TrustIV, a small performance testing consultancy. I've decided to share some of my experiences as a rookie tester. For anyone starting out in performance testing, and possibly more experienced testers, I'll do my best to keep it interesting.
James
This site was created with KompoZer and Screem. Contact me at james_pic@hotmail.com. Unless otherwise stated, all material on this website is released into the public domain.
For optimal viewing, Internet Explorer users should either download MathPlayer, or use a better browser.