Lies, damned lies, and benchmarks

[OC2PS]

Michael Armstrong:
We have been doing some benchmarks internally and LiteSpeed and Nginx have similar speeds in comparison to basic Apache, though our benchmarks suggest that LSWS is still faster than Nginx. We plan to release some new benchmarks soon, but we will probably not do a benchmark of WordPress installations — it's just too easy for someone to say, "Well, that's not what my WordPress installation looks like." and throw out the results. We feel it's better to keep the results as pure as possible and use them as just a proof of concept.

"Small static file" benchmarks have zero credibility and sully the whole science of performance benchmarking as a whole.

Benchmarking DEFAULT installs of php scripts like

• WordPress,
  
• Joomla,
  
• Drupal,
  
• phpBB,
  
• Simple Machines Forums,
  
• Menalto Gallery,
  
• Magento and
  
• OpenCart

provides a real-life like scenario...covering most applications that folks use. Sure individual installs differ, with different plugins, themes, data etc...but I can pretty much damn well guarantee that nobody will hold that against you....at least not as much as they would dismiss a "small static file" type benchmark.

I don't know if someone will be willing to benchmark the following, but I'd certainly be interested in seeing head-to-head comparisons of:

• Apache 2.4.x vs
• Apache 2.4.x + Varnish 3.0.x vs
• nginx 1.4.x vs
• nginx 1.4.x + Varnish 3.0.x vs
• OpenLiteSpeed 1.2.x vs
• OpenLiteSpeed 1.2.x + Varnish 3.0.x vs
• LiteSpeed Enterprise 4.2.x

 

[eva2000]

I agree to certain extend, but if you benchmark real apps such as listed above, then you are also benchmarking MySQL backend database configuration as well and not solely php/static images. So benchmark results reflect how well you properly tune your MySQL server settings and from my experience not many folks do the MySQL optimisation size properly which will definitely affect benchmark result outcome

And how you tune MySQL depends on size of database and it's work load.

Of course if you keep MySQL settings the same, they it's certainly comparable to an extent but you have to see that the resulting benchmark results might reflect more the MySQL backend performance, than the performance of the web server itself.

From personal experience pretty much from best listed at top to bottom, will be the following:

1. LiteSpeed Enterprise 4.2.x + Varnish 3.0
2. OpenLiteSpeed 1.2.x + Varnish 3.0.x <-- extrapolated from LiteSpeed Enterprise based performance
3. nginx 1.4.x + Varnish 3.0.x
4. Apache 2.4.x + Varnish 3.0.x
5. LiteSpeed Enterprise 4.2.x
6. OpenLiteSpeed 1.2.x <-- extrapolated from LiteSpeed Enterprise based performance
7. nginx 1.4.x
8. Apache 2.4.x

Personal experience is working with high user concurrency sites using drupal, wordpress, vbulletin etc up to 14,000+ php requests/s, 10 billion memcached server hits/month and 400,000+ unique ip visitors per day.

 

[OC2PS]

if you benchmark real apps such as listed above, then you are also benchmarking MySQL backend database configuration as well and not solely php/static images.

I don't see that as a bad thing at all. The fact is that I don't care how blazingly fast a webserver is in serving a text file. What I am interested in is whether it makes any difference in real-world throughput to real users.

Including mysql won't result in "raw" performance stats, and I feel that's fine. Maybe the raw stats are fine trophies for server developers, but what folks like me are interested in (and really the only reason we read benchmarks) are COMPARATIVE performance. I don't give a hoot about the absolute numbers (as long as they meet some basic threshold, which, it is reasonable to assume that most popular servers do), only about relative ones..i.e. ok, OpenLiteSpeed is faster than nginx at x concurrency level...but how much faster...5%, 10%, etc...

Ultimately, it boils down to the business problem of choosing a server/stack...and what the trade-offs are. And in solving that business problem, I think that real-app based benchmarks are helpful in a way that no "proof of concept" small static file benchmarks can be.

From personal experience...

Thanks so much for the insight and perspective. Very helpful.

This is the kind of comparison folks are after. We just hope that perspectives can be backed up by actual science...which is why we search for benchmarks.

2 questions about it:

• Why is nginx + Varnish better than nginx (or better than Apache + nginx)? Don't nginx and Varnish both excel at the same thing i.e. serving static files (while Apache is thought to be fairly adequate with PHP)
• You put LiteSpeed Enterprise + Varnish on top. Similar question to previous...isn't LiteSpeed Enterprise page cache supposed to do the same as Varnish?

 

[eva2000]

my thoughts of Varnish cache vs LiteSpeed Enterprise's LightSpeed Cache at http://vbtechsupport.com/33/ and http://vbtechsupport.com/26/

Varnish and LiteSpeed cache excel as they serve their cache from memory and not disk which Nginx, Apache or LiteSpeed/OpenLiteSpeed do not do by default.

I ranked them that way from my thoughts at http://vbtechsupport.com/33/, varnish cache and litespeed cache can not cache everything especially for dynamic php content which need low cache TTL, so ultimately performance is determined by how well non-cached requests get served and that ultimately comes down to the backend served (LS/OLS, Nginx or Apache) when paired with Varnish cache or LiteSpeed cache.

Same if you compared Varnish + Nginx > Nginx + Apache. As non cached requests hit backend where Nginx backend > Apache backend. Varnish cache hit rate can range from 5-10% sometimes, which means 90% of requests are non-cached and hit your backend

But now that OpenLiteSpeed is released, the choices aren't that hard as Varnish cache is free, Apache is free, OpenLiteSpeed is free, Nginx is free. So try them all and choose the best one for your site's required work loads

That's why I am working on integrating Apache 2.4 event mpm + php-fpm + OpenLiteSpeed + LSAPI PHP into my Centmin Mod Nginx installer project so eventually Centmin Mod users can pick and choose whatever web server they want https://plus.google.com/104831941868856035845/posts/UNj5sVYt6WF

Either use Nginx as load balancer or my favourite Haproxy as front end load balancer and I even break up a site to be served from different backend web servers if I want.

And not to forget about Node.js as well http://centminmod.com/siegebenchmarks/2013/020313/index.html

But these plans are a long way off while I do all the testing

Anyway, Nginx is great and why i developed http://centminmod.com the way it is and for most folks - especially single server users they won't reach the traffic and user concurrency loads on Nginx to tell much of a difference between LiteSpeed Enterprise (and suspect OpenLiteSpeed). That's where you see folks claim they don't see any speed difference although there's other metrics to look at i.e cpu load, memory usage etc. So ultimate choice comes down to personal preference and what you're comfortable working with. If you don't want to stray away from Apache rewrite syntax, then LS/OLS are the only real alternatives to Apache

 

[OC2PS]

Good stuff, George!

 

[lsmichael]

I don't know if someone will be willing to benchmark the following, but I'd certainly be interested in seeing head-to-head comparisons of:

• Apache 2.4.x vs
• Apache 2.4.x + Varnish 3.0.x vs
• nginx 1.4.x vs
• nginx 1.4.x + Varnish 3.0.x vs
• OpenLiteSpeed 1.2.x vs
• OpenLiteSpeed 1.2.x + Varnish 3.0.x vs
• LiteSpeed Enterprise 4.2.x

We're willing and I think you're right about a lot of things. It is very time consuming to do good benchmarks, though. We're hoping to have some out soon, and then I will keep pushing to have them meet your standards. Keep pushing for what you want. We can't spend all our time on benchmarks, but we'll do our best to have something relevant.

m

 

[OC2PS]

We're hoping to have some out soon, and then I will keep pushing to have them meet your standards.

Bated breath, etc.