Testing the effect of limiting connections & load testing

uselinux

uselinux

New Member
#1
#1
I switched to OLS from Apache a few months ago to nail down a shorter TTFB. Overall the switch has gone okay but recently (due to traffic) I've experienced instability.
In response to that I've limited the number of SSL connections and PHP processes that can be spun up per instance. Great, site doesn't go down anymore.

However I do need to be able to serve all visitors. I'm working on AWS autoscaling and that should help.

What, from a user perspective, is hitting a server that has hit its php/ssl connection limit going to look like?

Also, I've run ApacheBench and JMeter and these tests show me I can serve more visitors than I can in reality. JMeter worked well with apache but is a bit of a flop with OLS. Any suggestions on load testing approaches?

Thanks for any input.
 
Cold-Egg

Cold-Egg

Administrator
#2
#2
Hi,

May I ask limited number means how many connections to SSL and PHP?
Any cache solution on it? I am more curious about how's the site went down.

Auto-scaling now may not ideal for cache solution since the cache file may not be valid. To make cache solution work in this case, may need to put WebADC(LiteSpeed Loadbalancer) in the front and let it handle the cache part.

I am not a JMeter guy, but you can try h2load/wrk or online testing tool loader.io to help. Please share the number you got and whats your test target if possible.

Best,
Eric
 
uselinux

uselinux

New Member
#3
#3
I limited SSL connections to 15 (30) per server. PHP processes were 20 soft, 25 hard for each server.
I'm using the OLS cache plugin for WordPress, Cloudfront and S3 for images. I also use an Elastic Load Balancer behind Cloudfront.

A connection will spawn a php process which consumes x memory. When that runs out the server starts moving it to swap. Swappping has a huge cpu cost. As connections continue to poor in the whole thing locks up and that takes the site down. Under live traffic this will start happening after the instances have a constant 40+ connections.

My limit worked well to keep servers up. I'm now increasing the limit in dev while pounding it with loader.io (found that yesterday).
So far I can't cause things to spiral.

One thing I have noticed is that on my productions instances a lot of lswsphp processes spawn while on my dev environment I only see openlitespeed ones. I'm not sure what might be different.
 
Cold-Egg

Cold-Egg

Administrator
#4
#4
HI,

So if you have lscache setup correctly (see a cache hit from response header), then it general should be able to handle >1000 connections. So "15/30" is really way too low. I may guess cache not working in your production site, could you share the test link here so we can verify? If you want, you can also provide a report number from lscache here so we can check.

Correct, if cache works, you should not hit back-end php by public view.
 
Last edited:
uselinux

uselinux

New Member
#5
#5
I currently have my development site capped at 60x2 and am handling 1500 requests a second. Actually struggling to hit and keep a high cpu usage.

You're right. For whatever reason production isn't getting a cache hit. I'm not seeing why.
I turned debug log on and tapped it with some requests and saw lsphp spin up.

I'm specifically seeing: X-LiteSpeed-Cache: no-cache

Could a cache exclusions be the cause and can I determine that from the log?

### marker NOCACHE COOKIES start ###
RewriteCond %{HTTP_COOKIE} state|city|lat|lng|radius
RewriteRule .* - [E=Cache-Control:no-cache]
### marker NOCACHE COOKIES end ###

Some page content alters depending on state, city, etc. I am simply not caching if the cookies exist? Is there a way to cache each individually?
 
Last edited:
uselinux

uselinux

New Member
#7
#7
Much appreciated.
I removed the wordpress plugin rule with the excludes and updated .htaccess with vary rules.
It's definitely caching again. What i'm running into now is users are not able to change their location (set in a cookie).
The cookie will change but a request to a page with a differing cookie will just pull content cached from another location.

Is it possible (or even a good idea) to have a vary cache for query vars such as a search result page?

Now that I've learned auto scaling and sorted this out... big runway. I am happy.


edit: https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:lscwp:handle-cookie-vary
Similiar enough case to mine yet I have multiple cookies I need to vary on.

Should I be doing something like:
Code: 
<IfModule LiteSpeed>
  RewriteEngine on
  CacheLookup on
  RewriteRule .* - [E=cache-vary:state]
  RewriteRule .* - [E=cache-vary:city]
  RewriteRule .* - [E=cache-vary:lat]
  RewriteRule .* - [E=cache-vary:lng]
  RewriteRule .* - [E=cache-vary:radius]
</IfModule>
or:
Code: 
<IfModule LiteSpeed>
  RewriteEngine on
  CacheLookup on
  RewriteRule .* - [E=cache-vary:state,city,lat,lng,radius]
</IfModule>
 
Last edited:
uselinux

uselinux

New Member
#9
#9
I've encountered another issue.
We run some ad campaigns and so through the WordPress LSCache plugin I set up a couple of querystrings to drop.
In .htaccess this looks like:
Code: 
### marker DROPQS start ###
CacheKeyModify -qs:s
CacheKeyModify -qs:glcid
CacheKeyModify -qs:fbclid
CacheKeyModify -qs:utm*
### marker DROPQS end ###
However when I visit a page and change the value of any of these I always get a miss. So each variety is creating a cache entry.

Here is my full .htaccess in case anything may be conflicting
Code: 
<IfModule LiteSpeed>
  RewriteEngine on
  CacheLookup on
  RewriteRule .* - [E="Cache-Vary:state"]
</IfModule>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   # BEGIN LSCACHE                                                                                                                                                                                                                                                                ## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
<IfModule LiteSpeed>
  RewriteEngine on
  CacheLookup on
  RewriteRule .* - [E=Cache-Control:no-autoflush]
  RewriteRule ^min/\w+\.(css|js) - [E=cache-control:no-vary]
  ### marker CACHE RESOURCE start ###
  RewriteRule wp-content/.*/[^/]*(responsive|css|js|dynamic|loader|fonts)\.php - [E=cache-control:max-age=3600]
  ### marker CACHE RESOURCE end ###

  ### marker LOGIN COOKIE start ###
  RewriteRule .? - [E="Cache-Vary:wp-postpass_f86d1bf72bfdeb2f65cdac91fd3d3cda"]
  ### marker LOGIN COOKIE end ###
  ### marker FAVICON start ###
  RewriteRule favicon\.ico$ - [E=cache-control:max-age=86400]
  ### marker FAVICON end ###

  ### marker CORS start ###
  <FilesMatch "\.(ttf|ttc|otf|eot|woff|woff2|font\.css)$">
    <IfModule mod_headers.c>
      Header set Access-Control-Allow-Origin "*"
    </IfModule>
  </FilesMatch>
  ### marker CORS end ###
  ### marker WEBP start ###
  RewriteCond %{HTTP_ACCEPT} "image/webp" [or]
  RewriteCond %{HTTP_USER_AGENT} "Page Speed"
  RewriteRule .* - [E=Cache-Control:vary=%{ENV:LSCACHE_VARY_VALUE}+webp]
  ### marker WEBP end ###
  ### marker DROPQS start ###
  CacheKeyModify -qs:s
  CacheKeyModify -qs:glcid
  CacheKeyModify -qs:fbclid
  CacheKeyModify -qs:utm*
  ### marker DROPQS end ###
</IfModule>
## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
# END LSCACHE

# BEGIN NON_LSCACHE
## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
### marker MINIFY start ###
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} ^(.*)/min/(\w+)\.(css|js)$
  RewriteCond %1/wp-content/cache/$2/$1.$2 -f
  RewriteRule min/(\w+)\.(css|js) wp-content/cache/$2/$1.$2 [L]
</IfModule>
### marker MINIFY end ###
## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
# END NON_LSCACHE
[/code[
 
uselinux

uselinux

New Member
#10
#10
Also testing this method and getting the same result with a url of /?utm_source=anything
Code: 
RewriteCond %{QUERY_STRING} (utm_source|utm_medium|utm_campaign|gclid|fbclid|s)=[^&]+&*
RewriteRule  .* - [E=cache-key-mod:-qs:utm_source*]
Edit: I learned from support that dropping query strings is only for enterprise at the moment.
My working solution:
Code: 
RewriteCond %{QUERY_STRING} ^(.*)?gclid=[^&]+&?(.*)$ [NC, OR]
RewriteCond %{QUERY_STRING} ^(.*)?&gclid=[^&]+&?(.*)$ [NC]
RewriteRule ^/?(.*)$ /$1?%1%2 [R=301,L]

RewriteCond %{QUERY_STRING} ^(.*)?fbclid=[^&]+&?(.*)$ [NC, OR]
RewriteCond %{QUERY_STRING} ^(.*)?&fbclid=[^&]+&?(.*)$ [NC]
RewriteRule ^/?(.*)$ /$1?%1%2 [R=301,L]
I imaging regex could be better but it does work.
This will strip fbclid and gclid while preserving all other query strings. I'll cache a few different versions for campaigns but it won't be hundreds of versions as is seen with the click ids.
 
Last edited:
You must log in or register to reply here.
Top