OLS using header IP, not trusted Cloudflare IP, for connection limit?

mcbsys

mcbsys

New Member
#1
#1
Hi,

I run several WordPress web sites on OpenLiteSpeed 1.7.18 on CyberPanel 2.3 on an Azure virtual machine. It was working fine until I recently switched to Cloudflare for DNS and CDN caching. Since then, after loading a few web pages (or sometimes right away when loading the WordPress back end), Cloudflare returns Error code 520.

I found https://docs.litespeedtech.com/lsws/cp/cpanel/cloudflare/ which says Cloudflare is trusted automatically, but maybe that is only Enterprise Lightspeed, so I added this to Server Configuration > Security > Access Control:

ALL, 173.245.48.0/20T, 103.21.244.0/22T, 103.22.200.0/22T, 103.31.4.0/22T, 141.101.64.0/18T, 108.162.192.0/18T, 190.93.240.0/20T, 188.114.96.0/20T, 197.234.240.0/22T, 198.41.128.0/17T, 162.158.0.0/15T, 104.16.0.0/13T, 104.24.0.0/14T, 172.64.0.0/13T, 131.0.72.0/22T

I also set Configuration > General Settings > Use Client IP in Header to Trusted IP Only.

I thought this probably has something to do with OLS Anti-DDoS but increasing those limits doesn't help. And it shouldn't matter since the Cloudflare IPs are trusted. Then today I found this in the error log:

2023-12-26 22:46:43.164082 [NOTICE] [1290834] [166.216.158.18] reached per client soft connection limit: 20 for 49 seconds, close connection!
2023-12-26 22:46:43.164112 [WARN] [1290834] [166.216.158.18] bot detected for vhost [N/A], reason: OverConnSoftLimit, close connection!
2023-12-26 22:46:43.164119 [INFO] [1290834] [172.71.159.23:59872] Client IP from header: 166.216.158.18, cur conns: 1, access denied
2023-12-26 22:46:43.561406 [INFO] [1290834] [172.71.155.29:44858] Client IP from header: 166.216.158.18, cur conns: 2, access denied

166.216.158.18 is an IP address from my cell phone provider (currently tethered to my laptop). It is the IP address in the header (see third and fourth messages above). But it looks like this header IP, rather than the Cloudflare IP, is being used for connection limiting (see first message).

Shouldn't OLS allow the connection through Cloudflare, even if Cloudflare asks for a lot of connections at once?

How should I run OLS behind Cloudflare? Should I disable OLS throttling (set to 0?) and just let Cloudflare manage DDoS?
 
Cold-Egg

Cold-Egg

Administrator
#2
#2
Since v1.7.13, OLS whitelisted QUIC.cloud and Cloudflare IPs automatically. What if you clean up those IPs from the Access Control, does that help?
 
mcbsys

mcbsys

New Member
#3
#3
Yes, if it put just "ALL" in the Access Control, it seems better, even after setting Per Client Throttling back to previous values and purging the Cloudflare cache:

Static Requests/second 25
Connection Soft Limit 10
Connection Hard Limit 40

What is happening here? How does Access Control interact with the automatic whitelist?

Maybe I should mention that might whitelist also included my own static IP before ALL (e.g. 123.123.123.123):

123.123.123.123T, ALL, 173.245.48.0/20T, 103.21.244.0/22T, 103.22.200.0/22T, 103.31.4.0/22T, 141.101.64.0/18T, 108.162.192.0/18T, 190.93.240.0/20T, 188.114.96.0/20T, 197.234.240.0/22T, 198.41.128.0/17T, 162.158.0.0/15T, 104.16.0.0/13T, 104.24.0.0/14T, 172.64.0.0/13T, 131.0.72.0/22T

Is that the problem, putting the Trusted IP before ALL?
 
mcbsys

mcbsys

New Member
#4
#4
And now the error is back... received on the second click in a WordPress back end. This on a different server that I manage, running OLS 1.7.17 without CyberPanel:

20231229 Cloudflare error.png

Once again, it is showing my actual IP address (which I replaced here with 123.123.123.123):

2023-12-30 01:46:35.488569 NOTICE [1187829] [123.123.123.123] Reached per client hard connection limit: 40, current: 40, close connection!
2023-12-30 01:46:35.488597 WARN [1187829] [123.123.123.123] bot detected for vhost [N/A], reason: OverConnHardLimit, close connection!
2023-12-30 01:47:21.692917 NOTICE [1143908] [123.123.123.123] reached per client soft connection limit: 10 for 46 seconds, close connection!
2023-12-30 01:47:21.692952 WARN [1143908] [123.123.123.123] bot detected for vhost [N/A], reason: OverConnSoftLimit, close connection!

Why is it blocking based on the actual end-user IP rather than on the trusted Cloudflare IPs? What is the proper configuration to use OLS behind Cloudflare?
 
Last edited:
mcbsys

mcbsys

New Member
#5
#5
This hit us again this morning, blocking access to a the WordPress back end at a critical time. Very frustrating!

I just found the article https://docs.litespeedtech.com/lsws/cp/cpanel/antiddos/ which indicates that using the client IP (and not whitelisted CDN IPs) for throttling is expected behavior and recommends raising the connection limits to 100000/150000.

"If you use CDN services with real visitor IP enabled, the IP which is forwarded from the CDN may get blocked if the soft or hard limit is too low. One way to disable such blocking or per-client throttling is to set the connection limits to very large numbers, such as 100000 and 150000, respectively." (I assume that the other method, using an Apache DisableForwardedIpBan directive, would not work in OLS.)

It seems odd that LiteSpeed intentionally ignores CDN whitelist, but if that is the case, please update the OpenLiteSpeed documentation as well: https://openlitespeed.org/kb/per-client-throttling/. From the screen shot in the first article, it looks like the correct configuration behind a CDN is

20231231 OpenLiteSpeed Throttling.png
 
krlabs

krlabs

New Member
#10
#10
mcbsys said:
This hit us again this morning, blocking access to a the WordPress back end at a critical time. Very frustrating!

I just found the article https://docs.litespeedtech.com/lsws/cp/cpanel/antiddos/ which indicates that using the client IP (and not whitelisted CDN IPs) for throttling is expected behavior and recommends raising the connection limits to 100000/150000.

"If you use CDN services with real visitor IP enabled, the IP which is forwarded from the CDN may get blocked if the soft or hard limit is too low. One way to disable such blocking or per-client throttling is to set the connection limits to very large numbers, such as 100000 and 150000, respectively." (I assume that the other method, using an Apache DisableForwardedIpBan directive, would not work in OLS.)

It seems odd that LiteSpeed intentionally ignores CDN whitelist, but if that is the case, please update the OpenLiteSpeed documentation as well: https://openlitespeed.org/kb/per-client-throttling/. From the screen shot in the first article, it looks like the correct configuration behind a CDN is

View attachment 1328
Click to expand...
Tried today this settings and got 520 from Cloudflare, but all IP's was in Trust IP and put in Access Allow list.

So. I have not been able to completely block access to the SERVER IP.
 
Last edited:
krlabs

krlabs

New Member
#12
#12
LiteCache said:
You have a strange understanding of how the Internet works if a CDN is used.
Click to expand...
My understanding leads to the fact that I solve 90% of my tasks for Administration and Security VPS. So it's not a bad thing.

Offer your options and your vision if you have more professional experience/approach. I would be grateful for any advice. This forum was created for this purpose.
 
LiteCache

LiteCache

Active Member
#13
#13
krlabs said:
This forum was created for this purpose.
Click to expand...
This forum is created to support OLS, but not for any CDN related issues.

Using a CDN is a complete different thing, especially with CloudFlare. If you get a 520 status error you should firstly follow CloudFlare suggestions to solve such errors since status code 520 is a CF related error code.
 
Cold-Egg

Cold-Egg

Administrator
#14
#14
Hi, if you think the 520 is a backend server issue, please check the web server error log and see what's in there. FYI, you don't need to whitelist CloudFlare IP again since it's whitelisted by the web server automatically.
 
LiteCache

LiteCache

Active Member
#15
#15
Cold-Egg said:
Hi, if you think the 520 is a backend server issue, please check the web server error log and see what's in there. FYI, you don't need to whitelist CloudFlare IP again since it's whitelisted by the web server automatically.
Click to expand...
Of course, but CloudFlare status code 520 means there is an issue connecting the origin host. Maybe the origin host is down or cannot be connected by CF. If there were a block by the origin a CloudFlare status code page 520 would not be displayed.
 
You must log in or register to reply here.
Top