OLS using header IP, not trusted Cloudflare IP, for connection limit?

[mcbsys]

Hi,

I run several WordPress web sites on OpenLiteSpeed 1.7.18 on CyberPanel 2.3 on an Azure virtual machine. It was working fine until I recently switched to Cloudflare for DNS and CDN caching. Since then, after loading a few web pages (or sometimes right away when loading the WordPress back end), Cloudflare returns Error code 520.

I found https://docs.litespeedtech.com/lsws/cp/cpanel/cloudflare/ which says Cloudflare is trusted automatically, but maybe that is only Enterprise Lightspeed, so I added this to Server Configuration > Security > Access Control:

ALL, 173.245.48.0/20T, 103.21.244.0/22T, 103.22.200.0/22T, 103.31.4.0/22T, 141.101.64.0/18T, 108.162.192.0/18T, 190.93.240.0/20T, 188.114.96.0/20T, 197.234.240.0/22T, 198.41.128.0/17T, 162.158.0.0/15T, 104.16.0.0/13T, 104.24.0.0/14T, 172.64.0.0/13T, 131.0.72.0/22T

I also set Configuration > General Settings > Use Client IP in Header to Trusted IP Only.

I thought this probably has something to do with OLS Anti-DDoS but increasing those limits doesn't help. And it shouldn't matter since the Cloudflare IPs are trusted. Then today I found this in the error log:

2023-12-26 22:46:43.164082 [NOTICE] [1290834] [166.216.158.18] reached per client soft connection limit: 20 for 49 seconds, close connection!
2023-12-26 22:46:43.164112 [WARN] [1290834] [166.216.158.18] bot detected for vhost [N/A], reason: OverConnSoftLimit, close connection!
2023-12-26 22:46:43.164119 [INFO] [1290834] [172.71.159.23:59872] Client IP from header: 166.216.158.18, cur conns: 1, access denied
2023-12-26 22:46:43.561406 [INFO] [1290834] [172.71.155.29:44858] Client IP from header: 166.216.158.18, cur conns: 2, access denied

166.216.158.18 is an IP address from my cell phone provider (currently tethered to my laptop). It is the IP address in the header (see third and fourth messages above). But it looks like this header IP, rather than the Cloudflare IP, is being used for connection limiting (see first message).

Shouldn't OLS allow the connection through Cloudflare, even if Cloudflare asks for a lot of connections at once?

How should I run OLS behind Cloudflare? Should I disable OLS throttling (set to 0?) and just let Cloudflare manage DDoS?

 

[Cold-Egg]

Since v1.7.13, OLS whitelisted QUIC.cloud and Cloudflare IPs automatically. What if you clean up those IPs from the Access Control, does that help?

 

[mcbsys]

Yes, if it put just "ALL" in the Access Control, it seems better, even after setting Per Client Throttling back to previous values and purging the Cloudflare cache:

Static Requests/second 25
Connection Soft Limit 10
Connection Hard Limit 40

What is happening here? How does Access Control interact with the automatic whitelist?

Maybe I should mention that might whitelist also included my own static IP before ALL (e.g. 123.123.123.123):

123.123.123.123T, ALL, 173.245.48.0/20T, 103.21.244.0/22T, 103.22.200.0/22T, 103.31.4.0/22T, 141.101.64.0/18T, 108.162.192.0/18T, 190.93.240.0/20T, 188.114.96.0/20T, 197.234.240.0/22T, 198.41.128.0/17T, 162.158.0.0/15T, 104.16.0.0/13T, 104.24.0.0/14T, 172.64.0.0/13T, 131.0.72.0/22T

Is that the problem, putting the Trusted IP before ALL?

 

[mcbsys]

And now the error is back... received on the second click in a WordPress back end. This on a different server that I manage, running OLS 1.7.17 without CyberPanel:



Once again, it is showing my actual IP address (which I replaced here with 123.123.123.123):

2023-12-30 01:46:35.488569 NOTICE [1187829] [123.123.123.123] Reached per client hard connection limit: 40, current: 40, close connection!
2023-12-30 01:46:35.488597 WARN [1187829] [123.123.123.123] bot detected for vhost [N/A], reason: OverConnHardLimit, close connection!
2023-12-30 01:47:21.692917 NOTICE [1143908] [123.123.123.123] reached per client soft connection limit: 10 for 46 seconds, close connection!
2023-12-30 01:47:21.692952 WARN [1143908] [123.123.123.123] bot detected for vhost [N/A], reason: OverConnSoftLimit, close connection!

Why is it blocking based on the actual end-user IP rather than on the trusted Cloudflare IPs? What is the proper configuration to use OLS behind Cloudflare?

 

[mcbsys]

This hit us again this morning, blocking access to a the WordPress back end at a critical time. Very frustrating!

I just found the article https://docs.litespeedtech.com/lsws/cp/cpanel/antiddos/ which indicates that using the client IP (and not whitelisted CDN IPs) for throttling is expected behavior and recommends raising the connection limits to 100000/150000.

"If you use CDN services with real visitor IP enabled, the IP which is forwarded from the CDN may get blocked if the soft or hard limit is too low. One way to disable such blocking or per-client throttling is to set the connection limits to very large numbers, such as 100000 and 150000, respectively." (I assume that the other method, using an Apache DisableForwardedIpBan directive, would not work in OLS.)

It seems odd that LiteSpeed intentionally ignores CDN whitelist, but if that is the case, please update the OpenLiteSpeed documentation as well: https://openlitespeed.org/kb/per-client-throttling/. From the screen shot in the first article, it looks like the correct configuration behind a CDN is

 

[Cold-Egg]

From my test, the whitelist works, maybe you can submit the issue to support@litespeedtech.com for further support.
The connection set to 100000/150000 is similar to disabling the feature.

 

[mcbsys]

When you test it, do you have Use Client IP in Header is set to Trusted IP Only?

https://openlitespeed.org/kb/show-real-visitor-ip-instead-of-cloudflare-ips/

I use this so we can see "real" IPs in logs etc. but it seems it is also used for throttling, which is unexpected.

 

[Cold-Egg]

Yes, I did set it to the Trusted IP Only.

 

[mcbsys]

Okay I emailed them.