Cloudflare issue with LSCache crawler

[rsmith4321]

I'm using Cloudflare with the APO features. I've noticed if I'm in developer mode bypassing the Cloudflare cache, the LSCache crawler works fine. However if Cloudflare cache is enabled, there are lots of pages sent to the blacklist. I've checked the firewall rules in Cloudflare to see if something is triggering during the crawling, but I can't find anything. Does anyone have any ideas why the crawler would have issues with Cloudflare and some possible solutions? Thanks!

 

[raaec]

Hi,
I will try to reproduce the issue from my side and see if I can come up with a solution, and see what could be causing the problem. In the meantime, you could provide us with your domain and Report number? Also, you don't see anything in the firewall logs of cloudflare that could hint at what is causing the problem?

 

[rsmith4321]

Thanks for the help. I actually believe I figured out what was happening. I was using the automatic page optimization feature which is an additinal layer of page caching. So when the crawler would hit pages that were cached previously by Cloudflare, the headers would be incorrect and cause issues with the crawler, at least that is what I assume. After doing a full purge of my Cloudflare cache the crawler no longer puts pages in the blacklist. I'm actually thinking this might be nice functionality. The crawler will preload the Cloudflare cache and if the page has dropped of its cache then it will hit the LSCache directly on the server.

 

[ansys]

Thanks for the help. I actually believe I figured out what was happening. I was using the automatic page optimization feature which is an additinal layer of page caching. So when the crawler would hit pages that were cached previously by Cloudflare, the headers would be incorrect and cause issues with the crawler, at least that is what I assume. After doing a full purge of my Cloudflare cache the crawler no longer puts pages in the blacklist. I'm actually thinking this might be nice functionality. The crawler will preload the Cloudflare cache and if the page has dropped of its cache then it will hit the LSCache directly on the server.

when cache simply use one system .. disable litespeed cache if you prefer use cloudflare APO

 

[rsmith4321]

when cache simply use one system .. disable litespeed cache if you prefer use cloudflare APO

Based on what info? Cloudflare is a first caching layer, but it dumps the page cache pretty quickly and then hits the server. All the cache misses hit the Litespeed cache and it responds quickly without having to hit database and php. I've been using cloudflare for years with Nginx cache, I don't see why Litespeed cache can't work just as well. If you give me an actual reason that would be helpful.

 

[ansys]

Based on what info? Cloudflare is a first caching layer, but it dumps the page cache pretty quickly and then hits the server. All the cache misses hit the Litespeed cache and it responds quickly without having to hit database and php. I've been using cloudflare for years with Nginx cache, I don't see why Litespeed cache can't work just as well. If you give me an actual reason that would be helpful.

I'm not expert and I just tell you my experience using cloudflare FREE plan which is limiting page rules to 3 only + litespeed ... I wish you get solution and please share it with us.

Thank you

 

[raaec]

Hi,
I am not certain if there is a special setting for Nginx that allows compatibility with both Nginx cache and APO. For litespeed, we don't recommend having more than one caching solution at a time; duplicating functionality at best bogs down your server, and at worst breaks your site. APO is a page cache, so it must be turned off in order for litespeed cache to work correctly. It seems that you have had some sort of success with APO and Litespeed running simultaneously, which may be the case for certain websites, particularly if your site is not changed frequently. However, in most cases this will not work, nor is it recommended. This is actually the case for most caching solutions, not just litespeed. If your ultimate goal is greater optimization with OLS, I would recommend utilizing QUIC.Cloud CDN. You get a fair amount of free quota given that your site runs OLS, and it can be set up to serve cloudflare static cache