Alternative to LiteSpeed Cache Warmup Crawler

#1
This topic is about and for everyone who is using LiteSpeed LScache.

These users know, that it is important to always keep cache warmuped, but the problem is that all build-in crawlers from LiteSpeed to warmup the cache are not only sloww. They takea a long time until all URLs are crawled and cached. Furthermore too much resources are wasted and causes high load. Most of the time script time out prevents a complete crawl session if there are too much URLs that have to be crawled. The bash script version to be ran on CLI would be solve it, but it is not flexible and it is difficult to customize.

I call me an LiteSpeed expert ;) and worked on and with LiteSpeed since almost 10 years. I have developed a couple of cache plugins, so I think I know what I do. :) The big guys and nice girls from LiteSpeed team should know me. :) I follow many discusssions about LiteSpeed on different places and I noted a high need for an alternative. That's why I developed a crawler that can do the same as the existing, but faster and customized for the need of a specific application.

Faster means x-time faster. The current version for Wordpress crawls 1000 URLs within around 60 seconds without high load on shared hosting. Customized means the crawler respects settings in WP cache plugin for Mobile, Guest Mode and webp usage, so for every case my crawler crawls the right cache vary. My crawler is specialy made for users that don't have unlimited server resources, but want to get the maximum of LScache. Therefore I also made it to split a crawl session if there are too much URLs to be crawled and server limits for database connections time out prevents to open long sessions. This is not comparible with PHP max_execution time out. Special settings in my crawler makes script execution available for almost endless script execution.

My crawler is also made for non experts. That means, there is a control panel where all settings can be adjusted. No need for editing any configuration files. The current version only needs to be executed from CLI as cron. A second release will come soon and can be ran within control panel without almost any difference to CLI version. Both are fast, really fast!!

I have a log file attached where you can see how fast it is. Compare the time for the first and with the last request. You can also access to a demo installation of this crawler:

https://www.wordpress.litecache.de/litecache/
User: Demo
Pass: Demo

Enjoy ;)
 

Attachments

nvr

New Member
#3
Hey, your demo link is not working.
For how many websites can we use this script? Will be called via cron?
I suppose it is not a WordPress module right?
Thanks, waiting for your reply. All the best!
 
#4
Last edited:

nvr

New Member
#5
Thanks, I have filtered for Cache status: hit, but I couldn't see any results.
Can you check?
Also, I have found your website in another post signature. I have tried the free version and I have 2 questions:
1. Wasn't easier to include global $wpdb; instead of asking for DB user and pass?
2. Also in my case I got no hit even if the logs said that was crawled.
Thanks
 
#6
Thanks, I have filtered for Cache status: hit, but I couldn't see any results.
There is nothing wrong and correspondens with LScache behaviour. The view for cache inventory is the report of the last crawl session and is generated each time after a crawl session. After each request of a URL the crawler fetches the current cache status, but LiteSpeed response with a miss header if a URL isn't cached although the page has been cached. That may be irritating, but correspondens with LScache. If something goes wrong or a page couldn't be cached the cache header isn't miss and you see no-cache or caching fails in cache inventory.

1. Wasn't easier to include global $wpdb; instead of asking for DB user and pass?
Never ever, if we would do that our crawler would be as slow as LSCWP crawler.

2. Also in my case I got no hit even if the logs said that was crawled.
Read above
 
#8
We updated demo section. Now for the current version there are demos available for:

To access to these demos, login with:
User: Demo
Pass: Demo

Be aware that Demo account has restrictions, so you can't exexcute all features.
 
#11
Hi

I don't understand why I can't pass the tester, it kept saying wordpress and woocommerce is not detected. I dowloaded the wordpress/woocommerce version.
I installed the lscwpwc_tester.php under my public_html folder where wp-admin, wp-includes, and wp-content directories are located.

I ignored the tester.php results and go ahead with the application installation. When I click "start", it takes me to another URL that looks like this with a 404 error:
"https://www.sampledomain.com/home/milavgpi/domains/sampledomain.com/public_html/kitt/install/"

The attached screen shot and above URL has been edited with a different domain name for privacy reason.

What seems to be wrong here?
What is this file version_lscwpwc_1.51 file for inside the downloaded zip file?
 

Attachments

#12
@Spriteboy09

Please contact us cachecrawler.com. This forum is for OLS support.

What is this file version_lscwpwc_1.51 file for inside the downloaded zip file?
The only purpose of this empty file is to show you which version it is, otherwise you would have to install the software to find out the version number.

I ignored the tester.php results and go ahead with the application installation. When I click "start", it takes me to another URL that looks like this with a 404 error:
This requirement test also has its purpose and is not intended to uselessly waste your time. If you can find out immediately whether your server meets the necessary requirements with a simple call to the tester, you may save yourself time and effort if the test fails. In addition, you obviously only partially loaded the installation package onto the server. If the crawler is not installed, you will be automatically redirected to the installer, but you have not uploaded this directory,

I don't understand why I can't pass the tester, it kept saying wordpress and woocommerce is not detected. I dowloaded the wordpress/woocommerce version.
In your case, there can only be 2 reasons for this. You have either tried to install the crawler in a subdirectory, e.g. https://www.domain.com/subdirectory/wordpress_dir/ or|and you are using a server control panel, such as DirectAdmin, which uses private_html as well as public_html. But that is very easy to find out. Please contact the Cachecrawler Support.

FYI: You didn't use the current version!
 
Last edited:
Top