Alternative to LiteSpeed Cache Warmup Crawler

#1
This topic is about and for everyone who is using LiteSpeed LScache.

These users know, that it is important to always keep cache warmuped, but the problem is that all build-in crawlers from LiteSpeed to warmup the cache are not only sloww. They takea a long time until all URLs are crawled and cached. Furthermore too much resources are wasted and causes high load. Most of the time script time out prevents a complete crawl session if there are too much URLs that have to be crawled. The bash script version to be ran on CLI would be solve it, but it is not flexible and it is difficult to customize.

I call me an LiteSpeed expert ;) and worked on and with LiteSpeed since almost 10 years. I have developed a couple of cache plugins, so I think I know what I do. :) The big guys and nice girls from LiteSpeed team should know me. :) I follow many discusssions about LiteSpeed on different places and I noted a high need for an alternative. That's why I developed a crawler that can do the same as the existing, but faster and customized for the need of a specific application.

Faster means x-time faster. The current version for Wordpress crawls 1000 URLs within around 60 seconds without high load on shared hosting. Customized means the crawler respects settings in WP cache plugin for Mobile, Guest Mode and webp usage, so for every case my crawler crawls the right cache vary. My crawler is specialy made for users that don't have unlimited server resources, but want to get the maximum of LScache. Therefore I also made it to split a crawl session if there are too much URLs to be crawled and server limits for database connections time out prevents to open long sessions. This is not comparible with PHP max_execution time out. Special settings in my crawler makes script execution available for almost endless script execution.

My crawler is also made for non experts. That means, there is a control panel where all settings can be adjusted. No need for editing any configuration files. The current version only needs to be executed from CLI as cron. A second release will come soon and can be ran within control panel without almost any difference to CLI version. Both are fast, really fast!!

I have a log file attached where you can see how fast it is. Compare the time for the first and with the last request. You can also access to a demo installation of this crawler:

https://www.wordpress.litecache.de/litecache/
User: Demo
Pass: Demo

Enjoy ;)
 

Attachments

Top