Alternative to LiteSpeed Cache Warmup Crawler

#1
This topic is about and for everyone who is using LiteSpeed LScache.

These users know, that it is important to always keep cache warmuped, but the problem is that all build-in crawlers from LiteSpeed to warmup the cache are not only sloww. They takea a long time until all URLs are crawled and cached. Furthermore too much resources are wasted and causes high load. Most of the time script time out prevents a complete crawl session if there are too much URLs that have to be crawled. The bash script version to be ran on CLI would be solve it, but it is not flexible and it is difficult to customize.

I call me an LiteSpeed expert ;) and worked on and with LiteSpeed since almost 10 years. I have developed a couple of cache plugins, so I think I know what I do. :) The big guys and nice girls from LiteSpeed team should know me. :) I follow many discusssions about LiteSpeed on different places and I noted a high need for an alternative. That's why I developed a crawler that can do the same as the existing, but faster and customized for the need of a specific application.

Faster means x-time faster. The current version for Wordpress crawls 1000 URLs within around 60 seconds without high load on shared hosting. Customized means the crawler respects settings in WP cache plugin for Mobile, Guest Mode and webp usage, so for every case my crawler crawls the right cache vary. My crawler is specialy made for users that don't have unlimited server resources, but want to get the maximum of LScache. Therefore I also made it to split a crawl session if there are too much URLs to be crawled and server limits for database connections time out prevents to open long sessions. This is not comparible with PHP max_execution time out. Special settings in my crawler makes script execution available for almost endless script execution.

My crawler is also made for non experts. That means, there is a control panel where all settings can be adjusted. No need for editing any configuration files. The current version only needs to be executed from CLI as cron. A second release will come soon and can be ran within control panel without almost any difference to CLI version. Both are fast, really fast!!

I have a log file attached where you can see how fast it is. Compare the time for the first and with the last request. You can also access to a demo installation of this crawler:

https://www.wordpress.litecache.de/litecache/
User: Demo
Pass: Demo

Enjoy ;)
 

Attachments

nvr

New Member
#3
Hey, your demo link is not working.
For how many websites can we use this script? Will be called via cron?
I suppose it is not a WordPress module right?
Thanks, waiting for your reply. All the best!
 
#4
Last edited:

nvr

New Member
#5
Thanks, I have filtered for Cache status: hit, but I couldn't see any results.
Can you check?
Also, I have found your website in another post signature. I have tried the free version and I have 2 questions:
1. Wasn't easier to include global $wpdb; instead of asking for DB user and pass?
2. Also in my case I got no hit even if the logs said that was crawled.
Thanks
 
#6
Thanks, I have filtered for Cache status: hit, but I couldn't see any results.
There is nothing wrong and correspondens with LScache behaviour. The view for cache inventory is the report of the last crawl session and is generated each time after a crawl session. After each request of a URL the crawler fetches the current cache status, but LiteSpeed response with a miss header if a URL isn't cached although the page has been cached. That may be irritating, but correspondens with LScache. If something goes wrong or a page couldn't be cached the cache header isn't miss and you see no-cache or caching fails in cache inventory.

1. Wasn't easier to include global $wpdb; instead of asking for DB user and pass?
Never ever, if we would do that our crawler would be as slow as LSCWP crawler.

2. Also in my case I got no hit even if the logs said that was crawled.
Read above
 
Top