Alternative to LiteSpeed Cache Warmup Crawler

[LiteCache]

This topic is about and for everyone who is using LiteSpeed LScache.

These users know, that it is important to always keep cache warmuped, but the problem is that all build-in crawlers from LiteSpeed to warmup the cache are not only sloww. They takea a long time until all URLs are crawled and cached. Furthermore too much resources are wasted and causes high load. Most of the time script time out prevents a complete crawl session if there are too much URLs that have to be crawled. The bash script version to be ran on CLI would be solve it, but it is not flexible and it is difficult to customize.

I call me an LiteSpeed expert and worked on and with LiteSpeed since almost 10 years. I have developed a couple of cache plugins, so I think I know what I do. The big guys and nice girls from LiteSpeed team should know me. I follow many discusssions about LiteSpeed on different places and I noted a high need for an alternative. That's why I developed a crawler that can do the same as the existing, but faster and customized for the need of a specific application.

Faster means x-time faster. The current version for Wordpress crawls 1000 URLs within around 60 seconds without high load on shared hosting. Customized means the crawler respects settings in WP cache plugin for Mobile, Guest Mode and webp usage, so for every case my crawler crawls the right cache vary. My crawler is specialy made for users that don't have unlimited server resources, but want to get the maximum of LScache. Therefore I also made it to split a crawl session if there are too much URLs to be crawled and server limits for database connections time out prevents to open long sessions. This is not comparible with PHP max_execution time out. Special settings in my crawler makes script execution available for almost endless script execution.

My crawler is also made for non experts. That means, there is a control panel where all settings can be adjusted. No need for editing any configuration files. The current version only needs to be executed from CLI as cron. A second release will come soon and can be ran within control panel without almost any difference to CLI version. Both are fast, really fast!!

I have a log file attached where you can see how fast it is. Compare the time for the first and with the last request. You can also access to a demo installation of this crawler:

https://www.wordpress.litecache.de/litecache/
User: Demo
Pass: Demo

Enjoy

 

[LiteCache]

 

[nvr]

Hey, your demo link is not working.
For how many websites can we use this script? Will be called via cron?
I suppose it is not a WordPress module right?
Thanks, waiting for your reply. All the best!

 

[LiteCache]

Hey, your demo link is not working.

Sorry, link to demo has been changed to https://www.wp.cachecrawler.com/kitt/
User: Demo
Pass: Demo

For how many websites can we use this script?

There is no limitation.

Will be called via cron?

You can run the crawler via cron and from application.

I suppose it is not a WordPress module right?

Right

 

[nvr]

Thanks, I have filtered for Cache status: hit, but I couldn't see any results.
Can you check?
Also, I have found your website in another post signature. I have tried the free version and I have 2 questions:
1. Wasn't easier to include global $wpdb; instead of asking for DB user and pass?
2. Also in my case I got no hit even if the logs said that was crawled.
Thanks

 

[LiteCache]

Thanks, I have filtered for Cache status: hit, but I couldn't see any results.

There is nothing wrong and correspondens with LScache behaviour. The view for cache inventory is the report of the last crawl session and is generated each time after a crawl session. After each request of a URL the crawler fetches the current cache status, but LiteSpeed response with a miss header if a URL isn't cached although the page has been cached. That may be irritating, but correspondens with LScache. If something goes wrong or a page couldn't be cached the cache header isn't miss and you see no-cache or caching fails in cache inventory.

1. Wasn't easier to include global $wpdb; instead of asking for DB user and pass?

Never ever, if we would do that our crawler would be as slow as LSCWP crawler.

2. Also in my case I got no hit even if the logs said that was crawled.

Read above

 

[LiteCache]

@nvr

FYI: The version you see at https://www.wp.cachecrawler.com/kitt/ is not the same that can be downloaded at https://www.cachecrawler.com/Download:::25.html. There is huge difference between both version. The new version is already finished, but we can't release it for download, because the current version of LSCWP has a serious bug that exists since version 5.* or maybe longer.

 

[LiteCache]

We updated demo section. Now for the current version there are demos available for:

• Kitt Cache Crawler for LiteSpeed Cache Plugin for Wordpress
  https://www.wp.cachecrawler.com/kitt/
• Kitt Cache Crawler for W3 Total Cache
  https://www.w3tc.cachecrawler.com/kitt/
• Kitt Cache Crawler for LiteSpeed Cache Extension for OpenCart
  https://www.os.cachecrawler.com/kitt/
• Kitt Cache Crawler for LiteSpeed Cache Plugin for PrestaShop
  https://www.ps.cachecrawler.com/kitt/
• Kitt Cache Crawler for Shopware 6
  https://www.sw.cachecrawler.com/kitt/

To access to these demos, login with:
User: Demo
Pass: Demo

Be aware that Demo account has restrictions, so you can't exexcute all features.

 

[LiteCache]

An additional Kitt Cache Crawler version is available now for Shopware 6:

Demo to Kitt Cache Crawler for Shopware 6:

https://www.sw.cachecrawler.com

User: Demo
Pass: Demo

Be aware that Demo account has restrictions, so you can't exexcute all features.

 

[LiteCache]

Upcoming versions are on to do list:

- WP Rocket
- Magento

 

[Spriteboy09]

Hi

I don't understand why I can't pass the tester, it kept saying wordpress and woocommerce is not detected. I dowloaded the wordpress/woocommerce version.
I installed the lscwpwc_tester.php under my public_html folder where wp-admin, wp-includes, and wp-content directories are located.

I ignored the tester.php results and go ahead with the application installation. When I click "start", it takes me to another URL that looks like this with a 404 error:
"https://www.sampledomain.com/home/milavgpi/domains/sampledomain.com/public_html/kitt/install/"

The attached screen shot and above URL has been edited with a different domain name for privacy reason.

What seems to be wrong here?
What is this file version_lscwpwc_1.51 file for inside the downloaded zip file?

 

[LiteCache]

@Spriteboy09

Please contact us cachecrawler.com. This forum is for OLS support.

What is this file version_lscwpwc_1.51 file for inside the downloaded zip file?

The only purpose of this empty file is to show you which version it is, otherwise you would have to install the software to find out the version number.

I ignored the tester.php results and go ahead with the application installation. When I click "start", it takes me to another URL that looks like this with a 404 error:

This requirement test also has its purpose and is not intended to uselessly waste your time. If you can find out immediately whether your server meets the necessary requirements with a simple call to the tester, you may save yourself time and effort if the test fails. In addition, you obviously only partially loaded the installation package onto the server. If the crawler is not installed, you will be automatically redirected to the installer, but you have not uploaded this directory,

I don't understand why I can't pass the tester, it kept saying wordpress and woocommerce is not detected. I dowloaded the wordpress/woocommerce version.

In your case, there can only be 2 reasons for this. You have either tried to install the crawler in a subdirectory, e.g. https://www.domain.com/subdirectory/wordpress_dir/ or|and you are using a server control panel, such as DirectAdmin, which uses private_html as well as public_html. But that is very easy to find out. Please contact the Cachecrawler Support.

FYI: You didn't use the current version!

 

[LiteCache]

Are you unhappy how to warmup the cache of your applicattion?

Feel free to contact us at https//www.cachecrawler.com. We are certified official LiteSpeed Supporter and can help you in any case! LiteSpeed support

 

[LiteCache]

*** New Features *** *** New Features *** *** New Features *** *** New Features *** *** New Features ***

Since version 1.6 of the cache plugin for the LiteSpeed LScache plugin for Wordpress, there is a new function that dramatically reduces the crawl and warmup effort. While every Wordpress user benefits from it, WooCommerce users benefit even more.

What is the problem?

As is known, the cache is purged when changes are made to posts. This is particularly important when using WooCommerce, because purging inevitably means that almost the entire cache has to be purged. After all, product information is omnipresent. That's why purging the almost complete cache is a must. Otherwise, you run the risk of products being displayed with incorrect prices or stock levels. The downside is that the cache warmup, at least with the cache plugin's built-in crawler, can take a long time.

The new functions of the Kitt Cache Crawler offer a way out of this dilemma. What probably very few people know is that up to 70% of all URLs in the sitemap are either never or only very rarely requested. Therefore, the question inevitably arises, why warm up the entire cache of URls from the sitemap if this process not only takes a long time, but consumes a lot of resources? Perhaps this even disadvantages the loading time of the front end?!

The solution is:"URLs Most Wanted"

Urls Most Wanted means that only the cache of most wanted URLs is warmed up, where Most Wanted is relative. The new functions of the Kitt Cache Crawler allow you to determine whether a URL is most wanted, or in which period and how many hits a URL must have to be a most wanted URL. It may sound complicated at first, but experience has shown that the advantages of LScache can be used much faster and more effectively with this function. Above all, it is resource-friendly and at the same time more effective than the stupid crawling of the sitemap URLs.

The side effect of the Most Wanted function is that it warms up the cache of URLs that can never be warmed up by using the sitemap. This includes the URLs from the pagination, search results or from filters.

 

[pyotr]

Hello all!
I have Debian 10 with aapanel and openlitespeed and php 8.1.22 installed
Try to install this plugin - it returns

✘/proc/stat is not readable.
✘/proc/cpuinfo is not readable.

I checked permissions of this files - they are 444
What the problem is?
Please help!

 

[LiteCache]

What the problem is?

Insufficient server settings that you have changed in the meantime.

 

[albundy]

I use DirectAdmin and have private_html and public_html forward to the /magento2/pub directory from where magento is running iso /magento2 however the lcm_tester.php given me this as a result: LiteCache Requirement Tester can't be executed due to unsupported server configuration or wrong Magento 2 installation.

Hoe can i solve this?

 

[LiteCache]

Hoe can i solve this?

Create a php file, copy and paste this code in this file, upload this file to the /pub directory, run this file in browser and post what is displayed.

<?php
echo $_SERVER["DOCUMENT_ROOT"];

 

[albundy]

Create a php file, copy and paste this code in this file, upload this file to the /pub directory, run this file in browser and post what is displayed.

<?php
echo $_SERVER["DOCUMENT_ROOT"];

/home/name/domains/domainname.nl/private_html

I have changed my domain in domainname.nl for privacy reason

 

[LiteCache]

/home/name/domains/domainname.nl/private_html

I expected this answer and it confirms the failed test that DirectAdmin does not fix the well-known misconfiguration. There is a possibility to provide a workaround, although it would be better if DirectAdmin solved this problem. So please first ask DirectAdmin support if they have a solution for the problem. If DirectAdmin cannot or will not solve the problem, please contact me again, but please direct your request to LiteCache support. This forum is for OpenLiteSpeed only.