I came across an interesting pattern while analyzing traffic on several WordPress setups behind OLS.
AI crawlers don’t behave like normal users:
- they rarely hit warm cache
- they trigger full PHP + plugin stack
- they generate real CPU/load
At the same time, unlike traditional search engines, they don’t send measurable traffic back.
So the old model:
crawl → index → traffic → revenue
is slowly turning into:
crawl → consume → no return
From a pure server perspective, this raises a simple question:
why should we serve expensive dynamic requests if there is no clear benefit?
I’m curious if others here are seeing similar patterns on OLS setups.
(For context, I had an interesting discussion about this here)
AI crawlers don’t behave like normal users:
- they rarely hit warm cache
- they trigger full PHP + plugin stack
- they generate real CPU/load
At the same time, unlike traditional search engines, they don’t send measurable traffic back.
So the old model:
crawl → index → traffic → revenue
is slowly turning into:
crawl → consume → no return
From a pure server perspective, this raises a simple question:
why should we serve expensive dynamic requests if there is no clear benefit?
I’m curious if others here are seeing similar patterns on OLS setups.
(For context, I had an interesting discussion about this here)
Last edited: