Utilizing CSF to block all bots while allowing an exception for Google bots

A

admiral504

New Member
#1
#1
I'm inquiring about the possibility of utilizing CSF to block all bots while allowing an exception for Google bots. Google has kindly provided us with a list of whitelisted IPs at the following links:

Googlebot IPs:

https://developers.google.com/static/search/apis/ipranges/googlebot.json
https://developers.google.com/static/search/apis/ipranges/special-crawlers.json
https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json
https://www.gstatic.com/ipranges/goog.json

I've attempted to research this on various forums but haven't been able to find a solution. Most resources seem to focus on blocking specific IPs rather than implementing an exclusion for Google bots. Your insights would be greatly appreciated.
 
A

admiral504

New Member
#3
#3
Cold-Egg said:
Hope this helps https://forum.configserver.com/viewtopic.php?t=6998
Click to expand...
Thank you for your reply, but unfortunately it does not fully address my concern.

Since bots can easily spoof responses pretending to be from Google, the best solution would be to allow only requests from Google's verified IP addresses listed in their JSON metadata.

This would ensure responses are genuinely coming from Google servers. Please let me know if implementing IP allowlisting is feasible, as it would give me confidence in the authenticity of the data.
 
You must log in or register to reply here.
Top