nginx-module-bot-verifier - A search index bot verification module for NGINX
[BETA] This module has been tested on a handful of production
This is an NGINX module designed to validate actors claiming to be
search engine indexers. It is right to disable security mechanisms
for valid search engine bots to ensure your controls do not interfere
with page rankings on any of the search providers. The issue is
that the User Agent header cannot be trusted. In order to ensure you
are allowing only valid search engine indexers, you must validate
according to their published standards.
This module performs that validation and caches the results to ensure
you do not pay validation penalties on every request.
How to Install
sudo yum -y install https://extras.getpagespeed.com/release-latest.rpm
sudo yum -y install nginx-module-bot-verifier