nginxweb服务器 禁止垃圾搜索引擎爬虫访问网站的方法
直接修改网站配置文件，Root.txt只是防君子，意义不大：server {
listen 80;
server_name zzzyk.com www.zzzyk.com;
index index.html index.htm index.php default.html default.htm default.php;
root /home/zzzyk.com
#禁止垃圾蜘蛛抓取
if ($http_user_agent ~* (SemrushBot|GPTBot|python|Linespider|crawler|DingTalkBot|simplecrawler|ZoominfoBot|zoombot|Neevabot|coccocbot|Facebot|YandexBot|Adsbot|DotBot|Applebot|DataForSeoBot|MJ12bot|BLEXBot|trendictionbot0|trendictionbot|AhrefsBot|hubspot|opensiteexplorer|leiki|webmeup)) {
return 403;
}
access_log /home/log/zzzyk.com.log
}
代码里面的这些蜘蛛基本上都是无用蜘蛛，就像我的蜘蛛里的占最头的SemrushBot就是营销蜘蛛，还有一些是对网站没啥用的蜘蛛如Facebot、Applebot等。
您也可以根据网站日志自行配置
