看了下日志，发现好多垃圾蜘蛛在爬_美国VPS

这不用上CDN了，赶紧清理掉蜘蛛，

不然2天流量就给你爬光了，

以下垃圾蜘蛛，大佬们有什么要补充的吗

放 nginx 里面就可以用了
if ($http_user_agent ~* "Go-http-client|MauiBot|serpstatbot|FeedDemon|Bytespider|YisouSpider|GrapeshotCrawler|Seekport Crawler|Indy Library|YandexBot|DotBot|AhrefsBot|CrawlDaddy|CoolpadWebkit|UniversalFeedParser|Swiftbot|ZmEu|oBot|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|SemrushBot|Ezooms|^$" ) {
return 404;
}
云南网友:good
新疆网友:马一下
黑龙江网友:SemrushBot 太流氓了，

都设置

User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /

还来爬

58.251.121.28 – – [11/Nov/2019:06:29:07 +0800] "GET /robots.txt HTTP/1.1" 404 146 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)"
58.250.143.116 – – [11/Nov/2019:06:29:12 +0800] "GET /jingyan/135 HTTP/1.1" 404 146 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)"

复制代码

陕西网友:萌新疑问开盾拦不了？ υ?(????)υ? 技术贴还顶一下

未经允许不得转载：美国VPS_搬瓦工CN2 GIA VPS » 看了下日志，发现好多垃圾蜘蛛在爬