Posted on



(Note: this input "Bot" screening; foreign spiders usually called Bot, the domestic general called Spider

read seven or eight articles about foreign’s blog, does not have a clear view of the BOT, some think it is the Shanghai dragon Moz Bot, some think it is a collection of articles, but everyone comments on it are very good, such as water, is portrayed as the vampire leech etc.. Interception of a foreign comment:


The following is the

No. 14 web site log into Excel screening analysis, found in the day all access log (including real users and spiders), even as many as 342 visit records. In particular AhrefsBot &

general search engines to Robots.txt file has a valid, about 2-7 days. But the rapid development of, so I have to suspect that he is a violation of Robots agreement.

to share today is how to shield bad spider.Htaccess file.

) The

, however, from the beginning of 11, the website LOG log began to appear a number of non mainstream Chinese spider visit, which has many well-known Russian search engine spider Yandexbot, and "unidentified flying objects" AhrefsBot & According to the method of thinking in shielding spider, instinctively all spiders (on Chinese website Shanghai dragon, above the spider is junk spider) through the Robots.txt file disallow. The thought that fix, but this morning to open the last 3 days of the LOG log to see garbage spiders crawl more frequent and fierce, especially why. is? So through love Shanghai to search the related records, but it is not ideal, love Shanghai without any relevant records. No way, can only resort to Google, full length is English, big head, bite and chew slowly.

a week ago, I shared an article "Shanghai dragon diagnosis: Log log to find the site through the knot", and in the final with a two improvement suggestions. Due to the limitation of objective conditions, finally using robots shield method. First take a look at how spider changes after a week, the three major spider crawl total volume dropped, robots file comes into effect. From the figure on the number of visits, the total residence time and the total amount of capture, but progress is still a long way.

Leave a Reply

Your email address will not be published. Required fields are marked *