Bots. Bots. Bots. PLEASE EXPLAIN !!

[MichaelB]

OK, so I'm not a Pauline Hanson fan, and tbh, her voice grates, but at least she is better than that other lazy, greedy and shonky Queenslander Clive.

Oh, back on topic.

It's been mentioned before that Christopher has had to battle bots doing all sorts of nasty things and creating problems for us keen cyclists (but computer dullards) wanting to share news of the world, celebrate funny things and argue about meaningless things and opinions, but can anyone explain in plain English, what the hell these 'bots' are meant to achieve for whomever they do their dirty work for ?

It sounds like they create nothing but hassles for the good people like the Christopher's of this world.

Help !!

[RonK]

https://www.robotstxt.org/

[MichaelB]

https://www.robotstxt.org/

Thanks, but still a bit too 'nerd level' for me. Toldya I'm a computer dullard.

[g-boaf]

https://www.robotstxt.org/

Thanks, but still a bit too 'nerd level' for me. Toldya I'm a computer dullard.

The problem with them is that they pretty aggressively index a whole site (aka hammer it) and this drains the server resources completely so that actual valid requests get slowed down or the whole thing falls over in a heap because it cannot cope.

Robots.txt is a way of barring them from indexing a site. See the following discussions:

https://softwareengineering.stackexchan ... e-crawling
https://stackoverflow.com/questions/999 ... robots-txt

You do want your site indexed, but you don't want the crawlers to bring it down.

[ironhanglider]

I just picture Chris, alone with just his light sabre in hand, as the droid army advances like an incoming tide.

Cheers,

Cameron

[AUbicycles]

The comic book version is with the light sabre, but on this end it is watching graphs like the following and trying to determine who is hitting the site and how to block them.



Sometimes it is the blocked bots who still flood the site. Often there are waves when they suddenly start and it gets bad when multiple bots come at the same time. As there are tens of thousands of documented pages, there is a lot of material.

The robots.txt is a 'recommendation' and even Bing (Microsoft) don't abide by the rules. In fact, they were aggressively crawling (despite being instructed to crawl gently) and when they were blocked, the aggressive crawling continued. For some bots, they are just badly programmed and in one case I was in contact with the author who simply didn't realise it... which is a unbelievable.

This site has the ability to ramp-up when there is a load but still has limited resources so other big sites can solve it by throwing a lot of server power at it.

[Zippy7]

and in one case I was in contact with the author who simply didn't realise it... which is a unbelievable.

I prefer to believe that the author DID know, but simply does not care, because the problem doesn't really impact the author. The problem needs to be dealt with by the site and it's participants.... By the time the site is back, the bot is gone...