Bye Bye bing

[AUbicycles]

Guess what - I think I finally managed to get bing to stop overloading this site.

They have, what could be described as, an unethical bot. This crawls the site and saves it for their search engine. Yes, bing is a search engine from microsoft. The problem was, it ignore the search engine instructions and furthermore would frequently overload the site. When it visited, the site slows down for everyone else as bing was going overboard.

Though I finally got a contact partner and although they wanted me to make the site vulnerable and share user data (which I would never do), some persistence and escalations meant that they finally agrees to stop targeting.

The result is that the website is for the community who wont need to compete against a naughty bot that doesn't add any value to anyone.


If this is all a bit cryptic, don't worry, it juts feels like something to be happy about.

[Arbuckle23]

Well done

[b4igo]

Nice work! Appreciate all your effort to keep the site running smoothly for us all!!

[ValleyForge]

We need more bots to protect us from the bots.

[10speedsemiracer]

Well done Chris, many thanks.

[nickobec]

Had a similar problem a couple years back, found out the hard way trying to slow bingbot down that if msnbot and bingbot find a rule specific to itself it will ignore all other rules:

So my robots.txt now start with
User-agent: msnbot
Disallow: /
User-agent: bingbot
Disallow: /

[MichaelB]

You spoke to someone !!??????

[trailgumby]

You spoke to someone !!??????

Probably a robot.

[Thoglette]

Well done. MS should have better manners.

[AUbicycles]

Bingbot was ignoring their specified crawl rates and disallows in robots.txt . Eventually I had to block their IPs but when they were still responsible for up to 85% of traffic when their waves started, even that is still a load.

[nickobec]

Bingbot was ignoring their specified crawl rates and disallows in robots.txt . Eventually I had to block their IPs but when they were still responsible for up to 85% of traffic when their waves started, even that is still a load.

My problem was trying to slow their crawl rates, with a bingbot specific line in robots.txt

Which meant bingbot actually obeyed that rule, but ignored all the other rules in robots.txt

Which meant bingbot would crash the server by accessing huge number of pages normally blocked by robots.txt

So I removed the bingbot specific lines and just blocked bingbot and msnbot with
User-agent: msnbot
Disallow: /
User-agent: bingbot
Disallow: /

Have not had any issues since, and that was years ago

[AUbicycles]

I went so far, in frustration, to sign up to their webmaster tools and follow their guidance and set up the site but even that didn’t improve.

Very specifically, I used their tools to force them to ignore specific directories and their error was (to the effect) “these pages are too valuable to bing so can’t be removed”

[eeksll]

just interested,

was this something they did on their end or did you also have to do something?

[AUbicycles]

At their end- they wanted me to open up the site and make it vulnerable to their flooding but I refused. Contact partner was fast in responding and polite.

[Jmuzz]

Doesn't phpBB recognise all the bots as logged in users, and therefore let you assign them restricted permissions?
Or have they gone and defeated that by going stealth these days?

[Shred11]

Does anyone actually use Bing anyway? Once in a blue moon, I resort to trying Bing and it just throws up all sorts of rubbish search results that don't help at all. duckduckgo.com isn't a bad alternative if Google isn't giving good results.

[10speedsemiracer]

I'm over Google. Duckduckgo is my preference.

[mikesbytes]

Bing is popular in China, I'm sure there will be other demographic's where it has a client base

[AUbicycles]

phpbb by default can do some recognition, but doesn't actively identify and control or block. The other things is that bots can spoof - while bing wasn't spoofing, it wasn't ethical either.

Does anyone use it? well microsoft forced it on people and for those who couldn't bypass, it was used. But it was never really very important even though microsoft tried to compete and, as always will push it as far as they can. As Mikebytes suggests, there are probably some demographics who rely on it but Google was always a step ahead for quality results.

[kb]

I do like the use of quadtrees for mapping though.

[mikesbytes]

I suspect there are few searches done on Bing to find Australian bicycle info. A quick surf of the web has uncovered that 14% of searches in the US and Canada are done using Bing and that 85% of Bing's total searches are done by US users, however I think that statement has excluded China

https://www.aborg.com/2014/08/google-vs ... ers-using/

Most of the people using this site reside within Australia and the following page states that 98% of searches within Australia are thru Google, Yahoo and NineMSM. BUT what is underneath them? go to https://www.nine.com.au/?redirect=holdingpage and have a look at who's search it is.

BTW the remaining 2% includes specialty engines such as web wombat.

[Philistine]

I don't know if this is the right thread to bring this up, but here goes anyway....

I am a great believer in free speech, which includes comments that I might make that other people might not want to hear, and comments other people might make that I don't want to hear. If someone puts up a bad argument, the correct response should be to put up a better argument refuting it, rather than shutting down the person making the bad argument. I have been horrified by recent reports from the USA about Google, Facebook, etc. shadow banning and otherwise censoring people that they disagree with. I am not on Facebook and unlikely to be as long as my fundamental orifice points to the ground, but I have been using Google as a search engine. I thought it was time I voted with my feet (actually with my keyboard and mouse).

I tried Bing, then Yahoo, and found that I could not access some sites - including this one - that I use regularly. I used to type in "bna" and the forum would come up, but, when I tried this on Yahoo and Bing, I got literally dozens of sites that had the initials "bna" in their name, but which were nothing remotely to do with the cycling forum. I eventually tried Duckduckgo (horrible name, quite decent search engine), and it also gave me dozens of sites that didn't interest me, but it also gave me one which reads (paraphrasing) Google search of BNA Australian Cycling Forum, which gets me here with one extra mouse click. Several other sites that I visit are similarly labeled Google search of ***********.

Does this mean that, despite my best efforts to get rid of Google, it is refusing to go quietly? Should I bow down to my new Google overlord?

[Thoglette]

Does this mean that, despite my best efforts to get rid of Google, it is refusing to go quietly? Should I bow down to my new Google overlord?

No, you should use DuckDuckGo for the majority of searches, and use Google when nothing else works.

And the Google competitors need to stop behaving badly. For example, there's absolutely no excuse for Bing misbehaving. It's not like Microsoft is a start up with no experience in server management.bbb

[10speedsemiracer]

No, you should use DuckDuckGo for the majority of searches, and use Google when nothing else works....

+1

[mikesbytes]

The problem is that search engines are extremely heavy on the forum's resources when it gathering the information it uses within its search engine, hence why it had to be excluded. IE nothing to do with censorship