[Referrer-Domain] Microsoft Search Engine Spiders are blocked!

[HKPhysicist]

My sites have joined MS Clarity and MS search spiders began to crawl my sites frequently. nginxrepeatoffender began to jail their IPs.
Their general domain names are:
msnbot-xxx-xxx-xxx-xxx.search.msn.com

How do I whitelist them? In where xxx-xxx-xxx-xxx is a general IPv4 IP address.

[mitchellkrogza]

Please post some log line examples

[HKPhysicist]

Hello,
Here are some IPs from .search.msn.com which I saw today on fail2ban log files. They are not the same every time.

2024-03-30 21:48:43,595 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 40.77.167.41 - 2024-03-30 21:48:43
2024-03-30 21:53:24,688 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 52.167.144.20 - 2024-03-30 21:53:24
2024-03-30 23:12:03,360 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 52.167.144.20 - 2024-03-30 23:12:02
2024-03-30 23:12:51,766 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 52.167.144.20 - 2024-03-30 23:12:51
2024-03-30 23:12:52,299 fail2ban.actions [743]: NOTICE [nginxrepeatoffender] Ban 52.167.144.20

[HKPhysicist]

any one may try this regex for MS Clarity and MS Search but I am not sure it is 100% correct.

"~*(?:\b)msnbot-'\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(-|$)){4}\b'.search.msn.com(?:\b)" 0;

[HKPhysicist]

another one gives these to check IPv4
^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$

[HKPhysicist]

MS Search IPs are still banned today!!!

[mitchellkrogza]

Hello, Here are some IPs from .search.msn.com which I saw today on fail2ban log files. They are not the same every time.

2024-03-30 21:48:43,595 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 40.77.167.41 - 2024-03-30 21:48:43 2024-03-30 21:53:24,688 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 52.167.144.20 - 2024-03-30 21:53:24 2024-03-30 23:12:03,360 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 52.167.144.20 - 2024-03-30 23:12:02 2024-03-30 23:12:51,766 fail2ban.filter [743]: INFO [nginxrepeatoffender] Found 52.167.144.20 - 2024-03-30 23:12:51 2024-03-30 23:12:52,299 fail2ban.actions [743]: NOTICE [nginxrepeatoffender] Ban 52.167.144.20

I need to see your web access log not the fail2ban log

[HKPhysicist]

I need to see your web access log not the fail2ban

Hello, 2 more records from today.
40.77.167.1 - - [16/Sep/2024:19:59:12 +0800] "GET /search?c=37&l=35070 HTTP/2.0" 200 15725 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

40.77.167.50 - - [16/Sep/2024:19:59:41 +0800] "GET /search?l=20386&distance=300&c=1 HTTP/2.0" 200 16164 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

[mitchellkrogza]

20 more new records containing bingbot from today's log.

40.77.167.50 - - [16/Sep/2024:19:57:11 +0800] "GET /search?c=30&l=11890 HTTP/2.0" 200 15228 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.1 - - [16/Sep/2024:19:59:12 +0800] "GET /search?c=37&l=35070 HTTP/2.0" 200 15725 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.50 - - [16/Sep/2024:19:59:41 +0800] "GET /search?l=20386&distance=300&c=1 HTTP/2.0" 200 16164 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:02:41 +0800] "GET /search?l=46789&c=30&sc=31 HTTP/2.0" 404 3328 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.1 - - [16/Sep/2024:20:05:04 +0800] "GET /search?c=37&l=42770 HTTP/2.0" 200 15688 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:05:21 +0800] "GET /search?orderBy=date&c=114&l=26071&sc=121 HTTP/2.0" 200 15263 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 207.46.13.54 - - [16/Sep/2024:20:06:05 +0800] "GET /search?c=37&l=20521 HTTP/2.0" 200 15633 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:06:32 +0800] "GET /search?c=54&l=2596&distance=0 HTTP/2.0" 200 15198 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:06:34 +0800] "GET /search?l=46688&c=30&sc=34 HTTP/2.0" 404 3330 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 207.46.13.54 - - [16/Sep/2024:20:06:45 +0800] "GET /search?l=43079&orderBy=rating&c=62 HTTP/2.0" 200 15245 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.198 - - [16/Sep/2024:20:07:16 +0800] "GET /search?c=97&sc=113&l=7835 HTTP/2.0" 200 15365 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.198 - - [16/Sep/2024:20:07:17 +0800] "GET /search?l=46717&distance=500&c=114 HTTP/2.0" 404 3326 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:07:22 +0800] "GET /search?l=8224&c=97&sc=109 HTTP/2.0" 200 15365 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.198 - - [16/Sep/2024:20:08:05 +0800] "GET /search?c=62&sc=67&l=1279 HTTP/2.0" 200 16202 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.198 - - [16/Sep/2024:20:08:08 +0800] "GET /search?l=43079&distance=0&c=14 HTTP/2.0" 200 15993 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 207.46.13.54 - - [16/Sep/2024:20:08:10 +0800] "GET /search?c=73&l=2596&distance=500 HTTP/2.0" 200 15784 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:08:32 +0800] "GET /search?orderBy=date&c=114&l=26151&sc=121 HTTP/2.0" 200 15257 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:10:52 +0800] "GET /search?l=3889&c=97&sc=112 HTTP/2.0" 200 15380 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.173 - - [16/Sep/2024:20:10:53 +0800] "GET /search?c=62&l=46722 HTTP/2.0" 404 3329 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 52.167.144.198 - - [16/Sep/2024:20:11:07 +0800] "GET /search?l=6231&c=97&sc=109 HTTP/2.0" 200 15365 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

Bing Webmaster Tools

According to this log all the bingbots are getting a 200 OK and some are getting a 404 NOT FOUND - so according to this they are not being blocked unless they got a 444 error (default)

Bing Webmaster Tools

[HKPhysicist]

According to this log all the bingbots are getting a 200 OK and some are getting a 404 NOT FOUND - so according to this they are not being blocked unless they got a 444 error (default)

I see. Thanks for your reply.
I will report again when there is bingbot 444 code.

[HKPhysicist]

New MS Search 444 access today. Please advise.
40.77.167.55 - - [05/Oct/2024:00:34:13 +0800] "GET /globe-valves-dealers-in-kolkata/ HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.55 - - [05/Oct/2024:01:24:58 +0800] "GET /search?l=547&orderBy=distance&c=97 HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

[HKPhysicist]

This software is quite nice to web masters.

I recommend this to all web masters. Everybody should donate and keep it running. ^^

[hainguyen-cmd]

Today, I have checked and realized that bingbot was blocked by 444 error code.

The access_log sample is:
185.172.149.104 - - [10/Jul/2025:10:05:06 +0700] "GET /physician/shelby-township-mi/tel:5865663736 HTTP/1.1" 404 548 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.517 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:06 +0700] "GET /physician/bronx-ny/peter-m-bulow-493840.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:06 +0700] "GET /physician/texas-city-tx/jacklyn-morgan-2658991.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:06 +0700] "GET /physician/costa-mesa-ca/jeffrey-felgner-2122495.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:06 +0700] "GET /physician/williamsville-ny/paul-b-wirth-100640.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:35 +0700] "GET /physician/rochester-ny/vanessa-synowski-2125113.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:35 +0700] "GET /physician/richmond-va/augustine-w-lewis-284988.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:35 +0700] "GET /physician/albuquerque-nm/pamela-j-costello-489531.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:35 +0700] "GET /physician/miami-fl/francisco-pernas-1009909.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:36 +0700] "GET /physician/oakland-md/sarah-meredith-call-1313739.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:36 +0700] "GET /physician/los-angeles-ca/haroutun-v-abrahamian-1031831.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" 40.77.167.60 - - [10/Jul/2025:10:05:36 +0700] "GET /supplier/merrillville-in/wal-mart-stores-east-lp-24327.html HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

I have ran auto-updating daily for nginx-bad-bot-blocker!

[mitchellkrogza]

The 40.77.167.X range is explicitly whitelisted and so is bingbot

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/_generator_lists/bing-ip-ranges.list
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/_generator_lists/good-user-agents.list

A test against my own site(s) allows that User Agent through without issue

Test with

curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36" -I https://yourdomain.com

It's something else that's being detected

GitHub

nginx-ultimate-bad-bot-blocker/_generator_lists/bing-ip-ranges.list at master · mitchellkrogza/nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...

GitHub

nginx-ultimate-bad-bot-blocker/_generator_lists/good-user-agents.list at master · mitchellkrogza/nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...