Microsoft IP spider spam

[greektranslator]

I am getting massive traffic from multiple IP ranges tracing back to Microsoft. Is there any way to ban this? It is not easy to ban on IP range as they have 100s of IPs and they keep switching. I.e.

104.208.32.219
104.208.39.117
104.255.169.115
104.43.192.116
104.43.237.62
13.67.144.189
13.67.223.205

 

[cPRex]

Hey there! You're right that there isn't an easy way to ban such diverse IP ranges. Can you get us more details about what type of traffic you're seeing from these IP addresses? Is this bot traffic to a certain site, or some other type of traffic?

 

[greektranslator]

My guess is that this is typical bot traffic scanning whole sites. And when sites have million of pages they can be very aggressive for prolonged periods of time. The increase in traffic is 4-5 times the usual. I can provide further IPs if you need, but they all traced back to Microsoft. Here is a list with the traffic (you can see the IP patterns, out of these IPs maximum 100 are real visitors)

IP	Time
100.26.127.17	11:41:16
104.208.32.128	11:41:15
104.208.32.248	11:38:42
104.208.33.158	11:43:35
104.208.33.195	11:42:07
104.208.33.58	11:40:37
104.208.34.127	11:45:23
104.208.35.208	11:43:05
104.208.37.59	11:45:41
104.208.37.83	11:45:32
104.208.38.124	11:39:54
104.208.39.104	11:44:20
104.208.39.147	11:43:49
104.28.30.74	11:32:15
104.43.163.62	11:35:21
104.43.194.8	11:45:10
104.43.195.20	11:45:52
104.43.202.164	11:32:44
104.43.203.74	11:38:06
104.43.204.217	11:44:13
104.43.206.126	11:34:27
104.43.206.245	11:38:22
104.43.234.225	11:43:54
107.178.232.162	11:32:36
107.178.232.187	11:45:04
107.178.236.26	11:32:41
109.105.245.161	11:30:59
109.178.137.214	11:34:08
109.178.151.91	11:33:28
109.178.188.241	11:35:10
109.178.204.224	11:36:54
109.178.232.6	11:45:46
109.70.100.91	11:31:35
112.120.125.194	11:44:22
13.67.150.155	11:41:17
13.67.151.44	11:45:34
13.67.151.51	11:34:15
13.67.168.92	11:37:05
13.67.168.94	11:44:18
13.67.169.199	11:41:24
13.67.174.50	11:39:15
13.67.175.122	11:36:22
13.67.175.146	11:46:29
13.67.175.75	11:34:56
13.67.177.145	11:39:51
13.67.177.185	11:38:55
13.67.177.68	11:37:59
13.67.178.123	11:35:56
13.67.178.239	11:32:17
13.67.179.62	11:39:46
13.67.182.112	11:41:22
13.67.182.155	11:42:35
13.67.182.187	11:39:26
13.67.184.124	11:35:12
13.67.185.197	11:42:45
13.67.185.50	11:42:40
13.67.185.70	11:42:34
13.67.186.230	11:35:42
13.67.187.190	11:34:37
13.67.189.138	11:34:29
13.67.192.26	11:35:40
13.67.193.106	11:39:39
13.67.194.100	11:36:40
13.67.194.84	11:38:43
13.67.198.110	11:40:03
13.67.199.17	11:36:28
13.67.199.5	11:31:51
13.67.200.189	11:38:14
13.67.200.245	11:43:12
13.67.206.224	11:38:05
13.67.206.35	11:40:43
13.67.206.60	11:33:05
13.67.209.203	11:43:28
13.67.209.211	11:43:55
13.67.211.207	11:32:41
13.67.212.171	11:45:28
13.67.212.186	11:44:23
13.67.212.21	11:46:01
13.67.212.29	11:38:38
13.67.212.78	11:39:43
13.67.213.73	11:33:07
13.67.214.157	11:36:01
13.67.214.219	11:38:27
13.67.214.223	11:42:17
13.67.216.166	11:42:38
13.67.216.98	11:45:54
13.67.217.139	11:33:02
13.67.218.158	11:37:26
13.67.218.245	11:42:00
13.67.218.81	11:34:46
13.67.219.199	11:46:31
13.67.219.219	11:44:14
13.67.220.56	11:34:45
13.67.222.111	11:33:17
13.67.222.129	11:35:41
13.67.222.33	11:36:02
13.67.223.205	11:42:33
13.67.223.211	11:34:22
13.67.227.97	11:35:50
13.67.228.86	11:40:57
13.86.65.111	11:46:56
13.86.65.183	11:40:21
13.86.65.85	11:37:45
13.86.70.202	11:37:35
13.86.71.76	11:42:06
13.89.102.122	11:45:21
13.89.102.196	11:44:32
13.89.103.139	11:35:54
13.89.129.136	11:39:02
13.89.129.26	11:40:51
13.89.134.168	11:41:52
13.89.135.104	11:33:30
13.89.135.105	11:46:11
13.89.135.118	11:42:46
13.89.200.109	11:42:59
13.89.200.190	11:36:55
13.89.200.242	11:41:53
13.89.200.61	11:36:27
13.89.200.89	11:44:16
13.89.201.115	11:45:33
13.89.202.185	11:39:17
13.89.202.222	11:41:49
13.89.202.225	11:43:21
13.89.203.116	11:40:54
13.89.203.199	11:43:28
13.89.204.35	11:47:21
13.89.205.245	11:45:24
13.89.205.252	11:42:41
13.89.205.70	11:40:32
13.89.205.78	11:39:01
13.89.206.14	11:35:31
13.89.206.77	11:39:04
13.89.207.159	11:40:29
13.89.207.210	11:38:16
13.89.207.220	11:33:04
13.89.207.58	11:47:28
13.89.207.84	11:34:30
13.89.216.194	11:45:47
13.89.216.41	11:41:47
13.89.216.94	11:44:00
13.89.218.161	11:33:51
13.89.218.85	11:42:25
13.89.219.90	11:39:35
13.89.220.189	11:40:19
13.89.221.152	11:46:04
13.89.222.241	11:33:58
13.89.222.63	11:40:56
13.89.222.91	11:34:09
13.89.223.140	11:43:07
13.89.223.157	11:36:49
13.89.223.68	11:33:16
13.89.223.87	11:46:13
13.89.224.143	11:40:31
13.89.224.213	11:36:12
13.89.224.245	11:32:54
13.89.224.75	11:43:55
13.89.225.247	11:44:26
13.89.226.187	11:41:26
13.89.227.118	11:40:42
13.89.227.82	11:37:41
13.89.228.166	11:37:14
13.89.228.167	11:47:47
13.89.242.37	11:37:01
13.89.244.244	11:34:14
13.89.245.171	11:34:00
13.89.245.199	11:40:36
13.89.245.75	11:34:54
13.89.247.32	11:46:35
13.89.247.63	11:34:33
13.89.32.206	11:38:28
13.89.33.1	11:47:07
13.89.34.34	11:46:23
13.89.34.61	11:43:02
13.89.34.94	11:36:25
13.89.37.229	11:43:29
13.89.37.30	11:34:44
13.89.38.16	11:44:49
13.89.38.213	11:47:43
13.89.38.232	11:36:33
13.89.40.118	11:42:26
13.89.44.150	11:33:55
13.89.51.185	11:36:41
13.89.51.215	11:37:02
13.89.54.125	11:42:31
13.89.72.20	11:47:35
13.89.72.47	11:38:30
13.89.78.170	11:47:12
13.89.79.53	11:33:20
13.89.96.166	11:36:42
13.89.96.224	11:37:52
13.89.97.108	11:35:10
13.89.97.69	11:37:34
138.201.248.37	11:41:12
141.255.60.209	11:40:51
144.76.22.198	11:46:47
144.76.22.199	11:46:20
144.76.22.200	11:37:48
144.76.22.201	11:46:07
144.76.22.203	11:41:10
144.76.22.204	11:47:24
144.76.22.208	11:46:06
144.76.22.212	11:43:30
147.92.153.13	11:40:28
149.210.22.75	11:34:12
157.55.39.0	11:47:56
157.55.39.195	11:47:39
160.40.10.176	11:34:57
168.61.208.125	11:46:10
168.61.208.66	11:42:02
168.61.212.122	11:46:46
173.249.196.101	11:40:17
178.147.212.238	11:43:19
178.33.169.7	11:43:57
178.59.183.99	11:39:17
185.220.100.243	11:47:59
185.226.67.169	11:41:21
185.243.218.153	11:47:49
193.92.253.106	11:41:15
195.251.4.135	11:37:18
195.97.88.187	11:44:31
2.84.123.76	11:43:10
2.85.70.115	11:36:27
2.86.74.104	11:41:32
2.87.66.193	11:41:04
20.29.68.229	11:43:04
20.29.68.230	11:39:45
20.29.69.148	11:43:31
20.29.69.58	11:46:45
20.29.69.70	11:48:35
20.29.70.118	11:43:31
20.29.70.230	11:42:23
20.29.70.25	11:43:47
20.29.71.206	11:36:30
207.46.13.181	11:48:02
207.46.13.26	11:47:19
207.46.13.92	11:48:24
212.205.83.222	11:41:21
212.251.59.110	11:43:45
213.140.134.9	11:36:55
213.140.221.71	11:44:48
213.249.31.253	11:41:54
217.165.66.37	11:44:54
23.100.83.74	11:41:10
23.100.86.44	11:33:48
23.101.120.117	11:43:34
23.101.126.177	11:45:37
23.94.96.109	11:41:08
23.99.194.80	11:43:38
23.99.196.24	11:44:43
23.99.197.21	11:47:54
23.99.209.58	11:39:10
23.99.216.220	11:42:20
23.99.217.150	11:37:33
23.99.219.195	11:36:39
23.99.219.33	11:46:24
23.99.222.153	11:38:47
23.99.222.165	11:34:24
23.99.229.132	11:36:28
23.99.230.212	11:40:44
31.152.206.233	11:46:26
35.203.252.113	11:45:09
37.120.185.177	11:48:40
37.6.1.202	11:33:38
37.6.165.79	11:37:32
37.6.67.144	11:35:51
37.6.87.162	11:47:24
40.113.222.253	11:41:41
40.122.106.203	11:48:23
40.122.106.33	11:41:50
40.122.107.52	11:44:46
40.122.111.149	11:33:52
40.122.190.133	11:46:37
40.122.201.221	11:34:13
40.122.201.224	11:40:26
40.122.201.238	11:48:02
40.122.202.209	11:40:57
40.122.203.12	11:34:17
40.122.204.212	11:41:07
40.122.205.80	11:39:27
40.122.206.245	11:40:23
40.122.224.221	11:48:09
40.122.224.252	11:38:52
40.122.225.93	11:45:19
40.122.226.0	11:36:56
40.122.226.42	11:40:53
40.122.226.85	11:45:04
40.122.227.160	11:36:02
40.122.227.231	11:44:41
40.122.228.188	11:39:53
40.122.228.49	11:47:52
40.122.230.108	11:43:58
40.122.230.166	11:38:58
40.122.231.31	11:36:05
40.122.232.55	11:38:03
40.122.232.59	11:44:29
40.122.234.126	11:42:27
40.122.234.212	11:48:44
40.122.234.49	11:48:21
40.122.235.165	11:44:57
40.122.235.182	11:37:59
40.122.237.119	11:47:34
40.122.237.16	11:38:22
40.122.237.240	11:35:43
40.122.237.244	11:47:04
40.122.238.18	11:46:47
40.122.238.81	11:46:15
40.122.241.172	11:41:50
40.122.244.217	11:38:12
40.122.245.73	11:36:43
40.122.30.180	11:35:53
40.122.31.64	11:45:16
40.122.34.115	11:49:07
40.122.34.199	11:39:58
40.122.37.199	11:38:17
40.69.130.143	11:44:29
40.69.130.146	11:40:26
40.69.130.244	11:43:40
40.69.131.116	11:45:05
40.69.131.198	11:47:47
40.69.131.5	11:35:45
40.69.132.174	11:47:29
40.69.132.19	11:39:36
40.69.132.198	11:46:32
40.69.133.70	11:39:11
40.69.133.72	11:37:11
40.69.133.73	11:42:21
40.69.133.76	11:47:01
40.69.133.86	11:46:17
40.69.133.88	11:46:26
40.69.135.107	11:34:43
40.69.135.166	11:43:53
40.69.135.67	11:41:59
40.69.136.254	11:46:26
40.69.137.14	11:48:00
40.69.137.15	11:49:44
40.69.137.205	11:44:52
40.69.139.237	11:45:21
40.69.139.80	11:36:47
40.69.140.136	11:42:03
40.69.140.137	11:41:06
40.69.140.189	11:41:24
40.69.140.194	11:39:57
40.69.140.70	11:41:32
40.69.140.74	11:46:04
40.69.141.217	11:39:12
40.69.141.7	11:43:13
40.69.142.162	11:38:36
40.69.145.95	11:38:45
40.69.146.44	11:41:38
40.69.146.45	11:43:32
40.69.146.89	11:38:06
40.69.148.28	11:35:15
40.69.149.170	11:42:41
40.69.149.171	11:49:03
40.69.149.92	11:48:27
40.69.150.140	11:39:14
40.69.150.169	11:39:29
40.69.150.224	11:37:13
40.69.151.154	11:38:38
40.69.161.75	11:39:30
40.69.162.216	11:48:42
40.69.162.88	11:35:47
40.69.163.105	11:36:12
40.69.166.25	11:45:31
40.77.100.185	11:36:59
40.77.100.186	11:49:26
40.77.100.193	11:37:04
40.77.101.221	11:39:08
40.77.101.76	11:47:14
40.77.103.173	11:37:02
40.77.121.166	11:35:22
40.77.121.37	11:36:41
40.77.122.201	11:49:39
40.77.64.225	11:38:27
40.77.65.30	11:35:41
40.77.66.110	11:43:36
40.77.66.136	11:42:09
40.77.66.4	11:45:40
40.77.67.227	11:48:33
40.77.67.48	11:46:59
40.77.68.94	11:48:18
40.77.70.11	11:46:03
40.77.70.54	11:44:06
40.77.71.169	11:48:34
40.77.81.144	11:35:35
40.77.81.209	11:43:47
40.77.96.39	11:36:32
40.77.96.81	11:43:59
40.77.96.82	11:44:21
40.77.97.134	11:45:55
40.77.97.182	11:45:55
40.77.97.244	11:48:27
40.77.98.221	11:36:45
40.77.98.6	11:47:22
40.77.99.110	11:46:44
40.78.139.76	11:50:22
40.78.144.45	11:49:53
40.78.144.63	11:44:08
40.78.145.227	11:48:51
40.78.146.224	11:48:02
40.78.147.35	11:40:24
40.78.147.57	11:40:15
40.78.151.113	11:36:42
40.78.151.50	11:44:05
40.83.10.156	11:48:47
40.83.11.113	11:45:58
40.83.11.78	11:42:54
40.83.11.8	11:38:39
40.83.14.5	11:41:44
40.83.15.169	11:41:36
40.83.15.45	11:40:47
40.83.17.245	11:40:34
40.83.54.139	11:36:11
40.83.55.169	11:49:58
40.83.55.42	11:41:39
40.83.56.127	11:48:19
40.83.6.140	11:43:15
40.83.61.48	11:45:08
40.83.63.103	11:50:11
40.83.9.212	11:42:01
40.83.9.97	11:42:44
40.86.100.180	11:41:04
40.86.100.29	11:42:28
40.86.100.45	11:50:12
40.86.100.77	11:44:02
40.86.100.80	11:45:14
40.86.101.111	11:39:49
40.86.101.179	11:45:12
40.86.102.107	11:37:57
40.86.102.221	11:49:58
40.86.102.225	11:49:50
40.86.103.103	11:49:25
40.86.103.121	11:39:00
40.86.103.122	11:48:19
40.86.115.144	11:47:35
40.86.115.38	11:37:46
40.86.116.247	11:51:02
40.86.120.181	11:50:09
40.86.121.80	11:38:41
40.86.16.67	11:49:16
40.86.19.79	11:39:01
40.86.22.180	11:47:02
40.86.23.106	11:40:50
40.86.40.141	11:42:16
40.86.40.53	11:47:41
40.86.41.69	11:37:17
40.86.46.128	11:43:29
40.86.64.58	11:42:57
40.86.71.102	11:42:33
40.86.71.77	11:37:38
40.86.80.238	11:43:53
40.86.80.70	11:42:53
40.86.81.103	11:42:47
40.86.82.240	11:41:41
40.86.83.140	11:45:07
40.86.83.37	11:38:04
40.86.97.76	11:44:57
40.86.97.96	11:45:03
40.86.98.203	11:43:20
40.86.98.81	11:46:00
40.86.98.86	11:49:42
40.86.99.110	11:38:44
40.86.99.240	11:46:56
45.139.212.20	11:44:51
45.139.212.66	11:45:07
45.139.214.114	11:42:03
45.139.215.222	11:37:15
46.176.106.111	11:38:52
46.177.157.221	11:44:36
46.177.232.144	11:38:58
46.190.81.227	11:42:59
46.199.246.172	11:48:55
46.245.141.68	11:38:19
5.135.30.215	11:37:23
5.161.133.63	11:49:00
5.203.132.200	11:45:53
5.203.156.23	11:44:07
5.203.224.234	11:38:25
5.23.146.153	11:39:58
5.255.231.204	11:45:27
5.255.253.101	11:45:25
5.255.253.104	11:46:46
5.255.253.107	11:45:28
5.255.253.113	11:45:26
5.255.253.176	11:45:29
5.45.207.113	11:45:28
5.45.207.149	11:45:31
5.54.151.67	11:39:28
5.55.138.3	11:48:35
52.165.168.134	11:39:51
52.165.168.157	11:37:57
52.165.168.166	11:42:23
52.165.185.181	11:51:41
52.165.185.228	11:44:51
52.165.190.177	11:46:17
52.165.191.154	11:51:52
52.165.216.174	11:45:29
52.165.216.218	11:43:10
52.165.218.12	11:48:16
52.165.219.34	11:38:15
52.165.220.175	11:39:41
52.165.220.36	11:50:47
52.165.222.151	11:40:01
52.165.222.224	11:44:26
52.165.222.44	11:47:41
52.165.223.13	11:49:32
52.165.32.165	11:49:11
52.165.32.185	11:48:17
52.165.32.60	11:45:59
52.165.33.104	11:51:36
52.165.33.121	11:48:46
52.165.35.225	11:51:11
52.165.37.68	11:41:22
52.165.39.243	11:41:34
52.165.45.5	11:50:35
52.165.46.177	11:49:17
52.173.144.17	11:37:23
52.173.144.81	11:37:47
52.173.146.242	11:37:21
52.173.146.251	11:51:24
52.173.147.187	11:48:03
52.173.148.185	11:45:38
52.173.148.22	11:45:14
52.173.150.104	11:50:31
52.173.150.194	11:40:59
52.173.150.219	11:38:29
52.173.151.57	11:45:39
52.173.190.59	11:37:40
52.173.24.125	11:44:54
52.173.24.130	11:50:34
52.173.24.78	11:47:21
52.173.25.57	11:48:13
52.173.26.128	11:38:53
52.173.26.143	11:47:55
52.176.40.86	11:41:29
62.1.170.116	11:45:19
62.1.187.21	11:42:13
62.228.20.17	11:39:50
62.74.228.38	11:40:18
66.102.6.84	11:45:04
66.249.66.40	11:52:28
66.249.66.43	11:47:01
66.249.66.46	11:52:09
66.249.76.83	11:42:53
66.249.84.112	11:45:09
66.249.92.38	11:50:01
78.87.235.232	11:50:04
79.107.180.124	11:50:24
79.107.42.3	11:49:16
79.107.79.73	11:43:30
79.129.231.69	11:45:02
79.131.154.97	11:47:33
79.131.240.229	11:48:22
80.106.87.123	11:51:47
80.107.61.51	11:40:41
80.77.122.119	11:51:41
83.168.35.189	11:40:06
84.205.231.43	11:45:12
84.205.244.136	11:47:14
84.254.53.45	11:43:44
85.72.34.143	11:42:44
85.72.56.223	11:50:07
85.74.159.187	11:51:22
85.75.213.229	11:48:41
87.202.182.116	11:48:57
87.250.224.132	11:45:30
87.250.224.163	11:45:23
87.250.224.39	11:45:21
87.250.224.61	11:45:31
88.162.226.118	11:52:17
94.71.27.199	11:49:16
94.71.4.119	11:49:31
95.108.213.11	11:45:24
95.108.213.27	11:45:30
95.108.213.33	11:45:26
95.217.144.248	11:46:48

 

[cPRex]

Have you tried a robots.txt file to block the msnbot user-agent to see if that helps?

 

[greektranslator]

Yes, also I did that on .htaccess. But most of them simply spoof browser user agents, so it does not work.

Another example (not sure if this is Microsoft or just someone using Microsoft hosting or whatever):

ISP: MICROSOFT-CORP-MSN-AS-BLOCK

13.67.200.189 IP Location: Des Moines, United States US

IP 13.67.200.189, Hostname (Not set. Same as IP Address): 13.67.200.189, US IP Location: city Des Moines, state Iowa, country United States

www.ip-tracker.org

 

[cPRex]

That's the only thoughts I have on my end. Does your host possibly have an external firewall solution they could implement that has more advanced network filtering?

 

[greektranslator]

I have OVH bare metal with automatic mitigation turned on. Switching to permanent had no effect. As I can see enabling the firewall requires adding IPs which is not an option with these multiple varied ranges (if it were, I could have block them via CSF firewall).

Configuring the Network Firewall

docs.ovh.com