Question for Semrush users - GoFuckYourself.com

2MuchMark · 02-19-2023, 12:51 PM

Has anyone taken a look at their server access logs after using Semrush for a while? A client of mine seems to be getting a ton of Bot traffic (several dozens of file requests per minute) all from Semrush bots. What would this be for?

Klen · 02-19-2023, 01:38 PM

My server once crashed due their bot, better to block it on firewall level.

2MuchMark · 02-19-2023, 01:50 PM

Quote:

Originally Posted by Klen

My server once crashed due their bot, better to block it on firewall level.

Thats actually what I ended up doing for him. Do you know why Semrush sends all that bot traffic? I can understand them sending some once weekly or maybe once daily, but why so much per minute?

WiredGuy · 02-19-2023, 02:17 PM

They're doing link exploration to map out the internal links. I block them (and all other link exploration services).
WG

Klen · 02-19-2023, 02:20 PM

Quote:

Originally Posted by WiredGuy

They're doing link exploration to map out the internal links. I block them (and all other link exploration services).
WG

Yep, the reason why my server crashed because i had tube aggregator with millions of links and semrush bot decided to load each url several times at same time

WiredGuy · 02-19-2023, 02:24 PM

Quote:

Originally Posted by Klen

Yep, the reason why my server crashed because i had tube aggregator with millions of links and semrush bot decided to load each url several times at same time

Add to that Majestic, ahrefs, Semrush, MOZ and everything else that exists and there's more bot traffic than real surfers... block em all

WG

bns666 · 02-19-2023, 03:26 PM

use semrush to spy on uncareful competition, block on your sites.

redwhiteandblue · 02-19-2023, 04:21 PM

Are you sure they're Semrush bots, have you looked up the IP addresses? Could be scraper bots pretending to be Semrush and they won't care about your bandwidth.

2MuchMark · 02-19-2023, 05:53 PM

Quote:

Originally Posted by redwhiteandblue

Are you sure they're Semrush bots, have you looked up the IP addresses? Could be scraper bots pretending to be Semrush and they won't care about your bandwidth.

No I didn't think of that .... good idea - thanks for the tip!

Franklin765 · 02-20-2023, 12:04 PM

And, if anything, how can they be blocked?

Klen · 02-20-2023, 12:09 PM

Quote:

Originally Posted by Franklin765

And, if anything, how can they be blocked?

By blocking ip range on firewall or HTTPD level.

02-19-2023, 12:51 PM	#1
2MuchMark Videochat Solutions Industry Role: Join Date: Aug 2004 Location: Canada Posts: 48,715	Question for Semrush users Has anyone taken a look at their server access logs after using Semrush for a while? A client of mine seems to be getting a ton of Bot traffic (several dozens of file requests per minute) all from Semrush bots. What would this be for? __________________ Custom Software \| Server Management \| Integration and Technology Solutions https://www.2much.net

02-19-2023, 02:17 PM	#4
WiredGuy Pounding Googlebot Industry Role: Join Date: Aug 2002 Location: Canada Posts: 34,456	They're doing link exploration to map out the internal links. I block them (and all other link exploration services). WG __________________ I play with Google.

02-19-2023, 03:26 PM	#7
bns666 Confirmed Fetishist Industry Role: Join Date: Mar 2005 Location: Fetishland Posts: 11,523	use semrush to spy on uncareful competition, block on your sites. __________________ ⭐ CAM SODA ⭐ STRIPCHAT ⭐ ⭐ CHATURBATE ⭐ X LOVE CAM ⭐

02-19-2023, 04:21 PM	#8
redwhiteandblue Bollocks Industry Role: Join Date: Jun 2007 Location: Bollocks Posts: 2,792	Are you sure they're Semrush bots, have you looked up the IP addresses? Could be scraper bots pretending to be Semrush and they won't care about your bandwidth. __________________ Interserver unmanaged AMD Ryzen servers from $73.00

02-19-2023, 01:38 PM	#2
Klen Industry Role: Join Date: Aug 2006 Location: Little Vienna Posts: 32,235	My server once crashed due their bot, better to block it on firewall level.

02-20-2023, 12:04 PM	#10
Franklin765 Registered User Industry Role: Join Date: Feb 2023 Posts: 6	And, if anything, how can they be blocked?