Question for Semrush users

[2MuchMark]

Has anyone taken a look at their server access logs after using Semrush for a while? A client of mine seems to be getting a ton of Bot traffic (several dozens of file requests per minute) all from Semrush bots. What would this be for?

[Klen]

My server once crashed due their bot, better to block it on firewall level.

[2MuchMark]

My server once crashed due their bot, better to block it on firewall level.

Thats actually what I ended up doing for him. Do you know why Semrush sends all that bot traffic? I can understand them sending some once weekly or maybe once daily, but why so much per minute?

[WiredGuy]

They're doing link exploration to map out the internal links. I block them (and all other link exploration services).
WG

[Klen]

They're doing link exploration to map out the internal links. I block them (and all other link exploration services).
WG

Yep, the reason why my server crashed because i had tube aggregator with millions of links and semrush bot decided to load each url several times at same time

[WiredGuy]

Yep, the reason why my server crashed because i had tube aggregator with millions of links and semrush bot decided to load each url several times at same time

Add to that Majestic, ahrefs, Semrush, MOZ and everything else that exists and there's more bot traffic than real surfers... block em all
WG

[bns666]

use semrush to spy on uncareful competition, block on your sites.

[redwhiteandblue]

Are you sure they're Semrush bots, have you looked up the IP addresses? Could be scraper bots pretending to be Semrush and they won't care about your bandwidth.

[2MuchMark]

Are you sure they're Semrush bots, have you looked up the IP addresses? Could be scraper bots pretending to be Semrush and they won't care about your bandwidth.

No I didn't think of that .... good idea - thanks for the tip!

[Franklin765]

And, if anything, how can they be blocked?

[Klen]

And, if anything, how can they be blocked?

By blocking ip range on firewall or HTTPD level.