Quote:
Originally Posted by WiredGuy
They're doing link exploration to map out the internal links. I block them (and all other link exploration services).
WG
|
Yep, the reason why my server crashed because i had tube aggregator with millions of links and semrush bot decided to load each url several times at same time