|
robots.txt
i want that the spiders don't look automaticaly for domain.com/robots.txt file.. i DON"T want to block spiders.. i just don't want they looking for robots.txt.. how can i do that?
also, it is possible to change the robots.txt location from meta tags? for example to put robots.txt file in domain.com/xxx/spiders.txt file and tell spiders that there is the robots file
|