Re: IP address blocked on certain site

[cmeclax-sazri <cmeclax-sazri@xxxxxxxxxxxxxxxx>]

On Friday 04 February 2011 13:38:14 Joe Btfsplk wrote:
> No ideas yet on what "automated software that doesn't follow /robots.txt
> is forbidden," means?

robots.txt is a file put on some websites as a directive to robots. If you run 
a wiki, and you want only current versions, not the hundreds of previous 
versions of every page, indexed, you could put a directive in robots.txt, or 
label the pages themselves as "noindex nofollow". Automated software that 
ignores such directives is likely to eat up huge amounts of bandwidth and 
create copies that are many times bigger than the original.

cmeclax
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxxx with
unsubscribe or-talk    in the body. http://archives.seul.org/or/talk/