From Wikipedia:
Historically, Wayback Machine respected the robots exclusion standard (robots.txt) in determining if a website would be crawled or not; or if already crawled, if its archives would be publicly viewable. Website owners had the option to opt-out of Wayback Machine through the use of robots.txt. It applied robots.txt rules retroactively; if a site blocked the Internet Archive, any previously archived pages from the domain were immediately rendered unavailable as well. In addition the Internet Archive stated, “Sometimes a website owner will contact us directly and ask us to stop crawling or archiving a site. We comply with these requests.”[39] In addition, the website says: “The Internet Archive is not interested in preserving or offering access to Web sites or other Internet documents of persons who do not want their materials in the collection.”[40] [41]
Thank You :)
Such cleanup services are called reputation defenders but actually they just whitewash the past and bury the skeletons.
This is why you use Archive.is in addition to the WayBack Machine. And if it’s really important, archive it on your own hard drive, too.
This is why you use Archive.is in addition to the WayBack Machine. And if it’s really important, archive it on your own hard drive, too.