Archive.org no longer honors robot.txt
The other day while working on my wife’s website I realized that archive.org was making snapshots of the site when I had setup the robot.txt file to block that. Then I took a look at some other sites of mine and noticed all of them were being scraped. I’m not sure when it happen. But Continue Reading »
