Tags
Category
Created time
Oct 23, 2022 5:23 PM
Description
Martijn Koster presented the robots.txt standard (Robots exclusion standard or Robots exclusion protocol) as part of the W3C www-talk mailing list. The rules defined in the robots.txt file are used to prevent or restrict indexing robots from accessing a website.
Era
AD
Historical date
1994/07/03
Medias

Source
https://www.livescience.com/20727-internet-history.html https://www.webdesignmuseum.org/web-design-history/yahoo-1994
🧿 NEWS
Internet history timeline: ARPANET to the World Wide Web | Live Science