Google wants to formalize the robots exclusion protocol specification

Google has announced in a blog post that they want to formalize the robots exclusion protocol specification. "For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical...