Gary Illyes of Google confirmed that Google examines domain name records for the purpose of discovering new domains to include in the Google Search Index. This was stated at the 9:40 mark of the Search Off The Record podcast from yesterday.
Gary stated, “We are examining domain records for discovery, which also entails discovering the domain name.” Since 2005, when Google became a registrar (they didn’t begin selling domains until 2014), Google will know when new domain records are created.
Gary stated that Google may or may not be aware of subdomains and subfolders, depending on how the DNS is configured. In other words, Google is occasionally able to recognize it from the CNAME records, but more often it cannot. Gary penned:
“So for example if you are broadcasting in the DNS records the CNAMEs that you set up for… the CNAMEs are a special record for… in DNS which allows you to map custom authority bits to IP addresses. So you could do subdomain.example.com maps to example.com and then the server would know the authority bit is subdomain.example.com, then serve one specific resource on the server and you can set up your DNS to advertise these CNAME records and then we might be able to discover them.
But if the DNS is set up, I would say properly, then these are not advertised. So basically, you just get the A record, which is the main record for a domain name, basically maps the domain name to the server’s IP address. And that’s how our domain names typically are set up. Basically, just the A record, when you are asking for the domain record, not all the other subdomains as well.”
Since 2004, the SEO community has believed that Google finds new domains to index using this method. Therefore, this is not exactly new; it dates back roughly 20 years.
