.Google's John Mueller answered a question about why Google indexes web pages that are actually disallowed from crawling through robots.txt and why the it's safe to ignore the relevant Explore Console files regarding those crawls.Robot Website Traffic To Inquiry Specification URLs.The person talking to the concern recorded that bots were actually producing web links to non-existent query guideline URLs (? q= xyz) to web pages along with noindex meta tags that are actually also blocked out in robots.txt. What triggered the inquiry is that Google.com is crawling the hyperlinks to those pages, acquiring obstructed by robots.txt (without seeing a noindex robotics meta tag) at that point obtaining reported in Google Look Console as "Indexed, though obstructed through robots.txt.".The individual asked the adhering to concern:." Yet listed below is actually the big inquiry: why would Google.com index web pages when they can't also see the content? What's the conveniences during that?".Google.com's John Mueller confirmed that if they can not crawl the page they can't observe the noindex meta tag. He additionally makes an appealing acknowledgment of the web site: hunt driver, urging to overlook the results considering that the "common" users won't view those results.He wrote:." Yes, you are actually proper: if our team can not creep the webpage, our company can't view the noindex. That stated, if our company can't crawl the webpages, at that point there's certainly not a lot for our team to mark. So while you may see some of those pages along with a targeted web site:- query, the common customer won't see all of them, so I would not bother it. Noindex is actually likewise great (without robots.txt disallow), it only suggests the Links will certainly end up being crept (as well as wind up in the Explore Console file for crawled/not indexed-- neither of these standings result in problems to the rest of the internet site). The essential part is actually that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the limits being used the Web site: search progressed search operator for analysis causes. One of those reasons is given that it's certainly not attached to the regular hunt index, it is actually a separate factor entirely.Google's John Mueller discussed the website hunt operator in 2021:." The short answer is actually that a web site: concern is actually not implied to be total, neither utilized for diagnostics objectives.An internet site concern is a specific sort of search that confines the outcomes to a particular web site. It's essentially merely words web site, a bowel, and then the web site's domain name.This question limits the results to a particular internet site. It's not suggested to become an extensive collection of all the web pages coming from that site.".2. Noindex tag without making use of a robots.txt is great for these sort of conditions where a robot is linking to non-existent web pages that are obtaining discovered by Googlebot.3. URLs along with the noindex tag will generate a "crawled/not catalogued" entry in Explore Console and also those will not have an unfavorable effect on the remainder of the site.Go through the inquiry and also respond to on LinkedIn:.Why will Google.com mark web pages when they can't even observe the content?Featured Picture through Shutterstock/Krakenimages. com.