October 27, 2021
00 00 00 AM
Latest Post
Kairi Hôjô could play iconic role in Book of Tobit film Ryan Gosling will play Ken to Margot Robbie’s Barbie in live-action Barbie movie – LaineyGossip Lorrie Bartlett could play iconic role in Book of Tobit film The best horror movie tropes, picked by horror writers and directors Petition to ban Mel Gibson from playing the role of Tobit Harold Perrineau could play iconic role in Book of Tobit film – Screenplay News Tokyo festival features Yoshida’s films of hope amid despair – Screenplay News Saskia Mulder could play iconic role in Book of Tobit film – Screenplay News Lucas Till could play iconic role in Book of Tobit film – Screenplay News Who was Mamat Khalid and what was his cause of death? Malaysian screenwriter Kematian Mamat Khalid dies – Screenplay News

Should robots.txt support a feature for no indexation? Take the survey

I saw a discussion on Twitter this morning about the idea of having a feature in Robots.txt that would block both crawling AND indexing. It started with this tweet by Christian Thurston (@ct_oz):

“Hi John [Mueller], has Google considered making it so that the robots.txt file doesn’t just block crawling, but also blocks indexation? To quote @willcritchlow: “I can’t see many situations where I want to block crawling but don’t want to block indexing”.

“That would be a significant change in expectations (and yes, we do think about these things regardless). Do you have some examples where this would cause a visible improvement in search?” Mueller responded. “I’d like to avoid adding more directives. I’m still not aware of common issues caused by this documented functionality … SEOs worry about indexing, but usually these URLs only rank for site:-queries (or if there isn’t other, better content on the site), so it feels artificial?”

With over 20 years of experience in SEO, “I have never encountered a situation where a publisher wanted to have a page indexed that they block for crawling in robots.txt. Not even once have I seen that,” I tweeted in response. “It’s common practice for me to educate people that they have a choice: (1) block crawling, or: (2) prevent indexation, when what they want to do is both. Note: definitely more of an issue for larger sites where crawl budget is an issue.”

Will Critchlow of Search Pilot agreed, “100% agree. I can see the conservatism of not changing a long-standing standard, but I have never seen, and can’t think of a situation where you’d want to block crawling but allow indexation.”

But what do YOU think? Would it be helpful to have a feature in Robots.txt that allowed you to specify the pages you don’t want to have indexed? Take our quick three-question poll below and let us know what you think.

Please keep in mind that if you vote yes for doing this that you would need to accept ALL the risk that at some point your dev team might misapply the capability and NoIndex pages that you don’t want them to.

Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.

About The Author

Eric Enge is President of Pilot Holding. Previously, Eric was the founder and CEO of Stone Temple, an award-winning digital marketing agency, which was acquired by Perficient in July 2018. He is the lead co-author of The Art of SEO, a 900+ page book that’s known in the industry as “the bible of SEO.” In 2016, Enge was awarded Search Engine Land’s Landy Award for Search Marketer of the Year, and US Search Awards Search Personality of the Year. He is a prolific writer, researcher, teacher and a sought-after keynote speaker and panelist at major industry conferences.