Shopify stores are now able to edit their robots.txt file, which gives owners more control over how search engines crawl their site.
Tobi Lutke, Shopify CEO, broke the news this evening on Twitter while adding that it might take until Monday for this feature to roll out to everyone.
All Shopify stores start with the same robots.txt, which the company says works for most sites, but now the file can be edited through the robots.txt.liquid theme template.
Site owners can make the following edits to the robots.txt file:
- Allow or disallow certain URLs from being crawled
- Add crawl-delay rules for certain crawlers
- Add extra sitemap URLs
- Block certain crawlers
Shopify recommends using Liquid to add or remove directives from the robots.txt.liquid template, as it preserves the ability to keep the file automatically updated.
How To Edit Robots.txt in Shopify
Here are the steps to customizing a Shopify store’s robots.txt file:
- From your Shopify admin, go to Online Store > Themes.
- Click Actions, and then click Edit Code.
- Click Add a new template, and then select robots.
- Click Create template.
- Make the changes that you want to make to the default template.
- Save changes to the robots.txt.liquid file in your published theme.
Alternatively, site owners can delete the contents of the template and replace it with plain text rules, though Shopify doesn’t recommend it as it won’t be automatically updated.
ThemeKit or command line changes will preserve the robots.txt.liquid file. Uploading a theme from within the Themes section of the Shopify admin will not import robots.txt.liquid.
Shopify warns that editing the robots.txt.liquid file is an unsupported customization, which means their support team can’t help with edits.
“Incorrect use of the feature can result in loss of all traffic,” cautions Shopify. Though that’s true of the robots.txt file for any site.
Related: Shopify SEO – How to Optimize Your Shopify Site for Google
Additional Resources
Here are some resources from Shopify that may assist with editing robots.txt.
First, here’s the robots.txt liquid template referenced earlier.
Shopify also provides a robots.txt customization tutorial.
SEOs are quite happy with the announcement of this update, saying it’s about time Shopify offers the ability to edit robots.txt.
The feature is rolling out right now. As mentioned, if you do not yet have access to it, you likely will by Monday.
User-agent: everyone
Allow: /
Starting today, you have complete control over how search engine bots see your store. #shopifyseohttps://t.co/Hz9Ijj5h1y— Tobi Lutke (@tobi) June 18, 2021
