The robots.txt file is a standard text file that tells search engine crawlers which pages they can access, scrape, and ultimately list in their search engine results.
The file is listed on the root URL of your site, https://domain.com/robots.txt, and can easily be added to your Django project.
Be sure to use the correct casing and spelling for anything added to the robots.txt file or it will not work correctly.
User-agent: *
Disallow: /
This file will disallow all robots.
You have two options about where to create the robots.txt file in your Open edX project:
- In your base templates of LMS at the following path,
edx-platform/lms/templates/robots.txt
- In your theme directory (if you have enabled comprehensive theming),
edx-platform/themes/my-theme/lms/templates/robots.txt
Either way is correct but it is preferable to enable comprehensive theming and follow the second option.
Go to the edx-platform/lms/urls.py
file and add the following code:
from django.views.generic.base import TemplateView
# For koa or earlier releases
# urlpatterns = [
# url(r'^robots.txt', TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
# ]
# For lilac or later releases
urlpatterns = [
path("robots.txt", TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
]
Restart your LMS service and you can see the robots.txt file at: https://domain.com/robots.txt
https://searchfacts.com/robots-txt-allow-disallow-all/
https://www.ordinarycoders.com/blog/article/robots-text-file-django