WordPress Me Robot.txt Ka Use SEO Ke Liye Kaise Kare [Full Guide]

In this post we are going to tell you how to optimize Robot.txt file for SEO in WordPress blog? I am seeing from a few days that often the new blogger is worried that his blog is not getting indexed properly in the search engine. All bloggers should know that Robot.txt is very important part of SEO. If you misuse it, your blog may also get a penalty from the search engine. If you can use it properly, you can bring your blog to a good position in the search engine.

Wordpress Me Robot.txt Ka Use SEO Ke Liye Kaise Kare [Full Guide] 1

SEO is the #No 1 Key to success in blogging. But not all bloggers understand SEO properly and search engine optimization is not able to do it correctly. Due to which he is not able to achieve success in blogging. I have seen many such people who understood SEO very well from the beginning and also optimized it well. Due to which he became a successful blogger in a very short time. That is why if you are a new blogger then I would like to tell you one thing that if you want to become a successful blogger, first understand SEO well and then optimize using it properly. If there is no mistake in the SEO process of your blog, then gradually your blog will become famous.

I have been watching for a few days that many bloggers in social media are also talking about selling their blog. I can understand his feeling that if you work very hard and put a post in your blog and if he does not get any benefit from it, then you will be angry. But whatever my brothers do, I would like to say once again to them that the hard work does not go waste, if not today, then tomorrow you will benefit from that blog. Just don't lose your courage and keep on working regularly.

We are going to tell about Robot.txt in this post. Many new bloggers use Robot.txt in their blog incorrectly. So that his blog is not indexed in the search engine. If you also do not know about robot.txt then read this post. We will try to give full information related to this in this post.

Table of Contents

Robot.txt is a file, which is important for SEO in all blogs. In this, using some code, we tell the search engine which content of our site to crawl / index in the search engine and which content is not to be indexed. There are some files in our blog that are not for our readers and that file is indexed in the search engine by default. When the useless URLs of our blog will be more indexed in the search engine, then our blog may get penalty from the search engine. That is why we have to block it by Robot.txt to hide the useless file of our blog from search engine. The disallow: tag is used to block a file from indexing in robot.txt.
When the search engine crawl any website, first of all it checks the robot.txt file of that site which content to crawl and which content should not be crawled. All search engines (Google, Bing, Yahoo, Yandex, Ask.com etc) allow Robot.txt. If you do not want to show any post, page or image of your blog in the search engine, then you will not have to request it from the search engine, but you will have to disallow it in the robot.txt file of your blog. If we tell you in simple way, through Robot.txt we tell all the search engine which post, page, image etc. of the site. Who has to index and who does not have to index.

Friends you Readymade Be aware of that the design in this is only for show. If you want to make readymade better then you have to customize it. The same happens in our blog. The default codes are used in the robot.txt file of our blog i.e. only allow and disallow the main main files. Whereas for better SEO it is necessary to allow / disallow the smallest file. That is why if we want to make the performance of our blog much better, then we have to edit it and customize it accordingly. You can see the Robot.txt file of our blog from this link. www.BloggingHindi.com/robot.txt Similarly you can see this link robot.txt file of any wordpress blog. www.yourblog.com/robot.txt (add your blog's domain here instead of yourblog.com)

Just as tags are used in any coding language, similarly some tags are also used in robot.txt. By the way robot.txt is in txt file and some simple tags are used in it. We are telling you below about which tags are used for which work.

This is a very important tag of robot.txt. Through this tag, we tell the search engine which Kiss engine to do this. Like we use User-agent: googlebot for Google and for all search engines User-agent: * Let's use When we allow / disallow a URL or file in robot.txt, first of all we have to use the User-agent: tag. Because it only shows which search engine bot you are ordering for allow or disallow.

This tag also means a lot to the search engine. When we use disallow in a file, there will be no crawl or index in the search engine. As if we do not have to index the wp-admin file of our Blog Directory in the search engine, then we will write something like this. Remember one thing that we have to disallow the file of our blog directory. That is why we do not have to write the domain of our blog. (For example, if www.blogginghindi.com/wp-admin is there, only / wp-admin has to be written.)

User-agent: *
Disallow: / wp-admin /

You must have now understood that the way the disallow: tag is meant to not be indexed, similarly the allow tag is used to index a file in the search engine. Like if we have to index wp-content / uploads / file from our blog directory in search engine, then we will write something like this.

User-agent: *

Allow: / wp-content / uploads
Using this tag in a robot.txt file is very important. Because of this we allow the sitemap of our blog. The search engine bot indexes all the URLs of our blog's sitemap. When we allow sitemap in robot.txt, we write something like this.

User-agent: *
Sitemap: https://www.blogginghindi.com/post-sitemap.xml

Things to keep in mind while editing robot.txt.

1. You cannot use comment in robot.txt file. The way we write a comment like this in php coding. / ** comment here * / You cannot use comments like this in robot.txt.

2. Do not use unnecessary space. for example Dis allow: / wp-admin It went wrong Because Dis is different and allow different in it. His correct way would be something like this.

User-agent: *
Disallow: / wp-admin /

3. Do not break the rule of Command. Like we first User-agent: * Has to be written and then the use of Allow: or Disallow:. If you write something like this, it will be wrong.

Disallow: / wp-admin /
User-agent: *

The correct approach would be something like this.

User-agent: *
Disallow: / wp-admin /

4. If you have to allow or disallow the multi file of your blog, then you cannot write all the files in the same tag. Like what we are telling you below will be wrong.

User-agent: *
Disallow: / support / cgi-bin / images /

The correct approach would be something like this.

User-agent: *
Disallow: / support
Disallow: / cgi-bin
Disallow: / images

5. In Robot.txt capital And small Use letter properly. For example, if you have written Disallow: / Download / and downloaded in your blog directory, then the search engine will miss understanding.

How to create Custom Robot.txt for WordPress blog.

You must know that we do not have to index all the files of our wordpress blog in the search engine, but there are many such content, which have to be hidden from the search engine. If we do not hide those content from the search engine, then our blog can get the search engine which penalty. So let us tell which content of our blog we have to allow / disallow in robot.txt.

Disallow: / cgi-bin / – We have to disallow this, so that it is not indexed in the search engine. It is not useful for our readers, hence it has to be noindex in the search engine.

Disallow: / wp-admin / – You will know that from yourdomain.com/wp-admin we login to our blog. This is also not useful for our readers. So it is necessary to disallow this also.

Disallow: / archives / – It is also not useful for our visitors. Disallow this too.

Disallow: / comments / feed / – Google does not like to crawl comments in search engine. That's why disallow this too.

User-agent: Googlebot-Image
Allow: / wp-content / uploads /
– If you are a WordPress user, then perhaps you will know that when we upload image or other file in our blog, it is stored in yourdomain.com/wp-content/uploads/. You will also know that we have to index the image in Google. That is why it has to be allowed.

Disallow: / wp-content / plugins / – It is not useful for our visitors. So disallow this too.

Disallow: /readme.html – You will know that the readme file of any blog contains the blog version and some information, which is not for our visitor. Disallow this too.
Sitemap: https://www.blogginghindi.com/post-sitemap.xml –

It is very important to have this in the robot.txt file of our blog. Because this makes the search engine bot know about our sitemap file.

Copy These All Robot.txt Codes:

User-agent: *
Disallow: / cgi-bin /
Disallow: / wp-admin /
Disallow: / wp-content / plugins /
Disallow: /readme.html
Disallow: / archives /
disallow: / *? *
Disallow: *? replytocom
Disallow: / comments / feed /

User-agent: Mediapartners-Google *
Allow: /

User-agent: Googlebot-Image
Allow: / wp-content / uploads /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: *
Allow: /? Display = wide
Allow: /wp-admin/admin-ajax.php

sitemap: https://www.blogginghindi.com/sitemap.xml

Note: In the code mentioned above, replace the URL of your blog's sitemap in https://www.blogginghindi.com/sitemap.xml.

You have copied the code mentioned above. Now you have to update it in your blog. There is a very simple process for this. Before that, we would like to give you an important information about the below procedure that your blog should have a Yoast SEO plugin to follow the steps we are following below.

Step 1: First login to your blog and in the menu SEO Click on after that Tools Click on

Wordpress Me Robot.txt Ka Use SEO Ke Liye Kaise Kare [Full Guide] 2
Step 2: Now a new page will open. It will have 3 options but you File Editor Have to click on

Wordpress Me Robot.txt Ka Use SEO Ke Liye Kaise Kare [Full Guide] 3
Step 3: Now in this page Robot.txt will be written, there will be a box below it. You have to paste your copied Robot.txt code in the same box and Save Changes to Robot.txt Click on

Wordpress Me Robot.txt Ka Use SEO Ke Liye Kaise Kare [Full Guide] 4

What I hope is Robot.txt? And how to use Custom Robot.txt in WordPress. It would have been known. If you have any question related to this, then let us know by commenting. So that we will try to solve your problem. If you like the post, do share in social media.


Please enter your comment!
Please enter your name here