Google Sitemap and robots.txt Setup Techniques
Download as PPTX, PDF3 likes4,965 views
This document discusses sitemaps and robots.txt files. It explains that sitemaps help search engines and users navigate websites by providing a map of all pages. There are different types of sitemaps like HTML, XML, text, and special sitemaps for images, videos, news, and geo locations. The robots.txt file tells search engine crawlers which pages to index and which to ignore. It provides examples of how to write directives like User-agent, Disallow, and Allow in a robots.txt file.
Google Sitemap and robots.txt Setup Techniques
- 1. Google Site Map & Robots.txt File Nasir Uddin Shamim Co Founder DevsTeam
- 2. What The Hell A Sitemap Is? In the Visitors perspective: It gives user friendly navigation towards every corner of a site. In the Spider Bot perspective: Makes the crawling functionality better. Indexing have been much more faster and targeted.
- 3. Why You Should Have A Sitemap Each notifies the SE about each updates of your site. It helps the SE to easily find any content from the Web. Your Visitors can go through anywhere of your site as they want. You can have a clear idea where to improve more and what you are missing.
- 4. Types of Sitemaps HTML Sitemap XML Sitemap ?xml version="1.0" encoding="UTF- Code example of HTML: 8" !DOCTYPE html PUBLIC "- //W3C//DTD XHTML 1.0 Strict//EN" html lang="en" "http://www.w3.org/TR/xhtml1/DT head This is a site map head D/xhtml1-strict.dtd" body html h1 header of HTML site map xmlns="http://www.w3.org/1999/x h1 html" xml:lang="en" lang="en"> head This is a site map head p site map paragraph with body links h1 header of XHTML site map h1 body p site map paragraph with links p html body html
- 5. Types of Sitemap Text Sitemap Special Types Example of text sitemap file: Image Sitemap Video Sitemap http://www.example.com/ News Sitemap http://www.example.com/s ome-directory/ GEO Sitemap
- 6. Difference Between HTML & XML Sitemap HTML Sitemaps XML Sitemaps XML sitemaps are used for SEO. HTML sitemaps are user Makes it easier for search engine sitemaps. "spiders" to "crawl" through a HTML sitemaps enhance the look website. and feel of the website. Helps search engines to index the Communicates the overall theme content of a site. of the website. Communicates any website Organizes each section contained changes to search engines. in the website. External links are not relied on to Makes user navigation easier. index a site. Makes it easier for users to link All of the pages of a website have within the website. an opportunity to be indexed. Usually used for large websites. Helps search engine "spiders" to index a site faster.
- 7. What is robots.txt File? Robots.txt file tells the search engine spider or crawler, which Web pages or directory of your site should be indexed and which Web pages should be ignored!
- 8. Parameters To Be Used User-agent: Indicate Robots (Crawler or spider) * Indicate all robots Disallow: indicate not to crawl the selected directory or page (before writing a directory there should have a /). If there have no / before the selected directory that means it allow the page Allow: Indicate to crawl the directory # this sign is used to write a comment
- 9. A robots.txt File Example sitemap: http://www.example.com/sitemap.xml User-agent: * Disallow: /cgi-bin/ Disallow: /go/ Disallow: /wp-admin/ Disallow: /wp-includes/ User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Mediapartners-Google Allow: /
- 10. The End! Thanking You All……..!!! Shamim DevsTeam