Since 2012 · Online 24/7 TR Client panel

robots.txt generator

Create the file that tells search engine and AI bots which parts of your site they may crawl. Upload it to your site's root folder as robots.txt.

What platform is your site on?

Paths you don't want crawled (one per line)

Sitemap URL

Block AI training botsGPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended. ChatGPT and Perplexity search stay allowed.

Block the entire siteOnly for test or under-construction sites.

robots.txt

# robots.txt · created with the seomit.net robots.txt generator

User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.siteniz.com/sitemap_index.xml

Blocking ≠ hidingrobots.txt stops crawling, not indexing. To remove a page from Google completely, add a "noindex" tag to it and don't block it in robots.txt.

AI botsBlocking training bots limits the use of your content for model training. If you leave search bots (OAI-SearchBot, PerplexityBot) allowed, you'll keep appearing in assistants' answers.

Don't block CSS and JSGoogle renders your page like a browser. If you block theme and plugin files, it sees your site as broken.

More free tools

All tools →

Google previewContent Meta & Open GraphTechnical Schema (JSON-LD)Technical SEO-friendly URLContent Word counterContent UTM linkMarketing Google review linkLocal

Tools are just the start.Leave the backlinks and content to us: SEO packages from ₺3,000. See packages