Psilobase is a harm-reduction information portal dedicated to evidence-based guidance on psilocybin mushrooms, covering safety, science, culture, and therapeutic applications.

Is the information on Psilobase medically reviewed?

All content on Psilobase is written with reference to peer-reviewed research and harm-reduction principles; it is for educational purposes and does not constitute medical advice.

How can I use Psilobase safely?

Use Psilobase to learn about set and setting, dosage, contraindications, and integration before making any decisions; always consult a qualified healthcare professional for personal medical guidance.

Psilobase is a harm-reduction information portal dedicated to evidence-based guidance on psilocybin mushrooms, covering safety, science, culture, and therapeutic applications.

Is the information on Psilobase medically reviewed?

All content on Psilobase is written with reference to peer-reviewed research and harm-reduction principles; it is for educational purposes and does not constitute medical advice.

How can I use Psilobase safely?

Use Psilobase to learn about set and setting, dosage, contraindications, and integration before making any decisions; always consult a qualified healthcare professional for personal medical guidance.

Robots.txt Guide | Crawl Control SEO

📄 What is Robots.txt?

📄

Text File

A plain text file at your domain root

🤖

Bot Instructions

Tells crawlers what to access or avoid

📍

Location

yourdomain.com/robots.txt

⚠️

Advisory

Suggestions, not enforced rules

📝 Robots.txt Syntax

# This is a comment

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

User-agent: Googlebot
Disallow: /no-google/

Sitemap: https://example.com/sitemap.xml

User-agent:

Specifies which bot the rules apply to

Disallow:

Blocks access to specified paths

Allow:

Permits access to paths within disallowed directories

Sitemap:

Points to your XML sitemap location

💡 Common Examples

Allow All Bots Open

Allow all crawlers to access everything.

User-agent: *
Disallow:

Block All Bots Blocked

Block all crawlers from entire site.

User-agent: *
Disallow: /

Block Specific Directory Partial

Block crawlers from admin area.

User-agent: *
Disallow: /admin/
Disallow: /login/

Block Specific Bot Targeted

Block only a specific crawler.

User-agent: BadBot
Disallow: /

User-agent: *
Disallow:

⚠️ Common Mistakes

Blocking CSS/JS

Blocking CSS and JavaScript files prevents Google from rendering your page properly.

✓ Allow access to CSS and JS files

Blocking Entire Site

Using Disallow: / blocks your entire site from indexing.

✓ Only block specific private sections

Wrong File Location

Robots.txt must be at the root domain, not in subdirectories.

✓ Place at yourdomain.com/robots.txt

Using for Security

Robots.txt is not for hiding sensitive content - it's publicly visible.

✓ Use authentication for truly private content

🤖 Common User Agents

🔵

Google

Googlebot

🔷

Bing

Bingbot

🟣

Yahoo

Slurp

🟢

DuckDuckGo

DuckDuckBot

🖼️

Google Images

Googlebot-Image

🎥

Google Video

Googlebot-Video

📰

Google News

Googlebot-News

⭐

All Bots

*

🧪 How to Test Your Robots.txt

Visit yourdomain.com/robots.txt to verify file exists

Use Google Search Console's robots.txt Tester tool

Check URL Inspection tool for crawl issues

Monitor Coverage report for blocked pages

🤖 Robots.txt Guide