# SEO Steward - Full Documentation (AI-Optimized)
# https://www.seosteward.com/llms-full.txt
# This file contains comprehensive documentation for AI assistants
# Last updated: 2026-02-01

---

# SEO Steward Documentation

## What is SEO Steward?

SEO Steward is a website monitoring platform designed specifically for protecting search engine rankings. It monitors:

1. **Uptime** - Is your site accessible? What HTTP status codes is it returning?
2. **SSL Certificates** - When does your certificate expire? Is it valid?
3. **Robots.txt** - Have your crawler instructions changed? Is Googlebot blocked?
4. **X-Robots-Tag** - Are HTTP headers blocking indexing?
5. **Content** - Is expected content present on your pages?

---

## Getting Started

### Step 1: Create an Account
Visit https://www.seosteward.com/signup to create a free account. No credit card required for the 14-day trial.

### Step 2: Create Your First Project
Projects group related monitors together. Create a project for your website or client.

### Step 3: Add a Monitor
Enter your domain (e.g., "example.com"). SEO Steward will:
- Automatically detect HTTP vs HTTPS
- Find your robots.txt file
- Start monitoring immediately

### Step 4: Configure Alerts
Set up how you want to be notified:
- Email (default)
- Slack webhook
- Discord webhook
- Microsoft Teams
- Custom webhook URL

---

## Monitors - Detailed Guide

### Creating a Monitor

**URL Input**: Just enter the domain name (e.g., "example.com"). SEO Steward automatically:
- Detects if the site uses HTTPS or HTTP
- Follows redirects to find the final URL
- Locates the robots.txt file

**Check Interval**: How often to check your site
- Free Trial: 5 minutes minimum
- Starter: 60 seconds minimum
- Pro: 30 seconds minimum
- Business/Enterprise: 15 seconds minimum

**Check Regions**: Where to monitor from
- US East (Virginia) - Default
- EU West (Netherlands)
- Both regions simultaneously

### Monitor Settings

**Check SSL Certificate**: Enable/disable SSL monitoring (default: enabled)

**Check Robots.txt**: Enable/disable robots.txt monitoring (default: enabled)

**Treat 4xx as Down**: When enabled, 4xx errors (404, 403, etc.) trigger downtime alerts (default: enabled)

**Content Match**: Verify specific text appears on your page
- "Should contain" - Alert if text is missing
- "Should NOT contain" - Alert if text appears (e.g., error messages)

---

## SSL Certificate Monitoring

SEO Steward monitors your SSL certificates and alerts you before they expire:

### Alert Thresholds
- **30 days before expiry** - Early warning
- **7 days before expiry** - Time to renew
- **3 days before expiry** - Urgent
- **1 day before expiry** - Critical

### What We Track
- Expiration date
- Days remaining
- Certificate issuer (Let's Encrypt, DigiCert, etc.)
- Certificate subject
- Valid from date
- Certificate validity

### Certificate Renewal Detection
When you renew your certificate, SEO Steward detects the new expiration date and resets alert thresholds so you receive the full alert sequence for the new certificate.

---

## Robots.txt Monitoring

### What We Monitor
- Content changes (with diff comparison)
- Missing robots.txt file
- Sitemap declarations

### Indexability Analysis
SEO Steward analyzes your robots.txt to detect:

**Site-Wide Blocking**
```
User-agent: *
Disallow: /
```
This blocks ALL crawlers and will trigger an alert.

**Googlebot Blocking**
```
User-agent: Googlebot
Disallow: /
```
This specifically blocks Google and will trigger an alert.

**Bingbot Blocking**
```
User-agent: Bingbot
Disallow: /
```
This specifically blocks Bing.

### AI Crawler Tracking

SEO Steward tracks 20+ AI crawlers:

**OpenAI**
- GPTBot (training data)
- ChatGPT-User (real-time browsing)
- OAI-SearchBot (search results)

**Anthropic**
- ClaudeBot
- Claude-Web

**Google AI**
- Google-Extended (Gemini training)

**Other AI Bots**
- PerplexityBot
- Meta-ExternalAgent
- Amazonbot
- Bytespider (ByteDance)
- CCBot (Common Crawl)
- Applebot-Extended
- Cohere-ai
- FacebookBot
- Omgilibot

---

## PageSpeed & Core Web Vitals Monitoring

SEO Steward integrates with Google PageSpeed Insights to monitor your site's performance.

### Metrics Tracked

**Largest Contentful Paint (LCP)**
- Measures loading performance
- Good: ≤ 2.5 seconds
- Needs Improvement: 2.5 - 4.0 seconds
- Poor: > 4.0 seconds

**First Input Delay (FID)**
- Measures interactivity
- Good: ≤ 100 milliseconds
- Needs Improvement: 100 - 300 milliseconds
- Poor: > 300 milliseconds

**Cumulative Layout Shift (CLS)**
- Measures visual stability
- Good: ≤ 0.1
- Needs Improvement: 0.1 - 0.25
- Poor: > 0.25

**Time to First Byte (TTFB)**
- Measures server response time
- Good: ≤ 800 milliseconds
- Needs Improvement: 800 - 1800 milliseconds
- Poor: > 1800 milliseconds

**First Contentful Paint (FCP)**
- Measures when first content appears
- Good: ≤ 1.8 seconds
- Needs Improvement: 1.8 - 3.0 seconds
- Poor: > 3.0 seconds

### Configuration

**Enable PageSpeed Monitoring**: Toggle on/off per monitor

**Check Interval**: How often to run PageSpeed checks
- Hourly
- Every 6 hours
- Daily
- Weekly

**Device**: Mobile or Desktop analysis

**Thresholds**: Set custom alert thresholds for:
- Overall score (0-100)
- LCP (milliseconds)
- CLS (score)
- FID (milliseconds)

### Use Cases

**SEO Performance Monitoring**
Core Web Vitals are a Google ranking factor. Monitor them to protect your rankings.

### Improvement Opportunities

SEO Steward extracts actionable recommendations from Google Lighthouse audits:

- **Render-blocking resources** - JavaScript and CSS that delay page rendering
- **Unused CSS/JavaScript** - Code that can be removed or deferred
- **Image optimization** - Opportunities to use modern formats (WebP, AVIF)
- **Text compression** - Enable GZIP/Brotli compression
- **Efficient cache policy** - Improve browser caching headers

Each opportunity shows estimated time savings in milliseconds.

### Reports

Core Web Vitals data is included in:
- Daily digest emails (average score, poor performers)
- Weekly summary emails (best/worst performers)
- Performance trend charts on monitor pages

**Regression Detection**
Get alerted when a deployment degrades performance.

**Competitive Benchmarking**
Track performance over time to ensure you're improving.

---

## Incident Post-Mortems

Document and learn from incidents with built-in post-mortem support.

### Why Post-Mortems Matter
- Build institutional knowledge
- Prevent repeat incidents
- Improve team communication
- Track patterns in failures

### Root Cause Categories

SEO Steward provides standardized categories:
- **DNS** - DNS resolution failures or misconfigurations
- **Server** - Origin server issues
- **CDN** - CDN or edge network problems
- **Database** - Database connectivity or performance
- **Deployment** - Issues introduced by a release
- **Third Party** - External service dependencies
- **Network** - Network connectivity issues
- **SSL** - Certificate or TLS problems
- **Configuration** - Misconfiguration errors
- **Capacity** - Resource exhaustion
- **Unknown** - Cause not determined

### Post-Mortem Fields

**Root Cause Category**: Select from standardized categories

**What Happened**: Free-form description of the incident cause

**Resolution Notes**: How the issue was resolved and any follow-up actions

### Viewing Post-Mortems
Post-mortems appear in incident details and can be filtered/searched across all monitors.

---

## Status Badges

Embed status badges in your README, documentation, or status page.

### Badge Types

**Status Badge**
Shows current status: UP (green) or DOWN (red)
```
![Status](https://api.seosteward.com/api/badge/your-monitor-id)
```

**Uptime Badge**
Shows uptime percentage over the last 30 days
```
![Uptime](https://api.seosteward.com/api/badge/your-monitor-id/uptime)
```

**PageSpeed Badge**
Shows current PageSpeed score (0-100)
```
![Speed](https://api.seosteward.com/api/badge/your-monitor-id/speed)
```

### Badge Styles

Add `?style=` parameter to customize appearance:
- `flat` (default) - Standard shields.io style
- `flat-square` - Square corners
- `for-the-badge` - Large bold style

Example:
```
![Status](https://api.seosteward.com/api/badge/id?style=for-the-badge)
```

### Custom Labels

Add `?label=` parameter to customize the badge label:
```
![Status](https://api.seosteward.com/api/badge/id?label=My%20Site)
```

### Embedding

**Markdown (GitHub, GitLab, etc.)**
```markdown
![Status](https://api.seosteward.com/api/badge/your-id)
```

**HTML**
```html
<img src="https://api.seosteward.com/api/badge/your-id" alt="Status" />
```

---

## X-Robots-Tag Monitoring

The X-Robots-Tag HTTP header can block indexing without modifying your HTML:

### Headers We Detect
```
X-Robots-Tag: noindex
X-Robots-Tag: nofollow
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: noindex
```

### Why This Matters
A server misconfiguration or CDN setting can add X-Robots-Tag headers and de-index your entire site. SEO Steward catches this immediately.

---

## Content Matching

Verify specific content appears on your pages.

### Use Cases

**Detect Error Pages**
Match type: "Should NOT contain"
Text: "An error occurred"
Result: Alert when error message appears

**Verify Key Content**
Match type: "Should contain"
Text: "Free shipping on orders over $50"
Result: Alert when promotional text is missing

**Detect Maintenance Mode**
Match type: "Should NOT contain"
Text: "under maintenance"
Result: Alert when maintenance page activates

**Detect Login Walls**
Match type: "Should NOT contain"
Text: "Please sign in to continue"
Result: Alert when content becomes gated

---

## Alerts & Notifications

### Alert Types

1. **Downtime Alert** - Site returns error or is unreachable
2. **Recovery Alert** - Site is back up after downtime
3. **SSL Expiry Warning** - Certificate expiring soon
4. **Robots.txt Change** - Content changed
5. **Indexability Alert** - Googlebot or all crawlers blocked
6. **X-Robots-Tag Alert** - noindex header detected
7. **Content Match Alert** - Expected content missing/present
8. **PageSpeed Alert** - Performance dropped below threshold

### Notification Channels

**Email** (default)
Configure per-user notification preferences.

**Slack**
Add incoming webhook URL. Customize which alerts to send.

**Discord**
Add webhook URL. Same customization as Slack.

**Microsoft Teams**
Add incoming webhook URL.

**Custom Webhook**
POST to any URL with JSON payload containing alert details.

### Alert Frequency
- Downtime: Immediate, then every 15 minutes while down
- SSL: Once per threshold crossing (30, 7, 3, 1 days)
- Robots.txt: Once per change
- Content: Once per detection

---

## Organizations & Teams

### Creating an Organization
1. Go to Settings > Organizations
2. Click "Create Organization"
3. Enter name and optional domain for auto-join

### Inviting Team Members
1. Go to organization settings
2. Click "Invite Member"
3. Enter email address
4. Member receives invitation email

### Auto-Join by Domain
Enable domain auto-join to let anyone with your company email domain join automatically.

### Permissions
All organization members can:
- View all monitors
- Create monitors
- Edit monitors
- Manage projects

---

## Contact Groups

Route alerts to specific team members.

### Creating a Contact Group
1. Go to Settings > Contact Groups
2. Click "Create Group"
3. Add email addresses
4. Choose notification preferences

### Assigning to Monitors
When creating or editing a monitor, select which contact groups should receive alerts.

### Use Cases
- "On-Call Engineers" - Downtime alerts
- "SEO Team" - Robots.txt and indexability alerts
- "Security Team" - SSL expiry alerts

---

## Status Pages

Create public status pages for your services.

### Creating a Status Page
1. Go to Dashboard > Status Pages
2. Click "Create Status Page"
3. Enter name and slug (URL path)
4. Select monitors to include

### Customization
- Custom domain support
- Logo and branding
- Custom colors
- Incident history display

### Automatic Incidents
Status pages automatically show:
- Current status (Operational, Degraded, Down)
- Recent incidents
- Uptime percentage (24h, 7d, 30d)

---

## Maintenance Windows

Suppress alerts during planned maintenance.

### Creating a Maintenance Window
1. Go to Dashboard > Maintenance
2. Click "Create Window"
3. Select monitors
4. Set start and end time
5. Add optional description

### Behavior
During maintenance windows:
- Monitoring continues (data is recorded)
- Alerts are suppressed
- Status pages show "Under Maintenance"

### Recurring Windows
Set up recurring maintenance for regular deployment windows.

---

## Webhooks & Integrations

### Webhook Payload Format

```json
{
  "type": "DOWNTIME" | "RECOVERY" | "SSL_EXPIRY" | "ROBOTS_CHANGE" | "INDEXABILITY",
  "monitor": {
    "id": "...",
    "name": "Example Site",
    "url": "https://example.com"
  },
  "timestamp": "2026-01-10T12:00:00Z",
  "details": {
    "statusCode": 500,
    "latency": 1234,
    "error": "Connection timeout"
  }
}
```

### Slack Integration
1. Create Incoming Webhook in Slack
2. Copy webhook URL
3. Add to SEO Steward webhooks
4. Configure which alerts to send

### Discord Integration
1. Create webhook in Discord channel settings
2. Copy webhook URL
3. Add to SEO Steward (same as Slack)

---

## API Reference

Base URL: `https://api.seosteward.com/api`

### Authentication
Include JWT token in Authorization header:
```
Authorization: Bearer <token>
```

### Endpoints

**GET /monitors**
List all monitors for the authenticated user.

**POST /monitors**
Create a new monitor.
```json
{
  "url": "example.com",
  "interval": 60,
  "projectId": "optional-project-id"
}
```

**GET /monitors/:id**
Get monitor details including recent checks.

**PUT /monitors/:id**
Update monitor settings.

**DELETE /monitors/:id**
Delete a monitor.

**POST /monitors/:id/start**
Start monitoring.

**POST /monitors/:id/stop**
Stop monitoring.

---

## Troubleshooting

### "Site shows as DOWN but it's working"
- Check if your server blocks our User-Agent
- Verify no firewall rules blocking our IPs
- Check if you have geographic restrictions

### "Not receiving alerts"
- Check spam folder
- Verify email in notification preferences
- Check contact group assignments
- Ensure alerts aren't suppressed by maintenance window

### "SSL not showing"
- Site must use HTTPS
- Certificate must be valid (self-signed won't work)
- Check that SSL monitoring is enabled for monitor

### "Robots.txt changes not detected"
- Verify robots.txt monitoring is enabled
- Check that file is accessible (not blocked)
- Wait for next check interval

---

## Frequently Asked Questions

**Q: How often do you check my site?**
A: Depends on your plan. From every 15 seconds (Business) to every 5 minutes (Free Trial).

**Q: Where do you check from?**
A: US East (Virginia) and/or EU West (Netherlands). You choose per monitor.

**Q: Do you follow redirects?**
A: Yes, we follow up to 5 redirects to find the final URL.

**Q: What counts as "down"?**
A: HTTP 5xx errors, connection timeouts, DNS failures. Optionally, 4xx errors if "Treat 4xx as Down" is enabled.

**Q: How fast are alerts?**
A: Typically within seconds of detection. Email delivery depends on your email provider.

**Q: Can I monitor internal/private sites?**
A: No, we can only monitor publicly accessible URLs.

**Q: Is there an API?**
A: Yes, full REST API available. See API documentation.

**Q: Can I export my data?**
A: Yes, export monitors and history via API or dashboard.

---

## Contact & Support

- **Website**: https://www.seosteward.com
- **Email**: help@seosteward.com
- **Documentation**: https://www.seosteward.com/docs
- **Status**: https://status.seosteward.com