Skip to main content

WebScoop

WebScoop is a powerful web content extraction tool that automatically extracts web page content, converts it to Markdown format, and provides screenshot functionality.

Features​

  • 🚀 Automatic extraction of main web content
  • 📝 Convert HTML content to clean Markdown format
  • 🏷️ Retrieve page title and Open Graph tags
  • 📸 Automatic webpage screenshot functionality
  • 💾 Built-in caching system for improved response time

Service Tiers​

WebScoop is available in two tiers:

Free Tier​

  • Available to all users, including non-registered users
  • Rate limited to 1 request per minute per user
  • API Endpoint: https://free.web-scoop.cobbling.ai/
  • No API key required

Subscription Tier​

  • Unlimited requests
  • 10,000 website extractions included in monthly subscription
  • Additional usage charged at $0.0001 per website
  • API Endpoint: https://web-scoop.cobbling.ai/
  • Requires API key authentication

Important Notes​

  • WebScoop cannot guarantee successful content extraction from all websites
  • Extraction success depends on the target website's structure and content access policies
  • Screenshots are retained for a maximum of 5 days before being automatically deleted
  • Please respect website terms of service and robots.txt rules

API Usage​

Free Tier - Extract Web Content​

POST https://free.web-scoop.cobbling.ai/
Content-Type: application/json

{
    "url": "https://example.com"
}

Subscription Tier - Extract Web Content​

Getting an API Key​

To access the subscription tier:

  1. Sign up for a Cobbling AI account
  2. Subscribe to a paid plan
  3. Create a project from your dashboard
  4. Generate an API key for your project
  5. Use this key in your API requests
POST https://web-scoop.cobbling.ai/
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

{
    "url": "https://example.com"
}

Response Format​

{
    "url": "https://example.com",
    "content": {
        "title": "Page Title",
        "og_tags": {
            "title": "OG Title",
            "description": "OG Description",
            "image": "OG Image URL",
            "url": "OG URL",
            "type": "OG Type",
            "site_name": "OG Site Name"
        },
        "screenshot": "Screenshot URL",
        "markdown": "Converted Markdown Content"
    }
}

Troubleshooting​

If you encounter issues with content extraction:

  • Verify that the target website allows scraping
  • Check if the website uses client-side rendering, which may limit extraction
  • Ensure you're not exceeding rate limits (for free tier)
  • For subscription users, verify your API key is valid and correctly included