Does agentic_web_search support JavaScript-rendered pages?

No. It performs a standard HTTP fetch, so it works with server-rendered HTML. Pages that require JavaScript to render their content (SPAs, React apps without SSR) will return the raw HTML shell without the dynamic content.

Does it follow redirects?

Yes. agentic_web_search follows HTTP redirects automatically and returns data for the final destination URL, including the resolved canonical URL.

Is there a rate limit or cost for using it?

No. agentic_web_search is free and has no built-in rate limit. It makes standard HTTP requests from your machine — the same as curl. Be mindful of the target site's own rate limiting policies.

Can I use it to scrape content for training data or bulk indexing?

It's designed for single-URL SEO analysis and metadata extraction, not bulk scraping. For high-volume crawling, use a dedicated crawler. Always respect robots.txt and the target site's terms of service.

Why isn't the og:image or Twitter card showing up in the output?

Some pages don't set these tags. If the field is missing from the output, it means the tag wasn't present in the HTML of the page at the time of the fetch.

docs/tools/agentic_web_search

AgenticData

Web Search & Scraper MCP Tool

CLI Tool Name: agentic_web_search

Fetches any publicly accessible URL and returns a complete SEO signal report. Designed for AI agents that need to analyze web pages, audit competitor content, or extract structured metadata — without relying on search APIs or browser automation.

Parameters

Parameter	Type	Required	Description
url	string	yes	The full URL to fetch and analyze, including the protocol (https://).

What it extracts

SEO metadata

Title, meta description, canonical URL, robots directives, word count.

Open Graph tags

og:title, og:description, og:image, og:type — for social sharing previews.

Twitter Card

twitter:card, twitter:site, twitter:creator.

Heading structure

All H1–H6 headings extracted in order — reveals content architecture.

Link breakdown

Internal vs. external link counts, nofollow links.

HTTP status

Response code — detect redirects, 404s, and server errors.

Common use cases

→Audit competitor landing pages for SEO gaps

→Verify that your own pages have correct meta tags and canonical URLs

→Extract structured content from web pages for research

→Monitor pages for changes in title, description, or heading structure

→Check that Open Graph tags are set correctly before publishing

Example output

json

{
  "url": "https://example.com/article",
  "status_code": 200,
  "seo": {
    "title": "Example Article Title",
    "meta_description": "A concise description of the page content.",
    "canonical_url": "https://example.com/article",
    "robots": "index, follow",
    "word_count": 1842
  },
  "open_graph": {
    "og:title": "Example Article Title",
    "og:description": "A concise description of the page content.",
    "og:image": "https://example.com/og-image.jpg",
    "og:type": "article"
  },
  "twitter_card": {
    "twitter:card": "summary_large_image",
    "twitter:site": "@example"
  },
  "headings": {
    "h1": ["Example Article Title"],
    "h2": ["Introduction", "Key Concepts", "Conclusion"],
    "h3": ["Subpoint A", "Subpoint B"]
  },
  "links": {
    "internal": 12,
    "external": 5,
    "nofollow": 2
  }
}

[info]

This tool makes an outbound HTTP request to the URL you provide. It respects robots.txt directives and uses a neutral User-Agent string.

Explore other AgenticStore MCP tools

→ Python Code Linter MCP Tool → Secret & Repo Scanner MCP Tool → Dependency Vulnerability Scanner MCP Tool

← dependency_audit client setup →

Frequently asked questions