Sifthound offers open-source Tavily alternative for AI agents

Overview

An emerging open-source project called Sifthound is positioning itself as a privacy-focused, self-hosted alternative to Tavily, a popular web search API widely used by AI agents and large language model (LLM) applications. Developed by khsarvar, the project has been made public on GitHub under the MIT license, offering developers a way to maintain control over their data while accessing the same functional endpoints found in commercial offerings.

The core value proposition of Sifthound is its ability to serve as a drop-in replacement for the Tavily API. By replicating the same /search, /extract, /crawl, and /map endpoints with identical request and response structures, the tool allows existing codebases— including those using the official Tavily Python SDK and LangChain integrations—to function without modification. The only required change is updating the base URL to point to a user’s own Sifthound server instance.

Core Capabilities and Architecture

Sifthound is designed to operate without the need for proprietary search API keys, which are often required by other commercial services. Instead, it routes searches through a SearXNG metasearch instance. This architectural choice ensures that users can perform web searches without relying on or paying for external search engine APIs, thereby reducing both cost and dependency on third-party providers.

The platform includes several key features tailored for AI agents:

• Extraction: Uses the trafilatura library to extract clean markdown or text from web pages, removing clutter and noise to provide LLMs with readable content.
• Ranking: Employs a hybrid ranking system that blends BM25 relevance scores with the order provided by the upstream search engine to ensure high-quality results.
• Answers: An optional feature that can generate concise answers directly from retrieved results using Claude, when an Anthropic API key is configured.
• Crawling and Site Maps: Supports deep crawling with configurable depth, breadth, limits, and regex filters for paths and domains.

Security is a paramount concern for any service handling web requests. Sifthound includes robust SSRF (Server-Side Request Forgery) protection. The system blocks requests to private and internal IP addresses, safeguarding against attacks that attempt to access internal network resources. This protection extends to handling redirects and DNS rebinding attempts, making it safer to run in diverse deployment environments.

Integration with AI Ecosystems

One of the most significant differentiators for Sifthound is its native support for the Model Context Protocol (MCP). As MCP gains traction as a standard for connecting AI assistants to external tools, Sifthound provides an MCP server with four read-only tools: sifthound_search, sifthound_extract, sifthound_crawl, and sifthound_map.

This integration allows popular AI coding assistants and desktop applications to interact with web search capabilities seamlessly. For instance, users can connect Claude Code, Cursor, and other MCP clients to a running Sifthound server via HTTP at the /mcp endpoint, or run it locally over standard input/output (stdio).

For developers preferring traditional API integration, the transition from Tavily is streamlined. The official Tavily clients can be pointed at a Sifthound server by setting the api_base_url parameter. Authentication can be disabled entirely, allowing any string for the API key, or configured using user-defined API_KEYS. This compatibility has been tested with tavily-python version 0.8.4 and langchain-tavily version 0.2.18, covering both synchronous and asynchronous operations for search and extraction tasks.

Deployment Options and Comparison

Sifthound offers flexible deployment methods to suit different technical requirements and infrastructure preferences. The recommended approach is via Docker Compose, which includes a bundled SearXNG instance. This "all-in-one" stack allows users to get started quickly by cloning the repository, copying the environment example file, and launching the services with a single command.

Alternatively, developers can run the Sifthound Docker image independently, which is published for both linux/amd64 and linux/arm64 architectures. In this scenario, users must provide their own SearXNG instance by setting the SEARXNG_URL environment variable. For those who prefer installing directly from source, the project is available via pip, requiring only the SearXNG URL and a port configuration.

When compared to existing solutions, Sifthound occupies a specific niche:

Feature	Sifthound	Tavily	Firecrawl	crw
License	MIT	Proprietary	AGPL-3.0	AGPL-3.0
Self-hosted	Yes	No	Yes	Yes
Tavily-Compatible API	Yes	N/A	No	No
Search Source	SearXNG	Proprietary	N/A	N/A
JS Rendering	No	N/A	Yes	N/A
MCP Server	Yes	Yes	Yes	Yes

Note: While Sifthound is ideal for teams seeking a Tavily-compatible API under a permissive license, it does not currently support JavaScript rendering. Users requiring dynamic page scraping or advanced neural reranking may find hosted Tavily or Firecrawl more suitable for their needs.

The project represents a growing trend toward open, self-hosted infrastructure for AI development. By decoupling web search capabilities from proprietary APIs, Sifthound empowers developers to build more resilient, private, and cost-effective AI applications. As the AI agent ecosystem matures, tools that offer interoperability with existing standards while maintaining strict data sovereignty are likely to see increased adoption among enterprise and privacy-conscious users.