What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
Since SEO is big business, and agent driving search is probably the future, I suspect we might see the price of search API's going negative in the future.
Ie. Providers pay you to use their API over competitors, since then they get control of who has influence on the internet. They can probably get sites to pay to be up ranked for specific queries etc
DoctorOetker
@DeepSeek teams, why not train your next series of LLM models with source-aware training?
LLM's would be able to answer questions, delineating exactly which facts/assumptions/reasonings are prior work and from where (this URL, that book, etc.), and which parts are it's own deductions, or the user's.
show comments
mjmvisser
These services only exist because Google refuses to provide a paid search API.
show comments
thombles
For comparison that’s $5 with Brave Search API and $12 with Kagi API
show comments
frenchtoast8
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
show comments
vitorsr
Meant as a comparison given the discrepant pricing points:
I pay the Exa premium for semantic retrieval with far higher SNR, plus the academic index [1].
A field like objective can be up to 4096 characters [2] versus ori_question which is limited to 128 here.
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
show comments
suncemoje
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
show comments
ggnore7452
Ran a few quick tests on my web-fetch bench. As much as I like DeepSeek, its search and fetch aren't very good (is quite cheap tho). You'd probably be better off using free services like Jina directly, or if you're going paid, Parallel, Firecrawl, Tavily and Exa are all pretty good now.
dsco
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
I wonder if providers do this so they can 'scrape' for 'free' because the customers pay for it? They all seem to have an abstraction around search rather than suggesting the use of native search apis.
shaunpud
I "fetched" a website with hyperlinks and it only returned the text, which makes it a bit irrelevant?
thatxliner
they had this for a while now under the Anthropic API
show comments
hartator
How does the quality compare to SerpApi?
fullstackwife
I started to think differently about web search for agents, instead of traditional index, I would prefer to have an agentic WALKER that literally walks through pages and finds out whats needed, even if seeded with some initial web search.
show comments
gvfsa
Can someone explain why fetch costs money. I thought it was your locally-running harness doing it. How can they charge you for that?
show comments
apiclay
This is nice. Does search include their LLM data, or does it do real-time search like Perplexity?
We are building our own in-house search API/MCP too, with anti-bot and CAPTCHA workarounds, and we plan to launch Monday morning.
Jskewel
Is there any way to test this without an API key?
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
show comments
baalimago
I'm having trouble finding what this actually is
They're entering the search engine scene since they already have "all of the internet" stored due to the expansive LLM crawling -> training?
show comments
Havoc
As reference point exa charges 4 bucks
cdnsteve
I use to manage Google search appliances. This feels a lot like Google old search API product. Nice to see some new players here.
lifeisloving
Deepseeks browser platform has replaced my typical LLM search routines over the last few months.
Its extraordinary good. They arent trying to sell you things and I cab often find obscure sources/papers.
Its also great for finding research papers in chinese on a given topic.(also its free in the browser and doesn't slow my comptuer to a crawl like most webUIs by llm companies). I have it open all day nowadays.
Edit: How does this get downvoted immediately. Does Anthropic have bots just downvoting any positive sentiment about Chinese AI..
show comments
andai
Is AI search where the mysterious relentless crawler traffic is coming from lately?
show comments
larme
I wonder if this is censored. If not then users in China mainland can use the api to workaround gfw (though there are much better ways)
show comments
cyberax
Which ones of the search providers listed in this thread use ethical indexing and not outright DDoS?
czk
as expected, the typical "politically sensitive" search queries return 0 results
egress on fetch looks to be huawei cloud in hong kong
fetch tool uses a linux firefox 149 user agent with lang set to china. it runs js. could be using something like camoufox.
josteink
The rate-limit given there seems really low. 5 requests per second?
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
show comments
tinyhouse
DeepSeek models are amazing and this search pricing is fantastic. All these companies like Parallel, Exa and others are gonna have hard time justifying their hefty valuations. Add Perplexity and many others into the mix. The demand for what they are building is there, but not the margins for the amount of money they raised.
stavros
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
shevy-java
These prices will skyrocket eventually. AI companies need more money.
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
show comments
0-bad-sectors
Will DeepSeek pay the content creators whom pages were fetched?
What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
[1]: https://github.com/nikvdp/searxng-ai-kit
Since SEO is big business, and agent driving search is probably the future, I suspect we might see the price of search API's going negative in the future.
Ie. Providers pay you to use their API over competitors, since then they get control of who has influence on the internet. They can probably get sites to pay to be up ranked for specific queries etc
@DeepSeek teams, why not train your next series of LLM models with source-aware training?
https://arxiv.org/abs/2404.01019
LLM's would be able to answer questions, delineating exactly which facts/assumptions/reasonings are prior work and from where (this URL, that book, etc.), and which parts are it's own deductions, or the user's.
These services only exist because Google refuses to provide a paid search API.
For comparison that’s $5 with Brave Search API and $12 with Kagi API
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
Meant as a comparison given the discrepant pricing points:
I pay the Exa premium for semantic retrieval with far higher SNR, plus the academic index [1].
A field like objective can be up to 4096 characters [2] versus ori_question which is limited to 128 here.
[1] https://exa.ai/blog/publications-search
[2] https://exa.ai/docs/reference/search
Artificial analysis' compilation of search api costs per 1000 queries, for comparison:
https://artificialanalysis.ai/agents/search-api?search-api-c...
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
Ran a few quick tests on my web-fetch bench. As much as I like DeepSeek, its search and fetch aren't very good (is quite cheap tho). You'd probably be better off using free services like Jina directly, or if you're going paid, Parallel, Firecrawl, Tavily and Exa are all pretty good now.
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
Does anybody use https://www.anysearch.com/ ?
I wonder if providers do this so they can 'scrape' for 'free' because the customers pay for it? They all seem to have an abstraction around search rather than suggesting the use of native search apis.
I "fetched" a website with hyperlinks and it only returned the text, which makes it a bit irrelevant?
they had this for a while now under the Anthropic API
How does the quality compare to SerpApi?
I started to think differently about web search for agents, instead of traditional index, I would prefer to have an agentic WALKER that literally walks through pages and finds out whats needed, even if seeded with some initial web search.
Can someone explain why fetch costs money. I thought it was your locally-running harness doing it. How can they charge you for that?
This is nice. Does search include their LLM data, or does it do real-time search like Perplexity?
We are building our own in-house search API/MCP too, with anti-bot and CAPTCHA workarounds, and we plan to launch Monday morning.
Is there any way to test this without an API key?
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
I'm having trouble finding what this actually is
They're entering the search engine scene since they already have "all of the internet" stored due to the expansive LLM crawling -> training?
As reference point exa charges 4 bucks
I use to manage Google search appliances. This feels a lot like Google old search API product. Nice to see some new players here.
Deepseeks browser platform has replaced my typical LLM search routines over the last few months.
Its extraordinary good. They arent trying to sell you things and I cab often find obscure sources/papers.
Its also great for finding research papers in chinese on a given topic.(also its free in the browser and doesn't slow my comptuer to a crawl like most webUIs by llm companies). I have it open all day nowadays.
Edit: How does this get downvoted immediately. Does Anthropic have bots just downvoting any positive sentiment about Chinese AI..
Is AI search where the mysterious relentless crawler traffic is coming from lately?
I wonder if this is censored. If not then users in China mainland can use the api to workaround gfw (though there are much better ways)
Which ones of the search providers listed in this thread use ethical indexing and not outright DDoS?
as expected, the typical "politically sensitive" search queries return 0 results
egress on fetch looks to be huawei cloud in hong kong
fetch tool uses a linux firefox 149 user agent with lang set to china. it runs js. could be using something like camoufox.
The rate-limit given there seems really low. 5 requests per second?
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
DeepSeek models are amazing and this search pricing is fantastic. All these companies like Parallel, Exa and others are gonna have hard time justifying their hefty valuations. Add Perplexity and many others into the mix. The demand for what they are building is there, but not the margins for the amount of money they raised.
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
These prices will skyrocket eventually. AI companies need more money.
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
Will DeepSeek pay the content creators whom pages were fetched?