Meta is developing a search engine to support its AI chatbot

Could Meta develop its own search engine, and if so, what would it look like?

Meta reportedly plans to develop its own Meta AI search tool to reduce its reliance on Google and Microsoft's Bing for web searches within its AI chatbot.

There are a few possible aspects to the news, but it seems like Meta wants to expand its data collection process to continue developing more powerful real-time AI data tools. This could lead to a broader meta-search tool that enables real-time insights for its apps. This could be of particular value for Threads, as it provides up-to-date notes on developing news stories as a complement to in-app engagement.

When Meta first announced its Meta AI chatbot in September last year, the company explained how it would use web search results from Bing to augment its responses:

“For text-based chats, Meta AI has access to real-time information through our search partnership with Bing and provides an image generation tool.”

When Meta AI answers use Bing as contextual information, Meta provides a web link to the search query.

Then in April of this year, Meta announced another partnership with Google to expand its web search answers even further and provide a similar web link for Google-provided answers.

And now, according to The Information, Meta wants to develop its own web crawler to perform searches independently in order to reduce its dependence on its search partners for Meta AI.

According to the information:

“As meta platforms As Facebook tries to keep pace with OpenAI in artificial intelligence development, the Facebook owner is working on a search engine that will crawl the web to provide people with conversational answers to current events using its meta-AI chatbot.”

Conceptually, this would provide Meta with an alternative in the event that Microsoft or Google, both of which compete with Meta in the development of generative AI, decide to end their search contracts.

But that would also mean that Meta would have to build a complex search system that searches the web in a similar way to Google and Bing. This is a big undertaking and why Meta entered into these partnerships in the first place.

But could Meta actually create a web crawler that works in a similar way, and would that be good enough to provide adequate answers in Meta's chatbot?

Meta already has various web crawlers that collect information from external websites, and these crawlers have increased their activity in recent months as Meta looks to collect more data for its AI projects. They could already be collecting data for this new search engine, but a dedicated Facebook search engine would again be a large project.

However, in the broader sense of information retrieval, a dedicated search engine is a logical step that would also provide Meta with more data for its AI language models in the future. Maybe this is actually the bigger picture here, as opposed to just enabling in-stream search.

It's more likely that Meta is strengthening its data sources to power its AI tools and leveraging its existing crawlers to extract as much data as possible from external providers that have not updated their robots.txt protections. This would help Meta further expand its already vast data sets, and if it already collects this information, it makes sense to also reduce its reliance on external search providers where possible.

So perhaps it's less about building a competitor to these providers and more about making the most of your own data collection processes. Still, it's a significant undertaking that could provide expanded opportunities for Zuck and company if they get it right.

However, given the competition in this space, this is not an unexpected move. Again, Meta is fighting with these two vendors for AI supremacy, and as the race becomes more heated, it will be no surprise if they no longer support Meta in this regard.