Raphael ThysA holy grail of text AI: ChatGPT / LLM generative query on YOUR OWN unlimited custom data
·
Published in
·
6 min read
·
Jul 4
This is almost bigger than ChatGPT. So-called ‘vector databases’ now enable reliable ‘semantic’ search and generative AI on unlimited collections of your, or your organisation’s, own data & documents via LLMs (large language models) like GPT.
Crucially, that’s without re-training the LLM. And we’re not just passing a ChatGPT created SQL command to a normal DB.
Instead we’re (1) indexing our document fragments (or DB rows) as word-embeddings in a vector DB, (2) creating a compatible word-embedded query at search time, (3) receiving the hits as plain English results from our data and (4) the LLM is then generatively answering like usual, except (5) with references to document or DB row IDs.
This is game changing, ushering in true semantic search & generative AI on your own data that might be millions of times larger than the prompt size!
Early ‘hobby’ adopters or IT data managers do not need to change their data architectures at all, except in adding an additional indexing stage which feeds into an LLM pipeline and new query app.
Open source tools like LangChain and new Azure and AWS cloud services are available to implement such ‘vector’ database powered search.
Hobbyists can spin up solutions in just hours. IT departments can roll out mature solutions in just weeks.
Semantic search & generative AI
The result is a holy grail moment.
Instead of old style keyword searching or poorly performing ‘enterprise search’ or even prompting an LLM with a maximum of 25K words of your data at a time, now we can run true semantic search and generative AI queries against vast sets of our own data.
Semantic search means achieving both of
- asking in plain language AND
- getting results that genuinely match the meaning of our intent
And with the generative AI capabilities of LLMs, we can experience compositional AI, not just search hits, in response to generative requests about unlimited quantities of our own data.
- Find X will do a search