I hope GPT gets better at that. I tried asking it a few times to look up information for me in my draft, stuff I forgot if I mentioned it already or not. Or when. But it struggles a lot and comes back with wrong data if the document is too long, leaving me with manually searching for it after all.
Do you think there is enough interest in a document search assistant like that? I could probably cobble together a RAG/LLM hybrid thing for that in a month or so. I'm currently working with it in a different context so it probably wouldn't be too hard.
Then again, I'd probably have to consult with a friend who used to work at AWS in order to figurw out how to host a service to do it, since I would have no clue as to handle some of the llm backend for users without 16+ GB RAM on their local system to run it.
It wouldn't be pretty, but it would work (probably).
Thinking about it for a few seconds here, what you want probably doesn't even need the llm. It just needs an RAG search engine, which almost everyone can run on any device with more than 300MB of RAM. The llm would just be to pretty up the responses, but the RAG would just return results from documents you've uploaded after you ask a question.
LLMs aren't really meant to do that, they like to talk and chat. An RAG system quantizes entire documents into vectors, then finds the nearest vectors in the document compared to what the user typed. It's helpful because if you forget the phrasing you used, it still finds things.
toolbaz.com
