Today thanks to a NetworkChuck video I discovered OpenWebUl and how easy it is to set up a local LLM chat assistant. In particular, the ability to upload documents and use them as a context for chats really caught my interest. So now my question is: let’s say l’ve uploaded 10 different documents on OpenWebUl, is there a way to ask llama3 which between all the uploaded documents contains a certain information (without having to explicitly tag all the documents)? And if not is something like this possible with different local lIm combinations?

  • WalnutLum@lemmy.ml
    link
    fedilink
    English
    arrow-up
    2
    ·
    5 months ago

    Only if your model has a large enough token context to contain all the documents’ info would you be able to do something like that

      • General_Effort@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        5 months ago

        It probably says somewhere where you dled the model. It’s also in the metadata. I forget where it’s displayed. Maybe in the terminal window.

        Things you should know:

        • What a token is depends on the model.
        • Context takes a lot of (V)RAM.
        • People modify models to increase the context but that often doesn’t work well. Watch out for the model missing things, esp. in the middle of the document.

        L3 is probably not the right base for the task. Maybe Phi-3 or Cohere.