Build Custom LLM with Ollama

Installing Ollama

Install Ollama from https://ollama.com

Launching Ollama (Windows)

1. Launch the installed "OllamaSetup.exe" to install it.
   
   

2. After installation, launch Command Prompt and confirm the path is set correctly.
   Execute ollama --help in Command Prompt and confirm that the following help screen appears.
   

Large language model runner

                                        Usage:
                                        ollama [flags]
                                        ollama [command]

                                        Available Commands:
                                        serve       Start ollama
                                        create      Create a model from a Modelfile
                                        show        Show information for a model
                                        run         Run a model
                                        pull        Pull a model from a registry
                                        push        Push a model to a registry
                                        list        List models
                                        ps          List running models
                                        cp          Copy a model
                                        rm          Remove a model
                                        help        Help about any command

                                        Flags:
                                        -h, --help      help for ollama
                                        -v, --version   Show version information

                                        Use "ollama [command] --help" for more information about a command.
                                        

Downloading LLMs

1. Download the LLM to run locally using the ollama pull command. For example, to download Google's gemma-2-9B, enter ollama pull gemma2 in Command Prompt.
   
   

   • If you want other PCs to access this local LLM, execute set OLLAMA_HOST=0.0.0.0 in Command Prompt and set the OLLAMA_HOST environment to "0.0.0.0".

2. Execute ollama serve in Command Prompt to start the LLM.
   
   

   • If "Listening on [::]:11434" is displayed, it can be accessed from other PCs.
   • If "Listening on 127.0.0.1:11434" is displayed, it can only be accessed from that PC. The value "11434" is the port number to use in the ailia DX Insight settings.

Registering Custom LLM in ailia DX Insight

Refer to here for detailed settings.

• Name: The name of the LLM to be used (can be anything as it is used only for UI display)
• Description: Use as a memo as needed
• Model: The name of the downloaded model (in this example, gemma2)
• URL(*): The URL of the port displayed by ollama serve (in this example, http://localhost:11434)
• Maximum Token Length: For gemma-2-9B, it is "8192".
  

(*) If not filled, an HTTP connection error will occur.

Deleting Downloaded LLMs

If LLMs take up too much space or the wrong model is downloaded, you can delete them through the following steps.

1. Execute ollama list in Command Prompt to display the list of downloaded models.
2. Execute ollama rm "model_name" in Command Prompt to delete the downloaded model.
   
   

Next ＞