Project:LLMQueryService POC

This page describes how to install the proof-of-concept LLM-based query service.

Try it here: http://130.73.240.230/ (only from the ZIB network or VPN)

Using a OpenStack VM

1. Create a new instance (if you use Debian: at least 12)
   1. See Project:Docker_OpenStackVM
2. Install necessary libraries
   1. apt-get update
   2. apt-get install git python3-pip python3-venv
3. Clone the repository and follow the rest of the manual ( https://git.zib.de/bzfconra/mardi_llm_bottest )
4. Install Ollama ( https://ollama.com/download/linux )
   1. Check the logs: journalctl -u ollama -f
5. You might need ssh-port-forwarding
   1. Example: ssh -L 8000:127.0.0.1:8501 -i OPENSTACK_KEY_FILE.pem debian@130.73.240.230

The ZIB LLM Server

1. Reachable only from within the ZIB network
2. To see the installed models: curl https://SERVERNAME/api/tags | jq
3. To install a new model: curl https://SERVERNAME/api/pull -d '{"name": "qwen2.5:0.5b"}' | jq

Updating the ZIB LLM Server

1. SSH into it: ssh debian@IP -i key.pem
2. Kill currently running bot
3. Change to the mardi_llm_bottest directory
4. Pull newest version from Git
5. Change to webapp sub-dir
6. Activate virtual environment: source .venv/bin/activate
7. (Optional) Adjust config
8. Run bot: nohup sudo $(which python) -m streamlit run web_app.py > nohup.log &