Instructions to use ubergarm/Ling-1T-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ubergarm/Ling-1T-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ubergarm/Ling-1T-GGUF:Q2_K # Run inference directly in the terminal: llama cli -hf ubergarm/Ling-1T-GGUF:Q2_K
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ubergarm/Ling-1T-GGUF:Q2_K # Run inference directly in the terminal: llama cli -hf ubergarm/Ling-1T-GGUF:Q2_K
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf ubergarm/Ling-1T-GGUF:Q2_K # Run inference directly in the terminal: ./build/bin/llama cli -hf ubergarm/Ling-1T-GGUF:Q2_K
Use Docker
docker model run hf.co/ubergarm/Ling-1T-GGUF:Q2_K
- Llama new
- LM Studio
- Jan
- vLLM
How to use ubergarm/Ling-1T-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ubergarm/Ling-1T-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ubergarm/Ling-1T-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ubergarm/Ling-1T-GGUF:Q2_K
- Ollama
How to use ubergarm/Ling-1T-GGUF with Ollama:
ollama run hf.co/ubergarm/Ling-1T-GGUF:Q2_K
- Unsloth Desktop
- Pi
How to use ubergarm/Ling-1T-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ubergarm/Ling-1T-GGUF:Q2_K
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ubergarm/Ling-1T-GGUF:Q2_K" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use ubergarm/Ling-1T-GGUF with Docker Model Runner:
docker model run hf.co/ubergarm/Ling-1T-GGUF:Q2_K
- Lemonade
How to use ubergarm/Ling-1T-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ubergarm/Ling-1T-GGUF:Q2_K
Run and chat with the model
lemonade run user.Ling-1T-GGUF-Q2_K
List all available models
lemonade list
- Hermes Agent
How to use ubergarm/Ling-1T-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ubergarm/Ling-1T-GGUF:Q2_K
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default ubergarm/Ling-1T-GGUF:Q2_K
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ubergarm/Ling-1T-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ubergarm/Ling-1T-GGUF:Q2_K
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "ubergarm/Ling-1T-GGUF:Q2_K" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
403 Error
Sorry for the wonky org namespace situation, hopefully will have this cleared up and moved into proper https://huggingface.co/ubergarm namespace ASAP.
Quick note:
- Please don't follow
ubergarm2organization but you can follow ubergarm if you like. - Feel free to like this model, as hopefully the likes and discussion will transfer over when I can move it eventually.
Thanks @gghfez and @MikeRoz for some discussion on the original now deleted repo haha... Wasn't sure how to save that info...
Did you receive any official response from HF or the people tagged into the discussion?
If I delete enough to get below the 12TB, can I start uploading again automatically?
Or are our accounts stuck in the blocked state until they unlock it manually?
I created a post under models-and-benchmarks in the Aider discord called "ubergarm2/Ling-1T-GGUF" will hopefully be posting full aider polyglot benchmarks report for the IQ2_KS in the next day or so /neo
edit 2: nah it failed the next 4 test_cases completeley again.. hmm maybe its not just performing so good for aider polyglot. the aider benchmark they provided was for the older benchmark not the newer polyglot so we don't have a reference score either
another old message: edit: i updated Aider to latest and it nailed the first test_case. so I think we are back on track now! :).It also works in Roo Code!
old message: its failing every test and sub test within the each test without creating any errors i.e no malformed responses or error outputs other then the solutions being completely wrong. its weird. the responses are coherent. and it works in open-webui. i've only done 18/225 test cases so its possible although unlikely its very bad luck in the start. i don't think so tough since its failing every step within each test case as well.
i updated Aider to latest and it nailed the first test_case. so I think we are back on track now! :).It also works in Roo Code!
Wahoo! Great to hear, hopefully it keeps going lol... There may also be things like running with --jinja but its hot off the presses so I haven't fiddled with it much.
I have a few more quants landing now and have the initial perplexity vs size graph available if the relative "quality" of the quants is of interest to you. Of course the bigger ones will be slower too. The iq2_k is interesting in that it uses full q8_0 for attn.*/first 4 dense/shexp which does make it a bit slower for the size given the proportion of active tokens...