Last call for the 0.3.0 train

[eous]

If anyone has any code they want to get merged in try to get it finalized and ready for review. Thinking we tag the release towards the end of the week unless something exciting happens.

[emansom]

#47 is on pause for now, will continue next month

[emansom]

#47 is on pause for now, will continue next month

Reason being:

Currently experimenting with ggml-org/gemma-4-31B-it-GGUF:f16 with llama.cpp server and implementing patches to OpenCode and possibly llama.cpp itself for better Gemma 4 model compatibility, by emitting and parsing proper Gemma 4 formatting. And to enhance results, getting rid of the context bloat OpenCode currently emits by implementing proper model specific dynamic system prompt emitters; instead of the current static and bloated .txt files.

That's likely going to take a few more weeks to perfect and upstream ^{_{(mostly because running 31B on a 9070 XT + Ryzen 5 7600 + 96GB DDR5 isn't fast)}}. But definitely worth the time investment in the long run.

So far getting really great results with the dense 31B model at f16 precision, it seems to be one of the most capable models right now. Gemma 4 is highly optimized for agentic tool call usecases.

My environment for anyone stumbling on this wishing to replicate:

Pull the model:

$ llama-cli -hf ggml-org/gemma-4-31B-it-GGUF:f16
/exit [ENTER]

llama.cpp server arguments:

$ llama-server --ctx-size 131072 --offline --flash-attn on --cache-type-k f16 --cache-type-v f16 --threads 12 --temp 0.1 --top-p 0.1 --top-k 10

Using llama.cpp-vulkan package from the AUR with the following modifications:

diff --git a/PKGBUILD b/PKGBUILD
index 61a4474..c9d92d6 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -82,6 +82,10 @@ build() {
     # 本地构建, 针对当前设备优化
     _cmake_options+=(
       -DGGML_NATIVE=ON
+      -DGGML_AVX512=ON
+      -DGGML_AVX512_VBMI=ON
+      -DGGML_AVX512_VNNI=ON
+      -DGGML_AVX512_BF16=ON
     )
   fi

OpenCode config:
~/.config/opencode/opencode.json

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "ggml-org/gemma-4-31B-it-GGUF:F16": {
          "name": "Gemma-4-31B-it (local)",
          "limit": {
            "context": 128000,
            "output": 65536
          }
        }
      }
    }
  },
  "mcp": {
    "gopls": {
      "type": "local",
      "command": ["gopls", "mcp"],
      "enabled": true
    },
    "github": {
      "type": "local",
      "command": ["github-mcp-server", "stdio"],
      "enabled": true,
      "environment": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "sekrit"
      }
    }
  }
}

[eous]

Gemma4 has been running well in turnstone + vllm (not sure about amd support) since release fwiw if you wanted to try it out. (Turnstone is one of my other projects)

docker run -it --gpus all -p 8000:8000 --ipc=host     --ulimit memlock=-1 --ulimit stack=67108864     -v /home/ptrck/models:/models     --entrypoint bash     vllm/vllm-openai:gemma4 -c '
       PARSER=/usr/local/lib/python3.12/dist-packages/vllm/tool_parsers/gemma4_tool_parser.py &&
       sed -i "s/from vllm.tool_parsers.abstract_tool_parser import ToolParser/from vllm.tool_parsers.abstract_tool_parser import Tool, ToolParser/" $PARSER &&
       sed -i "s/def __init__(self, tokenizer: TokenizerLike):/def __init__(self, tokenizer: TokenizerLike, tools: list[Tool] | None = None):/" $PARSER &&
       sed -i "s/super().__init__(tokenizer)/super().__init__(tokenizer, tools)/" $PARSER &&
       python3 -m vllm.entrypoints.openai.api_server \
         --model /models/gemma-4-26B-A4B-it \
         --trust-remote-code \
         --port 8000 \
         --enable-auto-tool-choice \
         --tool-call-parser gemma4 \
         --reasoning-parser gemma4 \
         --chat-template /models/gemma-4-26B-A4B-it/chat_template.jinja --enable-thinking'

https://github.com/turnstonelabs/turnstone