User · Serve

Running the server

Run the serving server and talk to it over HTTP. Same binary as everything else: ./build/1bit zaya. It speaks an OpenAI-compatible API, so any client that knows the shape can use it.

Weights

Weights are flat float16 .bin files under /tmp/zaya_weights/, one file per tensor named after the HF-style key (e.g. model_embed_tokens_weight.bin, model_layers_0_self_attn_o_proj_weight.bin, x 40 layers). About 6 GB fp16 plus MoE params in GPU memory. Download from the releases page or export from an HF checkpoint.

Launch

$ source env.sh          # sets HSA_OVERRIDE_GFX_VERSION=11.5.1 etc.
$ ./build/1bit zaya                    # default port 8088
$ ./build/1bit zaya --port 8080        # custom port
$ ./build/1bit zaya --model /path/to/model.h1b

Health and generate

$ curl http://localhost:8088/                                    # {"status":"ok","model":"Zaya1-8B",...}
$ curl -X POST http://localhost:8088/completion -H 'Content-Type: application/json' \
  -d '{"prompt":"The future of AI is","n_predict":32}'
# -> {"tokens":[...],"text":"...","gen_ms":2850.12,"tok_s":11.2}

API surface

OpenAI-compatible: POST /v1/chat/completions, POST /v1/batch/completions, GET /v1/models, plus A2A agent-card endpoints.

Performance (measured)

Decode speed is bound by GPU memory bandwidth. MoE expert memory pressure (16 experts x 2 layers) matters.

Troubleshooting