Google Vertex AI

An llm · client with dialect: gcp-vertex sends requests to Vertex AI. Applications keep using the OpenAI, Anthropic, or Gemini SDK: Zilla Plus sends each request to the Vertex surface that hosts the model, authenticates it with a Google access token, and returns the reply in the caller's API. An llm · server can also accept requests in the Vertex AI APIs.

Available in Zilla Plus

Configuration

vertex_llm_client:
  type: llm
  kind: client
  options:
    dialect: gcp-vertex
    server: https://us-central1-aiplatform.googleapis.com/v1/projects/my-project/locations/us-central1
  telemetry:
    metrics:
      - llm.*
  exit: sys:http_client

The server URL names the Google Cloud project and location, the same way a Vertex AI SDK addresses them, so the location chosen here is where every request is processed. Unlike an AWS Bedrock client, a gcp-vertex client connects through sys:http_client, which uses TLS when the URL is https://.

Route Vertex AI models to it from llm · proxy:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - gemini-*
            - claude-*
            - meta/*
      exit: vertex_llm_client

Models and Surfaces

Vertex AI serves each kind of model from its own API. Zilla Plus picks the surface from the model the caller asked for, so one client serves every model routed to it:

Model	Vertex AI surface	Path appended to `server`
Starts with `gemini`	Gemini	`/publishers/google/models/{model}:generateContent` or `:streamGenerateContent`
Starts with `claude`	Claude	`/publishers/anthropic/models/{model}:rawPredict` or `:streamRawPredict`
Contains `/`, such as `meta/llama-3.3-70b-instruct-maas`	OpenAI-compatible endpoint, for open models named `publisher/model`	`/endpoints/openapi/chat/completions`

Requests for any other model are not served by the client. Model IDs are sent unchanged, so a route can choose the location for a model with when.model. Newer Claude models are served from the global endpoint, https://aiplatform.googleapis.com with location global, or from a multi-region endpoint such as https://aiplatform.us.rep.googleapis.com with location us, instead of from a regional endpoint. Add a second client for those, and route the models to it first:

north_llm_proxy:
  type: llm
  kind: proxy
  routes:
    - when:
        - model:
            - claude-*
      exit: vertex_llm_client_global
    - when:
        - model:
            - gemini-*
            - meta/*
      exit: vertex_llm_client

How Requests Are Sent

Streaming: Zilla Plus sends every request to the streaming method, :streamGenerateContent or :streamRawPredict, and builds the whole reply for a caller that didn't ask for a stream. A caller that already speaks the API of the surface, such as a Gemini caller for a Gemini model, keeps its own choice of method.
Gemini: the request is sent in Gemini's own format, with ?alt=sse added to a streaming request.
Claude: the request is sent in Anthropic's format with anthropic_version set to vertex-2023-10-16. The model member is removed from the body, because Vertex AI takes it from the path.
Open models: the request is sent in OpenAI's format to the OpenAI-compatible endpoint.

What Is Converted

When the caller's API differs from the API of the surface, each part of the conversation is converted section by section, as with any translation: system instructions, text, images, documents and audio sent inline, reasoning, tool definitions, tool calls, and tool results.

Some content can't be converted:

A url source in a request for a Claude model is rejected with 400 Bad Request, naming Vertex AI.
Other content the surface has no equivalent for is dropped or rejected with 400 Bad Request, naming the section, such as user-video cannot be converted from gemini to anthropic.

Credentials

Vertex AI takes a Google access token as Authorization: Bearer. Zilla Plus obtains the token itself from Google's application default credentials, keeps it in memory for each engine worker, and refreshes it before it expires, so options.authorization is not used with gcp-vertex and no key is stored in zilla.yaml. The credentials come from the first of these that is available:

The key file named by the GOOGLE_APPLICATION_CREDENTIALS environment variable, either a service account key or an authorized user file.
The file written by gcloud auth application-default login.
The metadata server of a Google Cloud VM, or of a GKE pod with workload identity.

The identity needs the Vertex AI User role, roles/aiplatform.user, on the project. When no credentials are available, requests fail with 503 Service Unavailable and the message Google credentials are not available.

A token that a caller sends is never forwarded to Vertex AI.

Accepting the Vertex AI APIs

An llm · server with dialect: gcp-vertex, or with no dialect, also accepts requests in the Vertex AI APIs, as a POST to the same paths:

Path	API
`/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:generateContent`, `:streamGenerateContent`	Gemini
`/v1/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:rawPredict`, `:streamRawPredict`	Claude
`/v1/projects/{project}/locations/{location}/endpoints/openapi/chat/completions`	OpenAI-compatible endpoint

The model is taken from the path, so llm · proxy routes these requests like any other, to Vertex AI or to another provider. A Vertex AI request resolves to the gemini, anthropic, or openai dialect, so match it with when.dialect using one of those values.

north_llm_server:
  type: llm
  kind: server
  options:
    dialect: gcp-vertex
  exit: north_llm_proxy

When the server has options.authorization, the request's Authorization header is checked against the guard with the credentials template, and a rejected request gets 401, in the error format of the API. Zilla Plus doesn't verify a Google access token on incoming requests unless a guard is configured. A request that isn't a POST to one of these paths gets 404, and a request whose content type isn't JSON gets 415.

Limits

A request must fit in one of Zilla's buffer slots, 32 KiB by default. Each event of a reply streamed from Vertex AI must also fit in a buffer slot.
A Gemini request whose content lists its parts before its role must fit in the decode window, so that the role is known before the content is converted.

Next Steps

llm · client: the full llm · client reference.
Model routing: route models to Vertex AI.