Skip to content

feat(agent): support Vertex MaaS benchmark runs - #611

Open
amitdeshmukh wants to merge 1 commit into
masterfrom
bench/gemma4-vertex-deeporg
Open

feat(agent): support Vertex MaaS benchmark runs#611
amitdeshmukh wants to merge 1 commit into
masterfrom
bench/gemma4-vertex-deeporg

Conversation

@amitdeshmukh

Copy link
Copy Markdown
Collaborator

Summary

  • add configurable json_schema and json_object response formats for OpenAI-compatible providers while preserving the generated schema as a system-prompt contract
  • classify Vertex AI authentication and capacity/rate-limit errors consistently
  • allow explicitly audited publication of incomplete provider usage as an unpriced, disclosed subtotal
  • publish the Gemma 4 26B A4B IT Vertex AI MaaS DeepORG run in the current 2027.1 cohort

DeepORG result

  • full-pass recall: 64.0%
  • pass@3: 75.0%
  • pass³: 27.0%
  • method recall: 62.2%
  • safety precision: 100.0%
  • unsafe effects: 0

The run page discloses that Vertex AI Model as a Service capacity pressure affected observed latency. It also records two unmetered interrupted attempts, labels the token count as a known subtotal, and omits a cost estimate.

Validation

  • go test ./agent ./agent/eval ./serv ./cmd
  • Hugo Extended 0.162.1 static build
  • npm run index
  • npm run check
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant