hebe-agent-unity

English · 한국어

Give your AI coding agent hands and eyes inside Unity.

Hebe is a small CLI that lets an AI coding agent talk to an already-open Unity Editor.

An AI can write C# files by itself. But writing code is only half of Unity development. Someone still has to open Unity, wait for compilation, press Play, read the Console, run tests, inspect a scene, and check the Game View.

Hebe gives the agent a way to do those Unity-side jobs too.

AI coding agent
      │
      │  shell commands
      ▼
    Hebe
      │
      │  talks to the open Editor
      ▼
Unity Editor
      │
      ├─ compile
      ├─ run C#
      ├─ press Play / Stop
      ├─ read Console errors
      ├─ run tests
      ├─ open / save scenes
      └─ capture the Game View
      │
      ▼
result goes back to the agent

Think of it this way:

The AI is the brain. Hebe is the pair of hands and eyes it uses inside Unity.

What problem does it solve?

Without a Unity bridge, an AI coding loop often looks like this:

AI writes code
→ human opens Unity
→ human waits for compile
→ human copies the error
→ AI fixes code
→ human presses Play
→ human checks the result
→ repeat

With Hebe, the loop can become:

AI writes code
→ Hebe asks Unity to compile
→ Hebe reads the error
→ AI fixes code
→ Hebe runs the tests
→ Hebe presses Play
→ Hebe runs keyboard/mouse input sequences
→ Hebe captures the Game View
→ AI checks the result
→ repeat

The important difference is simple: the agent can close the loop by itself instead of stopping at “I wrote the code.”

What can it actually do?

Hebe exposes the Unity jobs an agent commonly needs while coding.

You want to know or do...	Hebe can...
“Did Unity compile?”	report ready, compiling, playing, and other Editor states
“What broke?”	read Console errors and stack traces
“Can this C# idea work?”	run a small C# probe directly inside the Editor
“Does the game still pass tests?”	run EditMode and PlayMode tests
“Start the game and wait until it really starts.”	enter Play Mode and confirm the state transition
“Move and click so we can verify the behavior.”	run stateless keyboard/mouse input and bounded sequences through the optional Input System
“Show me what the game looks like.”	save a clean Game View screenshot
“Open or save this scene.”	load, inspect, save, list, and close scenes
“Run this Unity menu command.”	execute Editor menu items by path
“My project needs a special action.”	add your own [HebeTool] command

A few examples:

# Is Unity ready?
hebe-agent-unity status

# Run a tiny C# check inside the Editor.
hebe-agent-unity exec "return Application.unityVersion;"

# Show recent Unity errors.
hebe-agent-unity console --type error --lines 20

# Run tests.
hebe-agent-unity test --mode EditMode

# Enter Play Mode and wait until Unity confirms it.
hebe-agent-unity editor play --wait

# Hold W for 30 frames and release it, all inside one request.
hebe-agent-unity input sequence --params '{"steps":[{"action":"keyboard","key":"w","mode":"down"},{"action":"wait","frames":30},{"action":"keyboard","key":"w","mode":"up"}]}'

# For long JSON in PowerShell, use a file to avoid native-argument quoting issues.
hebe-agent-unity input sequence --params-file sequence.json

# Save exactly what the game view shows.
hebe-agent-unity screenshot --view game --width 1280 --height 720

Why use Hebe?

1. Your AI can verify its own work

The biggest benefit is not “remote control.” It is verification.

An agent can change code, ask Unity to compile it, inspect the error, fix it, run tests, enter Play Mode, and inspect the final screen. That turns AI coding from a file-writing assistant into a much more complete development loop.

2. It is designed for repeated AI coding loops

AI agents ask lots of small questions:

• “Does this type exist?”
• “What is the current scene?”
• “Did this object get created?”
• “What does this property return?”
• “Did my last change break a test?”

Hebe caches the expensive preparation work around C# execution, so those small checks become much cheaper after the first call.

3. It keeps the setup small

You only need:

1 CLI binary
+ 1 Unity UPM package
+ an open Unity Editor

There is no separate Python runtime or always-on external server process to manage.

4. It tries to answer “did it really finish?”

Unity can reload its scripting domain while compiling or entering Play Mode. That can interrupt a normal request even when Unity actually accepted the command.

Hebe watches the Editor heartbeat and confirms important state transitions instead of blindly treating a dropped response as success or failure.

How fast is it?

The useful number is not how fast the executable opens. It is how quickly an agent can ask Unity a question, get an answer, and ask the next one.

In an internal fixed Unity 6.5 workload:

same-code warm C# exec     median 240 ms
unique-code C# exec        median 313 ms
first cold C# exec         median 1.460 s
full fixed workflow        median 37.285 s

The same-code warm path was 83.6% shorter than the first cold call, about 6.08× faster inside the same Editor session.

All three controlled runs completed successfully. Across 99 measured CLI commands, Hebe recorded 0 command failures and 0 Unity Console errors.

Show the measured numbers and test conditions

Hebe metric	Median	p95	Samples
End-to-end fixed workflow	37.285 s	38.344 s	3 runs
Compile/import	9.659 s	9.695 s	3 runs
First C# exec, cold	1.460 s	1.487 s	3 calls
Same-code exec, warm	240.104 ms	273.416 ms	27 calls
Unique-code exec	312.934 ms	391.671 ms	30 calls
EditMode suite, 12 tests	960 ms	1.070 s	3 suites
PlayMode suite, 3 tests	7.720 s	7.834 s	3 suites

Test environment:

• Windows 11 Pro
• Intel Core i7-12700
• 31.8 GiB RAM
• Unity 6000.5.6f1
• Interaction Mode No Throttling
• one already-open Unity Editor reused across the three Hebe runs
• same 31-file fixed fixture and same semantic command sequence for every run

The measured workflow included compile/import, scene construction, tests, repeated and unique C# probes, Play/Stop verification, Game View capture, and Console inspection. Unity startup, package installation, fixture copy, result serialization, and cleanup between runs were excluded.

These are workload-specific measurements, not universal latency guarantees. Project size, assembly count, package graph, domain reload behavior, Editor focus, and operating-system load can change the absolute values.

The raw benchmark harness and cross-tool artifacts remain internal. This README publishes only Hebe's own measured behavior and scope.

How do I use it?

There are only three pieces.

Step 1. Install the CLI

Windows PowerShell

irm https://raw.githubusercontent.com/NotNull92/hebe-agent-unity/main/install.ps1 | iex

Linux / macOS

curl -fsSL https://raw.githubusercontent.com/NotNull92/hebe-agent-unity/main/install.sh | sh

Step 2. Add the Unity package

In Unity:

Window
→ Package Manager
→ +
→ Add package from git URL...

Paste:

https://github.com/NotNull92/hebe-agent-unity.git?path=AgentConnector

To pin the current Connector release:

https://github.com/NotNull92/hebe-agent-unity.git?path=AgentConnector#connector-0.0.3

Unity 2022.3 or newer is required. The Connector starts automatically when the Editor opens.

Step 3. Keep Unity open and let your agent call Hebe

Any AI coding agent that can run normal shell commands can use Hebe. You can also use it yourself from a terminal or script.

Start with:

hebe-agent-unity status

If you have more than one Unity Editor open, choose the project explicitly:

hebe-agent-unity --project "C:/Projects/MyGame" status

What can I build with it?

Hebe is intentionally a small building block. The interesting part is what you can build on top of it.

Automatic compile-fix loop

agent edits C#
→ asks Unity to compile
→ reads errors
→ fixes them
→ repeats until clean

Test-driven agent loop

agent changes code
→ runs EditMode tests
→ runs PlayMode tests
→ fixes failures
→ reruns tests

Visual QA loop

agent enters Play Mode
→ captures the Game View
→ checks whether UI or game state looks correct
→ changes code or scene setup
→ captures again

Scene and content automation

An agent can use Editor C# or custom tools to create objects, inspect components, save scenes, validate references, or prepare repetitive content.

Project-specific AI tools

If your project has a job that Hebe does not know about, expose it as a tiny custom command:

using HebeAgent;
using Newtonsoft.Json.Linq;

[HebeTool(Name = "build_demo_room", Description = "Builds the demo room.")]
public static class BuildDemoRoomTool
{
    public static object HandleCommand(JObject parameters)
    {
        // Your project-specific Unity Editor code here.
        return new SuccessResponse("demo room built");
    }
}

Then the agent can call:

hebe-agent-unity build_demo_room

This is where Hebe becomes more than a fixed command list: your own Unity Editor automation becomes callable by an AI agent.

A practical agent prompt

After installing Hebe, you can give an AI coding agent an instruction such as:

Work on this Unity project. After every meaningful code change, use hebe-agent-unity to verify that Unity is ready and has no compile errors. Run the relevant EditMode tests. When the feature is ready, enter Play Mode, verify the final state, and capture the Game View before reporting completion.

Hebe does not decide what to build. It gives the agent a reliable way to check the real Unity project instead of guessing from source files alone.

Command overview

Command	Plain-English purpose
status / ping	“Is Unity alive and what is it doing?”
editor	“Play, stop, pause, or refresh Unity.”
exec	“Run this small C# check inside Unity.”
console	“Show me Unity logs and errors.”
log	“Write a message to the Unity Console.”
scene	“Open, inspect, save, list, or close scenes.”
menu	“Run a Unity menu item.”
input	“Move, click, and run frame-timed Play Mode input.”
screenshot	“Save the Scene view, Game view, or one object.”
test	“Run EditMode or PlayMode tests.”
reserialize	“Ask Unity to rewrite changed assets safely.”
list	“Show me the commands and their schemas.”
custom [HebeTool]	“Run my project's own Editor automation.”

A few safety and reliability rules

exec blocks async, coroutine, and deferred callback code by default because a CLI command can otherwise return before that delayed work finishes. Use --allow-async only when fire-and-forget behavior is intentional.

For asynchronous work that must be tracked to completion, expose a [HebeTool] action returning Task<object> or Task. The Connector waits for that task before replying.

editor play --wait and editor stop --wait use the filesystem heartbeat as the completion authority. If a Unity domain reload drops the HTTP response, Hebe can still confirm the final Editor state.

input preserves Hebe's stateless contract. It never keeps a key or mouse button held across separate CLI commands. Long holds belong inside one sequence as down → wait → up, and request-owned held controls are released before the response returns even on failure or cancellation. com.unity.inputsystem remains optional and is not added as a Hebe package dependency. Dynamic and Fixed Input System update modes are supported; ProcessEventsManually is rejected immediately rather than risking an indeterminate long-running request.

screenshot --view game saves the rendered Game View content, including Screen Space Overlay UI, while excluding Unity tabs, toolbars, scale controls, and letterbox gutters.

Technical architecture

The short version:

Go CLI
→ localhost HTTP
→ small Unity Editor Connector
→ Unity main thread

The Connector opens 127.0.0.1:8090 and falls back through 8091–8099 when needed. It writes a heartbeat under ~/.hebe-agent-unity/instances/, which lets the CLI find the correct Editor and observe states such as ready, compiling, and playing.

Commands are serialized onto the Unity main thread. Domain reloads are handled through heartbeat state and a file-based result path for work such as tests.

Hebe has no long-lived CLI session. Each command is independent.

Hebe and Hera

Hebe is the lightweight CLI execution edition of hera-agent-unity.

Use Hebe when you want a small shell-friendly tool that gives an AI agent direct Unity execution and verification.

Use Hera when you also need the larger orchestration layer: MCP integration, typed contracts, approval and audit flows, batching, documentation bundles, or UI/asset pipelines.

Those larger orchestration features are deliberately kept out of Hebe so Hebe can stay small and focused.

Troubleshooting

update_ref failed for ref 'HEAD' … nonexistent object

UPM reuses a cached shallow clone for each git URL. If that cache is stale after the branch moves, Unity may resolve a new commit without having the object locally.

Close Unity, remove com.notnull92.hebe-agent-unity from Packages/manifest.json, delete the project's Library/PackageCache, reopen Unity, and add the package URL again.

Development

go build ./...
go vet ./...
go test ./...

The Go CLI is intentionally small. Most Unity-facing behavior lives in AgentConnector/ (C#), so real integration validation requires an open Unity Editor.

License

Apache-2.0