Skip to content

Commit dc9540c

Browse files
Merge pull request #274 from modelstudioai/feat/caller-telemetry
Attribute agent callers in telemetry
2 parents e89c331 + 646f4ea commit dc9540c

22 files changed

Lines changed: 541 additions & 36 deletions

File tree

‎docs/agents/skill-change.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ bailian-gen bailian-finetune bailian-managed-agent bailian-web-search
4040

4141
- [ ] **整包装齐**:安装 / 缺 skill 文案主推 `bl skill init`;升级同步用 `bl skill update`(只更新已安装,不装新 skill);业务 skill **不**声明 `companions`
4242
- [ ] **协议读取**:CRITICAL / references 可链 `../bailian-protocol/…`;若读不到 → 停止执行 `bl`,提示 `bl skill init`
43+
- [ ] **调用归因**:协议 `SKILL.md` 放 bash + PowerShell 双模板与协议样例;每个业务 skill **只留一条**填好的样例并链回协议;其余示例 / `assets/` / `reference/` 保持裸 `bl …`。Agent 执行时再套模板;人类终端不要设这些变量
4344
- [ ] **高风险确认**:统一由 `bailian-protocol` 定义;reference / leaf help 以 `risk: high` 明示风险,业务 skill 不得引导 Agent 自动补 `--yes`。遇到 exit code 7 / `requires_confirmation` 时停止执行并请求确认;目标或范围变化后重新确认
4445
- [ ] **正常控制流**:`requires_confirmation` 不是 CLI bug,`assets/issue-reporting.md` 必须将 exit code 7 保持在 EXCLUDE 范围
4546
- [ ] **软 hand-off**:兄弟业务 skill **只写 skill 名**;已安装则 Read,未安装则 `bl … --help` 或提示整包安装;**不要**把 `../bailian-gen/…` 等写成执行前提

‎docs/agents/telemetry-change.md‎

Lines changed: 25 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -95,9 +95,18 @@ source-config 只用于百炼 / DashScope API 侧消费,不发送到通用网
9595
- `command`、`timestamp`、`durationMs`、`success`
9696
- `cliVersion`、`nodeVersion`、`os`
9797
- `authMethod`
98+
- `callerType`(必填:`agent` 或 `unknown`);`callerType=agent` 时可选 `callerName`、`skillName`、`skillVersion`
9899
- 失败时的 `errorMessage`、`exitCode`、`httpStatus`、`requestId`
99100
- 安全 allowlist 过滤后的 `params`
100101

102+
调用方归因只认本次进程的 `BAILIAN_CALLER_*`(由 `readCaller` 归一化后写入事件):
103+
104+
- `BAILIAN_CALLER_TYPE` 为 `agent` / `ai_agent`(大小写不敏感)→ `callerType=agent`
105+
- 未设置、空白或其他值 → `callerType=unknown`;不要把该字段当成人工
106+
- `callerName` / `skillName` / `skillVersion` 仅在 `agent` 时发送;`kscli` 通常没有 skill 字段
107+
108+
新 CLI 每条进入 middleware 的命令事件都有 `callerType`。旧 CLI 没有该字段。看板只统计带 `callerType` 的事件,不能把字段缺失混进分母。`unknown` 只表示未识别为 agent,不是人工。新 CLI + 旧 skill / Agent 漏设变量时,会落进 `unknown`。`BAILIAN_CALLER_*` 是调用方声明,不是 `Identity`,不得用来猜 `bl` / `kscli`。
109+
101110
参数默认不上传,只有 `packages/core/src/telemetry/tracker.ts` 的 `PARAM_ALLOWLIST` 中字段会进入事件。不得加入 prompt、凭证、文件路径、URL、账号/租户/工作空间 ID 或其他用户内容。
102111

103112
`page` 仅记录有限数值页码;Webhook 等命令使用的不透明字符串游标不得进入本地日志或远端 AEM,即使字符串内容看起来是数字。
@@ -113,16 +122,16 @@ source-config 只用于百炼 / DashScope API 侧消费,不发送到通用网
113122

114123
AEM 映射:
115124

116-
| AEM 字段 | 内容 |
117-
| ---------- | ------------------------------------------------------------------ |
118-
| event name | 命令路径 |
119-
| `et` | `EXP` |
120-
| `ext` | 除 `command`、`params` 外的结构化事件字段,包含失败时的 `exitCode` |
121-
| `c1` | allowlist 参数 |
122-
| `c2` | `success` / `failure` |
123-
| `c3` | HTTP status |
124-
| `c4` | 错误文案,最多 500 字符 |
125-
| `c5` | request ID |
125+
| AEM 字段 | 内容 |
126+
| ---------- | -------------------------------------------------------------------------------- |
127+
| event name | 命令路径 |
128+
| `et` | `EXP` |
129+
| `ext` | 除 `command`、`params` 外的结构化事件字段,含 `callerType` 与失败时的 `exitCode` |
130+
| `c1` | allowlist 参数 |
131+
| `c2` | `success` / `failure` |
132+
| `c3` | HTTP status |
133+
| `c4` | 错误文案,最多 500 字符 |
134+
| `c5` | request ID |
126135

127136
远端发送是 best-effort,不得阻塞命令或改变退出码。正常退出最多等待 1 秒,SIGINT 最多等待 500 ms。
128137

@@ -140,6 +149,7 @@ AEM 映射:
140149

141150
- [ ] 同时核对 `packages/core/src/client/http.ts`、`mcp.ts`、`instrumented-fetch.ts`、`client.ts` 与 `files/upload.ts`
142151
- [ ] 产品身份必须来自 `Identity`;不得从命令路径、环境变量或 `process.argv` 猜测
152+
- [ ] `BAILIAN_CALLER_*` 是调用方声明,不得用来猜产品身份(`bl` / `kscli`)
143153
- [ ] `bl` 与 `kscli` 必须分别验证 `binName`、`clientName`、`version`
144154
- [ ] OSS、结果文件、npm、二进制和 Skill 下载不得为了业务渠道统计新增 source-config
145155
- [ ] 改 URL / host 范围时同时执行 [URL / 渠道变更](url-change.md) 清单
@@ -148,6 +158,7 @@ AEM 映射:
148158

149159
- [ ] 更新 `TrackingEvent`、`createTrackingEvent()` 与 `buildRemoteAemOptions()` 的字段映射
150160
- [ ] 本地 JSONL 与远端 AEM 必须基于同一结构化事件,不能维护两套字段口径
161+
- [ ] 调用方字段经 `readCaller` → `TrackingDeps.caller` → 事件;仅显式 `agent` / `ai_agent` 为 `agent`,其余为 `unknown`
151162
- [ ] 成功与失败均覆盖;遥测异常必须静默且不改变业务退出码
152163
- [ ] runtime 本地语义错误应记录 `exitCode`;新增字段默认随 AEM `ext` 上报,无需占用新的 `c1`—`c5`
153164
- [ ] 检查 `DO_NOT_TRACK=1` 与 `telemetry: false` 两个关闭入口
@@ -157,15 +168,17 @@ AEM 映射:
157168

158169
```sh
159170
rg -n "trackingHeaders|x-dashscope-source-config|User-Agent" packages --glob '*.ts'
160-
rg -n "trackCommandExecution|PARAM_ALLOWLIST|buildRemoteAemOptions" packages/core packages/runtime --glob '*.ts'
171+
rg -n "trackCommandExecution|PARAM_ALLOWLIST|buildRemoteAemOptions|readCaller" packages/core packages/runtime --glob '*.ts'
161172
vp check
162-
vp test packages/core/tests packages/commands/tests/e2e/auth.e2e.test.ts
173+
vp test packages/core/tests packages/runtime/tests/telemetry-stage.test.ts packages/commands/tests/e2e/auth.e2e.test.ts
163174
```
164175

165176
## 常见漏点
166177

167178
- ✗ 只看 AEM 命令事件,误以为它能替代网关侧请求渠道统计
168179
- ✗ 把 `authMethod` 当成实际凭证来源;它只是命令声明的鉴权域
180+
- ✗ 把旧 CLI「没有 `callerType`」或「`callerType=unknown`」当成人工占比
181+
- ✗ 用 `BAILIAN_CALLER_*` 猜测产品身份,或根据 TTY / 安装目录反推调用方
169182
- ✗ 新增 bypass `fetch` 后漏掉应由网关消费的 source-config,或把它发给 OSS / npm / 第三方下载地址
170183
- ✗ 只改 `bl` 入口,导致 `kscli` 的产品名或版本标签错误
171184
- ✗ 把帮助、版本或参数校验失败算进“全部命令”;这些路径当前没有进入 telemetry middleware

‎packages/cli/tests/skill-sandbox-reference.test.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,8 @@ test("Sandbox is a valid standalone skill aligned with the CLI version", () => {
2323
expect(frontmatter.metadata.version).toBe(cliPackage.version);
2424

2525
for (const link of skillMarkdown.matchAll(/\]\(([^)]+)\)/g)) {
26-
expect(existsSync(join(sandboxSkillDirectory, link[1]))).toBe(true);
26+
const target = link[1].split("#")[0];
27+
expect(existsSync(join(sandboxSkillDirectory, target)), target).toBe(true);
2728
}
2829
});
2930

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
/** Parse BAILIAN_CALLER_*. Only an explicit agent is `agent`; everything else is `unknown`. */
2+
3+
export type CallerAttribution =
4+
| { callerType: "unknown" }
5+
| {
6+
callerType: "agent";
7+
callerName: string;
8+
skillName?: string;
9+
skillVersion?: string;
10+
};
11+
12+
const CALLER_TYPE_ENV = "BAILIAN_CALLER_TYPE";
13+
const CALLER_NAME_ENV = "BAILIAN_CALLER_NAME";
14+
const SKILL_NAME_ENV = "BAILIAN_SKILL_NAME";
15+
const SKILL_VERSION_ENV = "BAILIAN_SKILL_VERSION";
16+
17+
const NAME_PATTERN = /^[a-z0-9][a-z0-9.-]{0,63}$/;
18+
const SKILL_NAME_PATTERN = /^bailian-[a-z0-9-]{1,64}$/;
19+
const SKILL_VERSION_PATTERN = /^[0-9]+\.[0-9]+\.[0-9]+(?:[-+][0-9A-Za-z.-]{1,20})?$/;
20+
21+
function readTrimmed(env: NodeJS.ProcessEnv, key: string): string | undefined {
22+
const raw = env[key];
23+
if (typeof raw !== "string") return undefined;
24+
const trimmed = raw.trim();
25+
return trimmed.length > 0 ? trimmed : undefined;
26+
}
27+
28+
/** Normalize host-reported name to a stable id, or `unknown`. */
29+
function normalizeCallerName(raw: string | undefined): string {
30+
if (!raw) return "unknown";
31+
const normalized = raw.toLowerCase().replace(/[\s_]+/g, "-");
32+
return NAME_PATTERN.test(normalized) ? normalized : "unknown";
33+
}
34+
35+
/** Env-only attribution; no TTY or install-path heuristics. */
36+
export function readCaller(env: NodeJS.ProcessEnv): CallerAttribution {
37+
const callerType = readTrimmed(env, CALLER_TYPE_ENV)?.toLowerCase();
38+
if (callerType !== "agent" && callerType !== "ai_agent") {
39+
return { callerType: "unknown" };
40+
}
41+
42+
const skillNameRaw = readTrimmed(env, SKILL_NAME_ENV)?.toLowerCase();
43+
const skillName =
44+
skillNameRaw !== undefined && SKILL_NAME_PATTERN.test(skillNameRaw) ? skillNameRaw : undefined;
45+
const skillVersionRaw = readTrimmed(env, SKILL_VERSION_ENV);
46+
const skillVersion =
47+
skillName !== undefined &&
48+
skillVersionRaw !== undefined &&
49+
SKILL_VERSION_PATTERN.test(skillVersionRaw)
50+
? skillVersionRaw
51+
: undefined;
52+
53+
return {
54+
callerType: "agent",
55+
callerName: normalizeCallerName(readTrimmed(env, CALLER_NAME_ENV)),
56+
...(skillName !== undefined ? { skillName } : {}),
57+
...(skillVersion !== undefined ? { skillVersion } : {}),
58+
};
59+
}

‎packages/core/src/telemetry/event.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import type { AuthRequirement } from "../types/command.ts";
2+
import type { CallerAttribution } from "./caller.ts";
23

34
export interface TrackingEvent {
45
command: string;
@@ -14,6 +15,11 @@ export interface TrackingEvent {
1415
os: string;
1516
authMethod?: AuthRequirement;
1617
params?: Record<string, unknown>;
18+
/** Always present on new CLI events: `agent` or `unknown`. */
19+
callerType: CallerAttribution["callerType"];
20+
callerName?: string;
21+
skillName?: string;
22+
skillVersion?: string;
1723
}
1824

1925
export function createTrackingEvent(opts: {
@@ -24,6 +30,7 @@ export function createTrackingEvent(opts: {
2430
cliVersion: string;
2531
authMethod?: AuthRequirement;
2632
params?: Record<string, unknown>;
33+
caller: CallerAttribution;
2734
}): TrackingEvent {
2835
const event: TrackingEvent = {
2936
command: opts.command,
@@ -33,8 +40,15 @@ export function createTrackingEvent(opts: {
3340
cliVersion: opts.cliVersion,
3441
nodeVersion: process.version,
3542
os: process.platform,
43+
callerType: opts.caller.callerType,
3644
};
3745

46+
if (opts.caller.callerType === "agent") {
47+
event.callerName = opts.caller.callerName;
48+
if (opts.caller.skillName !== undefined) event.skillName = opts.caller.skillName;
49+
if (opts.caller.skillVersion !== undefined) event.skillVersion = opts.caller.skillVersion;
50+
}
51+
3852
if (opts.authMethod) {
3953
event.authMethod = opts.authMethod;
4054
}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1+
export type { CallerAttribution } from "./caller.ts";
2+
export { readCaller } from "./caller.ts";
13
export type { TrackingEvent } from "./event.ts";
24
export { createTrackingEvent } from "./event.ts";
35
export { localSink, remoteSink, flushTelemetry } from "./sink.ts";
46
export { trackCommandExecution } from "./tracker.ts";
7+
export type { TrackingDeps } from "./tracker.ts";

‎packages/core/src/telemetry/tracker.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import type { Identity, Settings } from "../config/schema.ts";
22
import { BailianError } from "../errors/base.ts";
33
import type { AuthRequirement } from "../types/command.ts";
4+
import type { CallerAttribution } from "./caller.ts";
45
import { createTrackingEvent } from "./event.ts";
56
import { localSink, remoteSink } from "./sink.ts";
67

@@ -89,6 +90,8 @@ export interface TrackingDeps {
8990
identity: Identity;
9091
settings: Settings;
9192
authMethod?: AuthRequirement;
93+
/** From `readCaller(process.env)` in telemetryStage. */
94+
caller: CallerAttribution;
9295
}
9396

9497
export async function trackCommandExecution(
@@ -133,6 +136,7 @@ export async function trackCommandExecution(
133136
cliVersion: deps.identity.version,
134137
authMethod: deps.authMethod,
135138
params: extractParams(flags),
139+
caller: deps.caller,
136140
});
137141

138142
// Fire-and-forget — never block the CLI exit or mask errors
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
import { expect, test } from "vite-plus/test";
2+
import { readCaller } from "../src/telemetry/caller.ts";
3+
4+
test.each([
5+
{ label: "missing", env: {} },
6+
{ label: "blank", env: { BAILIAN_CALLER_TYPE: " " } },
7+
{ label: "explicit human", env: { BAILIAN_CALLER_TYPE: "human" } },
8+
{ label: "illegal false", env: { BAILIAN_CALLER_TYPE: "false" } },
9+
{ label: "illegal ai-agent", env: { BAILIAN_CALLER_TYPE: "ai-agent" } },
10+
{ label: "illegal zero", env: { BAILIAN_CALLER_TYPE: "0" } },
11+
])("maps $label to unknown without agent fields", ({ env }) => {
12+
expect(readCaller(env)).toEqual({ callerType: "unknown" });
13+
});
14+
15+
test.each([
16+
{ label: "agent", type: "agent" },
17+
{ label: "ai_agent", type: "ai_agent" },
18+
{ label: "Agent", type: "Agent" },
19+
{ label: "AI_AGENT", type: "AI_AGENT" },
20+
{ label: "padded agent", type: " agent " },
21+
])("accepts $label as agent", ({ type }) => {
22+
expect(
23+
readCaller({
24+
BAILIAN_CALLER_TYPE: type,
25+
BAILIAN_CALLER_NAME: "cursor",
26+
BAILIAN_SKILL_NAME: "bailian-cli",
27+
BAILIAN_SKILL_VERSION: "2.5.0",
28+
}),
29+
).toEqual({
30+
callerType: "agent",
31+
callerName: "cursor",
32+
skillName: "bailian-cli",
33+
skillVersion: "2.5.0",
34+
});
35+
});
36+
37+
test("normalizes callerName and falls back to unknown", () => {
38+
expect(
39+
readCaller({
40+
BAILIAN_CALLER_TYPE: "agent",
41+
BAILIAN_CALLER_NAME: " Claude Code ",
42+
}),
43+
).toEqual({ callerType: "agent", callerName: "claude-code" });
44+
45+
expect(
46+
readCaller({
47+
BAILIAN_CALLER_TYPE: "agent",
48+
BAILIAN_CALLER_NAME: "qwen_code",
49+
}),
50+
).toEqual({ callerType: "agent", callerName: "qwen-code" });
51+
52+
expect(
53+
readCaller({
54+
BAILIAN_CALLER_TYPE: "agent",
55+
BAILIAN_CALLER_NAME: "bad/name!",
56+
}),
57+
).toEqual({ callerType: "agent", callerName: "unknown" });
58+
59+
expect(readCaller({ BAILIAN_CALLER_TYPE: "agent" })).toEqual({
60+
callerType: "agent",
61+
callerName: "unknown",
62+
});
63+
});
64+
65+
test("drops illegal skillName and only keeps version when skillName is valid", () => {
66+
expect(
67+
readCaller({
68+
BAILIAN_CALLER_TYPE: "agent",
69+
BAILIAN_CALLER_NAME: "cursor",
70+
BAILIAN_SKILL_NAME: "not-bailian",
71+
BAILIAN_SKILL_VERSION: "2.5.0",
72+
}),
73+
).toEqual({ callerType: "agent", callerName: "cursor" });
74+
75+
expect(
76+
readCaller({
77+
BAILIAN_CALLER_TYPE: "agent",
78+
BAILIAN_CALLER_NAME: "cursor",
79+
BAILIAN_SKILL_NAME: "bailian-cli",
80+
BAILIAN_SKILL_VERSION: "not-a-version",
81+
}),
82+
).toEqual({
83+
callerType: "agent",
84+
callerName: "cursor",
85+
skillName: "bailian-cli",
86+
});
87+
88+
expect(
89+
readCaller({
90+
BAILIAN_CALLER_TYPE: "agent",
91+
BAILIAN_CALLER_NAME: "cursor",
92+
BAILIAN_SKILL_NAME: "bailian-cli",
93+
BAILIAN_SKILL_VERSION: "2.5.0-beta.1",
94+
}),
95+
).toEqual({
96+
callerType: "agent",
97+
callerName: "cursor",
98+
skillName: "bailian-cli",
99+
skillVersion: "2.5.0-beta.1",
100+
});
101+
});
102+
103+
test("lowercases skillName before validation", () => {
104+
expect(
105+
readCaller({
106+
BAILIAN_CALLER_TYPE: "agent",
107+
BAILIAN_CALLER_NAME: "cursor",
108+
BAILIAN_SKILL_NAME: "Bailian-CLI",
109+
BAILIAN_SKILL_VERSION: "2.5.0",
110+
}),
111+
).toEqual({
112+
callerType: "agent",
113+
callerName: "cursor",
114+
skillName: "bailian-cli",
115+
skillVersion: "2.5.0",
116+
});
117+
118+
expect(
119+
readCaller({
120+
BAILIAN_CALLER_TYPE: "agent",
121+
BAILIAN_CALLER_NAME: "cursor",
122+
BAILIAN_SKILL_NAME: "BAILIAN-GEN",
123+
BAILIAN_SKILL_VERSION: "2.5.0",
124+
}),
125+
).toEqual({
126+
callerType: "agent",
127+
callerName: "cursor",
128+
skillName: "bailian-gen",
129+
skillVersion: "2.5.0",
130+
});
131+
});
132+
133+
test("ignores agent detail fields when callerType is not agent", () => {
134+
expect(
135+
readCaller({
136+
BAILIAN_CALLER_TYPE: "human",
137+
BAILIAN_CALLER_NAME: "cursor",
138+
BAILIAN_SKILL_NAME: "bailian-cli",
139+
BAILIAN_SKILL_VERSION: "2.5.0",
140+
}),
141+
).toEqual({ callerType: "unknown" });
142+
});

0 commit comments

Comments
 (0)