Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ build-*
venv
.cache
*.pcm
*.mp3
*.o
third_party/*
!third_party/mbedtls
Expand Down
79 changes: 79 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Examples — the rtc-tcp-client POSIX demos share their parsing helpers.
- `demo_json.h` (JSON readers, base64, session-token parsing, bounded copy
into fixed-size config fields), `demo_text.h` (`tai_text_msg_t` handling
and stream reassembly) and `demo_mcp.h` (device-side MCP answering)
replace the copy of that code each of the five demos carried, alongside
the existing `demo_reconnect.h`.
- The shared JSON readers are string- and escape-aware: a `{`, `}`, `[`, `]`
or `"` inside a JSON string no longer terminates a span, and `\"`, `\/`
and `\uXXXX` (including surrogate pairs) decode instead of truncating the
value. A value that does not fit its buffer now reports failure rather than
being silently truncated, and `parse_token` names the field when that
happens — an empty `derived_client_id` / `agentToken` otherwise surfaced
only as an unexplained auth failure. An out-of-range port is rejected
instead of being truncated modulo 65536.
- The music demo leaves `session_attrs_json` / `event_user_data_json` NULL
instead of spelling out a subset of the built-in defaults. Setting either
replaces the default wholesale rather than merging, so the subset was
silently dropping `tts.order.supports`, `asr.enableVad`, `tts.alternate`
and `processing.interrupt`.

- iot-client — `iot_get_qrcode_info` and `iot_get_ca_certificate` now write
into caller-provided buffers (API break)(#10).
- The single-field `iot_qrcode_response_t` struct is removed and both APIs
Expand All @@ -52,6 +72,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- Examples — the tool-less rtc-tcp-client demos answer MCP requests correctly.
text_chat, audio_chat, edu_camera and music_play each answered every
`TAI_EVT_MCP_CMD` with one canned reply that hardcoded `"id":1` — JSON-RPC
correlates a response to its request by echoing the id — and always used the
`tools/call` result shape, so the `initialize` handshake and `tools/list`
were answered with the wrong body. Opting out is not an option: the SDK's
built-in default session attributes declare `deviceMcp.supportCustomMCP`, so
a device that passes no `session_attrs_json` is asked anyway.
- New `demo_mcp.h` answers as a device with an empty tool catalog: it echoes
the request id, returns the right result shape per method
(`initialize` / `tools/list` / `tools/call`), reports unknown methods as
JSON-RPC `-32601`, and stays silent for a request with no id, which is a
notification. Its `demo_mcp_copy_id()` also replaces `mcp_demo`'s local
copy, where an id too long for the buffer used to be spliced in truncated
— dropping its closing quote and producing unparseable JSON.
- `id` and `method` are read from the request's top-level members only: a
`tools/call` may carry an `"id"` of its own inside `params.arguments`,
which a document-order search finds first. An object or array id is
refused rather than spliced back unbalanced.
- `mcp_demo` implements real tools, and now stays silent for notifications
instead of answering `"id":null`.

- Examples — text streams that cannot be reassembled are reported, not dropped
in silence. Each loss is counted in `demo_textbuf_t.dropped`, and
`music_play_demo` exits non-zero on it rather than reporting "no music skill
response" and exiting 0 for a run that lost its payload. A stream displaced by
a new `START` used to vanish without a word. A `seq` gap now warns and keeps
accumulating — the empty frames the SDK swallows consume a seq while carrying
no bytes, so continuing reassembles the right document where dropping loses a
healthy one; `-DDEMO_TEXT_SEQ_CHECK=2` drops instead, `=0` skips the check.

- Examples — NLG prose is unescaped before printing. `nlg_print_content()`
decodes `\n` and `\uXXXX` (Chinese arrived on the terminal as escapes) and
claims the empty terminator line `{"content":""}`, which used to fall through
and dump a whole JSON envelope into the middle of the prose.

- Examples — a value too long for a field that is only printed truncates
instead of being emptied: `music_play_demo` showed `Song: (unknown)` for a
title past 255 bytes and dropped long cover URLs entirely. Credentials still
reject. `parse_token` tells capacity apart from a wrong type and a bad escape,
and an audio URL that does not fit is a parse failure, not a success with no
URL.

- Examples — out-of-bounds read on received text in the rtc-tcp-client demos.
`tai_text_msg_t.text` is a borrowed slice of the SDK receive buffer and is not
NUL-terminated, but the demos ran `strstr`/`strchr` over it — reading past
`msg->len` into the previous packet's bytes and, eventually, past the end of
the `tai_ctx_t` allocation. All text handling is now length-bounded or copies
the bytes out first. Reassembly also lets the music demo recognise a SKILL
response split across `TAI_STREAM_START`/`MIDDLE`/`END` — parsed per chunk
it never matched at all.

- Examples — stack overflow from `argv` credentials in the rtc-tcp-client demos.
`devid` / `secret_key` / `local_key` were `memcpy`'d into
`iot_client_config_t`'s 32-byte fields with no length check, so an over-long
value overwrote the adjacent fields and ran past the end of the stack-local
config. All five demos now go through `demo_copy_field()`, which rejects a
value that does not fit.

- iot-client — US region renamed to AZ(#7).
- The IoT DNS region string and token prefix for the US West (Oregon) data
center is `AZ`, not `US`. The enum member `US` is renamed to `AZ`,
Expand Down
24 changes: 19 additions & 5 deletions docs-site/docs/guides/device-mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,11 +140,23 @@ static void on_event(tai_ctx_t *ctx, const tai_event_msg_t *msg, void *ud)
static void handle_mcp_request(tai_ctx_t *ctx,
const char *payload, size_t len)
{
// 解析 method 和 id
char id[64] = "null";
// msg->data 借用自 SDK 接收缓冲区且没有 '\0' 结尾,而 demo_json.h
// 的所有函数都要求 NUL 结尾的缓冲区:先按 len 拷出一份
char *req = (char *)malloc(len + 1);
if (!req) return;
memcpy(req, payload, len);
req[len] = '\0';

// 解析 method 和 id —— 只取顶层成员:tools/call 的
// params.arguments 里可能自带 "id" / "method",按文档顺序
// 搜索会先命中那一个,回显错的 id 会让服务端无法关联响应
char id[64];
char method[64] = {0};
copy_id(payload, id, sizeof(id));
json_get_string(payload, "method", method, sizeof(method));
int have_id = (demo_mcp_copy_id(req, id, sizeof(id)) == 0);
json_object_get_string(req, "method", method, sizeof(method));

// 没有 id 就是通知(notification),JSON-RPC 2.0 规定不得应答
if (!have_id) { free(req); return; }

char resp[2048];
int resp_len = 0;
Expand All @@ -167,6 +179,7 @@ static void handle_mcp_request(tai_ctx_t *ctx,
if (resp_len > 0) {
tai_send_mcp_response(ctx, resp);
}
free(req);
}
```

Expand Down Expand Up @@ -283,7 +296,8 @@ static int tool_read_sensor(const char *args_json,
## 注意事项

- 所有回调(包括 `TAI_EVT_MCP_CMD`)在后台接收线程中执行,工具函数应避免长时间阻塞
- 响应的 `id` 必须与请求的 `id` 完全一致,否则云端无法匹配
- 响应的 `id` 必须与请求的 `id` 完全一致,否则云端无法匹配。这里有两个容易踩的坑:一是 `id` 只能取**顶层**的那一个,`params.arguments` 里业务自带的 `id`(灯的 id、歌曲 id)会被"取第一个匹配"的写法先命中;二是 `id` 只能原样回填,截断一个带引号的 id 会丢掉右引号,把对象或数组形式的值截一半更会产生括号不配对的 JSON。`demo_mcp.h` 的 `demo_mcp_copy_id()` 两者都已处理
- 请求里**没有** `id` 就是通知(notification),JSON-RPC 2.0 规定不得对其应答——包括不要回 `"id":null` 的错误响应
- `tai_send_mcp_response()` 的参数是完整的 JSON-RPC 2.0 响应字符串(非 `result` 部分)
- 工具输出中的双引号和反斜杠需要转义
- 响应缓冲区大小需根据工具输出长度合理设置,避免截断
Expand Down
2 changes: 1 addition & 1 deletion docs-site/docs/tutorials/edu-camera.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: 拍学机(图片理解)
sidebar_label: 图片理解
sidebar_position: 3
sidebar_position: 4
---

# 拍学机(图片理解)
Expand Down
71 changes: 61 additions & 10 deletions docs-site/docs/tutorials/music-play.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: 音乐播放
sidebar_label: 音乐播放
sidebar_position: 4
sidebar_position: 3
---

# 音乐播放
Expand Down Expand Up @@ -187,24 +187,74 @@ AI 触发音乐技能后,会通过 `on_text` 回调返回结构化的 SKILL
}
```

示例在 `on_text` 回调中检测 `"code":"music"`,然后逐层提取 `general.data.audios[0]` 中的歌曲字段
示例的 `on_text` 回调做两件事:NLG 文本逐片即时打印(保持流式体验),同时把所有分片累积起来,流结束后再解析 SKILL 结构

```c
static void on_text(tai_ctx_t *ctx, const tai_text_msg_t *msg, void *ud)
{
if (strstr(msg->text, "\"code\":\"music\"")) {
try_parse_music(msg->text, msg->len);
dc->got_music = 1;
return;
}
/* NLG 文本:仅打印 content 字段 */
...
demo_ctx_t *dc = (demo_ctx_t *)ud;

/* NLG 文本:每个分片自成一行 JSON,到达即打印(按 msg->len 截断,
并解码 \n / \" / \uXXXX 转义)。返回 1 表示这片是 NLG 且已处理,
包括 {"content":""} 这样的空结束片——它仍然是 NLG,不能再按原样打印 */
if (nlg_print_content(msg->text, msg->len))
dc->stream_printed = 1;

/* 同时累积整个流:SKILL 响应是一份 JSON,可能跨分片,拼完整才解析 */
if (demo_textbuf_accum(&dc->text, msg) == 1)
handle_complete_text(dc); /* is_music_response → try_parse_music */
}
```

服务端可能不发独立的文本 END 分片(SDK 会丢弃空文本帧),因此 `on_event` 在收到 `TAI_EVT_END`(回合结束)时调用 `demo_textbuf_flush()` 兜底交付缓冲中的流。

:::caution 两个必须注意的约束
- **`msg->text` 没有 `\0` 结尾**。`tuya_ai.h` 中明确标注该指针借用自 SDK 接收缓冲区且非 NUL 结尾,对它直接调用 `strstr` / `strchr` / `strcmp` 会越过 `msg->len` 读到上一个数据包的残留字节。所有解析都必须先按 `msg->len` 把数据拷出来。
- **文本按 `stream_flag` 分片下发**(`TAI_STREAM_START` / `MIDDLE` / `END`,或单个 `ONE_SHOT`)。只做打印的场景可以逐片处理,但解析 JSON 结构必须先重组整个流,否则 `"code":"music"` 与 `audios` 可能落在不同分片里。

这两件事由 `demo_text.h` 的 `demo_textbuf_accum()` / `demo_textbuf_flush()` 统一处理;断线重连前用 `demo_textbuf_reset()` 丢弃旧连接的半截流。
:::

:::info 缓冲区只装一条流
`demo_textbuf_t` 一次只重组一条文本流,也**无法**分离回合内交错的两条流——`tai_text_msg_t` 里没有可用于分路的字段:同一回合内所有文本包共享同一个 `event_id`(SDK 只 latch 一个回合 id,`TAI_EVT_END` 后清空)和同一个 `data_id`(`TAI_DATA_ID_TEXT_DOWN`)。

能做的是**察觉**,分两种情况:

**一条流被新流顶掉**——上一条流还没收到 END,就来了 `START` / `ONE_SHOT`。缓冲区只装一条流,旧的那条必然丢失,示例把它计入 `tb->dropped` 并打印告警,而不是无声丢弃:

```
[demo_text] a new stream started while 214 bytes of the previous one were still buffered: dropping those — ...
```

**`seq` 出现缺口**——`seq` 是回合内的文本包计数器,缺口说明有应用没看到的分片消耗了序号。但这个信号是**有歧义**的:SDK 自己会丢弃零长度文本帧(`tai_protocol.c` 的 `media_text()` 仅在 `payload_len > off` 时上抛),这类帧不携带任何字节,跨过它们拼出来的正是那份正确的文档;只有当缺失的分片属于另一条交错的流时,继续拼接才会把两份文档混在一起——而那种混合物随后会在 JSON 解析处被拒。因此默认策略是**打印告警后继续累积**:

```
[demo_text] text seq gap (11 -> 13): continuing — ...
```

若某个部署里交错才是更可能的原因,用 `-DDEMO_TEXT_SEQ_CHECK=2` 改为遇缺口即丢流;`-DDEMO_TEXT_SEQ_CHECK=0` 完全关掉该检查。

无论哪种丢失,`demo_textbuf_t.dropped` 都会累加(`demo_textbuf_reset()` 不会清零它),示例在退出前据此判定成败——丢了流却报告"本次查询没有音乐响应"并返回 0,会让脚本把丢数据的运行当成成功。
:::

另外,`code` 字段要在 SKILL 信封的 `data` 对象里取,而不是在整份文档里取第一个匹配——外层常见的 `{"code":0,"msg":"ok","data":{"code":"music",...}}` 结构会让"取第一个 code"拿到状态码 `0`,从而静默丢弃这条音乐响应。

## 公共辅助头文件

`examples/posix/ai/rtc-tcp-client/` 下的五个示例共用三个头文件,避免各自复制一份解析代码:

| 头文件 | 内容 |
|--------|------|
| `demo_json.h` | 极简 JSON 读取(字符串感知的括号配对、`\"` / `\/` / `\uXXXX` 转义解码)、Base64 解码、session token 解析、定长配置字段的有界拷贝 |
| `demo_text.h` | `tai_text_msg_t` 的安全处理:按长度截断的查找、NLG 正文解码打印、文本流重组与丢流记账 |
| `demo_mcp.h` | 设备端 MCP 应答:回显请求 `id`、按 method 返回正确形状;无工具设备用 `demo_mcp_reply_no_tools()` |
| `demo_reconnect.h` | 应用侧重连策略(指数退避 + 熔断器) |

`demo_json.h` 中的所有函数都要求传入 **以 `\0` 结尾** 的缓冲区;回调里的 `msg->text` / `msg->data` 需要先拷贝。

## NLG 文本输出

非音乐响应的 NLG 文本(AI 的语音回复文字)会流式打印。示例从 JSON 中提取 `content` 字段,仅输出文本内容
非音乐响应的 NLG 文本(AI 的语音回复文字)会按文本流逐段打印。示例用 `nlg_print_content()` 从 JSON 中提取 `content` 字段,**解码其中的 JSON 转义**后仅输出文本内容——服务端常把中文写成 `\uXXXX`,不解码的话终端上看到的是 `你好` 而不是「你好」

```
Response: 正在为您播放周杰伦的歌
Expand Down Expand Up @@ -245,3 +295,4 @@ AI 音乐功能默认返回的是**试听版**歌曲,存在时长限制(通
- 当前示例仅解析并展示第一首歌曲信息;如需播放完整音频,请在设备端实现音频播放器。
- 元数据展示框按**显示宽度**(中文字符占 2 列)对齐,而非字节数;过长的字段会在字符边界截断。
- `on_audio` 回调在本示例中为空实现,不处理 TTS 音频数据。
- 本示例声明了 MCP 支持但未实现任何工具,`on_event` 收到 `TAI_EVT_MCP_CMD` 时调用 `demo_mcp.h` 的 `demo_mcp_reply_no_tools()` 作答。注意 **SDK 的内置默认属性本来就打开 MCP**,所以不传 `session_attrs_json` 的设备同样会收到 MCP 请求,必须能正确应答。要实现真正的设备工具请参考 `mcp_demo.c`。
Loading
Loading