feat(worker): split ai workloads

[nguyenvantruc92]

Summary

• Splits RAM/CPU/I/O-intensive AI work out of the latency-sensitive integration worker into a new heavy BullMQ queue and dedicated worker.
• Keeps orchestration and conversational AI jobs in aiAgent, while moving file processing, image generation/editing, speech-to-text, text-to-speech, document extraction, and image analysis to the bounded heavy-worker path.
• Adds typed, Zod-validated job contracts and targeted test coverage across queue schemas, workers, retry handling, downloads, and flow-step execution.

Changes

• New heavy queue and worker, with per-action retry/backoff, retention, completion deadlines, metrics, and configurable concurrency.
• New heavy-job handlers for AI files, image generation/editing/analysis, speech-to-text, text-to-speech, and text extraction.
• Flow heavy steps now enqueue work and wait for its final terminal state, including retries, before advancing the flow.
• AI agent queue now owns automated responses, AI comment replies, and story-reply automation; job payloads are validated through a discriminated Zod union.
• AI-file embeddings use a repository boundary, and embeddings remain pending while a retried heavy job has not reached a terminal state.
• Provider rate limiting centralizes protection for expensive provider calls.
• Download handling is bounded and validates every redirect target before fetching, preventing redirects from bypassing the public-URL safety check.
• Image-model selection and flow-step schemas are updated to match the new heavy-job contracts.
• Worker boot and queue behavior remain safe in test/build environments without Redis.

Behavior changes

• Heavy AI work no longer occupies integration-worker capacity, reducing the risk that slow providers or large files delay inbound channel processing.
• A transient heavy-job failure is retried according to the action’s configured policy; callers wait through retry attempts rather than treating the first failed attempt as terminal.
• Redirected file downloads are rejected when any redirect resolves to a non-public URL.
• AI-file processing now executes asynchronously through the heavy queue.

Test plan

• pnpm --filter @chatbotx.io/worker-config check-types && test
• pnpm --filter @chatbotx.io/ai check-types && test
• pnpm --filter worker check-types && test
• pnpm --filter builder check-types && test
• pnpm lint
• Manual: process an AI file; run image generation/editing, STT, and TTS flow steps; verify transient provider failures retry and integration traffic remains responsive.