Skip to content
This repository was archived by the owner on Apr 13, 2026. It is now read-only.
This repository was archived by the owner on Apr 13, 2026. It is now read-only.

feat(ai): evaluate lighter LLM backend for general-purpose hardware #14

@db-tycoon-stephen

Description

@db-tycoon-stephen

Background

The current `tycoon ai` stack is hard-wired to LM Studio's local server at `localhost:1234`. LM Studio is a heavy desktop app — it's fine for development on a high-end machine but is not a realistic dependency for general-purpose users.

Research findings

  • OpenAI Codex CLI — cloud-only, conflicts with local-first design. Not worth pursuing.
  • OpenAI gpt-oss-20b — Apache 2.0 open-weight model, 16GB RAM, available on Ollama. Viable locally, but adds Ollama as a dependency.
  • Ollama — lighter than LM Studio, broader hardware support, CLI-first, no GUI. But still a separate install.

Goal

Find the single shortest path to running a capable model for the specific tasks tycoon needs:

  • `TestFixer` — fix a failing dbt test
  • `ColumnDocumenter` — generate schema.yml descriptions
  • `StagingImprover` — refactor a staging model

These are focused, single-turn tasks with small context windows. They do not need a general-purpose chat model or a large context.

Design questions to answer

  1. What is the lightest runtime that can serve a small model locally without a GUI dependency? (Ollama, llama.cpp server, mlx, transformers pipeline?)
  2. Is there a model small enough to run on CPU-only hardware that is still useful for these tasks? (e.g. Qwen2.5-Coder-1.5B, Phi-3-mini)
  3. Should `tycoon ai` ship with a recommended model + one-line install command, rather than requiring the user to set up LM Studio separately?
  4. Can we reduce `tycoon.ai.client` to a single, minimal HTTP call with no backend abstraction layer?

Out of scope

  • Multiple LLM backend options / provider abstraction layer
  • Cloud API fallbacks
  • OpenAI Codex CLI integration

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions