Skip to content

AGENTS.md

Latest commit

 

History

History
447 lines (369 loc) · 22.2 KB

AGENTS.md

File metadata and controls

447 lines (369 loc) · 22.2 KB
Raw

AGENTS.md

This file guides Codex (Codex.ai/code) when working in this repository. It is kept deliberately factual — every claim below was verified against the source tree. When you change something structural, update this file.

Project Overview

Backtrader is a Python algorithmic-trading backtesting framework supporting low-, mid-, and high-frequency strategy development, backtesting, and live trading. This repo is a performance-oriented fork of the original backtrader that removes metaclass-based metaprogramming in favor of explicit mixin + factory initialization while keeping the public API compatible.

  • Version: 1.4.0 (see backtrader/version.py)
  • License: GPLv3
  • Python: 3.8–3.13 (classifiers in setup.py; 3.11 recommended)
  • Not on PyPI — install from source only.

Branch context

This repository uses a three-branch model (authoritative source: docs/source/developer-guide/branch-governance.md):

  • dev — daily development entry. Routine features, ordinary bug fixes, docs, tests, refactors, and community contributions land here first.
  • development — improved & optimized version. Optimization capabilities, architecture improvements, and optimization-only regression fixes.
  • master — original Backtrader baseline. Only bug/compatibility/security fixes that reproduce on the original baseline, via hotfix/master-* PRs. Used as the correctness baseline (regression tests bake master's metrics as expected values).

Other branches (crypto, ctp, dev_cython, etc.) are feature/experiment branches; do not target them unless asked.

Do not push directly to master or development. Push to dev. git push is configured to push to both GitHub (cloudQuant/backtrader) and Gitee (yunjinqi/backtrader) remotes.

Implementation status / reality checks

These correct common stale assumptions — verify before relying on docs:

  • Pure Python today. Although cython>=0.29.0 is a declared dependency and older docs reference compile_cython_numba_files.py, there are currently no tracked .pyx files and no ext_modules in setup.py. pip install builds a pure-Python package. The only native-ish acceleration in the tree is numba used inside backtrader/utils/dateintern.py. Do not assume a Cython build step is required or present.
  • Metaclasses are gone. Object construction goes through metabase.ObjectFactory / BaseMixin.donew and ParamsMixin.__init_subclass__ (a patched_init wrapper), not a metaclass __call__.
  • File sizes below are real line counts, not the inflated numbers in earlier revisions of this doc.

Development commands

Install

pip install -r requirements.txt   # core + dev deps
pip install -U .                  # build & install
pip install -e .                  # editable/dev install

No separate Cython compile step is needed for a normal install.

Testing (tiered — see Makefile and conftest.py)

The strategy regression suite is large (~10 min full). Tests are split into tiers by measured per-file duration, applied dynamically at collection time (no test files are edited):

make test-fast        # parallel non-performance tests + serial wall-clock
                      #   microbenchmarks; excludes slowest ~65% of strategy tests.
                      #   Daily "did I break anything" loop.
make test-slow        # the slowest ~65% strategy tests test-fast skips
make test-strategies  # all 1,271 strategy regression tests (~9 min)
make test-all         # parallel functional suite + serial wall-clock microbenchmarks
make test-performance # wall-clock microbenchmarks without xdist
make test-coverage    # coverage report

# Single test, verbose:
pytest tests/path/to/test_file.py::test_name -v --tb=short

How the split works:

  • conftest.py::pytest_collection_modifyitems reads tests/functional/strategies/.test_durations.json (committed), computes the BT_SLOW_PERCENTILEth percentile (default 35) of recorded durations, and tags any strategy file at/above it with the existing slow marker.
  • Unknown/new files default to the FAST tier, so newly added or regenerated tests always run on test-fast — exactly what you want for catching new bugs.
  • Tune coverage vs speed: BT_SLOW_PERCENTILE=25 make test-fast (faster) … =50 (broader).
  • Refresh timings after adding/removing strategy tests: python scripts/refresh_strategy_durations.py.

Wall-clock microbenchmarks and time-bounded latency contracts have a separate serial lane: those tests are explicitly skipped under xdist or coverage tracing and make test-performance runs them without either. Its short RSS stress profile uses a separate fresh pytest process so suite-import RSS cannot be mistaken for the profile's process-tree budget. make test-fast and make test-all include that serial lane after their parallel functional tests, preserving each performance contract without treating worker scheduling noise as an application regression.

Choosing which backtrader to test against

Running pytest from the repo root resolves import backtrader to the local repo copy by default. To test the installed site-packages copy instead:

BACKTRADER_USE_INSTALLED=1 pytest ...        # env var
pytest ... --use-installed-backtrader        # CLI flag

The active backtrader.__file__ is printed in the pytest session header. The switch works under pytest-xdist parallel mode. Logic lives in conftest.py.

Code quality

make format         # black, line-length 100
make format-check
make lint           # ruff
make type-check     # mypy
make security       # bandit
make quality-check  # all of the above (no tests)
bash scripts/optimize_code.sh   # pyupgrade + isort + black + ruff + tests

Docs & utilities

make docs / docs-en / docs-zh   # Sphinx docs (English + Chinese)
make help                       # list all make targets
make clean                      # clean build artifacts

Architecture

Construction pipeline (replaces the old metaclass)

Object creation flows through backtrader/metabase.py:

  • ObjectFactory.create(cls, *args, **kwargs) runs the lifecycle hooks: doprenew → donew → dopreinit → doinit → dopostinit.
  • BaseMixin provides default donew/dopreinit/doinit/dopostinit.
  • ParamsMixin.__init_subclass__ installs a patched_init wrapper on each subclass's __init__ that wires up self.p/self.params, sets data0/data1 aliases, and runs the lifecycle. Most indicators are constructed through this patched_init path, not ObjectFactory.create directly.
  • Owner discovery uses metabase.OwnerContext (a context stack) and metabase.findowner() — the legacy stack-frame inspection is gone.

Strategy has a separate explicit path: Strategy.__new__() creates self.p and broker/analyzer state, and Strategy.__init__() creates datas/data aliases and _clock before calling the direct subclass's user __init__(). Therefore a class that directly subclasses bt.Strategy does not need to call super().__init__(); doing so currently re-enters Strategy.__init__(). Cooperative custom Strategy parents/mixins must still call super() when their own parent initialization is required. Indicators and other ParamsMixin objects follow their own patched-init lifecycle and must not be validated using the direct-Strategy exception. Never reintroduce a metaclass — use mixins + donew().

Line system (bottom-up)

LineRoot → LineBuffer → LineSeries → LineIterator

  • lineroot.py — base interfaces, period management, stage1/stage2.
  • linebuffer.py (~2,800 lines) — circular-buffer line storage; also defines LineActions / LinesOperation (the objects produced by expressions like (data.high + data.low) / 2.0).
  • lineseries.py (~2,450 lines) — Lines/LineSeries, LineSeriesStub, LineSeriesMaker.
  • lineiterator.py (~2,920 lines) — LineIterator, IndicatorBase, DataAccessor; iteration phases and the _clock resolution helpers (_line_like_source_clock, _resolve_authoritative_buflen, _ensure_lineactions_inputs_computed).

Access patterns: data.close[0] (current bar), data.close[-1] (previous).

Components (all extend LineIterator)

  • indicator.py (Indicator, _ltype=IndType=0) + indicators/ (50 files).
  • observer.py + observers/ — chart observers; notably observers/trade_logger.py (TradeLogger) for JSON order/trade/signal/ position logs (used by the branch-compare tooling).
  • analyzer.py + analyzers/ (17 files) — Sharpe, drawdown, returns, SQN, …
  • sizer.py + sizers/, signal.py + signals/, comminfo.py + commissions/.

Data, broker, engine

  • feed.py + feeds/ (17 files) — CSV, pandas, IB, CCXT, etc.; resamplerfilter.py for resample/replay.
  • broker.py + brokers/ — order matching and portfolio state.
  • cerebro.py (~830 lines, public facade) + _cerebro/ private mixin package (9 files, iteration 28 split) — orchestrator. The facade keeps the Cerebro class definition (params/descriptors/__init__/run/pickle protocol) and OptReturn; registry/notifications/lifecycle/channel/execution hold configuration, dispatch and orchestration; runnext/runonce hold the four engine loops (hot paths — verbatim-moved, see docs/_internal/opts/requirements/迭代28-Cerebro模块化拆分/). run() → runstrategies() → _runonce() (vectorized) or _runnext() (event-driven). Tick-level mode is also supported.

Indicator registration & multi-data clocks (high-bug-risk area)

  • An indicator registers with its owner via LineIterator.addindicator() (lineiterator.py:1584), appending to owner._lineiterators[ind._ltype]. If an indicator isn't registered it won't update during the run.
  • Multi-timeframe gotcha: an indicator built on a secondary feed — e.g. SMA((h1.high + h1.low)/2.0) or EMA(EMA(h4.close)) inside an M15 strategy — must advance on the secondary feed's clock, not the strategy's primary feed. In runonce mode this is handled in Strategy._periodset(), which resolves each indicator's data dependency to its concrete feed and pins indicator._resolved_secondary_clock; the post-phase advance loop in _oncepost() and Indicator.advance() honor that clock. See docs/DEV_REGRESSION_FAILURES.md for the full diagnosis of the bug class this fixes. When touching clock/minperiod logic, run make test-strategies — these multi-data cases are exactly what regress.

Execution phases

prenext (before minperiod) → nextstart (minperiod first met) → next (normal). Vectorized mode uses once() (preonce/oncestart/once) to fill whole line arrays in batch, then replays per bar.

Data flow

Data Feed(s) → Cerebro → Strategy → Indicators / Observers / Analyzers
                   ↓
                Broker ← Orders

Special modes

  • TS (time series) and CS (cross-section) modes for multi-asset portfolio backtests (utils/ helpers; some docs reference dedicated value calculators — confirm presence before relying on them).
  • Multiple plotting backends: Plotly (plot/), Bokeh (bokeh/), Matplotlib.
  • Report generation: reports/ (reporter.py, performance.py, charts.py).

Repository layout

backtrader/            core library
  cerebro.py (facade) + _cerebro/ (private engine mixins) strategy.py
  indicator.py analyzer.py observer.py broker.py feed.py
  metabase.py parameters.py
  lineroot.py linebuffer.py lineseries.py lineiterator.py dataseries.py
  indicators/ analyzers/ observers/ feeds/ brokers/ filters/ sizers/ signals/
  commissions/ stores/ channels/ mixins/ plot/ bokeh/ reports/ configs/ utils/
AI strategy products are maintained outside this repository:
  cloudQuant/backtrader-skills   standalone author/review/test skills product
  cloudQuant/backtrader-mcp      standalone local-stdio MCP product
  cloudQuant/backtrader-agent    standalone stateful agent product
tests/                 unit/ functional/ integration/ performance/ original_tests/
  add_tests/ strategies/ bench/ datas/ fixtures/ factories/ test_utils/
  functional/strategies/   1,271 inlined regression tests in ~30 categories
docs/                  Sphinx docs (EN + ZH) + design/bug notes
scripts/               optimize_code.sh, refresh_strategy_durations.py,
                       run_strategy_branch_compare.py, …
studies/               research/diagnostic scripts (e.g. branch_compare/)
examples/012_1_midfreq_cross_exchange/  mid-frequency OKX/Binance perpetual example
examples/012_2_event_driven_cross_exchange/ event-driven OKX/Binance perpetual candidate
examples/013_3_sa_midfreq_simnow/ controlled CTP/SimNow SA mid-frequency example
examples/strategy-candidate-manifest.json  hash-bound research/demo admission manifest
examples/strategy_candidate_approval.py  candidate-specific receipt/provenance policy
Makefile pyproject.toml setup.py pytest.ini requirements.txt conftest.py

The three AI products are not vendored and are not Git submodules. Make product changes, packaging releases, and product-specific acceptance changes in their respective repositories; this repository only links to them from its README.

The cross-exchange arbitrage examples use BtApiStore.getdata() / BtApiFeed with orderbook_as_ticks=True and TimeFrame.Ticks. Native notify_orderbook callbacks drive bt.Strategy.buy/sell and notify_order; BtApiBroker routes demo orders through public BtApi methods with normalized=True. The SDK owns venue schemas, request mapping and optional execution-session state (durable intents, unique client IDs, uncertain-order reconciliation and fees). bt_api_py.cross_venue owns only provider-neutral, stateless typed execution planning: quantity lattices, executable VWAP, cost accounting, funding schedule validation and normalized orderbook evidence. It consumes SDK contracts and does not own a client, account, order, pair state, alpha, or compensation policy. The store holds BtApi directly and only maps framework orders, references and native market-data objects; there is no second Backtrader trading client. examples/strategy_candidate_approval.py binds the two example candidates' manifest, offline receipt and source provenance. It is example admission policy, not a Backtrader utility or SDK protocol. OKX endpoint selection belongs to the SDK through api_region=global|eea|us|tr: REST plus public/private/business WebSockets use one atomic region/environment profile. Global/EEA/US support production and demo; TR currently supports only production, so tr+demo fails before network I/O. OKX 50119 proves that the selected credential/domain combination was rejected; by itself it does not distinguish region, key, secret, passphrase, expiry, or permission causes. Funding is a typed SDK read model. BtApiStore refreshes it on a separate single-concurrency read-only lane with request coalescing, TTL/schedule-boundary expiry, and generation fencing; strategy callbacks only read the local cache. This snapshot supports entry reserves and settlement-window risk. The SDK does not yet expose a unified, pagination-complete, account-bound OKX/Binance funding cashflow ledger, so a cycle crossing settlement cannot claim complete realized net PnL. The production status is PRODUCTION_BLOCKED_ACTUAL_FUNDING_CASHFLOW_LEDGER: venue-level single-page raw income/bills parsers do not prove pagination coverage, identity, deduplication, aggregation, settlement latency, or a complete empty result. Idle risk also advances without a new bar through notify_idle polling. The exchange_kwargs and symbol_routes configuration supports multiple providers in a single broker. Amounts remain native units (OKX contracts / Binance BTC); strategy sizing uses metadata multipliers. The examples require independently verified dual-side/hedge mode and maintain long/short legs separately; no net-position fallback is accepted. shadow uses public production books with zero orders/fills/PnL, paper-live uses public books with local hypothetical fills, and only demo can submit exchange orders. Demo writes additionally require a strategy-specific, hash-bound approval receipt. Both Iteration 21 frozen candidates failed their pre-OOS calibration cost screen, so their paper-live simulated-fill and demo order paths remain prohibited; read-only shadow and demo preflight remain available. Any new economic attempt requires a new candidate ID, preregistration, and untouched holdout. Credentials are kept in each example's ignored .env. Deterministic replay reports are formula fixtures with zero orders/fills and no PnL; the native Store/Feed/Cerebro/Broker path is tested separately. The second candidate is classified as event-driven and remains HFT FAIL/NOT_ADMITTED until end-to-end latency, queue and real-fill evidence exists.

The Iteration 22 SA example uses one authoritative BtApiFeed to dispatch CTP quote events and form watermark-closed one-minute bars. CTP trading admission requires typed terminal account/position/order/trade/reference queries bound to one stable connection generation and account fingerprint. replay is an offline zero-write path, shadow is read-only, and simnow additionally requires a hash-bound approval receipt plus a complete first-set observation gate. The example never treats replay output or a single SimNow day as evidence that the strategy is profitable.

Tests

  • tests/functional/strategies/ holds 1,271 inlined regression tests across ~30 categories (trend_following, mean_reversion, asset_allocation, machine_learning, options, pairs_trading, …). Each is self-contained: inline strategy + data loader + cerebro.run() + assertions against master-baselined metrics.
  • tests/unit/, tests/integration/, tests/performance/, tests/original_tests/, tests/add_tests/ cover the framework itself.
  • Config: pytest.ini (markers incl. slow, warning filters), conftest.py (temp cleanup, installed-vs-local switch, slow auto-marking).
  • tests/datas/ holds fixtures; MT5 daily CSVs in tests/datas/mt5_1d_data/.
  • New regression tests should pass on both dev and master (bake master's output as the expected values). Some tests/unit/brokers/*_performance tests are flaky under heavy -n 8 parallelism (timing-sensitive; pass in isolation).

Common tasks

Add an indicator

  1. New file in backtrader/indicators/; subclass bt.Indicator.
  2. lines = ('out',), params = (('period', 30),).
  3. Build the calculation in __init__ (assign self.lines.out = ...) and/or implement next() / once(start, end) for explicit modes.
  4. Register in indicators/__init__.py.
  5. If it consumes a secondary feed or a LinesOperation, test runonce vs runnext parity (multi-data clock alignment).

Add a strategy

  1. Subclass bt.Strategy; declare params.
  2. Build indicators in __init__; trading logic in next().
  3. Use self.buy()/sell()/close().

Debug line/indicator issues

  • len(obj), obj._minperiod, obj._owner, obj._ltype == 0 (IndType).
  • Confirm obj in owner._lineiterators[0].
  • For multi-data drift, inspect obj._clock and obj._resolved_secondary_clock and compare runonce vs runnext output (the branch-compare harness in studies/branch_compare/ + scripts/run_strategy_branch_compare.py with TradeLogger is the established way to localize divergences).

Logging (iteration 29)

  • Single entry point backtrader/utils/log_message.py (get_logger, configure_logging, throttled storm suppression). See docs/LOGGING_GUIDELINES.md; baseline catalogs are regenerable via python scripts/scan_logging_baseline.py --out <dir>.
  • Default silence: nothing is emitted or written until configure_logging(...) is called (protected by tests).
  • Split-file layout (opt-in): configure_logging(level="INFO", log_dir="logs") writes logs/<script>/<YYYY_MM_DD>/{error,warning,info}.log (level-exact routing, debug.log at DEBUG level). <script> auto-detects from sys.argv[0] (xxx/run.py -> xxx_run); script_name= overrides; retention_days=30 prunes only that script's expired date dirs; child processes (cerebro optimize) get .p{pid} suffixes.
  • Write backend: backend="auto" prefers the optional spdlog package (PyPI spdlog 2.0.6, sdist build — macOS/Python 3.11 verified working) and silently falls back to stdlib; "spdlog" raises ImportError if unavailable; "stdlib" forces pure stdlib. get_logger() always returns a stdlib Logger — spdlog is mounted as a logging.Handler.
  • Silent-exception policy: no new except: pass/continue without a log line; bare re-raises log an ERROR first; hot-loop repeats use throttled_error/throttled_warning. CLI tools (btrun, reports/reporter.py) and public APIs (Analyzer.print, Strategy.log) keep print on purpose.
  • Key lifecycle INFO (run start/finish, feed load, strategy nextstart/stop, order submit/fill/reject in bbroker) is per-run/per-order — never per bar; keep it that way in hot paths.

Code style & constraints

  • Line length 100 (black); ruff/isort at 121. Type hints encouraged.
  • Bilingual (EN/ZH) comments are normal in this codebase.
  • Never introduce new metaclasses — use mixins with the donew() pattern.
  • Preserve public API compatibility.
  • Minimize isinstance()/hasattr()/len() in hot paths.
  • Performance work already done: metaclass removal, broker __getattribute__/param-cache optimization, indicator once() tuning.

Config files

  • pyproject.toml — black, ruff, isort, mypy, bandit, coverage.
  • pytest.ini — discovery, markers, warning filters. No Kiro steering files are tracked; use this AGENTS.md, README.md, and the project configuration files as the current build/test/structure guidance.

外部文件加载

CRITICAL: 当你遇到文件引用时（例如 @rules/general.md），使用你的读取工具按需加载。它们与当前具体任务相关。

说明:

  • 不要预先加载所有引用 - 基于实际需求使用懒加载
  • 加载后，将内容视为强制性指令，覆盖默认设置
  • 在需要时递归地遵循引用

开发规范

当前项目下存在前端和后端项目，开发前请阅读并遵守以下开发规范

后端开发规范：@.joyincode/rules/backend.md 前端开发规范：@.joyincode/rules/frontend.md