diff --git a/tools/kiro-attribution/README.md b/tools/kiro-attribution/README.md new file mode 100644 index 00000000000..05fe7e1838c --- /dev/null +++ b/tools/kiro-attribution/README.md @@ -0,0 +1,148 @@ + +# Binding Kiro sessions to git commits + +## What this solves + +Usage data alone cannot answer whether AI helped a given person: high credit +consumption is compatible with both "the agent did a lot of work" and "the agent +went in circles". Answering it needs to know which commits the AI actually +contributed to. + +Kiro has no git-lifecycle hook (the feature request, kirodotdev/Kiro#6436, is +still open), and its S3 interaction logs carry no `conversationId`, so neither +source can bind a session to a commit on its own. + +## Why a trailer rather than a time window + +The obvious alternative - "mark a commit as AI-assisted if Kiro was active +within 30 minutes" - has a base-rate trap. For anyone who keeps Kiro open all +day, every commit qualifies. The attribution rate approaches 100%, and the label +stops distinguishing between people, which is exactly what it was for. Heavy +users are the population the measurement most needs to separate. + +A trailer records what happened instead of inferring it. That turns attribution +from probabilistic into deterministic, which in turn unlocks the strongest +comparison available from observational data: **the same person, in the same +week, on AI-assisted versus unassisted work**. That within-person comparison +holds individual skill, team environment and time period constant, so the +remaining difference can only come from AI involvement. + +Trailers are also a standard git mechanism - `Co-Authored-By:` and +`Signed-off-by:` use the same convention - so nothing here is bespoke, and the +value stays readable to anyone inspecting the log. + +## How it works + +``` +Kiro Stop hook ──► .git/kiro-sessions (epoch, session id, ISO time) + │ +git prepare-commit-msg ────┘ + ▼ + commit message: + fix: payment retry + + Kiro-Session-Id: sess-abc-123 + │ + ▼ + DevLake already collects commits.message, + so extraction needs no new collector. +``` + +## Installation + +Run the installer once per repository: + +```sh +tools/kiro-attribution/install.sh +``` + +It installs the git `prepare-commit-msg` hook in the directory Git actually +uses (respecting a writable `core.hooksPath`, or falling back to `.git/hooks` +for managed tooling that forwards there). It does not change `core.hooksPath`. +If the repository has a `.kiro/` directory, it also writes the current Kiro +hook schema to `.kiro/hooks/kiro-attribution.json`: + +```json +{ + "version": "v1", + "hooks": [ + { + "name": "kiro-session-record", + "description": "Records the Kiro session id so the next commit carries a Kiro-Session-Id trailer.", + "trigger": "Stop", + "action": { + "type": "command", + "command": "bash tools/kiro-attribution/kiro-session-record.sh" + }, + "timeout": 15, + "enabled": true + } + ] +} +``` + +The `bash` prefix is required by Kiro's command permission rules. Restart Kiro +after installation because hooks are loaded when a session starts. + +Check or remove the installation with: + +```sh +tools/kiro-attribution/install.sh --check +tools/kiro-attribution/install.sh --uninstall +``` + +## Verified behaviour + +| Scenario | Result | +|---|---| +| No AI session before the commit | No trailer | +| One session | One trailer line | +| Several sessions since the last commit | One line each, deduplicated | +| A commit with no new session | No trailer, and does **not** inherit the previous commit's | +| `--amend` | Trailer not duplicated | + +Two defects were found and fixed while testing these, both of which failed +silently rather than erroring: + +- **Timezone mismatch.** The recorder wrote UTC while the commit hook compared + against git's local-time output, an 8-hour gap here. The trailer simply never + appeared. Both sides now use epoch seconds, which has no timezone to disagree + about. +- **Boundary off by one.** With `>=`, a session recorded in the same second as a + commit was counted again by the *next* commit, so a commit containing no AI + work inherited the previous trailer. Now strictly `>`. + +## Known limitations + +**Client-side, therefore optional.** The hooks live on the developer's machine +and can be removed or disabled. That reintroduces self-selection: "no AI usage" +and "no hook installed" become indistinguishable. **Hook coverage must itself be +tracked as a metric**, and attribution figures from an unknown-coverage +population must not enter a conclusion. + +The S3 exports are the complement here: organization-controlled, tamper-proof, +and complete. Use S3 for the denominator (who uses Kiro, how much) and trailers +for attribution (which commits it touched). + +**No backfill.** Only commits made after installation carry a trailer. Team-level +analysis over history has to rest on the S3 usage data. + +**IDE session id unverified.** The CLI hook payload carries `session_id` (Kiro's +own docs, and confirmed by the existing adapter in `.kiro/hooks/`). The IDE +documentation lists only `USER_PROMPT`, so the IDE path needs a live probe +before it can be relied on. diff --git a/tools/kiro-attribution/install.sh b/tools/kiro-attribution/install.sh new file mode 100755 index 00000000000..8e70b8eb498 --- /dev/null +++ b/tools/kiro-attribution/install.sh @@ -0,0 +1,303 @@ +#!/bin/sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Installs Kiro commit attribution into a repository. +# +# Usage: +# ./install.sh install into the current repository +# ./install.sh /path/to/repo install into another repository +# ./install.sh --check report status without changing anything +# ./install.sh --uninstall remove what this script installed +# +# Where the hook goes depends on core.hooksPath, which git consults in place of +# .git/hooks whenever it is set. Two tools set it in practice and they behave +# differently, so the destination is decided by testing rather than by name: +# +# husky (repository scope, e.g. config-ui/.husky) does NOT forward to +# .git/hooks - its shim only re-executes itself. Verified by installing into +# .git/hooks in such a repo and watching the trailer never appear. So the hook +# must go into the husky directory itself. +# +# git-defender (system scope, a root-owned directory) DOES invoke +# .git/hooks/prepare-commit-msg - its own error text says "Your local +# prepare-commit-msg hook failed". Its directory is not writable anyway, so +# .git/hooks is both the only option and the correct one. +# +# The rule that covers both: install into the directory git reads, unless that +# directory is unwritable, in which case fall back to .git/hooks - which is +# exactly the case where the tooling forwards there. +# +# core.hooksPath itself is never modified. Redirecting it would be worse than +# useless under git-defender, which reports the event ("User had a non Code +# Defender hooks path value set"), tripping a security signal while solving +# nothing. +set -eu + +SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) +MODE=install +TARGET_REPO=. +MARKER='# kiro-attribution' + +for arg in "$@"; do + case "$arg" in + --check) MODE=check ;; + --uninstall) MODE=uninstall ;; + --help|-h) sed -n '2,8p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -*) echo "unknown option: $arg" >&2; exit 2 ;; + *) TARGET_REPO=$arg ;; + esac +done + +cd "$TARGET_REPO" 2>/dev/null || { echo "not a directory: $TARGET_REPO" >&2; exit 1; } +# --absolute-git-dir, not --git-dir: the latter returns a path relative to the +# current directory (".git" at the repo root), and the verification step below +# cd's elsewhere. Every path derived from a relative GIT_DIR would then resolve +# against the wrong directory - silently, since sed on a missing file just +# produces an empty hook that chmod happily marks executable. +GIT_DIR=$(git rev-parse --absolute-git-dir 2>/dev/null) || { + echo "not a git repository: $TARGET_REPO" >&2; exit 1 +} +REPO_ROOT=$(git rev-parse --show-toplevel) +SESSION_FILE="$GIT_DIR/kiro-sessions" + +# resolve_hook_dir prints the directory to install into, plus a short reason. +resolve_hook_dir() { + _hp=$(git config --get core.hooksPath 2>/dev/null || true) + if [ -z "$_hp" ]; then + echo "$GIT_DIR/hooks|git default" + return + fi + case "$_hp" in + /*) _abs=$_hp ;; + *) _abs="$REPO_ROOT/$_hp" ;; + esac + # Writability is the deciding test. An unwritable hooksPath belongs to managed + # tooling, and that tooling is the kind that forwards to .git/hooks. + if [ -d "$_abs" ] && [ -w "$_abs" ]; then + echo "$_abs|core.hooksPath ($_hp)" + elif [ ! -d "$_abs" ]; then + echo "$_abs|core.hooksPath ($_hp), creating it" + else + echo "$GIT_DIR/hooks|core.hooksPath ($_hp) is not writable, falling back" + fi +} + +RESOLVED_DIR=$(resolve_hook_dir) +HOOK_DIR=${RESOLVED_DIR%|*} +HOOK_WHY=${RESOLVED_DIR#*|} +HOOK_FILE="$HOOK_DIR/prepare-commit-msg" + +# ---------------------------------------------------------------- check -------- + +if [ "$MODE" = check ]; then + echo "repository: $REPO_ROOT" + + echo "hook dir: $HOOK_DIR" + echo " chosen by: $HOOK_WHY" + + if [ -f "$HOOK_FILE" ] && grep -q "$MARKER" "$HOOK_FILE" 2>/dev/null; then + if [ -x "$HOOK_FILE" ]; then + echo "commit hook: installed" + else + echo "commit hook: installed but NOT EXECUTABLE - git will skip it" + fi + elif [ -f "$HOOK_FILE" ]; then + echo "commit hook: a different prepare-commit-msg is present (not ours)" + else + echo "commit hook: not installed" + fi + + KIRO_HOOK="$REPO_ROOT/.kiro/hooks/kiro-attribution.json" + if [ -f "$KIRO_HOOK" ]; then + echo "kiro hook: registered" + else + echo "kiro hook: not registered" + fi + + if [ -f "$SESSION_FILE" ]; then + echo "session log: $(wc -l < "$SESSION_FILE" | tr -d ' ') entries" + else + echo "session log: none yet (the Kiro hook has not fired in this repo)" + fi + + # Coverage decides whether attribution can support a conclusion at all. + # Unknown coverage means unusable data, not partial data: a commit with no + # trailer is indistinguishable between "no AI was used" and "the hook was + # never installed on that machine". + TOTAL=$(git rev-list --count HEAD 2>/dev/null || echo 0) + TAGGED=$(git log --all --grep='^Kiro-Session-Id:' --oneline 2>/dev/null | wc -l | tr -d ' ') + echo "commits: $TOTAL total, $TAGGED with a Kiro-Session-Id trailer" + exit 0 +fi + +# ------------------------------------------------------------ uninstall -------- + +if [ "$MODE" = uninstall ]; then + if [ -f "$HOOK_FILE" ] && grep -q "$MARKER" "$HOOK_FILE" 2>/dev/null; then + if [ -f "$HOOK_FILE.kiro-backup" ]; then + mv "$HOOK_FILE.kiro-backup" "$HOOK_FILE" + echo "restored the pre-existing prepare-commit-msg" + else + rm -f "$HOOK_FILE" + echo "removed $HOOK_FILE" + fi + else + echo "no hook installed by this script" + fi + # Also clear a stale copy in .git/hooks. A repo can acquire core.hooksPath + # after the hook was installed (adding husky does exactly that), leaving an + # orphan behind that --check would no longer look at. + STALE="$GIT_DIR/hooks/prepare-commit-msg" + if [ "$STALE" != "$HOOK_FILE" ] && [ -f "$STALE" ] && grep -q "$MARKER" "$STALE" 2>/dev/null; then + rm -f "$STALE" + echo "removed a stale copy at $STALE" + fi + rm -f "$REPO_ROOT/.kiro/hooks/kiro-attribution.json" 2>/dev/null || true + echo "note: $SESSION_FILE is left in place; delete it manually if unwanted" + exit 0 +fi + +# -------------------------------------------------------------- install -------- + +mkdir -p "$HOOK_DIR" + +# Chain to an unrelated hook rather than replacing it: silently dropping another +# team's hook would break their workflow with nothing to point at. +CHAIN_TO="" +if [ -f "$HOOK_FILE" ] && ! grep -q "$MARKER" "$HOOK_FILE" 2>/dev/null; then + mv "$HOOK_FILE" "$HOOK_FILE.kiro-backup" + CHAIN_TO="$HOOK_FILE.kiro-backup" + echo "kept the existing prepare-commit-msg as prepare-commit-msg.kiro-backup" +fi + +{ + printf '#!/bin/sh\n' + printf '%s - appends a Kiro-Session-Id trailer.\n' "$MARKER" + printf '# Installed by tools/kiro-attribution/install.sh; safe to remove.\n' + printf 'set -eu\n\n' + if [ -n "$CHAIN_TO" ]; then + printf '# Run the pre-existing hook first. Its failure must still abort the\n' + printf '# commit, hence propagating the exit code.\n' + printf 'if [ -x "%s" ]; then "%s" "$@" || exit $?; fi\n\n' "$CHAIN_TO" "$CHAIN_TO" + fi + # Inlined rather than sourced: a developer's checkout of this repository can + # move or be deleted, and a hook pointing at a missing file would break every + # commit in that repo. + sed '1d' "$SCRIPT_DIR/prepare-commit-msg.sh" +} > "$HOOK_FILE" + +chmod +x "$HOOK_FILE" +echo "installed $HOOK_FILE" +echo " location chosen by: $HOOK_WHY" +[ -n "$CHAIN_TO" ] && echo " chaining to $CHAIN_TO" + +# Register the Kiro-side recorder for this workspace. +# +# The schema matters: Kiro loads only .kiro/hooks/*.json files shaped as +# {"version":"v1","hooks":[...]}. The older one-object-per-file format with a +# .kiro.hook extension is silently ignored - the log says "loaded 0 standalone +# hooks" and nothing else indicates why. Field names changed too: "when.type": +# "agentStop" became "trigger": "Stop", and "then" became "action". +# +# Hooks load at session start, so a running Kiro session must be restarted before +# a newly installed hook fires. +if [ -d "$REPO_ROOT/.kiro" ]; then + mkdir -p "$REPO_ROOT/.kiro/hooks" + # Invoked via bash, and with a repo-relative path. + # + # The bash prefix is required by Kiro's permission model: shell capability + # rules match on command prefix, and a bare script path matches nothing, so the + # hook is skipped with no error logged. "bash *" is a standard allowed pattern. + # + # Relative rather than absolute so the hook survives the repo being moved or + # re-cloned; Kiro runs hooks with the workspace root as cwd. + REL_RECORDER=$(printf '%s' "$SCRIPT_DIR/kiro-session-record.sh" | sed "s|^$REPO_ROOT/||") + cat > "$REPO_ROOT/.kiro/hooks/kiro-attribution.json" < .git/hooks/prepare-commit-msg + chmod +x .git/hooks/prepare-commit-msg + + echo x > x && git add . && git commit -q -m baseline + sleep 1 + printf '%s\tverify-session-id\t%s\n' "$(date -u +%s)" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + >> .git/kiro-sessions + echo y > y && git add . && git commit -q -m "with session" + + if ! git log -1 --format=%B | grep -q 'Kiro-Session-Id: verify-session-id'; then + echo " FAILED: no trailer written. Do not rely on this install." >&2 + exit 1 + fi + echo " a trailer is written when a session was recorded" + + echo z > z && git add . && git commit -q -m "no new session" + if git log -1 --format=%B | grep -q 'Kiro-Session-Id'; then + echo " FAILED: a commit with no new session inherited a trailer." >&2 + exit 1 + fi + echo " no trailer when no new session was recorded" +) +echo +echo "done. Check status any time with:" +echo " $SCRIPT_DIR/install.sh --check" diff --git a/tools/kiro-attribution/kiro-session-record.sh b/tools/kiro-attribution/kiro-session-record.sh new file mode 100755 index 00000000000..8bda2f8c888 --- /dev/null +++ b/tools/kiro-attribution/kiro-session-record.sh @@ -0,0 +1,55 @@ +#!/bin/sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Records the Kiro session id that last touched this repository. +# +# Registered as a Kiro Stop hook, which fires after the agent finishes a turn - +# by then the files are written and a commit is the likely next step. Recording +# at turn start instead would capture a session that went on to touch other +# files, or one the user abandoned. +# +# The payload arrives as JSON on stdin and carries session_id; see +# https://kiro.dev/docs/cli/hooks. Verified against kiro-cli 2.6.1. +set -eu + +GIT_DIR=$(git rev-parse --git-dir 2>/dev/null) || exit 0 +PAYLOAD=$(cat) + +# Extract session_id without depending on jq being installed. +SESSION_ID=$(printf '%s' "$PAYLOAD" \ + | sed -n 's/.*"session_id"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' \ + | head -1) + +[ -n "$SESSION_ID" ] || exit 0 + +# Append rather than overwrite: several sessions may contribute to one commit, +# and dropping the earlier ones would under-report AI involvement. +# +# Epoch seconds first, because that is what the commit hook compares against. +# An ISO timestamp would have to agree on a timezone with git's own output, and +# a mismatch there fails silently - the trailer simply never appears. The +# readable form is kept as a second column for humans debugging the file. +printf '%s\t%s\t%s\n' "$(date -u +%s)" "$SESSION_ID" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + >> "$GIT_DIR/kiro-sessions" + +# Keep only recent entries so the file cannot grow without bound. 200 turns is +# far more than any single commit spans. +if [ "$(wc -l < "$GIT_DIR/kiro-sessions")" -gt 200 ]; then + tail -100 "$GIT_DIR/kiro-sessions" > "$GIT_DIR/kiro-sessions.tmp" + mv "$GIT_DIR/kiro-sessions.tmp" "$GIT_DIR/kiro-sessions" +fi + +exit 0 diff --git a/tools/kiro-attribution/prepare-commit-msg.sh b/tools/kiro-attribution/prepare-commit-msg.sh new file mode 100755 index 00000000000..c33a29256ba --- /dev/null +++ b/tools/kiro-attribution/prepare-commit-msg.sh @@ -0,0 +1,75 @@ +#!/bin/sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Appends a Kiro-Session-Id trailer for sessions that touched this repo since +# the last commit. +# +# A git trailer is used rather than a time-window heuristic because the two +# differ in kind, not degree. Matching "was Kiro active within 30 minutes of +# this commit" hits a base-rate trap: for anyone who keeps Kiro open all day, +# every commit qualifies, the attribution rate approaches 100%, and the label +# can no longer distinguish between people - which is precisely what heavy +# users need it to do. A trailer records what actually happened. +# +# Trailers are a standard git mechanism (Co-Authored-By, Signed-off-by use the +# same convention), so nothing here is bespoke, and the value stays readable to +# a human inspecting the log. +set -eu + +MSG_FILE=$1 +COMMIT_SOURCE=${2:-} + +# Leave amends, merges and squashes alone: their message already carries +# whatever trailers belong to it, and appending again would duplicate them. +case "$COMMIT_SOURCE" in + merge|squash|commit) exit 0 ;; +esac + +GIT_DIR=$(git rev-parse --git-dir 2>/dev/null) || exit 0 +SESSION_FILE="$GIT_DIR/kiro-sessions" +[ -f "$SESSION_FILE" ] || exit 0 + +# Only sessions newer than the previous commit belong to this one. Without this +# cut, every future commit would inherit the whole history of session ids. +# +# Both sides are epoch seconds (%ct on git's side, date +%s on the recorder's). +# Comparing formatted timestamps instead requires the two to agree on a +# timezone, and when they disagree the hook fails silently - no trailer, no +# error. Epoch has no timezone to get wrong. +LAST_COMMIT_EPOCH=$(git log -1 --format=%ct 2>/dev/null || echo 0) +[ -n "$LAST_COMMIT_EPOCH" ] || LAST_COMMIT_EPOCH=0 + +# Strictly greater than, not >=: a session recorded in the same second as the +# previous commit already belongs to that commit. With >=, it would be counted +# again here, so a commit containing no AI work at all would inherit the +# previous one's trailer. +SESSIONS=$(awk -v since="$LAST_COMMIT_EPOCH" '$1 + 0 > since + 0 { print $2 }' \ + "$SESSION_FILE" 2>/dev/null | sort -u) + +[ -n "$SESSIONS" ] || exit 0 + +# Skip if a trailer is already present, so re-running the hook is harmless. +if grep -q '^Kiro-Session-Id:' "$MSG_FILE" 2>/dev/null; then + exit 0 +fi + +# A blank line before trailers is required for git to parse them as such. +printf '\n' >> "$MSG_FILE" +for s in $SESSIONS; do + printf 'Kiro-Session-Id: %s\n' "$s" >> "$MSG_FILE" +done + +exit 0