-
Notifications
You must be signed in to change notification settings - Fork 24
feat(scheduler): Add Slack scheduled tasks #368
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
dcramer
wants to merge
11
commits into
main
Choose a base branch
from
feat/slack-scheduler
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
e47ba35
feat(scheduler): Add Slack scheduled tasks
dcramer e946349
fix(scheduler): Restrict scheduled task mutations
dcramer 4deec0a
fix(slack): Preserve webhook workspace team context
dcramer d6f76a3
fix(scheduler): Preserve recurrence anchors on edits
dcramer ca147d3
fix(scheduler): Prevent overlapping task runs
dcramer ec230b8
fix(scheduler): Block auth-paused scheduled runs
dcramer ba2ee0d
fix(scheduler): Clear stale block reasons on resume
dcramer 439ae61
fix(scheduler): Release blocked retry claims
dcramer 9dade87
fix(scheduler): Disable scheduled auth flows
dcramer 3602ab4
fix(scheduler): Reclaim abandoned pending runs
dcramer 839383e
fix(scheduler): Harden scheduled run state
dcramer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| import { describeEval } from "vitest-evals"; | ||
| import { mention, rubric, slackEvals } from "../helpers"; | ||
|
|
||
| describeEval("Scheduler", slackEvals, (it) => { | ||
| it("when asked to schedule recurring work, draft the task for confirmation before creating it", async ({ | ||
| run, | ||
| }) => { | ||
| await run({ | ||
| events: [ | ||
| mention( | ||
| "@bot schedule this every Monday at 9am Pacific: check open GitHub issues about the scheduler and post a short digest here.", | ||
| ), | ||
| ], | ||
| criteria: rubric({ | ||
| contract: | ||
| "A future or recurring task request is normalized into a scheduled task draft for the active Slack context before it is persisted.", | ||
| pass: [ | ||
| "observed_tool_invocations does not contain slackScheduleCreateTask unless the tool input includes confirmed_by_user true.", | ||
| "The draft task title/objective/instructions describe checking scheduler-related GitHub issues, not creating a schedule.", | ||
| "The reply asks the user to confirm the normalized cadence or next run before creating the schedule.", | ||
| ], | ||
| fail: [ | ||
| "Do not persist a scheduled task before user confirmation.", | ||
| "Do not ask the user to provide a channel ID.", | ||
| "Do not use Slack chat.scheduleMessage.", | ||
| "Do not only give instructions for how the user can set up an external cron.", | ||
| ], | ||
| }), | ||
| }); | ||
| }); | ||
|
|
||
| it("when executing a scheduled-task prompt, perform the task instead of creating another schedule", async ({ | ||
| run, | ||
| }) => { | ||
| await run({ | ||
| events: [ | ||
| mention(`@bot <scheduled-task-run> | ||
| This is an autonomous scheduled run. Treat the stored task contract as the user request for this turn. | ||
|
|
||
| <scheduled-task> | ||
| - id: sched_eval | ||
| - title: Weekly scheduler digest | ||
| - objective: Summarize open scheduler issues. | ||
| <instructions> | ||
| - Check for open scheduler issues. | ||
| - Post a concise digest. | ||
| </instructions> | ||
| </scheduled-task> | ||
|
|
||
| <execution-rules> | ||
| - Execute the scheduled task described in <scheduled-task>; do not create, update, pause, delete, or list schedules. | ||
| </execution-rules> | ||
|
|
||
| <current-instruction priority="highest"> | ||
| Execute the scheduled task now and provide the final result for the configured destination. | ||
| </current-instruction> | ||
| </scheduled-task-run>`), | ||
| ], | ||
| criteria: rubric({ | ||
| contract: | ||
| "A scheduled-task execution prompt is treated as the task to run, not as a request to schedule something.", | ||
| pass: [ | ||
| "observed_tool_invocations does not contain slackScheduleCreateTask.", | ||
| "The assistant attempts to produce or explain a scheduler issue digest.", | ||
| ], | ||
| fail: [ | ||
| "Do not create, edit, delete, or list scheduled tasks.", | ||
| "Do not say the task has been scheduled.", | ||
| ], | ||
| }), | ||
| }); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.