You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: map octolens mentions from ten platforms per the ingestion spec - #4993
Aligns Octolens mention ingestion with the ingestion spec and extends it to every source with complete data: mentions from GitHub, X/Twitter, LinkedIn, Reddit, dev.to, Hacker News, Bluesky, TikTok, Instagram and YouTube become mention activities, each routed to its keyword's project and nothing else. It also fixes the poll for the live API, where a mention's keywords arrive only in keywords[]; before this, every mention would have been skipped.
Changes
Platforms. GitHub, X/Twitter and LinkedIn (the spec), plus Reddit, dev.to (Octolens dev → the existing devto platform), Hacker News, Bluesky, TikTok, Instagram and YouTube: in 2,000 live mentions every one of these carried an id, timestamp, URL and a matching profile link. New platform types bluesky, tiktok and instagram, plus mention activity types and activityTypes rows. Stack Overflow, Medium, Product Hunt and Facebook stay mapped but off behind the mapper's allow-list (Octolens hasn't sent any); podcasts, news, firehose, newsletter and review sites are never emitted because their author is a show or publication. Skipped sources are logged at debug.
Identities come from the author's profile link, as in the spec, for every platform: GitHub login, X/Twitter handle, LinkedIn /in/<handle>, Reddit /user/<name>, dev.to /<name>, Hacker News user?id=, Bluesky /profile/<handle>, TikTok /@<handle>, Instagram /<name>, YouTube /channel/<id>. Octolens's author is a display name on LinkedIn and YouTube, so it is not used. The link must point at the mention's own platform; bot links (github.com/apps/…), app routes and malformed handles are skipped (Reddit's [deleted] included). Identities stay verified usernames. Checked against 2,000 live mentions: every person mention on these platforms maps.
Activity fields.channel is the Octolens source, sourceId is the Octolens mention id, score is 0, isContribution/isCollaboration stay false. emit keeps a record's own channel when the connector's schema includes one (GitHub connector records are unchanged). Attributes keep the Octolens fields and add octolensSource, postId (the platform's post id), keyword, keywordId and, for GitHub, repoUrl.
Routing. A mention is routed by the first keyword in keywords[] that has a mapping. The data sink no longer re-routes GitHub activities from Octolens integrations through public.repositories; their segment always comes from the keyword's project. Every other integration is unchanged.
No backfill by default. The first poll starts at the current time, and keywords mapped later are no longer backfilled automatically. The per-keyword backfill queue is still processed when filled; a script to fill it follows separately.
Migration V1792094459__add_octolens_mention_activity_types.sql adds the mention rows (false/false, idempotent, the existing twitter row is skipped). Protected path.
Tests for the mapper (profile links, allow-list, attributes), the poll (keyword routing, first run, backfill queue) and the data sink opt-out.
Only a leading /r/ names a subreddit, more GitHub site pages (advisories,
security, copilot and others) no longer read as repos, and a mention whose
keywordId is missing from its keywords gets no keyword channel instead of a
different keyword's. The ignored identity platforms log as ignored rather
than unknown.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
Medium Risk
Touches activity ingestion, segment routing, and DB activity types across many platforms; incorrect identity or routing logic could mis-attribute members or drop mentions, though behavior is heavily tested and Octolens-specific paths are isolated.
Overview
Expands Octolens mention ingestion to ten person-attributed sources (GitHub, X/Twitter, LinkedIn, Reddit, dev.to, Hacker News, Bluesky, TikTok, Instagram, YouTube) with strict profile-link identity parsing, mention activities (score 0, channel = Octolens source, sourceId = mention id), and richer attributes (octolensSource, postId, keyword context, optional repoUrl/subreddit). Unsupported or non-profile sources are skipped; new PlatformType values and an idempotent activityTypes migration back the UI/scoring grids.
Poll and routing: Mentions route via the first mapped keyword in keywords[] (fixes live API shape); first sync starts at now with watermark committed before fetch; automatic unmapped-keyword backfill is removed (manual backfill queue only). The data sink keeps Octolens results on the keyword’s segment by skipping GitHub/GitLab repository segment lookup for PlatformType.OCTOLENS. emit preserves a record’s channel when the connector schema defines it.
Also adds mention display settings for additional platforms, ignores Bluesky/TikTok/Instagram in LFX org-enrichment identities, and broad test coverage for mapper, poll, emit, and sink behavior.
Reviewed by Cursor Bugbot for commit 6feba8e. Bugbot is set up for automated code reviews on this repo. Configure here.
…pository
The data sink resolved every github and gitlab activity's segment through
public.repositories. Activities from an Octolens integration now keep the
segment the connector set from the keyword's project mapping; every other
integration is unchanged.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
Only GitHub, X/Twitter and LinkedIn mentions are emitted for now; the other
sources keep their mapping so they can be enabled later. Their identities
come from the author's profile link (LinkedIn's author field is a display
name, and github.com/apps links are bots). The activity's channel is the
Octolens source, its sourceId the Octolens mention id and its score 0;
octolensSource, the post id, the matched keyword, keywordId and a GitHub
repo URL are kept in the attributes.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
Live mentions carry their keywords only in keywords[], so the poll skipped
every one waiting for a top-level keywordId. It now routes a mention by its
first mapped keyword. The first run starts at the current time instead of
fetching the whole history, and keywords are no longer queued for backfill
automatically; the backfill queue is left for a manual script.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
A mention is emitted only when its profile link points at the same platform
as its source, X/Twitter non-profile paths and malformed GitHub logins are
not read as usernames, percent-encoded handles are decoded, and a mention
without an Octolens id is skipped instead of sharing one sourceId. A first
run that stops early now keeps its start date, and a backfilled mention is
routed to the keyword being backfilled first.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
themarolt
changed the title
feat: map every person-handle octolens source with a per-mention channel
feat: map octolens github, twitter and linkedin mentions per the ingestion spec
Oct 16, 2026
…l entries
A first run that failed before committing a page left no start date, so the
retry picked a later time and skipped the mentions in between. The cutoff is
now saved before any fetch. Backfill pages now validate each entry's
timestamp too, so a null entry is skipped instead of blocking the page.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
This regex accepts malformed GitHub logins such as octocat- and octo--cat, although hyphens must be single and between alphanumeric characters. A GitHub author URL containing either value passes profileLinkIdentity and is emitted as a verified username instead of being skipped. Reject trailing and consecutive hyphens while retaining the 39-character limit, and add regression cases for both.
A backfill preferred the keyword being backfilled, so a mention mapped
under two keywords could land in a second project when replayed. Both paths
now use the first mapped keyword.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
This regex accepts user- and user--name, although GitHub logins cannot end with a hyphen or contain consecutive hyphens. These malformed profile links therefore produce verified username identities instead of being skipped. Require single hyphens between alphanumeric characters, preserve the 39-character limit, and add regression cases for both invalid forms.
Exclude X/Twitter application routes from profile identity parsing
The route filter accepts https://x.com/compose/post, https://twitter.com/login, and https://x.com/signup as verified users named compose, login, and signup. These are application routes, not author profiles, so mentions carrying these links would be attributed to incorrect shared identities. Exclude these routes and add regression cases that expect no identity.
…tities
GitHub logins with a trailing or doubled hyphen and X routes such as
/compose, /login and /signup are no longer read as author usernames.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
Reject GitHub usernames with trailing or consecutive hyphens · services/libs/connectors/src/connectors/octolens/mapper.ts:207
Fixed in 4f353d2: logins must be alphanumeric runs joined by single hyphens, up to 39 characters; user- and user--name now yield no identity (regression cases added).
Exclude X/Twitter application routes from profile identity parsing · services/libs/connectors/src/connectors/octolens/mapper.ts:220
Fixed in 4f353d2: compose, login, logout, signup and tos join the excluded routes; x.com/compose/post, twitter.com/login and x.com/signup now yield no identity (regression cases added).
…, instagram and youtube
These sources carry an id, timestamp, URL and a profile link in every live
mention, so they are emitted alongside GitHub, X/Twitter and LinkedIn.
Their identities also come from the profile link (reddit /user, dev.to,
news.ycombinator.com user?id, bsky.app /profile, tiktok @handle, instagram,
youtube /channel), which must belong to the mention's own platform; app
routes and malformed handles are skipped. Octolens's dev source maps to the
existing devto platform. Stack Overflow, Medium, Product Hunt and Facebook
stay mapped but off, since Octolens hasn't sent any.
Refs linuxfoundation/lfx-cdp-insights-ops#748
Signed-off-by: Uroš Marolt <uros@marolt.me>
themarolt
changed the title
feat: map octolens github, twitter and linkedin mentions per the ingestion spec
feat: map octolens mentions from ten platforms per the ingestion spec
Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
themarolt
changed the title
Summary
Aligns Octolens mention ingestion with the ingestion spec and extends it to every source with complete data: mentions from GitHub, X/Twitter, LinkedIn, Reddit, dev.to, Hacker News, Bluesky, TikTok, Instagram and YouTube become
mentionactivities, each routed to its keyword's project and nothing else. It also fixes the poll for the live API, where a mention's keywords arrive only inkeywords[]; before this, every mention would have been skipped.Changes
dev→ the existingdevtoplatform), Hacker News, Bluesky, TikTok, Instagram and YouTube: in 2,000 live mentions every one of these carried an id, timestamp, URL and a matching profile link. New platform typesbluesky,tiktokandinstagram, plusmentionactivity types andactivityTypesrows. Stack Overflow, Medium, Product Hunt and Facebook stay mapped but off behind the mapper's allow-list (Octolens hasn't sent any); podcasts, news, firehose, newsletter and review sites are never emitted because their author is a show or publication. Skipped sources are logged at debug./in/<handle>, Reddit/user/<name>, dev.to/<name>, Hacker Newsuser?id=, Bluesky/profile/<handle>, TikTok/@<handle>, Instagram/<name>, YouTube/channel/<id>. Octolens'sauthoris a display name on LinkedIn and YouTube, so it is not used. The link must point at the mention's own platform; bot links (github.com/apps/…), app routes and malformed handles are skipped (Reddit's[deleted]included). Identities stay verified usernames. Checked against 2,000 live mentions: every person mention on these platforms maps.channelis the Octolenssource,sourceIdis the Octolens mention id,scoreis 0,isContribution/isCollaborationstay false.emitkeeps a record's own channel when the connector's schema includes one (GitHub connector records are unchanged). Attributes keep the Octolens fields and addoctolensSource,postId(the platform's post id),keyword,keywordIdand, for GitHub,repoUrl.keywords[]that has a mapping. The data sink no longer re-routes GitHub activities from Octolens integrations throughpublic.repositories; their segment always comes from the keyword's project. Every other integration is unchanged.V1792094459__add_octolens_mention_activity_types.sqladds thementionrows (false/false, idempotent, the existing twitter row is skipped). Protected path.Type of change
Related issue
Refs linuxfoundation/lfx-cdp-insights-ops#748