Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@

- Add `LoginStore::list_candidates()` and `LoginStore::get_many()`, a pair of read APIs for consumers which filter logins on their unencrypted fields. `list_candidates()` returns a `LoginCandidate` per stored login - everything `Login` has except the secure fields (`username`/`password`), so searching by `origin`, `httpRealm` or `formActionOrigin` no longer forces a primary password prompt. `get_many()` then decrypts just the logins which matched. `list()` is unchanged, for callers who really do want every login in cleartext.

## ⚠️ Breaking Changes ⚠️

### Suggest

- `Suggestion.Amp` gained a new `suggestionId` field: a unique identifier for the sponsored suggestion assigned by the ingestion pipeline, deserialized from the `suggestion_id` field of the remote settings AMP data. ([#7554](https://github.com/mozilla/application-services/pull/7554))

# v155.0 (_2026-08-13_)

[Full Changelog](https://github.com/mozilla/application-services/compare/v154.0...v155.0)
Expand Down
10 changes: 8 additions & 2 deletions components/suggest/src/db.rs
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,7 @@ impl<'a> SuggestDao<'a> {
amp.iab_category,
amp.impression_url,
amp.click_url,
amp.moz_suggestion_id,
i.data AS icon,
i.mimetype AS icon_mimetype
FROM
Expand Down Expand Up @@ -368,6 +369,7 @@ impl<'a> SuggestDao<'a> {
raw_click_url,
score,
fts_match_info: None,
suggestion_id: row.get("moz_suggestion_id")?,
})
},
)
Expand Down Expand Up @@ -422,6 +424,7 @@ impl<'a> SuggestDao<'a> {
amp.iab_category,
amp.impression_url,
amp.click_url,
amp.moz_suggestion_id,
i.data AS icon,
i.mimetype AS icon_mimetype
FROM
Expand Down Expand Up @@ -463,6 +466,7 @@ impl<'a> SuggestDao<'a> {
raw_click_url,
score,
fts_match_info: Some(match_info),
suggestion_id: row.get("moz_suggestion_id")?,
})
},
)
Expand Down Expand Up @@ -1436,9 +1440,10 @@ impl<'conn> AmpInsertStatement<'conn> {
iab_category,
impression_url,
click_url,
icon_id
icon_id,
moz_suggestion_id
)
VALUES(?, ?, ?, ?, ?, ?, ?)
VALUES(?, ?, ?, ?, ?, ?, ?, ?)
",
)?))
}
Expand All @@ -1453,6 +1458,7 @@ impl<'conn> AmpInsertStatement<'conn> {
&amp.impression_url,
&amp.click_url,
&amp.icon_id,
&amp.suggestion_id,
))
.with_context("amp insert")?;
Ok(())
Expand Down
1 change: 1 addition & 0 deletions components/suggest/src/rs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,7 @@ pub(crate) struct DownloadedAmpSuggestion {
pub impression_url: String,
#[serde(rename = "icon")]
pub icon_id: String,
pub suggestion_id: String,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we already forwarding this to all users? If not this will probably need to be an Option to avoid a deserialization error if a client doesn't receive a suggestion_id

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To add to this (and I'm not entirely sure how this interacts with remote settings so forgive me if this doesn't apply) but if a user had really old RS artifacts still loaded for some reason (or an old enough version of FF that they are using the legacy RS collection) would this break things if suggestion_id is required?

@kalamazilla kalamazilla Aug 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My rust is very basic so I got some questions about this to Claude.
The problem is serialization errors are swallowed entirely:

match serde_json::from_slice::<SuggestAttachment<T>>(&attachment_data) { Ok(attachment) => ingestion_handler(dao, &record.id, attachment.suggestions()), // If the attachment doesn't match our expected schema, just skip it. It's possible // that we're using an older version. If so, we'll get the data when we re-ingest // after updating the schema. Err(_) => Ok(()), }

An attachment is a JSON array of thousands of suggestions, so a single record missing suggestion_id causes the entire attachment to be dropped — no error, no log, no metric. The user just silently gets zero sponsored suggestions for that region/form-factor.

Instead of using Option is recommends using #[serde(default)].
The cost of an issue is a silent, total loss of sponsored suggestions with no telemetry to detect it. It also protects against a mars rollback and against stale non-prod collections. Importantly, #[serde(default)] keeps the Rust type as String and leaves the public API exactly as this PR has it — so unlike switching to Option

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging this! It's really good to think through. I think it is okay to have it required but you let me know if the following makes sense. Being required is an important property since this is load-bearing billing telemetry, if we don't do it this iteration, we'll have to come back in again and do it later. And I think leaving it required is low risk.

The field has been populated in production for about a month, and Claude can check that it hasn't been missing in any payload -- the deserialization risk happens if some records lack the field. In this case the client would only lack sponsored suggestions until the next resync, which Claude tells me happens about daily for active clients, and client that are dormant refresh when they start up again. And afaict the way the migrations work with the clear_database() call, it forces a fresh re-ingestion.

Please see if you can confirm, I'm new to rust, also relying on Claude. (Side question: Are our Claudes just talking to each other and are we vibe coding too close to the sun?)

Today I'll post the PR for review by disco team

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using Claude on Claude is still useful. The questions we ask are different and that is where the real value comes from is as it triggers different paths within Claude. Its like having a Claude with multiple personalities (that never ends well in the movies though!)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems okay to me as-is assuming that a) serialization errors cause records to be skipped rather than errors to be thrown and b) The production RS data has had this field for about a month. In that case, there's not really a risk of "total loss of sponsored suggestions". I think the only effect is that users who have extremely stale data in the RS cache and who can't download new data won't see the stale suggestions.

I do agree that silently ignoring the errors feels wrong. You could consider adding an error_support::report_error! call to this PR or filing a issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed with bendk, this should be fine, i.e., no need to make this an Option. IIRC we typically don't worry about outdated RS data being ingested by updated clients. It looks like the suggestions in the quicksuggest-amp collection already include suggestion_id and probably have for some time I imagine?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the suggestions in the quicksuggest-amp collection already include suggestion_id and probably have for some time I imagine?

Yes, the suggestion_ids have been included in the prod RS collection since MARS deploy on 7/22/2026, so almost a month now. From what you're both saying that sounds like a sufficient amount of time.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only effect is that users who have extremely stale data in the RS cache and who can't download new data won't see the stale suggestions.

I think there's also a good chance we wouldn't have gotten paid for many of those suggestions even if we were able to show them anyway, since those ad campaigns have probably ended by now...so just piling on reasons that this is probably okay to ship as required.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do agree that silently ignoring the errors feels wrong. You could consider adding an error_support::report_error! call to this PR or filing a issue.

Yes, agreed, the whole point of having it required is to fail loudly and early when it's missing, so that feels incomplete without an error. I'll take a swing at implementing this

}

/// A Wikipedia suggestion to ingest from a Wikipedia attachment.
Expand Down
23 changes: 22 additions & 1 deletion components/suggest/src/schema.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ use sql_support::{
/// `clear_database()` by adding their names to `conditional_tables`, unless
/// they are cleared via a deletion trigger or there's some other good
/// reason not to do so.
pub const VERSION: u32 = 45;
pub const VERSION: u32 = 46;

/// The current Suggest database schema.
pub const SQL: &str = "
Expand Down Expand Up @@ -108,6 +108,7 @@ CREATE TABLE amp_custom_details(
impression_url TEXT NOT NULL,
click_url TEXT NOT NULL,
icon_id TEXT NOT NULL,
moz_suggestion_id TEXT NOT NULL,
FOREIGN KEY(suggestion_id) REFERENCES suggestions(id) ON DELETE CASCADE
);

Expand Down Expand Up @@ -835,6 +836,26 @@ impl ConnectionInitializer for SuggestConnectionInitializer<'_> {
)?;
Ok(())
}
45 => {
clear_database(tx)?;
tx.execute_batch(
"
DROP TABLE amp_custom_details;
CREATE TABLE amp_custom_details(
suggestion_id INTEGER PRIMARY KEY,
advertiser TEXT NOT NULL,
block_id INTEGER NOT NULL,
iab_category TEXT NOT NULL,
impression_url TEXT NOT NULL,
click_url TEXT NOT NULL,
icon_id TEXT NOT NULL,
moz_suggestion_id TEXT NOT NULL,
FOREIGN KEY(suggestion_id) REFERENCES suggestions(id) ON DELETE CASCADE
);
",
)?;
Ok(())
}

_ => Err(open_database::Error::IncompatibleVersion(version)),
}
Expand Down
16 changes: 12 additions & 4 deletions components/suggest/src/store.rs
Original file line number Diff line number Diff line change
Expand Up @@ -808,10 +808,18 @@ where
context.measure_download(|| self.settings_client.download_attachment(record))?;
match serde_json::from_slice::<SuggestAttachment<T>>(&attachment_data) {
Ok(attachment) => ingestion_handler(dao, &record.id, attachment.suggestions()),
// If the attachment doesn't match our expected schema, just skip it. It's possible
// that we're using an older version. If so, we'll get the data when we re-ingest
// after updating the schema.
Err(_) => Ok(()),
// If the attachment doesn't match our expected schema, just skip it and emit an error.
// It's possible that we're using an older version. If so, we'll get the data when we
// re-ingest after updating the schema.
Err(e) => {
error_support::report_error!(
"suggest-attachment-deserialize",
"Failed to deserialize attachment for record {}: {}",
record.id,
e
);
Ok(())
}
}
}

Expand Down
1 change: 1 addition & 0 deletions components/suggest/src/suggestion.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ pub enum Suggestion {
raw_click_url: String,
score: f64,
fts_match_info: Option<FtsMatchInfo>,
suggestion_id: String,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above as to whether we think this should be an Option or not

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need, we can/should expect suggestions to have IDs now.

},
Wikipedia {
title: String,
Expand Down
8 changes: 6 additions & 2 deletions components/suggest/src/testing/data.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ pub fn los_pollos_amp() -> JsonValue {
"icon": "los-pollos-favicon",
"impression_url": "https://example.com/impression_url",
"click_url": "https://example.com/click_url",
"score": 0.3
"score": 0.3,
"suggestion_id": "11111111-1111-1111-1111-111111111111"
})
}

Expand Down Expand Up @@ -53,6 +54,7 @@ pub fn los_pollos_suggestion(
score: 0.3,
full_keyword: full_keyword.to_string(),
fts_match_info,
suggestion_id: "11111111-1111-1111-1111-111111111111".into(),
}
}

Expand All @@ -67,7 +69,8 @@ pub fn good_place_eats_amp() -> JsonValue {
"url": "https://www.lasagna.restaurant",
"icon": "good-place-eats-favicon",
"impression_url": "https://example.com/impression_url",
"click_url": "https://example.com/click_url"
"click_url": "https://example.com/click_url",
"suggestion_id": "22222222-2222-2222-2222-222222222222"
})
}

Expand Down Expand Up @@ -99,6 +102,7 @@ pub fn good_place_eats_suggestion(
raw_click_url: "https://example.com/click_url".into(),
score: 0.2,
fts_match_info,
suggestion_id: "22222222-2222-2222-2222-222222222222".into(),
}
}

Expand Down