Stop bots from creating a guest customer on every request

[KrzysztofPajak]

Type: bugfix

Issue

Every storefront request that no resolver claims ends in WorkContextSetter.CreateCustomerGuest, which inserts a Customer document and sets the guest cookie. Clients that never send the cookie back create a new document on every request. Crawler detection relied on Wangkanai.Detection, which only recognised user agents containing "bot" plus a few named crawlers, so Bytespider, ...Spider, ...crawler and every scraper with a browser user agent created a guest per request. Detected crawlers caused a second problem: CustomerActivityAttribute wrote LastActivityDateUtc, LastIpAddress and LastVisitedPage to the single shared SearchEngine document on each of their requests.

To reproduce on develop: request the home page several times without a cookie jar (curl -A "Mozilla/5.0 (compatible; Bytespider)" http://localhost:<port>/) and count the documents in the Customer collection.

Solution

• Unsaved guest. A guest created on a GET/HEAD/OPTIONS request without a guest cookie stays in memory (IGuestCustomerPersistence, request scope) and gets its cookie and the Guests group. It is inserted when the browser returns the cookie (promotion in WorkContextSetter), when something writes to it (an ICustomerService decorator registered after the business services saves it before the first write), or immediately on any other HTTP method. A cookie GUID that belongs to any existing customer is never promoted; ids are never derived from the cookie.
• Unique CustomerGuid_1. Concurrent promotions of one cookie collapse into one document. The installer creates the index unique; a 2.5 migration recreates it unique on MongoDB (duplicates are logged and the index stays non-unique; the lookup index is never left missing).
• Crawler detection. Wangkanai.Detection is replaced by IUserAgentClassifier (crawler names: bot, crawl, spider, slurp, Bytespider, facebookexternalhit, meta-externalagent; mobile/tablet for the admin tab strip). Requests with an Authorization header are never treated as crawlers.
• Guest cookie. Resolves only to an active guest that is not registered and not a system account; the cookie must hold a GUID in "D" format.
• Activity tracking skips system accounts and the unsaved guest; recently viewed products are not recorded for the unsaved guest.
• rel="nofollow" on the language, currency, tax type and theme selector links.

Breaking changes

• Wangkanai.Detection is removed: IDetectionService is no longer registered and UseDetection() no longer runs. Plugins that injected IDetectionService must use Grand.Web.Common.UserAgent.IUserAgentClassifier.
• Constructor changes: SearchEngineCustomerResolver (adds IHttpContextAccessor, takes IUserAgentClassifier), AdminTabStripTagHelper (takes IUserAgentClassifier), ProductController (adds IGuestCustomerPersistence). Code that constructs these directly must be updated.
• ICustomerService is resolved as a decorator (TransientGuestCustomerService); a host that registers ICustomerService through a factory instead of a type now fails at startup with an explanatory exception.
• Behaviour: a visitor whose first request is a GET and who never returns is no longer stored, so one-request visitors disappear from the guest list and the online customers report.

Testing

1. Run the storefront on an installed database and note the number of documents in the Customer collection.
2. curl -s -o /dev/null -H "Accept: text/html" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120" http://localhost:<port>/ twice without a cookie jar: both return 200, the count does not change.
3. curl -s -o /dev/null -A "Mozilla/5.0 (compatible; Bytespider)" http://localhost:<port>/: the count does not change and the SearchEngine document's LastVisitedPage is unchanged.
4. curl -c jar -b jar -H "Accept: text/html" http://localhost:<port>/ twice: the count grows by one after the second call and the new document's CustomerGuid equals the cookie value.
5. With a fresh cookie jar, open /?affiliate=<friendly name of an active affiliate> once: one new guest with that AffiliateId.
6. With a fresh cookie jar, open /changelanguage/<seo code>?returnUrl=/: redirect to / (not /login) and one new guest with the LanguageId user field.
7. Send the home page with the guest cookie set to an administrator's CustomerGuid: a new guest cookie is issued and the administrator's document is unchanged.
8. Call a web API endpoint with a valid bearer token and user agent python-requests/2.31.0: the call is authorised as the API customer.
9. Upgrade a 2.4 MongoDB database: db.Customer.getIndexes() lists CustomerGuid_1 with unique: true.
10. dotnet test GrandNode.slnx: all projects pass.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.