doc: explain how to compare strings with a custom key

A new how-to guide shows how to give a policy to the storage, and what each database supports.

Author: azmeuk

doc/_examples/intranet/scim.py

@@ -58,14 +58,15 @@
 )
 
 
-def create_provider():
+def create_provider(policy=None):
     return ScimProvider.from_discovery(
         [*USERS.schemas(), *GROUPS.schemas()],
         [
             ResourceType.from_resource(USERS.model),
             ResourceType.from_resource(GROUPS.model),
         ],
         config=load_default_service_provider_config(),
+        policy=policy,
     )
 
 

doc/explanation.rst

@@ -41,9 +41,10 @@ differs from a direct translation in several places:
   while ``emails[value ne "x"]`` holds when one email is not ``x``.
 - A value selection, such as ``emails[type eq "work" and primary eq true]``, applies all its
   conditions to the same entry.
-- A string compares in the form the comparison key of the policy gives it. By default, the key
-  normalizes the string to NFC, and lowers it unless the schema declares it ``caseExact``. Each
-  database reproduces the key its own way, as the next section describes.
+- A string compares in the form the comparison key of the policy gives it. By default, the
+  comparison key normalizes the string to NFC, and lowers it unless the schema declares it
+  ``caseExact``. Each database reproduces the comparison key its own way, as the next section
+  describes.
 
 An attribute the resource type does not declare matches no resource
 (:rfc:`RFC 7644 §3.4.2.1 <7644#section-3.4.2.1>`). An attribute it declares but the mapping does
@@ -54,35 +55,35 @@ Strings compare as the policy says
 ----------------------------------
 
 The :attr:`comparison_key <scim2_models.ScimPolicy.comparison_key>` of a
-:class:`~scim2_models.ScimPolicy` gives the form strings are compared in. scim2-models applies it
-in Python, to filters, sorts and PATCH operations. For the reasons behind the key, read
+:class:`~scim2_models.ScimPolicy` gives the form strings are compared in. scim2-models applies it in
+Python, to filters, sorts and PATCH operations. For the reasons behind the comparison key, read
 :doc:`scim2_models:explanation/comparisons`. The storage follows the policy of the provider it
-receives, or the default policy without a provider. It reproduces the key in SQL, as far as the
-database allows:
-
-- SQLite runs Python functions. The storage gives each connection a SQL function that calls the
-  key, so filters, sorts and uniqueness checks compare exactly as in Python, whatever the key.
-  The function runs once for each row, and no index serves it. The ``co``, ``sw`` and ``ew``
-  filters use ``instr()`` and ``substr()``, since the ``LIKE`` of SQLite ignores the case of ASCII
-  letters, even for a ``caseExact`` attribute.
+receives, or the default policy without a provider. It reproduces the comparison key in SQL, as far
+as the database allows:
+
+- SQLite runs Python functions. The storage gives each connection a SQL function that calls the key,
+  so filters, sorts and uniqueness checks compare exactly as in Python, whatever the comparison key.
+  The function runs once for each row, and no index serves it. The ``co``, ``sw`` and ``ew`` filters
+  use ``instr()`` and ``substr()``, since the ``LIKE`` of SQLite ignores the case of ASCII letters,
+  even for a ``caseExact`` attribute.
 - PostgreSQL normalizes both sides to NFC with ``normalize()``, and lowers them with ``lower()``.
-  This approaches the default key without reaching it. ``lower()`` follows the locale of the
-  database, and only lowers ASCII letters under the ``C`` locale. Even under a UTF-8 locale, a
-  few letters differ, such as ``İ`` and the final sigma. ``normalize()`` needs PostgreSQL 13 and
-  a database encoded in UTF-8. The comparison uses the ``C`` collation, which orders strings by
-  code point, as scim2-models does, whatever the collation of the column. An index that serves
-  these filters is an index on the same expression, with the same collation. With a key other
-  than the default one, the storage warns that it only approaches the key.
+  This approaches the default comparison key without reaching it. ``lower()`` follows the locale of
+  the database, and only lowers ASCII letters under the ``C`` locale. Even under a UTF-8 locale, a
+  few letters differ, such as ``İ`` and the final sigma. ``normalize()`` needs PostgreSQL 13 and a
+  database encoded in UTF-8. The comparison uses the ``C`` collation, which orders strings by code
+  point, as scim2-models does, whatever the collation of the column. An index that serves these
+  filters is an index on the same expression, with the same collation. With a comparison key other
+  than the default one, the storage warns that it only approaches it.
 - Other databases lower both sides with their own ``lower()``. MySQL and MariaDB are not
   supported: their default collations ignore the case and the accents, even for a ``caseExact``
   attribute.
 
 Where ``LIKE`` serves the ``co``, ``sw`` and ``ew`` filters, the ``%`` and ``_`` characters of the
 value are escaped, so that they match themselves.
 
-A string the key refuses is equal to no other. A filter comparing an attribute with such a
-string holds only for ``ne``. On SQLite, a stored string the key refuses also matches ``ne``
-only, and sorts with the missing values.
+A string the comparison key refuses is equal to no other. A filter comparing an attribute with such
+a string holds only for ``ne``. On SQLite, a stored string the comparison key refuses also matches
+``ne`` only, and sorts with the missing values.
 
 Sorting follows RFC 7644
 ------------------------
@@ -132,13 +133,12 @@ refuses an entry without ``value``: storing nothing for it would lose the link.
 Versions protect concurrent writes
 ----------------------------------
 
-The version of a resource comes from the
-:ref:`version counter <sqlalchemy:mapper_version_counter>` of its SQLAlchemy model. SQLAlchemy
-increments the column on every write, and adds the version it read to the ``WHERE`` clause of the
-``UPDATE``. The storage first compares the stored version with the one the client sent in
-``If-Match``, and answers 412 when they differ. A write by another transaction between the read
-and the write matches no row: SQLAlchemy raises :exc:`~sqlalchemy.orm.exc.StaleDataError`, which the storage turns into
-a 412 too.
+The version of a resource comes from the :ref:`version counter <sqlalchemy:mapper_version_counter>`
+of its SQLAlchemy model. SQLAlchemy increments the column on every write, and adds the version it
+read to the ``WHERE`` clause of the ``UPDATE``. The storage first compares the stored version with
+the one the client sent in ``If-Match``, and answers 412 when they differ. A write by another
+transaction between the read and the write matches no row: SQLAlchemy raises
+:exc:`~sqlalchemy.orm.exc.StaleDataError`, which the storage turns into a 412 too.
 
 Unique values are checked twice
 -------------------------------
@@ -153,15 +153,16 @@ taken. Any other :exc:`~sqlalchemy.exc.IntegrityError` goes through unchanged.
 The application owns the transaction
 ------------------------------------
 
-The storage :ref:`flushes <sqlalchemy:session_flushing>` each write, so that the database fills the identifiers and checks its
-constraints, and never commits. The application, or its web framework, already decides when a
-request ends and whether its changes are kept. Each operation runs in a :ref:`savepoint <sqlalchemy:session_begin_nested>`, so that a
-failed operation leaves the session usable, and a bulk request goes on after a failure.
+The storage :ref:`flushes <sqlalchemy:session_flushing>` each write, so that the database fills the
+identifiers and checks its constraints, and never commits. The application, or its web framework,
+already decides when a request ends and whether its changes are kept. Each operation runs in a
+:ref:`savepoint <sqlalchemy:session_begin_nested>`, so that a failed operation leaves the session
+usable, and a bulk request goes on after a failure.
 
 The asynchronous storage loads every collection of a record with the record. An
-:class:`~sqlalchemy.ext.asyncio.AsyncSession` cannot load a relationship
-:ref:`lazily <sqlalchemy:asyncio_orm_avoid_lazyloads>`, on first access. Both storages load the same way, so
-they send the same queries.
+:class:`~sqlalchemy.ext.asyncio.AsyncSession` cannot load a relationship :ref:`lazily
+<sqlalchemy:asyncio_orm_avoid_lazyloads>`, on first access. Both storages load the same way, so they
+send the same queries.
 
 Limits
 ------

doc/how-to/compare-strings.rst

@@ -0,0 +1,100 @@
+Compare strings with a custom comparison key
+============================================
+
+This guide makes the storage compare strings with a comparison key other than the default one, such
+as the PRECIS rules that :rfc:`RFC 7644 §5 <7644#section-5>` requires for ``userName``. A comparison
+key is a function that returns the form a string is compared in: two strings are equal when their
+forms are equal. The guide is for developers who serve a mapping with scim2-server, as in the
+:doc:`../overview`. It does not cover the rules a comparison key must follow: for them, read
+:doc:`scim2_models:how-to/compare-values`.
+
+Give the policy to the storage
+------------------------------
+
+The storage compares strings with the :attr:`~scim2_models.ScimPolicy.comparison_key` of the policy
+of its provider. Build the provider with the policy, and pass the same provider to the storage and
+to the application. The intranet of the :doc:`../overview` builds its provider with
+``create_provider``, which takes the policy:
+
+.. doctest::
+
+   >>> from unicodedata import normalize
+   >>> from scim2_models import ScimPolicy
+   >>> from scim2_models import default_comparison_key
+   >>> from intranet.scim import create_provider
+
+   >>> def compatibility_key(binding, value):
+   ...     return default_comparison_key(binding, normalize("NFKC", value))
+
+   >>> provider = create_provider(ScimPolicy(comparison_key=compatibility_key))
+
+Without a provider, the storage compares strings with the default comparison key.
+
+The comparison key of this example applies the compatibility normalization of Unicode, ``NFKC``,
+before the default comparison key. It replaces the characters that only differ by their presentation
+with their plain form:
+
+- the fullwidth letters of East Asian keyboards: ``ＢＪＥＮＳＥＮ`` becomes ``BJENSEN``;
+- the ligatures: ``fi`` becomes ``fi``;
+- the superscripts and the subscripts: ``x²`` becomes ``x2``.
+
+The stored values keep their form: only the comparisons change. More values then compare equal. A
+filter on ``ＢＪＥＮＳＥＮ`` finds the user ``bjensen``, and a second user named ``ｂｊｅｎｓｅｎ`` is refused:
+
+.. doctest::
+   :hide:
+
+   >>> from scim2_server.applications.wsgi import WSGIApplication
+   >>> from sqlalchemy import create_engine
+   >>> from sqlalchemy.orm import Session
+   >>> from werkzeug.test import Client
+   >>> from intranet.models import Base
+   >>> from intranet.scim import create_storage
+   >>> engine = create_engine("sqlite://")
+   >>> Base.metadata.create_all(engine)
+   >>> session = Session(engine)
+   >>> client = Client(WSGIApplication(create_storage(session, provider), provider))
+
+.. doctest::
+
+   >>> client.post("/v2/Users", json={"userName": "bjensen"}).status_code
+   201
+   >>> response = client.get("/v2/Users", query_string={"filter": 'userName eq "ＢＪＥＮＳＥＮ"'})
+   >>> response.json["totalResults"]
+   1
+   >>> client.post("/v2/Users", json={"userName": "ｂｊｅｎｓｅｎ"}).status_code
+   409
+
+.. doctest::
+   :hide:
+
+   >>> session.close()
+   >>> engine.dispose()
+
+The comparison key of this example applies to every string attribute, such as ``displayName``, where
+``x²`` and ``x2`` become equal too. To limit it to some attributes, test the URN of the binding,
+``binding.urn``, and return ``default_comparison_key(binding, value)`` for the others.
+
+Check what the database supports
+--------------------------------
+
+The comparisons happen in two places:
+
+- :doc:`scim2-models <scim2_models:index>` and :doc:`scim2-server <scim2_server:index>` compare in
+  Python, with the comparison key itself. They do so for the path filters of PATCH requests, such as
+  ``emails[value eq "…"]``, and to find the values a PATCH request adds or removes.
+- The storage compares in SQL, for the filters and the sorts of searches, and for the uniqueness
+  checks.
+
+Both give the same results only when the database follows the comparison key:
+
+- On SQLite, the storage calls the comparison key for each row, and follows it exactly. No index
+  serves these filters.
+- On PostgreSQL, the storage cannot call the comparison key. It compares the strings in the form of
+  the default comparison key, and warns when the policy sets another one. With the comparison key of
+  this guide, a search on ``ＢＪＥＮＳＥＮ`` finds nothing, and ``ｂｊｅｎｓｅｎ`` can be created next to
+  ``bjensen``. A PATCH request still compares in Python, and takes the two names as equal.
+
+On PostgreSQL, keep the default comparison key. Even then, a few letters differ between Python and
+the database, such as ``İ`` and the final sigma. For the details of each database, read
+:doc:`../explanation`.

doc/how-to/index.rst

@@ -10,3 +10,4 @@ the :doc:`../overview`.
    map-a-data-model
    publish-the-stored-attributes
    manage-the-transactions
+   compare-strings

doc/index.rst

@@ -40,8 +40,8 @@ Start with the page that matches the need:
 
 - To build a SCIM server over the tables of an existing application, follow the
   :doc:`overview`.
-- To map other data models, publish the stored attributes, or commit the changes of each
-  request, read the :doc:`how-to guides <how-to/index>`.
+- To map other data models, publish the stored attributes, commit the changes of each request,
+  or compare strings with a custom comparison key, read the :doc:`how-to guides <how-to/index>`.
 - For the reasons behind the behavior of the storage, read :doc:`explanation`.
 - For the public API, the supported searches and the errors, read the :doc:`reference <reference/index>`.
 

doc/overview.rst

@@ -249,4 +249,6 @@ The server now serves the accounts and the teams of the intranet:
   :doc:`how-to/publish-the-stored-attributes`.
 - To commit the changes of each request, in a synchronous or an asynchronous application, read
   :doc:`how-to/manage-the-transactions`.
+- To compare strings with the PRECIS rules or another custom comparison key, read
+  :doc:`how-to/compare-strings`.
 - For the reasons behind the behavior of the storage, read :doc:`explanation`.