Skip to content

gh-157710: Add more asserts to internal PyUnicode operations - #159033

Open
encukou wants to merge 1 commit into
python:mainfrom
encukou:more-unicode-protection
Open

encukou wants to merge 1 commit into
python:mainfrom
encukou:more-unicode-protection

Conversation

@encukou

@encukou encukou commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Add checks based on the review on #159028:

  • _Py*UnicodeObject_CAST macros assert they have the right bits for the shape
    (this means that _PyUnicode_UTF8 asserts it's not called on a compact ASCII object.)

  • _PyUnicode_IsModifiable & _PyUnicodeWriter_CanWrite assert that UTF-8 storage hasn't been allocated yet

  • Move _PyUnicode_UTF8 to the internal header for use in _PyUnicodeWriter_CanWrite

- `_Py*UnicodeObject_CAST` macros assert they have the right bits for the shape
- `_PyUnicode_IsModifiable` & `_PyUnicodeWriter_CanWrite` assert that
  UTF-8 storage hasn't been allocated yet
assert(!_Py_IsImmortal(buffer));
assert(PyUnicode_IS_COMPACT_ASCII(buffer)
|| _PyUnicode_UTF8(buffer) == NULL
|| _PyUnicode_UTF8(buffer) == PyUnicode_DATA(buffer));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer moving _PyUnicode_HAS_UTF8_MEMORY() static inline function to pycore_unicodeobject.h and just test:

assert(!_PyUnicode_HAS_UTF8_MEMORY(buffer);

_Py_CAST(PyASCIIObject*, (op)))
#define _PyCompactUnicodeObject_CAST(op) \
(assert(PyUnicode_Check(op)), \
(assert(!_PyASCIIObject_CAST(op)->state.ascii || !_PyASCIIObject_CAST(op)->state.compact), \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: you might write the test as "make sure that it's not a compact ASCII string":

Suggested change
(assert(!_PyASCIIObject_CAST(op)->state.ascii || !_PyASCIIObject_CAST(op)->state.compact), \
(assert(!(_PyASCIIObject_CAST(op)->state.ascii \
&& _PyASCIIObject_CAST(op)->state.compact)), \

It's just a minor coding style suggestion, feel free to ignore it.

return 0;
}

static inline char* _PyUnicode_UTF8(PyObject *op)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
static inline char* _PyUnicode_UTF8(PyObject *op)
static inline char*
_PyUnicode_UTF8(PyObject *op)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting core review type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants