iso-2022-cn-ext encoding silently returns b'' for non-ASCII text

[fedonman]

Bug description:

Encoding non-ASCII text to ISO-2022-CN-EXT returns b'' with no error. The ASCII part is
lost with it, and the codec reports the whole input as consumed:

>>> 'ABC中DEF'.encode('iso-2022-cn-ext')
b''
>>> codecs.iconv_encode('ISO-2022-CN-EXT', '中')   # (output, consumed)
(b'', 1)
>>> codecs.iconv_encode('ISO-2022-CN', '中')       # without EXT
(b'\x1b$)A\x0eVP\x0f', 1)

No error handler changes it: strict, replace, ignore and xmlcharrefreplace all give
b''. Only strings stored as 2-byte characters are affected; a pure-ASCII string encodes
normally, and an unencodable character in a 1-byte or 4-byte string raises
UnicodeEncodeError as expected.

This is not an iconv limitation. iconv -f UCS-2LE -t ISO-2022-CN-EXT, which is the
conversion the codec asks for on a 2-byte string, produces the expected
\x1b$)A\x0eVP\x0f.

The iconv codecs are new in 3.16 (gh-152997), so no released version is affected.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-154863: Fix iconv encoding of ISO-2022-CN-EXT returning empty bytes #154899

[tekinertekin]

@fedonman - before anyone else repeats efforts: are you planning to tackle this one too, based on #154862? Happy to give it up for you, just want to avoid two of us writing equivalent patches.

A couple of things I observed while reading, in case they help either way:

The "reports the whole input as consumed" part is definitely not iconv — _codecs_iconv_encode_impl inside Modules/_codecsmodule.c returns codec_tuple(..., PyUnicode_GET_LENGTH(str)), meaning the reported consumed count is always the string length, regardless of what iconv produced as output. The empty output would still need explanation though.

Your observation regarding "only 2-byte strings" is consistent with the kind dispatch inside _PyUnicode_EncodeIconv in Objects/unicodeobject.c: the 1-byte kind feeds ISO-8859-1, the 4-byte kind feeds native-endian UTF-32, and the 2-byte kind feeds UCS-2 — but only where ICONV_UCS2_PIVOT is defined, i.e. glibc or GNU libiconv. Elsewhere the 2-byte kind is widened to UTF-32 instead. So the failing kind is the only one that feeds that source encoding. If the fault really is on the UCS-2 → ISO-2022-CN-EXT path, that predicts it should not reproduce on a non-glibc build, which would be a quick way to confirm the direction.

That same function has the explicit flush stage whose comment mentions ISO-2022 shift sequences, which aligns with the state handling in #154862 — that might be a good place to look.

I wasn't able to build 3.16 locally, so I haven't reproduced it by hand; the observations above are from reading main.