Pure Python JSON decoder reports the error position one character past an invalid control character

[yangbaechu]

Bug report

Bug description:

The pure Python JSON decoder reports JSONDecodeError.pos one character after a literal control character in a JSON string. The C decoder reports the position of the control character itself.

For example, the following document contains an actual newline character, not the valid JSON escape sequence \n:

document = '"a\nb"'

Its character positions are:

0  "
1  a
2  \n
3  b
4  "

The difference can be reproduced on the CPython main (b3be16d) branch:

from test.support import import_helper

pyjson = import_helper.import_fresh_module(
    "json",
    blocked=["_json"],
)
cjson = import_helper.import_fresh_module(
    "json",
    fresh=["_json"],
)

document = '"a\nb"'

for name, module in [("Python", pyjson), ("C", cjson)]:
    try:
        module.loads(document)
    except ValueError as error:
        print(
            name,
            error.pos,
            error.lineno,
            error.colno,
            repr(document[error.pos]),
        )

Current output:

Python 3 2 1 'b'
C 2 1 3 '\n'

The parsing failure is caused by the newline at position 2, so the pure Python decoder should report pos=2, lineno=1, and colno=3, as the C decoder does. Instead, it points to the following valid character, b. These are public attributes of JSONDecodeError, so this is observable beyond a difference in error message formatting.

The same one-character offset occurs for every literal control character from U+0000 through U+001F.

The relevant pure Python scanner regular expression consumes both the ordinary string content and its terminator. For this input, the terminator is the newline at position 2, while Match.end() returns 3. That value is currently passed to JSONDecodeError.

I am working on a fix and plan to submit a PR with regression tests.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-154936: Fix JSON control character error position #154941