Add post-processing hook for custom token transformations

[hedgert]

Summary

unasync currently provides excellent support for single-token replacements
during async-to-sync conversion.

Some transformations, however, require replacing one token with multiple
tokens or vice versa. Examples include:

• asyncio.current_task → current_thread
• asyncio.Task → Thread
• asyncio.sleep → time.sleep

Solution

This change introduces an optional post-processing hook that runs after
the existing token replacement stage.

The hook receives the token stream and allows more complex transformations
to be implemented without affecting the existing replacement mechanism.

Existing users are unaffected unless they choose to provide a
post-processor.

Motivation

This extension makes it possible to implement more sophisticated language
transformations while keeping the existing replacement system simple.

[pquentin]

Thanks! Do you mind adding a test to clarify how this would be used?

[hedgert]

(I think I overwrote my earlier reply with my comment about the token to tokens change - so repeating it). Re your request for an example:

Thanks for the review. While putting together the test, I realised there are two possible directions:

• a minimal example that demonstrates the new hook, or
• extending unasync with a reusable implementation so users can express these more complex transformations through configuration rather than writing code (as I ended up writing code similar to your original to walk the token list but comparing consecutive tokens to look for matches instead of comparing single tokens - and it might be beneficial for others to be able to use that rather than have to write it as an extension using my hook).

Do you have a preference? I'm happy to take whichever approach best fits the project's direction.

[pquentin]

Thanks for asking! Assuming the reusable implementation is built on top of this change, I would start with the minimal test, merge this PR, and then open a new PR with the reusable implementation.

[hedgert]

Thanks for the suggestion. I've added a regression test for the new hook. The test demonstrates a subclass overriding _postprocess_tokens() to perform a context-sensitive token transformation and verifies that the transformed output is written correctly.

[hedgert]

Sorry please pause - although my test ran successfully standalone I hadn't run the full test suite and it's not running in the test suite - I will fix and update here.

[hedgert]

OK (I had lost the sort directories change that was the subject of a separate - overlapping - pull request). Given the tests here in Github worked, it maybe that has no effect on any environment but mine - but the sorted directories fix is required in my environment or the project structure test fails. My new test is now ready for review.