feat: enable QASYMM8_SIGNED to F32 assembly dequantization

[morgolock]

Enable QASYMM8_SIGNED input and weights to use the F32 DequantizeFloat assembly output stage, including the direct convolution selection path.

Propagate input and weight zero-points plus the unrounded mathematical K depth so asymmetric offset correction uses the real GEMM/convolution depth rather than arm_gemm's padded internal K.

Add NEON validation coverage for the direct I8S8F32 convolution path.

Performance was checked on a A76, pinned to CPU 4 with one thread. The change is neutral on large workloads and improves the small NHWC signed int8 to F32 convolution path: QASYMM8_SIGNED/RunSmallDequantizeF32 NHWC/no-activation cases show a 1.36x geomean speedup over github/main, with the tiniest single-batch cases improving by roughly 1.45x to 3.13x.

Change-Id: Ie723d3da629d48de6de737c425bf7ad48e0f7feb

[Passavee-Losripat]

Hi, I've been testing this PR for OpenVINO's ARM CPU plugin int8 enablement and found a possible validation gap in the new QASYMM8_SIGNED->F32 dequantization path.

QSYMM8_PER_CHANNEL weights are correctly rejected by validate() on this path. However, QASYMM8_SIGNED weights whose QuantizationInfo carries a per-channel scale vector are accepted and the kernel then dequantizes every output channel with scale[0] (the scalar uniform().scale product in create_arm_gemm_dequant).

I create minimal repo to test this: all-ones i8 input, all-ones 3×3×3 i8 weights, bias 0, distinct per-channel scales s[c] -> each interior output should be 27·s[c]:

channel    got      expected(27*s[c])    ratio
c00      0.0270        0.0270            1.0000
c01      0.0270        0.0540            0.5000
c02      0.0270        0.1080            0.2500
...
c11      0.0270       55.2960            0.0005

validate() returns OK, but every channel is computed with s[0]. Since quantization frameworks (NNCF, PyTorch, TF) produce per-channel weight scales by default, this input shape is what integrations like OpenVINO naturally pass, so it fails silently on real models rather than erroring out.

I am thinking of rejecting scale().size() > 1 on this path the same way QSYMM8_PER_CHANNEL is rejected, unless per-channel support is planned. I can share the full standalone test program if useful.

[morgolock]

Hi, I've been testing this PR for OpenVINO's ARM CPU plugin int8 enablement and found a possible validation gap in the new QASYMM8_SIGNED->F32 dequantization path.

QSYMM8_PER_CHANNEL weights are correctly rejected by validate() on this path. However, QASYMM8_SIGNED weights whose QuantizationInfo carries a per-channel scale vector are accepted and the kernel then dequantizes every output channel with scale[0] (the scalar uniform().scale product in create_arm_gemm_dequant).

I create minimal repo to test this: all-ones i8 input, all-ones 3×3×3 i8 weights, bias 0, distinct per-channel scales s[c] -> each interior output should be 27·s[c]:

channel    got      expected(27*s[c])    ratio
c00      0.0270        0.0270            1.0000
c01      0.0270        0.0540            0.5000
c02      0.0270        0.1080            0.2500
...
c11      0.0270       55.2960            0.0005

validate() returns OK, but every channel is computed with s[0]. Since quantization frameworks (NNCF, PyTorch, TF) produce per-channel weight scales by default, this input shape is what integrations like OpenVINO naturally pass, so it fails silently on real models rather than erroring out.

I am thinking of rejecting scale().size() > 1 on this path the same way QSYMM8_PER_CHANNEL is rejected, unless per-channel support is planned. I can share the full standalone test program if useful.

Hi @Passavee-Losripat

Good catch. I added validation to reject QASYMM8_SIGNED weights with multiple scales on this dequantized path, since the current output stage only supports a single scalar scale.

Please try the latest patchset and let us know if this works for you.

Hope this helps

[gunes-arm]

Quite confused here. When we have per channel quantized weights, the data type should be provided as QSYMM8_PER_CHANNEL. This should be the correct usage assumption throughout the library and how we validate against. We may need to think about trying to prevent this misuse in another way.

@Passavee-Losripat would you mind sharing your reproducible? And, do we have any workloads/integration points that OpenVino relies on ACL accepting a per-channel quantized weights tensor with QASYMM8_SIGNED/QASYMM8? Because, to me, we might prevent some TensorInfo being constructed this way, as it's not the correct usage of the types, but that's going to cause crashes instead of silently validating inside a function/operator.

[Passavee-Losripat]

Hi, @gunes-arm and @morgolock. I attached acl_per_channel_dequant_repro.cpp here as a self-contained repro: all-ones i8 input, all-ones 3*3*3 i8 weights, zero bias, distinct per-channel scales, so each interior output must equal 27*s[c] with no reference implementation needed. I re-ran it against the latest patchset and the per-channel vector case now fails validate() (raised from CpuConv2d::validate), and the per-tensor scalar case still validates and computes bit-exactly, so the added check resolves the issue I reported without over-rejecting.

On your question about integration points, OpenVINO relies on ACL accepting QASYMM8_SIGNED weights with a per-channel scale vector today, in two places:

1. The int8 FullyConnected path that is already merged in OpenVINO master (src/plugins/intel_cpu/src/nodes/executors/acl/acl_lowp_fullyconnected.cpp) constructs exactly this shape: QASYMM8_SIGNED weights with QuantizationInfo(dequantizationScales) where the scale vector is per-output-channel.
2. The int8 Convolution enablement in progress (which this PR would serve) follows the same pattern.

On the dtype question, OpenVINO's quantization pipeline produces symmetric per-channel i8 weights (zero-point 0), which semantically maps to QSYMM8_PER_CHANNEL but we got two reasons why we currently used QASYMM8_SIGNED + vector instead:

• The scale vector OpenVINO attaches is not strictly the weights' own quantization scale but it's the combined dequantization scale (src_scale * weight_scale, per output channel, folded together by our graph-level transformations), with the src tensor's QuantizationInfo set to trivial (scale=1) to compensate. It's effective scale attached to the weights tensor rather than a literal property of the weights data.
• On the standard requantize path, this already works correctly: quantization::compute_quantized_multipliers_and_shifts (src/core/utils/quantization/AsymmHelpers.cpp) sizes its multiplier array from weights->quantization_info().scale().size() and iterates every entry, with no dtype check at all. So the library's own output-stage math already reads and applies every entry in the scale vector, regardless of whether the tag is QSYMM8_PER_CHANNEL or QASYMM8_SIGNED. The FC integration above already relies on exactly this behavior, merged in OpenVINO master today.

Because of that, rejecting this combination when constructing TensorInfo could break an already-working OpenVINO integration on the standard requantization path, where the per-channel vector is handled correctly. I think new rejection at validate() looks quite right to me. It would reject only the output stage that cannot actually consume the vector, while allowing the existing supported path to continue working. It also turns the current silent incorrect result into a clear validation failure at the point where the unsupported combination is used.

Happy to help test further if useful.

[gunes-arm]

Hi @Passavee-Losripat, I agree that breaking what's already working is undesirable and we can keep the validation checks introduced in this PR because they are not harmful to current functionality in any sense.

But, for the per-channel quantized tensors, we expect the data type to be QSYMM8_PER_CHANNEL. Although this coupling is not ideal, for us it's too late to change this :) So, using QASYMM8_SIGNED or any other type with a quantization scale vector of length > 1 is an API misuse we couldn't prevent.

Many different places in the library rely on this type. We check is_data_type_quantized_per_channel in both Cpu and Gpu backends, including matrix multiplicaton, convolution and depthwise routines. So, I expect we can find more cases where validation doesn't fail but functional correctness does if the types aren't used properly. Also, having these validations in only some places is another validation gap for us as it seems like we are accepting length > 1 quantization info with QASYMM8_SIGNED in other operators.

As I said, I think the current set of additional checks can be added, but I recommend using QSYMM8_PER_CHANNEL where applicable. Could you check if this is achievable for a future release so that we can add guards in TensorInfo as well?

[gunes-arm]

There is a bit of a weird situation here.

I'd remove this check because that's what the switch/case is for and remove the S8 case from the switch. Would we breaking anything? I'm not too sure. Only if there is some misuse of data types. Same situation goes with comments and U8 type.

So, I'd suggest

• Ignore S8 was in the case statement above and just remove a->data_type() == DataType::QASYMM8_SIGNED from the ifs here as it's already checked.
• Create a ticket to resolve the S8/U8 confusion with tests involved.

[morgolock]

Thanks, agreed. I’ll keep this PR scoped to the QASYMM8_SIGNED -> F32 path, remove the redundant a->data_type() check inside the QASYMM8_SIGNED case, and avoid changing the S8/U8 handling here. I’ll create a follow-up MLCE/Jira ticket for the broader S8/U8 vs QASYMM8/QASYMM8_SIGNED validation cleanup with dedicated tests.

[gunes-arm]

I think we inadvertently added S8 handling as well. Could you remove that?

[gunes-arm]

Can you make the commit & PR title start with uppercase letter? Thanks :)

[gunes-arm]

I think we should use SmallConvolutionLayerDataset here as well.

[gunes-arm]

Why don't we instantiate and use a convolution_layer with halfs?

[gunes-arm]

I think we inadvertently added S8 handling as well. Could you remove that?

[gunes-arm]

Hi @Passavee-Losripat, I agree that breaking what's already working is undesirable and we can keep the validation checks introduced in this PR because they are not harmful to current functionality in any sense.

But, for the per-channel quantized tensors, we expect the data type to be QSYMM8_PER_CHANNEL. Although this coupling is not ideal, for us it's too late to change this :) So, using QASYMM8_SIGNED or any other type with a quantization scale vector of length > 1 is an API misuse we couldn't prevent.

Many different places in the library rely on this type. We check is_data_type_quantized_per_channel in both Cpu and Gpu backends, including matrix multiplicaton, convolution and depthwise routines. So, I expect we can find more cases where validation doesn't fail but functional correctness does if the types aren't used properly. Also, having these validations in only some places is another validation gap for us as it seems like we are accepting length > 1 quantization info with QASYMM8_SIGNED in other operators.

As I said, I think the current set of additional checks can be added, but I recommend using QSYMM8_PER_CHANNEL where applicable. Could you check if this is achievable for a future release so that we can add guards in TensorInfo as well?

[gunes-arm]

Can we create a ticket for adding some runtime validation checks to TensorInfo class so that it throws an exception if per-channel quantized data type is not used together with length > 1 quantization info?

And, after that, can we add a code comment just before every validation we did for temporarily handling this case, saying that these checks should be removed once the mentioned ticket is resolved?

It'd be a good idea if we also add to the desc. of the ticket that we'd like these comments gone as part of DoD.