Various other tests involving message passing were failing in the past, and #5754 was intended to fix that. It may be worth looking over the fix to see whether it properly handles the flow graphs in the FEC async tests.

Some observations:

• In the CI qa_fecapi_dummy failure, the output was 240 bits short, which is exactly one frame. This suggests a message was dropped somewhere along the line.
• I tried to induce failures by adding sleeps to the message outputs of FEC Async Encoder, FEC Async Decoder, and Tagged Stream to PDU, but this didn't work.
• In the Debian qa_fecapi_repetition failures, the output was 512 bits (= 64 bytes) short in the packed case, and 64 bits short in the unpacked case. This is not a multiple of the frame size, which suggests that some other problem occurred.
• The Debian qa_fecapi_repetition failures occurred for all four async test cases in two consecutive builds. This suggests that that the failure may not be intermittent. Strangely, the failures did not occur for qa_fecapi_cc, qa_fecapi_dummy, or qa_fecapi_ldpc, which use the same setup.

It turns out there are two separate bugs here:

1. The flow graph sometimes completes with messages still in Tagged Stream to PDU's input queue. This can readily be reproduced by increasing rep here:

   gnuradio/gr-fec/python/fec/qa_fecapi_repetition.py

   Line 161 in 4d3eb9f

   rep = 3

2. FEC Async Decoder calculates the frame length in floating point, then converts it to an integer without rounding. Floating point error can cause it to be one bit/byte too low.

I'll reopen #7022 to track bug 2.

I was able to track down the bug after adding some debug logging. Here's what goes wrong:

• In the flow graph, an "Async Encoder" block produces PDU messages and sends them to a "PDU to Tagged Stream" block.
• The "Async Encoder" block sends along a burst of PDUs, then finishes its work and sends a system message to the "PDU to Tagged Stream" block indicating that it is "done". This marks the "PDU to Tagged Stream" block as "done" receiving messages.
• The "PDU to Tagged Stream" writes PDUs to its stream output, but eventually its output buffer fills to the point that there is no longer room for the next PDU to be copied in full. When this happens, it refuses to produce output. (More specifically, it uses calculate_output_stream_length to inform its parent class (tagged_stream_block) that it requires noutput_items to be at least as large as the next PDU, and the parent's general_work method returns no output.)
• The scheduler sees that the "PDU to Tagged Stream" block is "done" receiving messages, and refuses to fill its output buffer. The scheduler concludes that the block must be "done" and stops invoking it.

Prior to #323, the PDU to Tagged Stream block would output partial PDUs. I think we should restore this behaviour so that run-to-completion flow graphs will finish properly.

Unfortunately, it's the tagged_stream_block class that insists on having sufficient space to write the entire PDU in a single shot, so this couldn't be fixed in the PDU to Tagged Stream block unless it is changed to no longer be a Tagged Stream Block.

It seems likely that other Tagged Stream Blocks could confuse the scheduler in a similar way when used in run-to-completion flow graphs.

FWIW, this exact issue caused me to immediately abandon tagged stream blocks back in the start of my PDU work. As you have identified its a property of TSBs, which as expounded in #1446 are inherently evil.

TSBs [...] are inherently evil

I'm starting to see that! They make unreasonable demands of the scheduler, so it's not a great surprise that it responds badly.

Maybe the right path forward for the FECAPI tests is to rewrite them to avoid the "Tagged Stream to PDU" and "PDU to Tagged Stream" blocks.

If that turns out to be impractical, then I suppose it would be possible to make the "PDU to Tagged Stream" block behave by setting its output multiple to the FEC frame size.

Maybe the right path forward for the FECAPI tests is to rewrite them to avoid the "Tagged Stream to PDU" and "PDU to Tagged Stream" blocks.

On second thought, rewriting tests to avoid these blocks wouldn't fix the root cause. The "Tagged Stream to PDU" and "PDU to Tagged Stream" blocks don't need to use the scheduler-hostile tagged_stream_block class; they could instead be adjusted to handle partial PDUs.