3.13.2 upgrade "x-opt-rabbitmq-received-time" only available for new messages

[gblessas]

Describe the bug

In a context where messages are publicated from MQTT to a topic exchange with timestamp in ms enabled with 3.12.x version of rabbit.
Messages are stored in a stream binded to the topic.

before upgrade to 3.13.2 :

• i can consume messages from stream in amqp (pika) and get timestamp_in_ms field
• i can consume messages from stream in stream protocol but can't get this field available.

after upgrade to 3.13.2:

• i can consume messages from stream in amqp (pika) and get timestamp_in_ms field
• i can consume existing messages from stream in stream protocol (rstream) but can't get this field availble even if reading x-opt-rabbitmq-received-time annotation.
• if i publish new messages in mqtt, i can consume new messages from stream in stream protocol (rstream) and get the field available in x-opt-rabbitmq-received-time annotation

Reproduction steps

1. in 3.12.x with timestamps enabled in config
2. create a stream
3. bind stream to a topic
4. publish mqtt messages to this topic
5. read stream in amqp from first offset to check timestamp in ms is available (pika)
6. upgrade to 3.13.2
7. read stream in amqp from first offset to check timestamp in ms is still available (pika)
8. read stream in stream protocol from first offset to check timestamp in ms is NOT available (rstream, msgss.message_annotations.get(b'x-opt-rabbitmq-received-time'))
9. publish new messages in mqtt
10. read stream in stream protocol from last offsets to check timestamp in ms is available (rstream, msgss.message_annotations.get(b'x-opt-rabbitmq-received-time'))

...

Expected behavior

after upgrade timestamp_in_ms field should be available in stream protocol for already existing messages through x-opt-rabbitmq-received-time. not only for new messages.

Additional context

No response

[lukebakken]

Hello, thanks for using RabbitMQ.

Please, at the very least, you MUST provide code we can run to observe this issue. Right now you're asking us to take a significant amount of time writing code based on your description, when in fact you could provide code for us to run.

Provide a git repository with the necessary code to run to observe this issue. Please also include all of your RabbitMQ configuration, and anything else we need to reproduce this issue, including instructions for setting up the code if necessary.

Thank you. If we find an actual bug, we will open an issue to fix it.

[lukebakken]

This is what a colleague has to say -

It doesn't make sense to retro-actively add this field to messages which have been sitting in the Stream given that this field denotes when RabbitMQ received the message.

So, the behavior you report is expected behavior.

[gblessas]

I do not completely agree with you when standing at feature level and not implementation. producer timestamp is stored in stream and available in amqp. It is not completely equivalent with stream protocol features. Ils a good thing it is available now for New piblished datas but if we want existing clients/data set to be migrated to stream protocol we have to forget older datas.
Regards,

[lukebakken]

@gblessas - let me re-iterate what I stated in this comment - we are willing and able to assist you, if you are willing to put in some effort to make it easy for us to reproduce this issue.

Think about all of the free software you use every single day. Providing code and whatever else is necessary to make it easy for maintainers to assist you is the least you can do if you expect to get support for any of it.

[lukebakken]

I think this is the same person as @gblessas -

https://discord.com/channels/1092487794984755311/1240568845333299280/1242839767176183869

Here is the message:

I tried to provide code joined:

My process is the current
1) start rmq 3.12 by up312.sh script
2) publish 10 messages in mqtt with publishMqtt.py
3) read messages with readAmqp.py timestamp is available for all messages
4) read messages with readStream.py timestamp is not available
5) down server with down312.sh
6) upgrade server with up313.sh
7) read messages with readAmqp.py timestamp is available for all messages
8) read messages with readStream.py timestamp is not available
9) publish 10 messages in mqtt with publishMqtt.py
10) read messages with readAmqp.py timestamp is available for all messages
11) read messages with readStream.py timestamp is  only available for 10 last messages

Code - https://github.com/lukebakken/rabbitmq-server-11262

[lukebakken]

@gblessas - #11300