Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add error message for topic query parameters with Pulsar sink and test other options #124356

Closed
kathancox opened this issue May 17, 2024 · 1 comment · Fixed by #124581
Closed
Assignees
Labels
A-cdc Change Data Capture C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-cdc

Comments

@kathancox
Copy link
Contributor

kathancox commented May 17, 2024

Describe the problem

While writing docs examples for the apache pulsar sink, found that some of the options that are listed in an issue as "unsupported" seem to work. The testing for docs hasn't been exhaustive, but found the following:

  1. resolved, updated, and mvcc_timestamp do send out timestamps to the pulsar sink, and they seem to be correct.
  2. topic_prefix and topic_name do not work; however, the changefeed creates successfully and there is no error that these parameters do not work. Pulsar will instead automatically create the table name based on the target table.

To Reproduce

Create changefeed with prefix:

create changefeed for table movr.promo_codes into 'pulsar://{host_IP}:6650?topic_prefix=foo_' with updated;

Try to start a consumer to listen to message for the foo_promo_codes topic:

sudo docker run -it --network="host" apachepulsar/pulsar:latest bin/pulsar-client consume foo_promo_codes -s sub1 -n 0

Consumer listens, but does not get any messages:

2024-05-17T17:59:56,048+0000 [pulsar-client-io-1-3] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [foo_promo_codes][sub1] Subscribing to topic on cnx [id: 0xcf3bd466, L:/127.0.0.1:37036 - R:localhost/127.0.0.1:6650], consumerId 0
2024-05-17T17:59:56,058+0000 [pulsar-client-io-1-3] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [foo_promo_codes][sub1] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0

Take away the prefix from the command:

sudo docker run -it --network="host" apachepulsar/pulsar:latest bin/pulsar-client consume promo_codes -s sub1 -n 0

Receive messages:

----- got message -----
key:[null], properties:[], content:{"Key":["cafe0bb2-4379-44be-9046-7f561af90c8b"],"Value":{"after": {"code": "cafe0bb2-4379-44be-9046-7f561af90c8b", "creation_time": "2019-01-11T03:04:05", "description": "However list project across family computer. Structure eight draw black modern. Focus certain job ok step detail hundred listen. Water technology station report.", "expiration_time": "2019-01-27T03:04:05", "rules": {"type": "percent_discount", "value": "10%"}}, "updated": "1715968562184554000.0000000000"},"Topic":"promo_codes"}
----- got message -----
key:[null], properties:[], content:{"Key":["e272c8f5-9498-4b10-8537-991bb7aaf733"],"Value":{"after": {"code": "e272c8f5-9498-4b10-8537-991bb7aaf733", "creation_time": "2018-12-11T03:04:05", "description": "Edge pay those level admit cut. Career total better course professor society. His these glass body account play. Water walk list rich recent.", "expiration_time": "2019-01-02T03:04:05", "rules": {"type": "percent_discount", "value": "10%"}}, "updated": "1715968562184554000.0000000000"},"Topic":"promo_codes"}
----- got message -----
key:[null], properties:[], content:{"Key":["e8c0f884-a4cb-4036-838b-a0961376076e"],"Value":{"after": {"code": "e8c0f884-a4cb-4036-838b-a0961376076e", "creation_time": "2019-01-10T03:04:05", "description": "There true without must. Mr different goal machine control. Response either above know generation agent. Show town quality later prevent.", "expiration_time": "2019-01-17T03:04:05", "rules": {"type": "percent_discount", "value": "10%"}}, "updated": "1715968562184554000.0000000000"},"Topic":"promo_codes"}

Environment:
This was testing on a ubuntu server running pulsar in Docker with a local cockroachdb cluster v24.1.0-beta.1.

Jira issue: CRDB-38845

@kathancox kathancox added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-cdc Change Data Capture labels May 17, 2024
@blathers-crl blathers-crl bot added the T-cdc label May 17, 2024
Copy link

blathers-crl bot commented May 17, 2024

cc @cockroachdb/cdc

craig bot pushed a commit that referenced this issue May 24, 2024
124581: changefeedccl: add error for unsupported pulsar params r=asg0451 a=rharding6373

This PR adds an error message when unsupported pulsar params are used in the sink URI.

Epic: CRDB-9180
Fixes: #124356

Release note(enterprise change): Adds an error message for unsupported pulsar sink parameters.

Co-authored-by: rharding6373 <harding@cockroachlabs.com>
@craig craig bot closed this as completed in 586e34d May 24, 2024
blathers-crl bot pushed a commit that referenced this issue May 24, 2024
This PR adds an error message when unsupported pulsar params are used in
the sink URI.

Epic: CRDB-9180
Fixes: #124356
Release note(enterprise change): Adds an error message for unsupported pulsar
sink parameters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-cdc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants