Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Features: Batching and graphile-worker upgrade to 0.16.0-rc.1 #524

Open
wants to merge 102 commits into
base: main
Choose a base branch
from
Open

Features: Batching and graphile-worker upgrade to 0.16.0-rc.1 #524

wants to merge 102 commits into from

Commits on Oct 2, 2023

  1. Upgrade graphile-worker

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    3eea949 View commit details
    Browse the repository at this point in the history

  2. CronItem.pattern to match

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    e4630f6 View commit details
    Browse the repository at this point in the history

  3. Update graphile job schema

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    8e887af View commit details
    Browse the repository at this point in the history

  4. Add FAIL_LOCKED_JOBS_ON_STARTUP

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    0400e02 View commit details
    Browse the repository at this point in the history

  5. Add to .env.example

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    6440e30 View commit details
    Browse the repository at this point in the history

  6. Detect migration errors

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    e3c9559 View commit details
    Browse the repository at this point in the history

  7. Rename migration env var

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    25d6478 View commit details
    Browse the repository at this point in the history

  8. Improve migration helper, don't require restart

    @nicktrn
    nicktrn committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    00eade2 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. Remove restart comment for now

    @nicktrn
    nicktrn committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    8efb0a5 View commit details
    Browse the repository at this point in the history

  2. Add migration docs

    @nicktrn
    nicktrn committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    d20a49e View commit details
    Browse the repository at this point in the history

  3. Link to migration docs

    @nicktrn
    nicktrn committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    aab2c8b View commit details
    Browse the repository at this point in the history

  4. Lowercase

    @nicktrn
    nicktrn committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    b45f9fb View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    6c3921b View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    4f1c52f View commit details
    Browse the repository at this point in the history

  2. Fix merge

    @nicktrn
    nicktrn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    86d6343 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    823506c View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    07af335 View commit details
    Browse the repository at this point in the history

  2. Don't fail locked jobs, add migration delay

    @nicktrn
    nicktrn committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    1219cbf View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2023

  1. Shut down existing workers prior to new migrations

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    0c449bf View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    d156d62 View commit details
    Browse the repository at this point in the history

  3. Latest lockfile

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    fd2ddb7 View commit details
    Browse the repository at this point in the history

  4. Replace migration error screenshot

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    6be9909 View commit details
    Browse the repository at this point in the history

  5. Remove redundant env var

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    c2dce5b View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    37e25ac View commit details
    Browse the repository at this point in the history

  7. Check graphile schema exists

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    4fa7e96 View commit details
    Browse the repository at this point in the history

  8. Upsert batch job function

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    59e16a9 View commit details
    Browse the repository at this point in the history

  9. Abstract migration helper into service

    @nicktrn
    nicktrn committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    3eecc27 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. Tweak add_batch_job function

    @nicktrn
    nicktrn committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    7cbc928 View commit details
    Browse the repository at this point in the history

  2. Fix batch function, more logging

    @nicktrn
    nicktrn committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    2070b14 View commit details
    Browse the repository at this point in the history

  3. Fix batch function.. again

    @nicktrn
    nicktrn committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    288341d View commit details
    Browse the repository at this point in the history

  4. Add enqueueBatch method

    @nicktrn
    nicktrn committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    d35ddfe View commit details
    Browse the repository at this point in the history

  5. Simulate batch jobs

    @nicktrn
    nicktrn committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    284acb2 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    ab17db0 View commit details
    Browse the repository at this point in the history

  7. Notify before migration delay

    @nicktrn
    nicktrn committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    d4a5922 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2023

  1. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    82163c5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    6b12a92 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    546588d View commit details
    Browse the repository at this point in the history

  2. Fix migration error detection

    @nicktrn
    nicktrn committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    1637d25 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    5e37d95 View commit details
    Browse the repository at this point in the history

  4. Batch enqueue types

    @nicktrn
    nicktrn committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    f648fb6 View commit details
    Browse the repository at this point in the history

  5. Upgrade graphile to latest rc

    @nicktrn
    nicktrn committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    285e7bb View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. Make batch enqueue only accept batch tasks

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    8ac89a0 View commit details
    Browse the repository at this point in the history

  2. Fix cleanup and simulate batch

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    01bd1af View commit details
    Browse the repository at this point in the history

  3. Fix custom batch job function

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    84c28bc View commit details
    Browse the repository at this point in the history

  4. Add batch opts to event trigger

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    4eef340 View commit details
    Browse the repository at this point in the history

  5. Lint

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    de428f0 View commit details
    Browse the repository at this point in the history

  6. Expose batch function on even triggers only

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    f7872a4 View commit details
    Browse the repository at this point in the history

  7. Add batching to event dispatch

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    f0d7b63 View commit details
    Browse the repository at this point in the history

  8. Event trigger batching tests

    @nicktrn
    nicktrn committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    175b58c View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. Pass jobKey to queueName func

    @nicktrn
    nicktrn committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    0d53568 View commit details
    Browse the repository at this point in the history

  2. Batch support for Job Runs

    @nicktrn
    nicktrn committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    5495f93 View commit details
    Browse the repository at this point in the history

  3. Update UI to support batch payloads

    @nicktrn
    nicktrn committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    497bbbd View commit details
    Browse the repository at this point in the history

  4. Update batch send example

    @nicktrn
    nicktrn committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    57fe918 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    b26bf16 View commit details
    Browse the repository at this point in the history

  6. Shorten redisinsight container name

    @nicktrn
    nicktrn committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    d10a7d0 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. Improve KV get types

    @nicktrn
    nicktrn committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    531b9e7 View commit details
    Browse the repository at this point in the history

  2. Webhook delivery batching and some fixes

    @nicktrn
    nicktrn committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    ccb6208 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. Fix payload size chunking

    @nicktrn
    nicktrn committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    5aa448b View commit details
    Browse the repository at this point in the history

  2. Fix batched payloads for n = 1

    @nicktrn
    nicktrn committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    32c6815 View commit details
    Browse the repository at this point in the history

  3. Start fixing batched webhook delivery

    @nicktrn
    nicktrn committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    9971267 View commit details
    Browse the repository at this point in the history

  4. Lower default max interval

    @nicktrn
    nicktrn committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    127514d View commit details
    Browse the repository at this point in the history

  5. Rename dispatch batcher task

    @nicktrn
    nicktrn committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    563736d View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    df92240 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2023

  1. Fix webhook delivery

    @nicktrn
    nicktrn committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    6d914b6 View commit details
    Browse the repository at this point in the history

  2. Batched run UI

    @nicktrn
    nicktrn committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    5bd1c20 View commit details
    Browse the repository at this point in the history

  3. Add read replica support

    @nicktrn
    nicktrn committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    ad1b512 View commit details
    Browse the repository at this point in the history

  4. Use read replica for jobs view queries

    @nicktrn
    nicktrn committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    fb81d90 View commit details
    Browse the repository at this point in the history

  5. Fix debug log

    @nicktrn
    nicktrn committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    60f47c4 View commit details
    Browse the repository at this point in the history

  6. Simplify batch options

    @nicktrn
    nicktrn committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    cb09814 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2023

  1. Add shopify delete all webhooks job

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    5324f9a View commit details
    Browse the repository at this point in the history

  2. Batched payload types

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    6006ba0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    0227271 View commit details
    Browse the repository at this point in the history

  4. Improve tests and add one for batch chunking

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    da439ce View commit details
    Browse the repository at this point in the history

  5. Batched webhook delivery examples

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    16ca511 View commit details
    Browse the repository at this point in the history

  6. Batched webhook trigger catalog entry

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    3738794 View commit details
    Browse the repository at this point in the history

  7. Add total events to trigger card

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    a9754a0 View commit details
    Browse the repository at this point in the history

  8. Migration comments and logging

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    6b702da View commit details
    Browse the repository at this point in the history

  9. Adjust batch defaults and limits

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    ac8d301 View commit details
    Browse the repository at this point in the history

  10. Document current event payload limit

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    5f94edf View commit details
    Browse the repository at this point in the history

  11. Batching docs

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    dca8f40 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    1d677e1 View commit details
    Browse the repository at this point in the history

  13. Fix merge

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    93d32d1 View commit details
    Browse the repository at this point in the history

  14. Fix rerun

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    3db8feb View commit details
    Browse the repository at this point in the history

  15. Consider test and external account when batching

    @nicktrn
    nicktrn committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    d5ac342 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2023

  1. Context on job run

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    9ea24f8 View commit details
    Browse the repository at this point in the history

  2. Fix testing package

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    61b7a7a View commit details
    Browse the repository at this point in the history

  3. Remove redundant batch param

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    d330134 View commit details
    Browse the repository at this point in the history

  4. Batch example respects event payload limit

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    b926ce8 View commit details
    Browse the repository at this point in the history

  5. Move testing package paths to compilerOptions

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    61cfe97 View commit details
    Browse the repository at this point in the history

  6. Changeset

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    d8698e2 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    2afa199 View commit details
    Browse the repository at this point in the history

  8. Consider total code length for highlight limit

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    9c6f6aa View commit details
    Browse the repository at this point in the history

  9. Use eventId not eventRecordId

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    c250eb6 View commit details
    Browse the repository at this point in the history

  10. Only include single event metadata when unbatched

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    4cb87c5 View commit details
    Browse the repository at this point in the history

  11. Include run id in batch trigger example

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    59a0370 View commit details
    Browse the repository at this point in the history

  12. Release worker utils on shutdown

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    7a1485b View commit details
    Browse the repository at this point in the history

  13. Use reschedule helper to handle full batches

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    8415eac View commit details
    Browse the repository at this point in the history

  14. Safely get queue name from ID

    @nicktrn
    nicktrn committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    a051fac View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2023

  1. Backward compatible webhook delivery

    @nicktrn
    nicktrn committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    9cb70df View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    d2cf1cc View commit details
    Browse the repository at this point in the history

  3. Remove dead code

    @nicktrn
    nicktrn committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    8edd2c4 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2023

  1. Merge branch 'main' into worker-upgrade

    @nicktrn
    nicktrn committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    f130fbd View commit details
    Browse the repository at this point in the history