Features: Batching and graphile-worker upgrade to 0.16.0-rc.1

[nicktrn]

Closes #396
Closes #348

✅ Checklist

• I have followed every step in the contributing guide
• The PR title follows the convention.
• I ran and tested the code works
• More testing just to be safe
• Create migration docs

---

Testing

Migration

1. Create this docker-compose.yml

version: "3"

volumes:
  data:

networks:
  app_network:
    external: false

x-db-url: &db-url postgresql://postgres:postgres@db:5432/postgres

x-env: &env
  LOGIN_ORIGIN: http://localhost:3030
  APP_ORIGIN: http://localhost:3030
  PORT: 3030
  REMIX_APP_PORT: 3030
  MAGIC_LINK_SECRET: secret
  SESSION_SECRET: secret
  ENCRYPTION_KEY: ae13021afef0819c3a307ad487071c06
  DATABASE_URL: &db-url
  DIRECT_URL: &db-url
  GRACEFUL_SHUTDOWN_TIMEOUT: 5000
  NODE_ENV: production
  RUNTIME_PLATFORM: docker-compose
  TRIGGER_LOG_LEVEL: debug

services:
  old-webapp:
    image: ghcr.io/triggerdotdev/trigger.dev:v2.2.21
    restart: "no"
    environment:
      <<: *env
      WORKER_ENABLED: false
      EXECUTION_WORKER_ENABLED: false
      TASK_OPERATION_WORKER_ENABLED: false
    ports:
      - 3065:3030
    depends_on:
      - db
    networks:
      - app_network

  old-workers:
    image: ghcr.io/triggerdotdev/trigger.dev:v2.2.21
    restart: "no"
    environment:
      <<: *env
      HTTP_SERVER_DISABLED: true
    depends_on:
      - db
    networks:
      - app_network

  new-webapp:
    image: ghcr.io/triggerdotdev/trigger.dev:batching-rc.2
    restart: always
    environment:
      <<: *env
      WORKER_ENABLED: false
      EXECUTION_WORKER_ENABLED: false
      TASK_OPERATION_WORKER_ENABLED: false
    ports:
      - 3030:3030
    depends_on:
      - db
    networks:
      - app_network

  new-workers:
    image: ghcr.io/triggerdotdev/trigger.dev:batching-rc.2
    restart: always
    environment:
      <<: *env
      HTTP_SERVER_DISABLED: true
    depends_on:
      - db
    networks:
      - app_network
      
  db:
    image: postgres:14
    restart: always
    volumes:
      - data:/var/lib/postgresql/data/
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres
    networks:
      - app_network
    ports:
      - 3085:5432

2. Start up the DB

docker-compose up -d db

3. Start up the old webapp and workers

docker-compose up old-webapp old-workers

4. In another window, start up the new webapp and workers

docker-compose up new-webapp new-workers

5. To delete the DB and start again, maybe after altering some env vars

docker-compose down -v

Batching

trigger.dev/references/job-catalog/src/events.ts

Lines 156 to 211 in 2afa199

	const maxPayloads = 10;
	const maxInterval = 10;
	client.defineJob({
	id: "batch-trigger-receive",
	name: "Batch Trigger Receive",
	version: "1.0.0",
	trigger: eventTrigger({
	name: "batch.trigger",
	batch: {
	maxPayloads,
	maxInterval,
	},
	}),
	run: async (payload, io) => {
	await io.logger.info(`Should at most receive ${maxPayloads} payloads per batch`);
	await io.logger.info(`Should wait no more than ${maxInterval} seconds between batches`);
	const totalPayloadSize = payload.reduce((sum, p) => sum + JSON.stringify(p).length, 0);
	return `Received ${payload.length} payloads. Total size in bytes: ${totalPayloadSize}`;
	},
	});
	const getLargeString = (bytes: number) => {
	return Array(bytes).fill("F").join("");
	};
	client.defineJob({
	id: "batch-trigger-send",
	name: "Batch Trigger Send",
	version: "1.0.0",
	trigger: eventTrigger({
	name: "batch.trigger.send",
	schema: z.object({
	payloads: z.number().default(12),
	// expect batches to be chunked as they will exceed the server limit
	oversized: z.boolean().default(false),
	}),
	}),
	run: async (payload, io) => {
	for (let i = 0; i < payload.payloads; i++) {
	await io.sendEvent(`send-${i}`, {
	name: "batch.trigger",
	payload: {
	count: i,
	...(payload.oversized
	? {
	largePayload: getLargeString(250 * 1000), // 250KB
	}
	: undefined),
	},
	});
	}
	},
	});

---

Changelog

• Upgrade to v0.16.0-rc.1
• Detect migration errors and point to docs
• Add migration delay to allow graceful shutdown to complete (rolling deploy)
• Insert custom add_batch_job function
• Add batched event triggers and webhook delivery
• Read replica support via optional DATABASE_READ_REPLICA_URL env var

---

Screenshots

💯

[changeset-bot]

🦋 Changeset detected

Latest commit: f130fbd

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[nicktrn]

0.15.0 was released yesterday. The only change being that migration files are now embedded as strings. Just in case of unforeseen bugs, we should stay on 0.14.0-rc.0 for now.

[nicktrn]

This helps decrease payload size for batch jobs

[nicktrn]

We'll be able to simplify this in a lot of places once we migrate existing data and make JobRun.payload non-optional.

[nicktrn]

Takes an array now, but only batches if the dispatcher has batching enabled.

[nicktrn]

Cheating here - timestamp comes from the first event. Timestamps would either be another thing to attach to JobRun, get for each execution, or remove.

Alternatively, we could create many-to-many relations between JobRun and EventRecord. This would be a big change.