Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: cli,events: speed up backfill with temporary index #11953

Merged
merged 2 commits into from
Jun 5, 2024

Conversation

rvagg
Copy link
Member

@rvagg rvagg commented May 2, 2024

From experience running this:

  1. Add a warning about running this against a node that's actively collecting events, you'll have trouble
  2. Use IMMEDIATE transactions so we know as soon as we start a transaction whether we can get an exclusive lock for a write or not so we can error more easily.
  3. Add an optional temporary index for our query which significantly speeds up this process (a lot).
  4. Run an optional vacuum after we're done.

Copy link
Contributor

@ZenGround0 ZenGround0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some suggestions that are probably not feasible or worth the hassle given that this is a shed command.

Unfortunately I'm web3 native so SQL is all magic to me but I trust that what you're doing is fine.

chain/events/filter/index.go Show resolved Hide resolved
cmd/lotus-shed/indexes.go Show resolved Hide resolved
@rvagg rvagg force-pushed the rvagg/backfill-improvements branch from 27402a8 to 6e571ab Compare May 6, 2024 03:12
@rvagg rvagg force-pushed the rvagg/backfill-improvements branch from 6e571ab to 1c6330b Compare June 5, 2024 03:44
Copy link

github-actions bot commented Jun 5, 2024

All checks have passed

@rvagg rvagg merged commit 423d8a7 into master Jun 5, 2024
94 checks passed
@rvagg rvagg deleted the rvagg/backfill-improvements branch June 5, 2024 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants