Version 5.3.0

What's Changed

This new minor version addresses an issue raised by @nalepae in #17, related to differences between complete and minimal implementations.

The previous v5.2.1 release had mistakenly assumed a minimal implementation in some places, by setting should_succeed: false in some cases where a complete database would be capable of signing. This caused erroneous failures on complete implementations.

To address this with the least disruption to existing test runners, we have kept the values of should_succeed to mean "can a minimal implementation sign this?", while adding a new field should_succeed_complete to carry the information about whether a complete implementation should be capable of signing.

Therefore the changes required by maintainers of minimal implementations are:

• Update deserialisation to allow for the existence of the should_succeed_complete field. It should be otherwise ignored.
• Update the test runner to forbid false positives. With the new field it should no longer be necessary to fail signing when should_succeed is true but signing is blocked by the watermark. These cases are now covered by should_succeed=false, should_succeed_complete=true.
• Report any failures or issues in case further adjustments are required to accommodate all implementations.

The changes required by maintainers of complete implementations are similar, but should use the should_succeed_complete field.

For full semantics please see the README: https://github.com/eth-clients/slashing-protection-interchange-tests#how-to-run.

For more detail of the rationale for the changes see #17 & #18.

Full Changelist

• Updating URL for eth-clients by @fredriksvantes in #13
• Clarify that signed messages should be remembered by @michaelsproul in #14
• Update tests with should_succeed_complete by @michaelsproul in #18

New Contributors

• @fredriksvantes made their first contribution in #13

Full Changelog: v5.2.1...v5.3.0

Version 5.2.1 - Merry Christmas

What's Changed

• Add a test case with multiple blocks out of order by @michaelsproul in #12

Full Changelog: v5.2.0...v5.2.1

Version 5.2.0

Add some more tests involving multiple interchange files. Please see #10 for details + diff.

No breaking changes to the test format.

Version 5.1.0

This release includes expanded testing of slashable data imports. There is one breaking change to the test format: the replacement of allow_partial_import with contains_slashable_data.

The semantics for contains_slashable_data are described in README.md, and included below for clarity:

• If should_succeed is false, then contains_slashable_data is irrelevant
• If contains_slashable_data is false, then the given interchange must be imported
  successfully, and the given block/attestation checks must pass.
• If contains_slashable_data is true, then implementations have the option to do one of two
  things:
  • Import the interchange successfully, working around the slashable data by minification
    or some other mechanism. If the import succeeds, all checks must pass and the test
    should continue to the next step.
  • Reject the interchange (or partially import it), in which case the block/attestation
    checks and all future steps should be ignored.

For more rationale behind the change, please see the v5.1.0 alpha release notes: https://github.com/eth2-clients/slashing-protection-interchange-tests/releases/tag/v5.1.0-alpha.1

Expand testing of slashable inputs

This is a draft release seeking feedback on some proposed changes to the semantics of the tests.

The main change is to the meaning of allow_partial_import. I no longer think that a boolean for indicating "partial import" is useful, because it doesn't convey enough information. Consider the following example, from duplicate_pubkey_slashable_attestation.json:

• There is a single interchange containing two records for the same pubkey. One record contains a single attestation 0=>3, and the other contains an attestation 1=>2.

If we assume a complete-style import (e.g. Lighthouse/Prysm), then only one of these records can be imported, because the two attestations are mutually slashable. If we then want to test whether an attestation like 0=>4 is safe to sign, then depending on which attestation was imported, it may not be correctly identified as unsafe. I.e. we've gotten into a bizarre state in which the meaning of the test is no longer well-defined (and no sensible interpretation exists).

The reason to care about cases like the one above is that they are handled just fine by "minifying" imports (e.g. Teku). A minifying import will shrink the two attestations into one attestation 1=>3, against which it can safely identify 0=>4 as unsafe.

In light of Lighthouse being slow to import large interchanges, and choking on seemingly-slashable data, I've implemented minification by default for Lighthouse. In the process of testing it, I started to expand the standard tests and found the issue described above.

Proposed Change

Rename allow_partial_import to contains_slashable_data with the following meaning:

• If should_succeed is false, then contains_slashable_data is irrelevant
• If contains_slashable_data is false, then the given interchange must be imported
  successfully, and the given block/attestation checks must pass.
• If contains_slashable_data is true, then implementations have the option to do one of two
  things:
  • Import the interchange successfully, working around the slashable data by minification
    or some other mechanism. If the import succeeds, all checks must pass and the test
    should continue to the next step.
  • Reject the interchange (or partially import it), in which case the block/attestation
    checks and all future steps should be ignored.

I think this change allows us to expand testing coverage for minifying clients (Teku & new Lighthouse), while not imposing anything odd on other implementations. I've made the above change to Lighthouse's test runner, and it is able to pass the tests both with and without minification. Note that I haven't actually done the field rename yet, it is still called allow_partial_import, but should be interpreted as above.

Separately, I've also made the decision to stop supporting partial imports in Lighthouse. It will have two modes of operation:

• Minify (default): flexible, allow import of slashable data, similar to Teku.
• Don't minify: strict, requires the entire input to be valid, will revert atomically if
  slashable data is found.

The logic for removing partial imports is that they're dangerous and provide a false sense of security (per above). Further, users likely want to see either "success" or "failure", and do not want to sift through logs to work out whether their specific botched partial import was safe.

Version 5.0.0

This marks the first tagged release of test cases for EIP-3076. The version number is expected to follow the interchange_format_version, with 5.x.x tests corresponding to the version 5 format. No further breaking changes are expected before EIP finalisation, so future releases will likely be v5.x.x.

The attached tarball is suitable for downloading and extracting in CI. It contains just the tests directory from this repository.