Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

re-think the results output #55

Open
pygy opened this issue Jan 26, 2023 · 0 comments
Open

re-think the results output #55

pygy opened this issue Jan 26, 2023 · 0 comments

Comments

@pygy
Copy link
Member

pygy commented Jan 26, 2023

We currently shove every kind of problems into the results array as assertions, even timeouts and bail outs.

The latter two should be tagged as such IMO.

Also, we shouldn't format the output on error, but in the reporter (with a possible exception for custom assertions).

Here's the current format:

{
	pass: null,
	message: "Incomplete assertion in the test definition starting at...",
	error: $$_testOrHook.error,
	task: $$_testOrHook,
	timeoutLimbo: $$_timedOutAndPendingResolution === 0,
	// Deprecated
	context: ($$_timedOutAndPendingResolution === 0 ? "" : "??? ") + $$_testOrHook.context,
	testError: $$_testOrHook.error
}

We should aim for something like this:

{
	kind: "assertion" | "error" | "timeout",
        pass: boolean,
	message?: {value, methodName, reference} | string
        error?: Error // or maybe just a stack trace?
	task: Task
}

This is a prerequisite for #31.

The "incomplete assertion" checks should run on task finalization time, and on timeout, and cause a hard error to be thrown.

Likewise, syntax errors while parsing a test file should cause hard errors, not bailouts.

At long last, it would be useful to be able to bulk-add results, such that we can parallelize running the tests and merge results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant