ci: add gomu gomu no gatling reports #1501
Conversation
isavov
force-pushed
the
gomu-gomu-ci-reports
branch
from
e05a1f1
to
88c30b8
Compare
|
@apoorvsadana what is our current situation regarding gomugomu? Is this PR still relevant? |
|
Ya it's relevant. The last PR created by Limechain on gomu gomu has to be merged. There was a bug before rebase in the PR which is solved now. So need to review it now, once we merge that, we can re run this and ensure everything works fine and finally merge |
| - name: Process results | ||
| run: | | ||
| set -v -x +e | ||
| cp ../gomu-gomu-no-gatling/report.json ./ | ||
| #### Extract TPS, UOPS, Extrinsics and Steps from the report #### | ||
| jq ' | ||
| .benches[] |= (.name as $benchName | .metrics[] |= (.name = .name + " ("+ $benchName + ")")) # Append benchmark name to each metric name | ||
| | .benches[].metrics[].extra = .extra # set extra field for each metric | ||
| |.benches # extract only benches | ||
| |map(.metrics)|add # merge metrics from different benches | ||
| |map(select(.name | test("Average TPS|Average UOPS|Average Extrinsics|Average Steps"))) #filter only the metrics we want | ||
| ' report.json > processed_report.json | ||
| - name: Check for failed requests | ||
| run: | | ||
| set -v -x +e | ||
| #### Extract Failed from the report #### | ||
| jq ' | ||
| .benches[] |= (.name as $benchName | .metrics[] |= (.name = .name + " ("+ $benchName + ")")) # Append benchmark name to each metric name | ||
| | .benches[].metrics[].extra = .extra # set extra field for each metric | ||
| |.benches # extract only benches | ||
| |map(.metrics)|add # merge metrics from different benches | ||
| |map(select(.name | test("Failed"))) # filter only the metrics we want | ||
| |any(.[]; .value > 0) | if . then halt_error(1) else {} end # fail if there are failed requests | ||
| ' report.json |
There was a problem hiding this comment.
I have no idea what I'm reading.
Is it some standard thing you found online, or do you write this yourself all custom?
There was a problem hiding this comment.
no problems here, it's just jq processing. I didn't tested them, so don't know if it's the desired output
There was a problem hiding this comment.
Unfortunately I had to write it custom, since the benchmark processing action doesn't support our output format. I've commented it extensively in the hopes of making it a bit more readable.
There was a problem hiding this comment.
Could we add a gh-compatible output format to gomu-gomu?
|
maybe @d-roak and @apoorvsadana will do a better job reviewing this than me |
| - name: Process results | ||
| run: | | ||
| set -v -x +e | ||
| cp ../gomu-gomu-no-gatling/report.json ./ | ||
| #### Extract TPS, UOPS, Extrinsics and Steps from the report #### | ||
| jq ' | ||
| .benches[] |= (.name as $benchName | .metrics[] |= (.name = .name + " ("+ $benchName + ")")) # Append benchmark name to each metric name | ||
| | .benches[].metrics[].extra = .extra # set extra field for each metric | ||
| |.benches # extract only benches | ||
| |map(.metrics)|add # merge metrics from different benches | ||
| |map(select(.name | test("Average TPS|Average UOPS|Average Extrinsics|Average Steps"))) #filter only the metrics we want | ||
| ' report.json > processed_report.json | ||
| - name: Check for failed requests | ||
| run: | | ||
| set -v -x +e | ||
| #### Extract Failed from the report #### | ||
| jq ' | ||
| .benches[] |= (.name as $benchName | .metrics[] |= (.name = .name + " ("+ $benchName + ")")) # Append benchmark name to each metric name | ||
| | .benches[].metrics[].extra = .extra # set extra field for each metric | ||
| |.benches # extract only benches | ||
| |map(.metrics)|add # merge metrics from different benches | ||
| |map(select(.name | test("Failed"))) # filter only the metrics we want | ||
| |any(.[]; .value > 0) | if . then halt_error(1) else {} end # fail if there are failed requests | ||
| ' report.json |
There was a problem hiding this comment.
no problems here, it's just jq processing. I didn't tested them, so don't know if it's the desired output
isavov
force-pushed
the
gomu-gomu-ci-reports
branch
from
19d1786
to
c4958ef
Compare
|
https://github.com/keep-starknet-strange/madara/actions/runs/9062710154/job/24897705455?pr=1501 The CI seems to be stale @isavov |
|
@isavov still red 🥲 |
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
Signed-off-by: Iliya Savov <isavov@users.noreply.github.com>
isavov
force-pushed
the
gomu-gomu-ci-reports
branch
from
b3e05b7
to
89d6467
Compare
|
@isavov the gomu run fails |
Yes, it appears the current version of gomu-gomu is not compatible with madara's current version. Will try to debug further in the next week. |
Pull Request type
What is the current behavior?
Currently gomu gomu benchmark is run, but the results are just output to console.
Resolves: #NA
What is the new behavior?
Benchmark results are processed and compared with previous runs. Regression in performance of over 20% causes the pipeline to fail and any failed requests to madara during the benchmark cause the pipeline to fail too.
Does this introduce a breaking change?
No
Other information