fix(launch): overwrite partial job metadata instead of building a new job version

[bcsherma]

Description

• Fixes WB-18438

This PR modifies the core and legacy job builder to never create a new job version when a partial job is used.

Partial jobs are created by the wandb job create or wandb launch -u commands. They include source information but no input or output type info. When a run created from a partial job finishes, the job builder constructs a new, non-partial job version with the input and output info saved into the wandb-job.json file.

Jobs now support having i/o types stored in artifact version metadata, which can be mutated without producing a new job version. Partial jobs now write _partial key to the metadata that can be used to identify partial jobs. If a partial job is detected the job builder will overwrite the artifact metadata with the i/o types.

• I updated CHANGELOG.md, or it's not applicable

Testing

How was this PR tested?

[bcsherma]

• fix(launch): partial git jobs dont require codePath in metadata #7692
• fix(launch): only mark job as partial in artifact version metadata #7652
• fix(launch): overwrite partial job metadata instead of building a new job version #7651 👈
• chore(artifacts): add updateArtifact mutation to wandb-core code gen #7650
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bcsherma and the rest of your teammates on Graphite

[codecov]

Codecov Report

Attention: Patch coverage is 64.70588% with 30 lines in your changes are missing coverage. Please review.

Project coverage is 75.78%. Comparing base (dd2628d) to head (0f5ac92).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7651      +/-   ##
==========================================
+ Coverage   74.26%   75.78%   +1.51%     
==========================================
  Files         500      500              
  Lines       55803    54133    -1670     
==========================================
- Hits        41444    41026     -418     
+ Misses      13946    12691    -1255     
- Partials      413      416       +3     

Flag	Coverage Δ
func	41.34% <66.66%> (+0.06%)	⬆️
system	63.54% <61.17%> (+0.03%)	⬆️
unit	56.20% <44.70%> (+0.91%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
wandb/sdk/launch/create_job.py	75.23% <ø> (ø)
wandb/sdk/internal/sender.py	90.32% <50.00%> (ø)
wandb/sdk/internal/internal_api.py	81.92% <25.00%> (-0.21%)	⬇️
wandb/sdk/internal/job_builder.py	91.26% <81.81%> (+3.36%)	⬆️
core/pkg/server/sender.go	71.04% <10.00%> (+8.45%)	⬆️
core/pkg/launch/job_builder.go	70.51% <69.44%> (+7.86%)	⬆️

... and 104 files with indirect coverage changes

[dmitryduev]

Do you have a (system) test for the added logic?

[KyleGoyette]

The overall logic seems correct to me. However I'd prefer to refactor some of the inherenet job builder logic into job builder methods to keep sender cleaner

[KyleGoyette]

A few comments, but mostly LGTM.

[KyleGoyette]

This LGTM, but codecov is saying there are no tests covering many of the new methods added. Should be pretty straight forward to get tests in tests/pytest_tests/unit_tests/test_job_builder.py

[dmitryduev]

LGTM, just a couple nits