Rename DefaultTarget to Target
#1674
Conversation
Signed-off-by: Mattt Zmuda <mattt@replicate.com>
|
max concurrency can't overriden in web though, cog currently reads it from the cog.yaml file inside the container. I could restore the COG_CONCURRENCY_OVERRIDE environment variable (currently removed) and plumb setting it through web and cluster, but that's not currently there and it's not obvious that would be good (it certainly can't be increased past the maximum with which the engine was built). on the other hand, target is purely a consideration for the autoscaler and doesn't need to be read from inside the container |
|
How about the value sent to director from cluster? I can imagine situations where it'd be nice to lower an ambitious max value for a model. For example, when running on a different hardware. |
|
there are some occasional models that could potentially run on different hardware (not triton), but what value does director send to cluster...? |
Follow-up to #1672
Both max and target concurrency levels can be overridden by web, so in effect both are default values.