Enhanced GPU discovery and multi-gpu support with concurrency #4517
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dhiltgen
force-pushed
the
gpu_incremental
branch
4 times, most recently
from
05ba1ca
to
91be1fa
Compare
dhiltgen
force-pushed
the
gpu_incremental
branch
4 times, most recently
from
ecde7d9
to
d788717
Compare
dhiltgen
force-pushed
the
gpu_incremental
branch
2 times, most recently
from
f02b076
to
076450a
Compare
dhiltgen
force-pushed
the
gpu_incremental
branch
3 times, most recently
from
12b47a0
to
bfbb50e
Compare
The amdgpu drivers free VRAM reporting omits some other apps, so leverage the upstream DRM driver which keeps better tabs on things
Now that we call the GPU discovery routines many times to update memory, this splits initial discovery from free memory updating.
This worked remotely but wound up trying to spawn multiple servers locally which doesn't work
Still not complete, needs some refinement to our prediction to understand the discrete GPUs available space so we can see how many layers fit in each one since we can't split one layer across multiple GPUs we can't treat free space as one logical block
Our default behavior today is to try to fit into a single GPU if possible. Some users would prefer the old behavior of always spreading across multiple GPUs even if the model can fit into one. This exposes that tunable behavior.
adjust timing on some tests so they don't timeout on small/slow GPUs
This was referenced Jun 1, 2024
jmorganca
reviewed
Jun 2, 2024
|
|
||
| switch runtime.GOOS { | ||
| case "windows": | ||
| oneapiMgmtName = "ze_intel_gpu64.dll" |
There was a problem hiding this comment.
This DLL gets installed on Windows with Intel iGPUs as part of the OS base install and doesn't always open reliably – it seems to be causing some crashes on both Win10 and Win11 and so we may want to put this behind a flag until we resolve those issues
jmorganca
reviewed
Jun 2, 2024
|
|
||
| type RocmGPUInfo struct { | ||
| GpuInfo | ||
| usedFilepath string // nolint: unused |
There was a problem hiding this comment.
Suggested change
| usedFilepath string // nolint: unused | |
| usedFilepath string |
I believe this is used now below
jmorganca
reviewed
Jun 2, 2024
| type RocmGPUInfo struct { | ||
| GpuInfo | ||
| usedFilepath string // nolint: unused | ||
| index int // nolint: unused |
There was a problem hiding this comment.
Suggested change
| index int // nolint: unused | |
| index int |
jmorganca
reviewed
Jun 2, 2024
| @@ -232,6 +228,10 @@ func NewLlamaServer(gpus gpu.GpuInfoList, model string, ggml *GGML, adapters, pr | |||
|
|
|||
| params = append(params, "--parallel", fmt.Sprintf("%d", numParallel)) | |||
|
|
|||
| if estimate.TensorSplit != "" { | |||
| params = append(params, "--tensor-split", estimate.TensorSplit) | |||
There was a problem hiding this comment.
This is super cool! Can't wait to try it more on 2x, 4x and 8x gpu systems
jmorganca
reviewed
Jun 2, 2024
| #ifndef GGML_USE_CUBLAS | ||
| fprintf(stderr, "warning: llama.cpp was compiled without cuBLAS. Setting the split mode has no effect.\n"); | ||
| #endif // GGML_USE_CUBLAS | ||
| #ifndef GGML_USE_CUDA |
jmorganca
reviewed
Jun 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Carries (and obsoletes if we move this one forward first) #4266 and #4441
This refines our GPU discovery to split it into bootstrapping where we discover information about the GPUs once at startup, and then incrementally refresh just free space information, instead of fully rediscovering the GPUs over and over.
Fixes #3158
Fixes #4198
Fixes #3765