Daniel Hiltgen 517807cdf2 perf: build graph for next batch async to keep GPU busy (#11863) ...

* perf: build graph for next batch in parallel to keep GPU busy

This refactors the main run loop of the ollama runner to perform the main GPU
intensive tasks (Compute+Floats) in a go routine so we can prepare the next
batch in parallel to reduce the amount of time the GPU stalls waiting for the
next batch of work.

* tests: tune integration tests for ollama engine

This tunes the integration tests to focus more on models supported
by the new engine.

2025-08-29 14:20:28 -07:00

..

ggml

perf: build graph for next batch async to keep GPU busy (#11863)

2025-08-29 14:20:28 -07:00

backend.go

next ollama runner (#7913)

2025-02-13 16:31:21 -08:00