services.tabbyapi.settings.memory.cuda_malloc_async
NixOS option
Use the cudaMallocAsync allocator backend in Torch. When false, the allocator is left to the environment: unless PYTORCH_CUDA_ALLOC_CONF is set, ExLlamaV3 enables expandable segments in Torch’s native allocator, which performs better than cudaMallocAsync. Enable this to force the cudaMallocAsync backend instead.
type: booleanDefault
declared in: nixos/modules/services/web-apps/tabbyapi.nixView source on NixOS/nixpkgs →false