llm-cold-start-benchmark

简介

LLM Container Cold-Start Benchmark Measurements of how long it takes to bring a language model from cold storage to a state where it can serve its first token, across 25 open-weight models spanning 17 architecture families and 100.9 GiB of checkpoints, on a single NVIDIA T4. Cold start is the latency a serverless or scale-to-zero inference platform pays when it has no warm replica. It decomposes into weight transfer from storage, deserialization into host memory, transfer to the… See the full description on the dataset page: https://huggingface.co/datasets/ArchCoder/llm-cold-start-benchmark.