Training · Containers and Kubernetes
The analogy. Shipping containers. The box is a standard size, the crane does not care what is inside, and the port decides where each box goes. A Service is the berth number painted on the dock.
From image to replica: the lecture behind this lab covers the mental model, the commands you are about to use, and the four things that usually go wrong.
Read the lecture before you startAI Platform: Serve Llama 3 with vLLM
Stand up an OpenAI-compatible inference server: the Hugging Face token in a Secret, the model selected with a vLLM argument, and a Service on 8000. Graded against the cluster state - the model itself is not downloaded here.
Activities
01Put the HF token in a Secret
A Secret named hf-token must hold a token key.
02Run vllm/vllm-openai
A Deployment named llama3-serve must run the vLLM OpenAI-compatible image.
03Select the model with an argument
The container must pass --model= with the Llama 3 8B model id (meta-llama/...). That is how vLLM is told what to serve.
04Serve it on 8000
The llama3-serve Deployment must be ready and a Service named vllm must route 8000.