The Vibe Father
AWS has published two concrete deployment paths for Kimi K3. SageMaker HyperPod with its Inference Operator, or a self-managed Amazon EKS cluster. Both paths use a p6-b300.48xlarge instance with eight NVIDIA B300 Blackwell Ultra GPUs, the model’s MXFP4 weights, and a Kimi-specific vLLM container.
This answers a different question from whether Kimi K3 has open weights or can run locally. It explains the infrastructure an organization needs to serve the full model. In a U.S. Google Trends snapshot taken August 3, Kimi K3 held the strongest average interest among five newly announced AI products, while related queries for vLLM and SGLang rose. Trends values are relative, dynamic, and not raw search volume.
Why Kimi K3 needs this much hardware
Kimi K3 is a 2.8-trillion-parameter mixture-of-experts model. It contains 896 experts and activates 16 per token, leaving roughly 104 billion parameters active during a forward pass. That sparse activation improves compute efficiency, but the expert weights still need to be available across the serving system.
| Attribute | AWS deployment value |
|---|---|
| Total parameters | 2.8 trillion |
| Active per token | About 104 billion |
| Experts | 896 total, 16 active per token |
| Context window | 1 million tokens |
| Weights | MXFP4 from moonshotai/Kimi-K3 |
| GPU node | p6-b300.48xlarge with 8 B300 GPUs |
HyperPod, the managed-operations path
SageMaker HyperPod uses EKS orchestration underneath, but its Inference Operator manages model download, container scheduling, health checks, and endpoint readiness. AWS’s manifest requests all eight GPUs, sets tensor parallelism to eight, enables prefix caching and automatic tool choice, and exposes an OpenAI-compatible chat-completions endpoint.
Capacity comes through a Flexible Training Plan. That committed reservation is the tradeoff for a more streamlined deployment experience, teams get the instance availability and more managed lifecycle behavior, but should plan capacity and duration before the rollout.
EKS, the operator-control path
The standalone EKS route uses EC2 Capacity Blocks for p6-b300 capacity. AWS’s AI on EKS recipe provisions networking and node groups, installs NVIDIA drivers and the device plugin, deploys the vLLM server, and exposes it through a LoadBalancer or ingress on port 8000.
Teams can point vLLM at Hugging Face or copy the weights to Amazon S3 for faster model loading. They also own more of the operational surface. Terraform, Helm or manifests, GPU scheduling, ingress, upgrades, health behavior, and cleanup.
| Decision | HyperPod | Standalone EKS |
|---|---|---|
| Capacity mechanism | Flexible Training Plan | EC2 Capacity Block |
| Endpoint lifecycle | Inference Operator manages more | Platform team manages it |
| Customization | Opinionated manifest path | Full Kubernetes control |
| Best fit | Teams prioritizing managed inference | Teams with mature GPU Kubernetes operations |
The serving stack is day-zero software
At publication time, Kimi K3 support lived in the vllm/vllm-openai:kimi-k3 container, with changes expected to merge into the main vLLM image later. That detail matters for production risk. Pin the tested image digest, validate tool calls and reasoning output, and plan an upgrade path rather than tracking a floating tag.
The resulting endpoint is OpenAI-compatible, so existing clients can change the base URL and model name. Compatibility at the HTTP layer does not prove behavioral compatibility. Structured outputs, tool schemas, reasoning modes, token accounting, and error handling still need integration tests.
A deployment checklist
- Reserve p6-b300 capacity in the correct Availability Zone.
- Pin model weights, vLLM image, and serving arguments.
- Keep the endpoint private and add authentication, the AWS sample uses a placeholder API key.
- Benchmark latency, throughput, and context length with your actual workload.
- Test tool calling and structured output before routing agents to it.
- Monitor GPU health, model loading, queue depth, and failed generations.
- Delete clusters and release reservations when an evaluation ends.
Bottom line
Kimi K3 is open-weight, but full-model self-hosting is still a large infrastructure project. HyperPod is the shorter path for teams that want AWS to manage more of inference lifecycle. Standalone EKS is appropriate when a platform team already operates GPU Kubernetes and values control. In both cases, eight B300 GPUs and reserved capacity make this an enterprise deployment—not a casual local experiment.
