Implementationmedium~35 min
Objective
Implement a DynamicBatcher that collects requests, flushes on batch_size or timeout, and wire it into a prediction endpoint with health checks.
Background
Your team is deploying a sentiment classifier behind a FastAPI server. Single-request inference wastes GPU cycles: the model processes one input at a time while the GPU can handle 8-32 in parallel. You need a request batcher that collects incoming requests and flushes them as a batch when either the batch is full or a timeout expires. This amortizes the fixed inference overhead across all requests in the batch.
Requirements
- 1.Initialize the DynamicBatcher with a buffer/queue to hold pending requests
- 2.Implement submit() to create a Future for each request and trigger flush when batch is full
- 3.Implement _flush() to drain the buffer, run model.predict(), and resolve each Future with its result
- 4.Implement the prediction endpoint that converts input to embedding, submits to batcher, and returns results with latency
- 5.Implement a health check that reports model status and configuration
Evaluation (100 points)
Batcher buffer initialized
DynamicBatcher.__init__ creates a buffer/queue for pending requests
Submit creates Future and buffers request
submit() creates a PendingRequest with Future, adds to buffer, and awaits result
Flush processes batch and resolves Futures
_flush() calls model.predict with collected inputs and sets Future results
Prediction endpoint submits to batcher
predict_endpoint calls batcher.submit and returns dict with latency
Health check returns status
health_check returns dict with status, model_loaded, and batch_size