Optimize Slow Inference Pipeline

Optimizationhard~45 min

Objective

Reduce per-request latency from ~250ms to under 100ms through batching, caching, and query optimization.

Background

Your team's inference API is too slow for production. Profiling shows three bottlenecks: each request runs model inference individually (no batching), repeated identical inputs aren't cached, and the preprocessing step makes one database call per feature instead of batching them. Fix all three to hit the latency target.

Requirements

1.Add request batching to process multiple inputs in a single model forward pass
2.Implement a cache for repeated inputs to avoid redundant inference
3.Fix the N+1 query pattern in preprocessing to use a single batch query

Evaluation (100 points)

Batch inference implemented

predict_batch should use model.predict_batch instead of looping over predict_single

30pt

Cache is functional

InferenceCache should store and retrieve cached predictions

25pt

Cache is used in predict()

The predict method should check cache before running inference

10pt

N+1 query pattern fixed

preprocess_features should batch feature lookups instead of one-at-a-time

25pt

Server structure supports low latency

Overall server architecture should support <200ms latency

10pt

Hints

Select a file to start editing

Terminal

AI Assistant50K tokens left

Ask me about the code, bugs, or concepts.
I'll guide you in plain English, no code output.
Budget: 50K tokens per lab