Implement LoRA from Scratch

Implementationmedium~35 min

Objective

Complete the LoRALayer class so it wraps a frozen linear layer with a trainable low-rank adapter that starts as an identity function.

Background

You're building a parameter-efficient fine-tuning system. Instead of fine-tuning all 768x768 = 590K parameters of a linear layer, LoRA freezes the original weights and adds a low-rank update through two small matrices A (in x rank) and B (rank x out). At initialization, B is zeros so the adapter has no effect. During training, only A and B are updated, reducing trainable parameters by 50-100x. Your task is to implement this from scratch using NumPy.

Requirements

1.Initialize A with small random values (Gaussian/Kaiming) and B with zeros
2.Compute the scaling factor as alpha / rank
3.Implement forward pass: frozen_output + (x @ A @ B) * scaling
4.Implement weight merging: W_frozen + A @ B * scaling
5.Count only the trainable parameters (A and B matrices)

Evaluation (100 points)

A and B matrices initialized

LoRALayer must create self.A (random) and self.B (zeros)

25pt

Scaling factor computed

scaling = alpha / rank

20pt

Forward pass implemented

Combines frozen base output with low-rank LoRA path

25pt

Weight merging implemented

get_merged_weight returns W_frozen + A @ B * scaling

15pt

Trainable parameter count

count_trainable_params returns size of A + size of B

15pt

Hints

Select a file to start editing

Terminal

AI Assistant50K tokens left

Ask me about the code, bugs, or concepts.
I'll guide you in plain English, no code output.
Budget: 50K tokens per lab