SwiGLU Feed-Forward Block

Task

Implement swiglu_feed_forward(inputs: list[list[float]], w_gate: list[list[float]], w_up: list[list[float]], w_down: list[list[float]]) -> list[list[float]]. Apply a complete bias-free SwiGLU feed-forward block: gate and up projections, stable SiLU gating, elementwise multiplication, and the down projection.

Requirements

• inputs has shape tokens by d_model.
• w_gate and w_up both have shape d_model by d_hidden, while w_down has shape d_hidden by d_model.
• Compute gate = inputs @ w_gate and up = inputs @ w_up independently.
• Apply SiLU elementwise to gate using SiLU(x) = x * sigmoid(x).
• Evaluate sigmoid stably for large positive and negative values without overflowing exp.
• Multiply the activated gate and up projection elementwise.
• Multiply the gated hidden rows by w_down and return shape tokens by d_model.
• Do not add biases, residual connections, dropout, or normalization.
• Preserve an empty input matrix by returning an empty list, and do not mutate any input.

Example

swiglu_feed_forward(
    [[1.0, -1.0]],
    [[1.0, 0.0], [0.0, 1.0]],
    [[1.0, 0.0], [0.0, 1.0]],
    [[1.0, 0.0], [0.0, 1.0]],
)
# [[0.73105858, 0.26894142]]