Search and recommendation are one big matmul.
Strip a search engine or a recommender down to its hot loop and one operation is left: score a batch of query embeddings against a corpus of item embeddings, then keep the best few per query. That scoring step is a single matrix multiply, and a consumer RTX 3080 runs it at half its available rate by default. Flipping one accumulator setting recovers the other half. Every number below was measured on that one GPU.
Written to be read cold. Each term, tool, and technique is explained where it comes up, with the mechanics behind the expandable panels and a glossary at the end.
The hardware. One NVIDIA RTX 3080 (10 GB), a consumer GeForce card on the Ampere architecture (compute capability sm_86). Consumer Ampere runs the tensor-core matmul with a 16-bit (fp16) accumulator at full rate and a 32-bit (fp32) accumulator at half rate. That split is the lever this whole page turns.
The change. A single PyTorch toggle, torch.backends.cuda.matmul.allow_fp16_accumulation = True. It is a cuBLAS setting, not a custom kernel, so there is nothing to build. Measured on torch 2.12 with CUDA 13; the toggle needs PyTorch 2.7 or newer.
01 the hot loop
Why retrieval and ranking reduce to a matmul
Retrieval scoring is the step where a search engine, given your query, rates how well every candidate document matches it, so the best can be returned. A recommender does the same with a user in place of the query and items in place of documents. Both represent the query, user, and every item as an embedding.
An embedding is a list of a few hundred numbers (a vector) that a trained model assigns to a piece of content, placed so that related things land near each other. A query and a matching document get similar vectors; the match score is how aligned two vectors are, measured as their dot product (multiply the numbers position by position, add them up). A bigger dot product means a closer match. The dimension d is how many numbers each embedding holds; here d = 512, mid-range for real models (384–1536).
Scoring one query against the whole corpus is one dot product per item. Scoring a batch of n_q queries against all n_c items is every query against every item, which is exactly a matrix multiply. Stack the query embeddings into a matrix Q and the item embeddings into a matrix C, and the full score table is:
[n_q × d] [d × n_c] → S is [n_q × n_c], every query-item score n_q = 8,192 queries · d = 512 · n_c = 2,097,152 items (~2.1M). Then take the top-k highest scores in each row.
Top-k is the last step: from each query's row of 2.1M scores, keep the k highest (here k = 10). Those ten items are what the user sees. The score table itself is the matrix multiply; top-k is a separate selection over the numbers it produces, and that difference matters for the measured speedup later.
Brute force, two-tower recommenders, and ANN re-ranking are all this same matmul
The benchmark scores all 2.1M items for every query, so nothing relevant can be missed. This is brute-force maximum-inner-product search (MIPS), the exact-answer baseline. The 2.1M item embeddings do not fit in one allocation comfortably, so the corpus is streamed in tiles of 65,536 items and each tile scored in turn, which keeps memory flat while the matmul runs at full size.
A two-tower recommender is the same shape: one tower turns the user into an embedding, another turns each item into an embedding, and serving a recommendation scores the user vector against the item corpus. Large systems first use an approximate nearest-neighbor (ANN) index to cut the corpus to a few thousand candidates, then re-rank those candidates with the exact dot product. That re-rank is the same Q @ Cᵀ on a smaller C. Whichever path you take, the scoring arithmetic is a general matrix multiply (a GEMM), and at realistic embedding dimensions the GEMM is where the time goes.
Why the scoring step is compute-bound, and top-k is not
A GPU operation is either compute-bound (limited by how fast the math units run) or memory-bound (limited by how fast data moves from memory). The scoring matmul reads each embedding once and does d multiply-adds against every item, so it does a lot of arithmetic per byte read. At d = 512 that ratio is high enough to keep the math units busy, which is what makes it compute-bound and what makes a faster math path help.
Top-k is the opposite. It scans the score table and tracks the ten largest values per row, which is comparisons and bookkeeping over a large array with almost no arithmetic. It runs on the general-purpose cores, not the matmul units, so a faster matmul path does nothing for it. The two stages respond to different levers, which is why the results section reports them separately.
02 the lever
Full-rate fp16 accumulation
The scoring matmul runs on the GPU's tensor cores, dedicated units that multiply small blocks of a matrix in one shot. As they multiply, they add the partial products into a running total called the accumulator. The accumulator can keep that total in 32-bit (fp32) or 16-bit (fp16) precision, and on this GPU the choice sets the speed.
On consumer Ampere (sm_86, the RTX 3080 among them) the tensor cores run fp16-input matmuls at full rate when the accumulator is also fp16, and at half rate when the accumulator is fp32. The data-center Ampere parts carry no such cap. PyTorch and cuBLAS default to fp32 accumulation for safety, which on this consumer card leaves roughly a factor of two on the table. The hardware ceiling for flipping to fp16 accumulation is about 2×; what a real scoring workload gets is measured below.
Tensor cores, and why the accumulator precision changes the rate
A tensor core executes a small matrix multiply-and-accumulate (the mma instruction): it multiplies a tile of one matrix by a tile of another and adds the result into an accumulator tile, many multiply-adds per clock. A big matmul is thousands of these tiles summed together. The inputs here are 16-bit; the question is what precision holds the running sum.
Keeping the sum in 32 bits is safer against rounding but, on consumer Ampere, the fp32-accumulate mma is clocked at half the throughput of the fp16-accumulate one. This is a limit on the GeForce parts, one of the ways they are separated from the data-center cards; the instruction set exposes both paths and the fp16-accumulate path is the fast one. The toggle allow_fp16_accumulation tells cuBLAS to pick it.
Why fp16 accumulation is safe for embedding scores
Summing thousands of products in 16 bits loses low-order precision; each score comes out off by roughly 1e-3 relative to the fp32-accumulate value. That would be a problem if the scores were exact physical quantities. They are not. Embeddings are learned and approximate: the model already places items with far more than 1e-3 of slack, and two items whose scores sit within 1e-3 of each other were interchangeable to begin with.
What matters is whether that error reorders the ranking, since the ranking is all the user sees. That is a measurable question, answered in the quality section by recall@k. The toggle is a cuBLAS setting flipped on around the scoring matmul and flipped back after, with no change to the model or the embeddings.
This full-rate fp16-accumulate path is the same lever measured in nerf-frontier and physics-frontier. Here it is applied to dense retrieval and ranking, and stress-tested against the conjecture that it would also help sparse workloads (section 05).
03 measured · dense scoring
The scoring matmul gets the full lever
Brute-force scoring over the 2.1M-item corpus, d = 512, 8,192 queries, streamed in tiles of 65,536. Two numbers, because they answer different questions. First the scoring matmul alone (Q @ Cᵀ, no top-k), which is exactly what the accumulator setting touches. Higher throughput is faster; the tail is wall-clock time for the full corpus pass.
scoring throughput · recall@10 0.987
End to end, with top-k
The full retrieval also selects the top-10 per query. That selection is not a matmul and is not accelerated by the accumulator, so it dilutes the lever. The tail is wall-clock time for the whole scoring-plus-select pass.
End to end lands at 1.25×, below the 1.59× of the scoring stage, because the top-k merge runs at the same speed either way and takes a fixed share of the time. Production retrieval fuses the top-k selection into the scoring kernel; the more of the pipeline that is the matmul, the closer the end-to-end number moves back toward the scoring lever.
04 quality · recall@k
Does the ranking survive the switch?
The speedup is worth nothing if it returns different results. The metric for that is recall@k: of the k items a query should return, how many actually appear in the returned top-k. A recall@10 of 1.000 means the fast path returns the same ten items; 0.900 means one of the ten is displaced on average.
Recall needs a reference ranking to score against. Here it is the fp32-accumulate ranking of the exact same fp16 embeddings. Nothing changes between the two runs except the accumulator precision, so recall@k isolates the effect of the accumulator alone rather than mixing in any other approximation. Measured this way, recall@10 is 0.9872: on average 9.87 of every query's true top-10 survive the switch to fp16 accumulation.
Why a 1e-3 score change barely moves recall
Ranking only changes when two items swap order, and two items swap only if the score error is larger than the gap between them. Across a 2.1M-item corpus the scores near the top-10 boundary are usually separated by much more than 1e-3, so a 1e-3 perturbation leaves the order intact almost everywhere. The rare flips happen between items whose scores were within 1e-3 of each other, meaning a near-tie whose two orderings are equally defensible.
This is why the accumulator lever is a fit for retrieval specifically. The output is a ranking, not the scores themselves, and rankings absorb small score noise. A workload that consumed the exact score values would not tolerate the same error.
Scoring is the matmul
The scoring stage takes the full 1.59× lever at recall@10 0.987. The more of the pipeline that is the matmul, the closer end-to-end gets to it.
Top-k dilutes, not disqualifies
The top-k merge is not a matmul, so end-to-end is 1.25×. Fusing top-k into the scoring kernel recovers the gap.
The change is near-free
One cuBLAS toggle, no kernel to build. The ~1e-3 score noise leaves ranking almost untouched.
Wider embeddings gain more
Speedup grows with d as arithmetic per byte rises. At d=512, mid-range for real models, the scoring stage already hits 1.59×.
05 negative result · the sparse hypothesis
The sparse-matrix conjecture, tested
A fair question is whether this lever also speeds up the sparse side of search, the part that works with matrices that are mostly zeros. It does not, and the repo measures why rather than asserting it. This section reports that negative result plainly.
A sparse matrix is one where almost every entry is zero, so it is stored as just the nonzero positions and values. Lexical search (matching words), co-occurrence tables, and graph adjacency are all sparse: each row has a handful of nonzeros out of millions. The two operations that matter are SpMM (a sparse matrix times a dense one) and SpGEMM (sparse times sparse). The test below runs both at 1% density, alongside a dense tensor-core matmul for reference.
Throughput on the same RTX 3080, higher is faster. The dense tensor-core matmul is the ceiling the lever reaches; the unstructured sparse path is measured against it.
SpMM below the dense tensor-core ceiling
Why unstructured sparse is memory-bound and skips the tensor cores
The tensor cores multiply dense tiles. A sparse matrix has no dense tiles to feed them; its nonzeros scatter unpredictably, so the work becomes chasing indices and gathering scattered values from memory, with little arithmetic between the loads. That makes SpMM and SpGEMM memory-bound, and the library that runs them, cuSPARSE, dispatches to the general-purpose CUDA cores, not the tensor cores. Since the fp16-accumulate lever is a tensor-core setting, it cannot apply to code that never reaches a tensor core.
The one place sparsity and tensor cores meet on Ampere is 2:4 structured sparsity: a rigid pattern of exactly two nonzeros in every contiguous group of four along the contracting dimension, which the hardware can feed to the tensor cores through CUTLASS. Measured here it comes out at 1.04× over dense, marginal for this shape on a 3080. The general lesson holds: the lever's home is the dense stage, not the sparse one.
Where each stage belongs
The result points at a clean split. Sparse candidate generation and dense re-ranking want different hardware paths, so a serving pipeline runs them in sequence.
Sparse candidate generation
Lexical matching or an ANN index cuts millions of items to a few thousand candidates. This work is memory-bound and runs on the CUDA cores, so the fp16-accumulate lever does nothing for it, and that is fine.
cuSPARSE / ANN · CUDA cores
Dense re-ranking
Scoring the candidates with the exact dot product is a dense GEMM on the tensor cores. This is where flipping the accumulator to fp16 buys the 1.59×, at recall the ranking absorbs.
Q @ Cᵀ · tensor cores · 1.59×
Unstructured sparse gets nothing
SpMM at 1% density is 0.93 TFLOP/s, 120× below the dense ceiling. cuSPARSE stays on the CUDA cores.
Memory-bound, not compute-bound
Scattered nonzeros make the work about fetching values, not multiplying them, so a faster math path cannot help.
2:4 structured is the exception
The only sparse path to the tensor cores on Ampere. Even so it is 1.04× here, marginal for this shape on a 3080.
The lever lives in the dense stage
Generate candidates sparse and memory-bound, then re-rank dense on the tensor cores where the 1.59× applies.
06 glossary
Every term and tool, defined
- Retrieval scoring
- Rating how well every candidate item matches a query so the best can be returned. The hot loop of a search engine.
- Recommender
- The same scoring with a user in place of the query and items in place of documents. Two-tower models embed user and items separately.
- Embedding
- A vector of a few hundred numbers a model assigns to content, placed so related things sit near each other. Match score is the dot product of two embeddings.
- Dot product
- Multiply two vectors position by position and sum. Higher means more aligned, so a closer match.
- d (embedding dimension)
- How many numbers each embedding holds. Here 512; real models run 384–1536. Larger d does more arithmetic per byte, so the lever grows with it.
- Q @ Cᵀ
- The scoring matmul. Query matrix Q [n_q×d] times the transpose of the item corpus C, giving every query-item score [n_q×n_c].
- Top-k
- Keeping the k highest scores in each query's row (k=10 here). A selection over the score table, not a matmul, so the accumulator lever does not touch it.
- MIPS
- Maximum-inner-product search: finding the items with the highest dot-product score. Brute-force MIPS scores the whole corpus for an exact answer.
- ANN
- Approximate nearest-neighbor search. An index that returns likely candidates fast, usually followed by an exact dense re-rank.
- GEMM / matmul
- General matrix multiply. The scoring step is a GEMM, and at realistic d it dominates the run time.
- Tensor cores
- Dedicated GPU units that multiply small matrix tiles and accumulate the result in one instruction (mma). Where a dense matmul runs.
- CUDA cores
- The GPU's general-purpose arithmetic units. Where sparse and top-k work runs, since neither maps to tensor-core tiles.
- Accumulator
- The running total the tensor cores sum partial products into. Its precision (fp16 or fp32) sets the rate on consumer Ampere.
- fp16 / fp32 accumulation
- Keeping that running sum in 16-bit or 32-bit. On sm_86 fp16 runs full rate, fp32 half rate; fp16 costs ~1e-3 of score precision.
- sm_86 / consumer Ampere
- The compute capability of GeForce Ampere cards like the RTX 3080, where fp32 accumulation is capped at half rate.
- allow_fp16_accumulation
- The PyTorch/cuBLAS toggle that selects the full-rate fp16-accumulate path. A setting, not a custom kernel; needs PyTorch 2.7+.
- cuBLAS
- NVIDIA's dense linear-algebra library. It runs the scoring matmul and honors the accumulator toggle.
- recall@k
- Of the k items a query should return, the fraction that appear in the returned top-k. The ranking-quality metric; 0.9872 here.
- Compute-bound vs memory-bound
- Limited by math-unit speed vs by data movement. Dense scoring is compute-bound; sparse and top-k are memory-bound.
- Sparse matrix
- A matrix that is almost all zeros, stored as only its nonzero positions and values. Lexical, co-occurrence, and graph data are sparse.
- SpMM / SpGEMM
- Sparse-times-dense and sparse-times-sparse matrix multiply. Both memory-bound on the CUDA cores, far below dense tensor-core throughput.
- cuSPARSE
- NVIDIA's sparse linear-algebra library. Runs SpMM and SpGEMM on the CUDA cores, so the tensor-core lever does not reach it.
- 2:4 structured sparsity
- Exactly two nonzeros per group of four along the contracting dimension. The one sparse pattern the tensor cores accept; 1.04× here via CUTLASS.
- CUTLASS
- NVIDIA's template library for building tensor-core matmul kernels. The backend for the 2:4 structured path.
- TFLOP/s
- Trillions of floating-point operations per second. The throughput number for a matmul; higher is faster.