Local AI Glossary
Terms and definitions of local artificial intelligence, open source models, and computing.
11 terms in Techniques
The splitting of a document into small text blocks before indexing. Each block becomes a searchable entry in the vector database: block size is one of the first parameters to tune when RAG results are not convincing.
Technique to shrink the part of the context sent to the model at a reasoning step, dropping or summarizing retrieved or historical portions the current step does not use. Examples: summaries generated by another model (ACON, Context-Folding), token removal via a learned policy (LLMLingua-2), or chunk selection based on the model's own attention (QRAE). On multi-step agents it cuts tokens and latency without, when done well, degrading quality.
Family of speculative decoding techniques (EAGLE paper, arXiv:2401.15077): a small draft model reads the big model's hidden states and predicts the next tokens from there. Sampling stays lossless, like classical speculative sampling. Official EAGLE checkpoints (for example Mistral-Medium-3.5-128B-EAGLE) are served natively by vLLM and SGLang.
A technique to transfer the capabilities of a large model (the teacher) to a smaller one (the student), which imitates the teacher's behavior instead of repeating its training. The result is a smaller, faster model with a small quality drop: in an extreme case, distilling a 35-step bidirectional video generator into a 4-step causal student is what lets Cosmos3-H-Dreams run in real time.
The art of writing effective instructions to get better results from a language model. Includes techniques like few-shot prompting (providing examples in the prompt), chain-of-thought (asking the model to reason step by step) and role prompting (defining a role for the model).
Training that simulates quantization during training, so the model learns to work with low-precision weights. Unlike post-training quantization (applied cold, e.g. Q4_K_M), QAT preserves much more quality at the same bit width. Google released official QAT checkpoints for the Gemma 4 family in June 2026.
Mathematical relationship between model or data scale and resulting quality: parameters, tokens and compute grow predictably relative to loss. In the synthetic-data context, scaling laws measure how many tasks it takes to reach a given success rate.
The closeness between two texts' vectors based on meaning, not on literal match. It is the principle behind search in a vector database: "brake" and "braking system" are not the same word, but their vectors are neighbors.
Data generated by an artificial process instead of collected from the real world: synthetic tasks, synthetic images, simulation traces. Used for training when real data costs money, changes, or can't be touched; quality depends on how well the synthetic distribution imitates the real one.
The set of instructions that defines a model's permanent behavior: tone, role, response format, constraints. Unlike the user prompt (a single question), the system prompt governs every response of the session and is not shown to the model as a request.
Voice Activity Detection: a technique that identifies which portions of an audio stream contain speech, so the audio can be segmented into relevant chunks and silence ignored. Essential in real-time transcription pipelines.

