Memory is your budget. The model is only part of it.
Parameters, quantization and context without magic promises. What actually needs to fit in memory.
More than a model file
Memory holds model weights, computation and the running state of a conversation. macOS and other applications also need their share. Apple silicon uses unified memory across CPU and GPU, but its entire capacity is not available to one model.
Choose using the exact model variant, inference engine requirements and measurements on your machine. A universal “parameter count equals memory” table would be misleading.
Quantization is a trade-off
Quantization stores numbers at lower precision and can substantially reduce weight size. Its quality impact depends on the model, method and task. A smaller variant may be more practical than a large model that forces continuous disk swapping.
Change one variable at a time. The same prompt and settings on two quantizations of one model tell you more than unrelated screenshots.
Context has a cost too
Long prompts and conversation history are not free. Increasing context typically requires more cache memory and more work to process the input. Exact behavior depends on the architecture and implementation.
Start with a shorter context, then gradually add real documents. Watch memory pressure, swap and responsiveness in Activity Monitor.
Measure usefulness, not just tokens
Generation throughput is only one metric. Time to first answer, output quality, context length and resource use also matter.
The right model for your setup handles your actual work reliably with acceptable latency. A result from one Mac or prompt does not automatically apply to another configuration.