Files that can't leave
Mark needs AI to summarize internal files, but his company forbids sending documents to outside services.
OFFLINE
A model on your own machine
Bit suggests: run an open model right on your computer. Everything is processed locally, it works with the internet off, and your data never leaves the machine.
TOOLS
Install it like an app
There are now apps like Ollama or LM Studio that install like normal software: pick a model, download it, and chat.
WHAT YOU NEED
What your computer needs
A laptop with 16 GB of RAM runs a model of about 8 billion parameters, quantized to 4 bits, quite well. A good GPU makes it faster.
LIMITS
The trade-off
In return, a small local model is weaker than the biggest cloud models, and slower. It's good for simple jobs like summarizing, translating and drafting.
This article is based on the video Running AI offline on your own machine from the Mark học AI channel. Watch the video (in Vietnamese) to see the animations.