🤖

Gemma3n Local RAG

Offline Document Chat Assistant

Chat with your PDFs using a fully local AI stack — zero cloud, zero data leaks

100% Offline Flask Backend FAISS Vector Search llama.cpp GGUF Models LangChain

Local

Inference Engine

Streaming

Responses

Multi-PDF

Per Project

github.com/sidx007/Gemma3n-Local-Rag

Overview

What is it?

A fully offline RAG (Retrieval-Augmented Generation) app. You drop PDFs in, it chunks them, creates FAISS vector embeddings, and lets you ask questions in plain English — answered using a local GGUF model via llama.cpp with real-time streaming. Nothing leaves your machine.

Project Management

Organize documents into named projects. Each project gets its own vector store. Filter by category, search by name, delete when done.

Smart Chunking

800-char chunks with 100-char overlap. Tunable. HuggingFace embeddings with optional Gemini metadata enrichment for better retrieval quality.

Local Inference

llama.cpp runs GGUF models (Q4/Q5 quantized). 4-8 GB RAM needed depending on model size. Streaming token-by-token responses via Flask.

Modern UI

Glassmorphism dark/light theme. Responsive for desktop and mobile. Vanilla JS + Flask — no heavy frontend framework.

How It Works

Ingestion → Retrieval → Generation

📥 Ingestion Pipeline

Upload PDFs

Drop one or more PDFs into a named project

Chunk + Embed

LangChain splits text → HuggingFace embeddings → stored in FAISS

Persist Locally

FAISS index saved to projects/ folder — ready for offline use

💬 Query Pipeline

Ask a Question

User types a query in the chat interface

Semantic Retrieval

FAISS finds the most relevant document chunks

Local Generation

Chunks + query → llama.cpp → streamed answer in the browser

Privacy & Setup

Fully offline by design

Privacy Guarantees

🔒

No data leaves your machine

All embeddings and inference run 100% locally

📁

Local storage only

Documents and FAISS index stored in projects/ folder

🚫

No hardcoded API keys

Optional Gemini metadata enrichment only — disabled by default

Quick Start

pip install -r requirements.txt

# Put your GGUF model in models/
# Update model_path in rag.py

cd chatbot
python rag_backend.py

# Open rag-index.html in browser
# Create a project, drop PDFs, chat

Requires Python 3.8+. Works on Windows, Linux, macOS. Use Q4/Q5 quantized models for best performance on CPU.