Building Agentic Inbox: Self-Hosted Multimodal Email AI to Replace $200/Yr SaaS

Modern email copilot tools promise radical productivity gains, but their price tags—routinely running $20 to $35 per user every month ($240 to $400 annually)—and aggressive data access policies make them untenable for independent developers, founders, and privacy-conscious teams. More critically, most cloud email tools only read plaintext, completely ignoring attached PDF proposals, balance sheets, and invoice receipts where actual work happens.

We built Agentic Inbox as an open-source, self-hosted autonomous email management agent. It monitors your inbox via IMAP or Gmail APIs, inspects multimodal attachments, categorizes message priorities, prepares contextual reply drafts, and schedules calendar holds—all while keeping your email data strictly within your own infrastructure.

Here is how we designed and implemented the agentic workflow.

The System Architecture: Autonomous Triage with Human Guardrails

The single biggest mistake in automated email agents is allowing the AI to hit "Send" autonomously. Hallucinations or misunderstood tone can destroy client relationships in seconds.

Agentic Inbox enforces a Zero-Accidental-Send invariant:

1. Perception: Ingest email headers, body text, and attachments.
2. Analysis: Extract core intents, determine urgency, and extract action items.
3. Execution: Apply mailbox labels, schedule tentative calendar events, and save high-fidelity reply drafts to your Drafts folder for a 1-click human review.

[ Incoming Email Stream ]
          │
          ├── Plaintext & HTML Body
          └── Multimodal Attachments (PDFs, Images, Invoices)
          │
          ▼
[ Agentic Ingestion Pipeline ]
          │
          ├── 1. Vision & Document Parser (OCR & Table Extraction)
          ├── 2. Vector Context Retriever (Past Thread History)
          └── 3. LLM Intent & Priority Classifier
          │
          ▼
   ┌────────────────────────────────────────────────────────┐
   │                  Triaged Action Set                    │
   │  - Apply Priority Tag: [URGENT_CLIENT] / [NEWSLETTER]  │
   │  - Sync Extracted Calendar Invites to CalDAV / Google  │
   │  - Stage Contextual Reply Draft in User's Outbox       │
   └────────────────────────────────────────────────────────┘
          │
          ▼
   [ 1-Click Human Approval Before Sending ]

1. Multimodal Document Parsing for Invoices and Contracts

When a client sends: "Please check attached proposal and let me know if we can proceed with Milestone 2," a traditional text-only LLM fails because the milestone deliverables and pricing reside entirely inside the attached PDF.

Agentic Inbox intercepts PDF attachments, converts pages into high-resolution bitmap buffers, and feeds them into multimodal vision models:

// Multimodal Attachment Ingestion Pipeline
import { PDFExtract } from 'pdf.js-extract';

export interface EmailMessage {
  id: string;
  from: string;
  subject: string;
  bodyText: string;
  attachments: { filename: string; mimeType: string; buffer: Buffer }[];
}

export async function extractEmailContext(msg: EmailMessage): Promise<string> {
  let context = `Subject: ${msg.subject}\nFrom: ${msg.from}\nBody:\n${msg.bodyText}\n\n`;

  for (const file of msg.attachments) {
    if (file.mimeType === 'application/pdf') {
      const extractedText = await parsePdfBuffer(file.buffer);
      context += `\n--- ATTACHED PDF: ${file.filename} ---\n${extractedText.slice(0, 3000)}\n`;
    }
  }

  return context;
}

By providing the LLM with direct visibility into contracts and invoice tables, the generated reply drafts address the specific numerical milestones rather than issuing empty acknowledgments.

2. Priority Scoring & Anti-Spam Classification

Conventional spam filters look at keyword density and domain reputation. Agentic Inbox evaluates semantic importance based on your historical interactions:

// Semantic Priority Evaluation Prompt Matrix
export async function triageEmail(context: string): Promise<TriageResult> {
  const prompt = `
Analyze the incoming email below. Output strictly JSON with keys:
- priority: "HIGH" (clients, urgent deadlines), "NORMAL", or "LOW" (newsletters, cold pitches)
- actionNeeded: boolean
- draftReplyNeeded: boolean
- suggestedDraft: string (polite, direct, professional, acknowledging specific attachments)
- extractedDeadlines: array of ISO date strings

EMAIL CONTENT:
${context}
  `;

  const response = await llmGateway.completeJson<TriageResult>(prompt);
  return response;
}

If an email is cold outreach from an SDR selling unwanted lead generation software, it is quietly moved to a Low-Priority/Screened folder without buzzing your phone. If it is an active client reporting a bug or signing a scope of work, it is pinned to the top of your inbox with an immediate draft reply ready for review.

3. What We Learned Replacing SaaS with Self-Hosted Agents

1. Deterministic Guardrails Protect Trust: The moment an AI agent sends an email without human review, anxiety increases. Staging drafts in the native Drafts folder gives you all the time savings of automated writing with 100% human editorial control.
2. Local Multimodal Is the New Baseline: Modern open-source vision-language models can read invoices and parse tables locally on consumer GPUs. You do not need to ship sensitive financial contracts to external proprietary APIs.
3. Open-Source Ownership Beats Recurring SaaS: Paying $300 every year to manage an inbox that you already own makes little sense. Self-hosting your own agent gives you unlimited volume, custom workflows, and total privacy.