AI Book Companion | Vapi, ElevenLabs

An AI-powered platform that transforms your books into interactive voice conversations.

Table of Contents

1. About
2. How It Works
3. Architecture
4. Tech Stack
5. Features
6. Getting Started
   • Prerequisites
   • Installation
   • Environment Variables
   • Running the Project
7. Project Structure
8. Contributing
   • Code of Conduct
   • How to Contribute
   • Development Workflow
9. License
10. Acknowledgments

---

About

BookNote is an AI-powered platform that lets you have real-time voice conversations with your books. Upload any PDF, and our AI will process it into an interactive entity that you can chat with using natural voice synthesis.

Key Highlights

• Voice-First Experience: Have natural conversations with your books using AI-powered voice technology
• Smart Document Processing: Automatic text extraction, intelligent chunking, and embeddings for precise context retrieval
• Multiple AI Personas: Choose from various AI personalities powered by ElevenLabs
• Real-time Transcripts: Get live transcripts of all your conversations
• Modern Tech Stack: Built with Next.js 16, TypeScript, and modern UI components

---

How It Works

1. PDF Upload & Processing

User Upload → PDF Parsing → Text Extraction → Chunking → Embedding Generation → Vector Storage

When a user uploads a PDF:

1. The PDF is processed using pdfjs-dist for text extraction
2. Text is split into meaningful chunks (overlapping for context)
3. Each chunk is converted to vector embeddings using Mistral AI API
4. Embeddings are stored in Convex for similarity search

2. Voice Conversation Flow

User Voice Input → Vapi (Voice AI) → Query Processing → Similarity Search → Context Retrieval → AI Response → Voice Output

1. User speaks to the AI via Vapi's voice interface
2. The spoken query is transcribed
3. Convex performs similarity search against book embeddings
4. Relevant context is retrieved and sent to the AI
5. ElevenLabs generates the voice response
6. User hears the AI's response in real-time

3. System Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Frontend      │────▶│   Next.js API   │────▶│   Convex        │
│   (Next.js 16)  │     │   Routes        │     │   (Database)    │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │                                               │
        │                                               ▼
        │                                     ┌─────────────────┐
        │                                     │   Vector Store  │
        │                                     │   (Embeddings)  │
        │                                     └─────────────────┘
        ▼
┌─────────────────┐     ┌─────────────────┐
│   Vapi          │────▶│   ElevenLabs    │
│   (Voice AI)    │     │   (TTS)         │
└─────────────────┘     └─────────────────┘

4. Data Models

Book

{
  _id: Id<"books">,
  title: string,
  author: string,
  description: string,
  coverURL?: string,
  coverBlobKey?: string,
  userId: string,
  persona?: string,
  isPublic: boolean,
  totalSegments: number,
  _creationTime: number
}

BookSegment

{
  _id: Id<"bookSegments">,
  bookId: Id<"books">,
  content: string,
  embedding: number[],
  _creationTime: number
}

---

Tech Stack

• Next.js 16 - Full-stack React framework with App Router
• TypeScript - Type-safe JavaScript
• Convex - Real-time database and serverless functions
• Vapi - Voice AI platform for real-time conversations
• ElevenLabs - AI-powered text-to-speech
• Clerk - Authentication and user management
• Tailwind CSS - Utility-first CSS framework
• Shadcn UI - Accessible UI component library
• Mistral AI - AI embeddings for semantic search

---

Features

• PDF Upload & Ingestion: Seamlessly upload PDF books with automated text extraction
• Voice-First Conversations: Engage in natural, real-time voice dialogues with your books
• AI Voice Personas: Choose from distinct AI personalities with ElevenLabs voices
• Smart Summaries: Extract key insights and summaries from any chapter
• Session Transcripts: Auto-generated text transcripts of all conversations
• Library Management: Organize and search through your personal collection
• Authentication: Secure access via Clerk with social login support

---

Getting Started

Prerequisites

Ensure you have the following installed:

• Git
• Node.js (v18 or later)
• npm or yarn
• Convex CLI (npm install -g convex)

Installation

1. Clone the repository:

git clone https://github.com/yourusername/booknote.git
cd booknote

2. Install dependencies:

npm install

3. Set up Convex:

npx convex dev

Environment Variables

Create a .env.local file in the root directory:

# Deployment (from npx convex dev)
CONVEX_DEPLOYMENT=your_convex_deployment
NEXT_PUBLIC_CONVEX_URL=your_convex_url

# Base URL
NEXT_PUBLIC_BASE_URL=http://localhost:3000

# Clerk Authentication
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=your_clerk_publishable_key
CLERK_SECRET_KEY=your_clerk_secret_key
NEXT_PUBLIC_CLERK_SIGN_IN_URL=/sign-in
NEXT_PUBLIC_CLERK_SIGN_UP_URL=/sign-up
NEXT_PUBLIC_CLERK_SIGN_IN_FALLBACK_REDIRECT_URL=/
NEXT_PUBLIC_CLERK_SIGN_UP_FALLBACK_REDIRECT_URL=/

# Vercel Blob (legacy - using Convex Storage)
BLOB_READ_WRITE_TOKEN=your_blob_token

# Vapi Voice AI
NEXT_PUBLIC_VAPI_API_KEY=your_vapi_key
VAPI_SERVER_SECRET=your_vapi_secret

# AI Services
MISTRAL_API_KEY=your_mistral_key
ELEVENLABS_API_KEY=your_elevenlabs_key

Getting API Keys:

Service	Sign Up Link
Clerk	clerk.com
Vapi	vapi.ai
ElevenLabs	elevenlabs.io
Mistral AI	console.mistral.ai

Running the Project

Start the development server:

npm run dev

Open http://localhost:3000 in your browser.

---

Project Structure

booknote/
├── app/                    # Next.js App Router
│   ├── (root)/            # Main routes
│   │   ├── page.tsx       # Home/Library page
│   │   └── books/
│   │       ├── new/       # Add new book
│   │       └── [slug]/    # Book landing page
│   ├── api/               # API routes
│   │   └── upload/        # File upload handler
│   ├── read/
│   │   └── [id]/          # Reading/chat page
│   ├── layout.tsx         # Root layout
│   └── globals.css        # Global styles
├── components/            # React components
│   ├── ui/               # Shadcn UI components
│   ├── BookCard.tsx      # Book display card
│   ├── Navbar.tsx        # Navigation
│   ├── HeroSection.tsx   # Landing hero
│   ├── Search.tsx        # Book search
│   ├── UploadForm.tsx    # PDF upload form
│   └── VapiControls.tsx  # Voice controls
├── convex/               # Convex backend
│   ├── books.ts          # Book queries/mutations
│   └── ...               # Other Convex functions
├── lib/                  # Utility functions
│   ├── actions/         # Server actions
│   ├── constants.ts     # App constants
│   └── utils.ts         # Helper utilities
├── public/              # Static assets
│   └── assets/          # Images, icons
├── types/               # TypeScript types
├── .env.local           # Environment variables
├── package.json
├── tailwind.config.ts
└── tsconfig.json

---

Contributing

We welcome contributions! Here's how you can help:

Code of Conduct

By participating in this project, you agree to follow our Code of Conduct. Please be respectful and inclusive.

How to Contribute

1. Fork the Repository Click the "Fork" button on GitHub or run:

   git fork https://github.com/yourusername/booknote

2. Clone Your Fork

   git clone https://github.com/yourusername/booknote.git
   cd booknote

3. Create a Feature Branch

   git checkout -b feature/your-feature-name
   # or
   git checkout -b fix/your-bug-fix

4. Make Changes

   • Follow the existing code style
   • Write meaningful commit messages
   • Add tests if applicable

5. Push to GitHub

   git push origin feature/your-feature-name

6. Create a Pull Request

   • Go to the original repository
   • Click "New Pull Request"
   • Fill out the template
   • Link any related issues

Development Workflow

1. Setup: Follow the Getting Started guide
2. Coding Standards:
   • Use TypeScript for all new code
   • Follow ESLint rules (npm run lint)
   • Use meaningful variable and function names
3. Testing:
   • Test your changes locally
   • Verify no linting errors
4. Commit Messages:

   feat: add new voice persona selection
   fix: resolve PDF upload timeout issue
   docs: update API documentation
   refactor: simplify book chunking logic
   

Types of Contributions

• 🐛 Bug Reports: Found a bug? Open an issue
• 💡 Features: Suggest new features
• 📖 Documentation: Improve docs
• 🎨 UI/UX: Improve the interface
• 🔧 Code: Submit pull requests

---

License

This project is licensed under the MIT License.

MIT License

MIT License

Copyright (c) 2024 BookNote

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

---

Acknowledgments

• JavaScript Mastery for the original inspiration
• Vapi for the voice AI infrastructure
• ElevenLabs for exceptional text-to-speech
• Clerk for authentication
• Convex for the backend infrastructure
• Shadcn for beautiful UI components

---

Built with ❤️ using Next.js, Vapi, and ElevenLabs

Star us on GitHub if you find this project useful!