VectorPipe

Ingestion pipelines for restless builders

GraphQL360 begat Twirrl; Twirrl is morphing into VectorPipe—a rapid-to-market way to upload anything and push it into vector stores without bespoke glue.

One app, one deploy

VectorPipe is a unified SvelteKit+tRPC app born from the Twirrl GraphQL stack. No more juggling Render + Vercel deployments.

Pipeline-first projects

Projects own their ingestion pipelines—upload PDFs, podcasts, YouTube channels, or web archives and send them through Pinecone/Chroma embeddings.

API keys + fair-use storage

Users bring their own OpenAI/Pinecone keys. Docs land in S3 during beta, with a “fair use” note so experiments stay cheap.

UI lineage

Twirrl screens → VectorPipe console

The UI mocks below evolved from capture workflows into ingestion pipelines. Same philosophy: keep the work visible, one project at a time.

Capture selector UI
Twirrl capture selector reused for pipeline project creation.
Project workspace mock
Project workspace concept that now stores ingestion status and pipeline runs.
List view mock
List view mock showing how multiple ingestion projects are tracked.
Room alignment experiment
Earlier computer-vision tooling feeding into today’s document alignment + QA ideas.

Build log

From walking notes to action items

May 2024 voice memos set the tone: consolidate deployments, drop GraphQL baggage, and expose the ingestion pipeline as fast as possible.

  • • Strip remaining scene/image GraphQL types and replace with resource entities.
  • • Add project-level pipeline configuration and API key vault.
  • • Switch to tRPC for typed endpoints; keep Prisma + Clerk auth wiring.
  • • Ship a basic Pinecone query console so people can sanity-check embeddings.

Why it matters

Clients keep asking for repeatable ingestion + retrieval setups. VectorPipe packages the lessons from Lighten, Zolidar, and Twirrl so every LLM residency starts with working plumbing.

Back to lab timeline