labs vectorpipe
The unglamorous middle of every RAG system
build log · May 2024
VectorPipe takes a pile of resources — documents, podcasts, YouTube channels, whole websites — and pipes them into a vector store without bespoke glue. There are a lot of intermediate steps between "here's my stuff" and "query it"; that middle is the product.
One app, one deploy
A single SvelteKit app — tRPC endpoints, Prisma, Clerk auth, BullMQ for the background jobs. One deploy instead of a split frontend/API pair.
Pipeline-first projects
Projects own their ingestion pipelines — upload PDFs, podcasts, YouTube channels, or web archives and send them through Pinecone or Chroma embeddings. Grouped resources handle a whole channel or site; a pipeline can split to multiple outputs.
API keys + fair-use storage
Users bring their own OpenAI/Pinecone keys, reusable across projects. Docs land in S3 during beta, with a fair-use note so experiments stay cheap.
Build log
From walking notes to action items
May 2024 voice memos set the tone: consolidate to one deployable app, take the rapid-to-market path, and expose the ingestion pipeline as fast as possible.
- Move the API to tRPC for typed endpoints; keep Prisma + Clerk auth wiring.
- Add project-level pipeline configuration and an API key vault.
- Publish per-resource status as the pipeline processes.
- Ship a basic Pinecone query console so people can sanity-check embeddings.
Why it matters
Clients keep asking for repeatable ingestion + retrieval setups. The same backend pieces were carved out — as non-exclusive code — for client work at Lighten and Zolidar, so an LLM engagement starts with working plumbing instead of a blank repo.