Company Intel Scraper (RAG Pipeline)

Enterprise-grade company research pipeline. Crawls sitemaps, chunks content for RAG, generates OpenAI embeddings, stores in Supabase vector DB with semantic search for AI email copy generation.

View on GitHub

workflowEnrichment & ResearchAI-PoweredSales IntelligenceOutbound

Original

by Roheel Jain

Nodes

Triggers

Platform

n8n

Tech Stack

OpenAI Embeddings

Supabase Vector DB

Sitemap Crawling

RAG

HTTP API

Workflow

Manual Trigger

manualTrigger

Set - Config

set

Parse Email Doc

code

IF - Skip Summary

Supabase - Get Step Stats

httpRequest

Enrich with Performance

code

OpenAI - Generate Embedding

httpRequest

Prepare Supabase Upsert

code

Supabase - Upsert Email Copy

httpRequest

HTTP Request

httpRequest

Merge

merge

Supabase - Get Step Stats1

httpRequest

Set - Unique ID

set

Fetch Campaign Request

httpRequest

Map Fields

code

OpenAI - Embed Request

httpRequest

Supabase - Match Email Copies

httpRequest

Build AI Prompt

code

OpenAI - Generate Copy

httpRequest

Format Output

code

Fetch Clients

httpRequest

Parse Form + Match Client

code

Supabase - Upsert Campaign Request

httpRequest

Webhook1

webhook

Webhook

webhook

Normalize Input

code

Scrape - Key Pages

httpRequest

Aggregate Scraped Content

code

OpenAI - Extract Profile

httpRequest

Format Profile

code

Supabase - Upsert Profile

httpRequest

Set - Config1

set

Fetch robots.txt1

httpRequest

Extract Sitemap URL1

code

Fetch Sitemap1

httpRequest

Filter Key Pages1

code

Delete Old Scraped Assets

httpRequest

Chunk Pages for RAG

code

OpenAI - Embed Chunk

httpRequest

Format Asset Row

code

Supabase - Insert Assets

httpRequest

Code in JavaScript

code

Fetch Sub-Sitemaps

httpRequest

Code in JavaScript1

code

Fetch Page Free

httpRequest

Has Content?

Check Content

code

How It Works

This automation turns any company's website into a smart research database that writes personalized email copy for you. Simply enter a company URL and get AI-generated, contextually relevant email content based on deep analysis of their business.

Submit company website for analysis

Enter the target company's website URL into the system. The automation begins by identifying all the important pages on their site.

Scan and collect company content

The system automatically crawls through key pages like About Us, Services, and Product pages to gather comprehensive information about the company.

Build intelligent company profile

AI analyzes all collected content to create a detailed profile including the company's services, value propositions, industry focus, and key messaging.

Store information for smart search

All company data is organized and stored in a searchable database that understands context and meaning, not just keywords.