AI-Powered Article Generator

An intelligent content tool that scrapes Google, extracts live web data with Bright Data MCP, and uses AI to generate articles from real-time research.

Features

• 🔍 SERP Scraping: Automatically extract relevant URLs from Google search results
• 📄 Content Extraction: Scrape and clean content from web pages using Bright Data MCP tools
• 🧠 AI Analysis: Process content using OpenAI embeddings and vector similarity search
• ✍️ Content Generation: Create article outlines or full articles using LangChain and OpenAI
• 📊 Research Metrics: View detailed analysis of scraped content and identified themes

Prerequisites

• Python 3.8+
• OpenAI API key
• Bright Data API token
• Node.js (for MCP tools)

Installation

1. 1.Clone the repository

```bash

git clone cd article-generator

```

1. 1.Install Python dependencies

```bash

pip install -r requirements.txt

```

1. 1.Install Node.js MCP tools

```bash

npx @brightdata/mcp

```

1. 1.Set up environment variables

Create a .env file in the project root:

```env

OPENAI_API_KEY=your_openai_api_key_here BRIGHT_DATA_API_TOKEN=your_bright_data_api_token_here WEB_UNLOCKER_ZONE=your_web_unlocker_zone_here BROWSER_ZONE=your_browser_zone_here

```

Usage

1. 1.Start the application

```bash

streamlit run article_generator.py

```

1. 1.Open your browser

The app will automatically open at http://localhost:8501

1. 1.Generate content

• Enter your research keyword (e.g., "artificial intelligence in healthcare")
• Configure settings in the sidebar:
• Maximum sources to scrape (5-20)
• Output type (Article Outline or Full Article)
• Target word count for full articles (800-3000)
• Click "🚀 Generate Content"

How It Works

1. 1.Search: Scrapes Google search results for your keyword
2. 2.Extract: Downloads and cleans content from relevant web pages
3. 3.Analyze: Uses AI embeddings to identify key themes and insights
4. 4.Generate: Creates structured content based on the research analysis

Configuration

Sidebar Options

• Maximum sources: Number of web pages to scrape (5-20)
• Output type: Choose between article outline or full article
• Target word count: Desired length for full articles (800-3000 words)

Environment Variables

• OPENAI_API_KEY: Your OpenAI API key for content generation
• BRIGHT_DATA_API_TOKEN: Your Bright Data API token for web scraping
• WEB_UNLOCKER_ZONE: Bright Data web unlocker zone (default: mcp_unlocker)
• BROWSER_ZONE: Bright Data browser zone (default: scraping_browser1)

Output Features