AI-Cursor-Scraping-Assistant

A powerful tool that leverages Cursor AI and MCP (Model Context Protocol) to easily generate web scrapers for various types of websites. This project helps you quickly analyze websites and generate proper Scrapy or Camoufox scrapers with minimal effort.

Project Overview

This project contains two main components:

1. 1.Cursor Rules - A set of rules that teach Cursor AI how to analyze websites and create different types of Scrapy spiders
2. 2.MCP Tools - A collection of Model Context Protocol tools that enhance Cursor's capabilities for web scraping tasks

Prerequisites

• Cursor AI installed
• Python 3.10+ installed
• Basic knowledge of web scraping concepts

Installation

Clone this repository to your local machine:

git clone https://github.com/TheWebScrapingClub/AI-Cursor-Scraping-Assistant.git
cd AI-Cursor-Scraping-Assistant

Install the required dependencies:

pip install mcp camoufox scrapy

If you plan to use Camoufox, you'll need to fetch its browser binary:

python -m camoufox fetch

Setup

Setting Up MCP Server

The MCP server provides tools that help Cursor AI analyze web pages and generate XPath selectors. To start the MCP server:

1. 1.Navigate to the MCPfiles directory:

```bash

cd MCPfiles

```

1. 1.Update the CAMOUFOX_FILE_PATH in xpath_server.py to point to your local Camoufox_template.py file.

1. 1.Start the MCP server:

```bash

python xpath_server.py

```

1. 1.In Cursor, connect to the MCP server by configuring it in the settings or using the MCP panel.

Cursor Rules

The cursor-rules directory contains rules that teach Cursor AI how to analyze websites and create different types of scrapers. These rules are automatically loaded when you open the project in Cursor.

Detailed Cursor Rules Explanation

The cursor-rules directory contains a set of MDC (Markdown Configuration) files that guide Cursor's behavior when creating web scrapers:

prerequisites.mdc

This rule handles initial setup tasks before creating any scrapers:

• Gets the full path of the current project using pwd
• Stores the path in context for later use by other rules
• Confirms the execution of preliminary actions before proceeding

website-analysis.mdc

This comprehensive rule guides Cursor through website analysis:

• Identifies the type of Scrapy spider to build (PLP, PDP, etc.)
• Fetches and stores homepage HTML and cookies
• Strips CSS using the MCP tool to simplify HTML analysis
• Checks cookies for anti-bot protection (Akamai, Datadome, PerimeterX, etc.)
• For PLP scrapers: fetches category pages, analyzes structure, looks for JSON data
• For PDP scrapers: fetches product pages, analyzes structure, looks for JSON data
• Detects schema.org markup and modern frameworks like Next.js