How to Scrape Competitor Pricing Data with Claude Code

If you've ever tried to keep track of competitor pricing manually, you know how fast it becomes a full-time job. Prices change daily. New products get added. Old ones go on sale. By the time you've updated your spreadsheet, half the data is already stale. I've watched Vancouver clients lose deals because they didn't realize a competitor had dropped prices two days earlier.

The fix is automated web scraping. With Claude Code, I've built pricing intelligence systems that monitor dozens of competitor sites and alert my clients the moment something changes. No manual checking. No spreadsheet rot. Just clean data that updates itself. Here's exactly how I do it.

Why Scrape Competitor Pricing Data

Before we get into the how, let's talk about why this matters. Pricing intelligence isn't just for e-commerce businesses undercutting each other on Amazon. I've built scrapers for:

SaaS companies tracking feature pricing and packaging changes from competitors
Service businesses monitoring hourly rates and project minimums in their local market
Retailers watching MAP violations and promotional windows across channels
B2B distributors keeping tabs on volume discount tiers and shipping costs

The common thread: in every case, pricing isn't static. Competitors adjust rates based on demand, seasonality, inventory levels, or just to see what sticks. If you're making pricing decisions based on data from three months ago, you're flying blind.

The Basic Web Scraping Workflow with Claude Code

Here's the simplest version of a pricing scraper. This is what I build for most clients as a starting point, and it usually takes about an hour to get running.

The workflow has four steps:

Fetch the HTML from the competitor's product or pricing page
Parse the HTML to extract the price element (and any relevant metadata like SKU, availability, or shipping cost)
Store the extracted data in a database or spreadsheet with a timestamp
Compare the new data to the previous scrape and flag any changes

Claude Code handles all of this in a single script. The key is teaching it where to look on each page. That's the only manual step — pointing out the CSS selector or XPath for the price element.

Step 1: Identify the Target Data

Before you write any code, you need to inspect the competitor's page and figure out where the price lives in the HTML. Most modern sites use consistent class names or IDs for product pricing. Open the browser's developer tools (right-click → Inspect Element), hover over the price on the page, and look at the HTML structure.

You're looking for something like:

<span class="product-price">$49.99</span>

Or:

Once you have the selector (e.g., .product-price or #priceblock_ourprice), you can tell Claude Code to extract it.

Step 2: Build the Scraper Script

I usually start with a prompt like this:

Write a Python script that fetches the HTML from [URL], extracts the text content of the element with selector [SELECTOR], parses the price as a float, and outputs it as JSON with the following fields: url, price, currency, timestamp.

Claude Code will generate a script using a library like BeautifulSoup or Playwright. For static sites, BeautifulSoup is faster. For JavaScript-heavy sites (like most e-commerce platforms), Playwright is more reliable because it renders the page like a real browser.

Here's a simplified version of what the output looks like:

import requests
from bs4 import BeautifulSoup
import json
from datetime import datetime

url = "https://competitor.com/product/widget"
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

price_element = soup.select_one('.product-price')
price_text = price_element.get_text(strip=True).replace('$', '').replace(',', '')
price = float(price_text)

output = {
    "url": url,
    "price": price,
    "currency": "CAD",
    "timestamp": datetime.now().isoformat()
}

print(json.dumps(output, indent=2))

Run this once manually to confirm it works. If the selector is wrong or the page structure is different than expected, adjust it. Once it's pulling the right data, you're ready to scale it.

Step 3: Store the Data

For most clients, I store pricing data in Google Sheets because it's easy to share and visualize. For larger datasets or when you need historical analysis, a proper database (SQLite, Postgres, or even Airtable) makes more sense.

The script appends each new scrape result as a new row. That way you get a full history of pricing changes over time, not just the current value. If you only care about the latest price, you can overwrite the previous row instead.

Step 4: Automate and Alert

A scraper that runs once isn't useful. You need it to run on a schedule — daily at minimum, hourly if pricing is volatile. I usually set this up as a cron job on a small VPS or use GitHub Actions for free scheduled runs.

The second part is alerting. When a price changes, the script should notify someone. I typically integrate with Slack, email, or a Discord webhook. The logic is simple: compare the new price to the most recent stored price. If they differ by more than a threshold (say, 5%), send an alert.

A Vancouver e-commerce client saved about $18,000 in lost margin over six months by catching a competitor's temporary price drop and adjusting their own pricing within 24 hours. The scraper cost them $1,200 to build.

Handling the Common Challenges

Web scraping is fragile. Sites change their HTML structure. They add anti-bot measures. They rate-limit requests. Here are the issues I run into most often and how I handle them.

Dynamic Content (JavaScript-Rendered Pages)

If the competitor's site loads prices via JavaScript after the initial page render, BeautifulSoup won't see them. You need a headless browser. I use Playwright for this. It's slower than static scraping but much more reliable for modern SPAs and e-commerce platforms.

The trade-off: headless browsers use more resources, so you can't run hundreds of scrapers simultaneously without spinning up multiple servers. For most pricing intelligence use cases, scraping 20–50 products once per day is plenty, so this isn't a real constraint.

Rate Limiting and Bot Detection

If you hit a site too hard, it will block your IP or serve you CAPTCHAs. The simplest fix: add random delays between requests (2–5 seconds) and rotate user agents. For more aggressive anti-bot measures, you can use residential proxies, but that adds cost and complexity. Most mid-market competitors don't have sophisticated bot detection, so basic throttling is usually enough.

Inconsistent HTML Structure

Sometimes a site will use different selectors for different product categories. When that happens, I build a fallback chain: try selector A, if it fails, try selector B, if that fails, log an error and alert a human. This keeps the system resilient to minor site changes without breaking completely.

Real Use Cases from My Client Work

Let me share three actual implementations to make this concrete.

SaaS competitor monitoring: A Vancouver software company wanted to track pricing changes across 12 direct competitors. I built a scraper that checked their pricing pages daily and logged any changes to plan names, feature inclusions, or monthly costs. Within the first month, they caught two competitors raising prices and used that intel to justify their own price increase without losing market positioning.

E-commerce margin protection: An online retailer was selling products that were also available on Amazon and three other Canadian marketplaces. They needed to ensure they weren't undercut while still staying competitive. The scraper checked all four channels twice daily and alerted them when any competitor dropped below their floor price. They adjusted automatically using a spreadsheet-linked pricing tool.

Service business local pricing: A commercial cleaning company in Metro Vancouver wanted to know what competitors were charging for office cleaning contracts. Most competitors listed per-square-foot rates on their websites. I scraped six competitor sites weekly and charted the distribution. The client realized they were underpricing by about 15% and adjusted their proposals accordingly.

Legal and Ethical Considerations

I'm not a lawyer, but I've worked with enough clients on this to know the basics. Scraping publicly available pricing data from a competitor's website is generally legal in Canada, as long as you're not violating their terms of service, circumventing paywalls, or causing harm to their infrastructure through excessive requests.

The safe approach: scrape sparingly, respect robots.txt, don't republish the data publicly, and use it only for competitive analysis. If a site explicitly prohibits automated access in their ToS, either get permission or don't scrape it.

Getting Started with Your Own Pricing Scraper

If you want to build this for your own business, here's the fastest path:

Pick 3–5 competitor products or pages to start with
Inspect the HTML and write down the CSS selectors for price, product name, and any other relevant fields
Use Claude Code to generate a scraper script for one page, test it, and confirm the output is correct
Scale it to all target pages with a loop or batch script
Set up automated daily runs and a simple alert system (email or Slack)

Most of my clients get a working system within a week. The build itself takes a few hours; the rest of the time is testing edge cases and making sure the alerts are actionable.

For more on how I use Claude Code for web scraping automation, I've written a separate deep dive. And if you're interested in other ways to monitor competitors automatically, check out my post on automated competitor analysis workflows.

Pricing intelligence used to require a full-time analyst. Now it's a script that runs in the background. If you're still checking competitor prices by hand, you're leaving money on the table. The only question is which products you want to start tracking first.