I used to spend every Monday morning doing the same thing: opening twelve competitor websites in separate tabs, checking their pricing pages, skimming their latest blog posts, and screenshotting anything that looked new. Then I'd compile it all into a Google Doc and send it to clients as a "weekly competitive intelligence report." It took between six and eight hours every week. That's 32 hours a month doing work a machine could handle.
Six months ago I built a Claude Code competitor analysis automation system that does most of it for me. It checks competitor sites every night, flags meaningful changes, and generates a summary report I can review in 15 minutes. The clients still get their insights — they just don't know I'm not manually pulling them anymore.
Here's exactly how I built it, what it tracks, and where the human judgment still matters.
What the System Actually Monitors
Before I automated anything, I had to get clear on what mattered. Not every change on a competitor's site is worth flagging. A new blog post might be noise. A pricing increase is a signal. The system I built tracks four categories of competitive intelligence:
- Pricing and packaging changes — any shift in listed prices, plan names, or feature inclusions
- Product launches and updates — new services, discontinued offerings, or major feature announcements
- Content and messaging — changes to homepage copy, value propositions, or positioning language
- SEO and keyword targeting — new landing pages, meta tag shifts, or blog topic patterns
The script runs every 24 hours. It scrapes the target pages, compares them to the previous snapshot, calculates a "change score" based on how much text has shifted, and only alerts me if the score crosses a threshold. Small tweaks get logged but not flagged. Major overhauls trigger an immediate Slack notification.
How the Web Scraping Works
The core of the system is a web scraping workflow built in Claude Code. I'm using Puppeteer to handle the browser automation — it loads each competitor page, waits for the DOM to settle, and extracts the HTML. Then Claude Code parses the HTML and pulls out the specific elements I care about: pricing tables, H1 tags, meta descriptions, and any structured data in JSON-LD format.
The first version of the script was brittle. If a competitor changed their CSS class names, the scraper would break. So I added a fallback layer: if the primary selector fails, Claude Code uses a fuzzy match to find elements by visible text patterns instead. It's not perfect, but it catches about 90% of changes even when site structure shifts.
One thing I learned the hard way: don't scrape too aggressively. I initially set the script to check every site every six hours. Two competitors blocked my IP within a week. Now I run it once a day and randomize the request timing to look more like a human visitor.
The extracted data gets saved to a JSON file with a timestamp. Each new scrape compares the current snapshot to the most recent prior snapshot and generates a diff report. If the diff shows more than 15% text change on a pricing page or a new H1 on the homepage, it flags it for review.
Tracking Pricing Changes Automatically
Pricing is the easiest thing to automate because it's structured. Most SaaS companies and service businesses display their pricing in tables or cards with consistent markup. I wrote a prompt that tells Claude Code to extract every visible price, plan name, and feature bullet from the pricing page and normalize it into a standard format.
When a price changes, the system logs the old value, the new value, the percentage change, and the date. Then it adds a one-line summary to the weekly report: "Competitor X raised their Pro plan from $99/mo to $129/mo (+30%) on July 3."
This has caught several pricing moves that would have taken me weeks to notice manually. One Vancouver-based competitor quietly raised their entry-level service package by 18% in May. My system flagged it the next morning. I sent the report to a client who was about to lose a deal to that competitor on price — they adjusted their pitch and closed it.
Monitoring Content and Messaging Shifts
Pricing is objective. Messaging is subjective. But there are still patterns you can automate. I track the homepage H1, the first paragraph of body copy, and any hero section CTAs. When one of those elements changes significantly, Claude Code flags it and shows me a side-by-side comparison of the old and new text.
The system also monitors blog publishing frequency and topic patterns. If a competitor suddenly starts publishing three posts a week about AI automation after six months of silence, that's a signal they're shifting focus. Claude Code pulls the title, publish date, and meta description of every new post and groups them by keyword theme.
I don't read every competitor blog post. But I do read the summary Claude Code generates: "Competitor Y published 4 posts this week, all targeting 'marketing automation' keywords. Estimated traffic potential: 2,400 monthly searches." That's enough to decide whether it's worth a closer look.
SEO and Keyword Strategy Tracking
The third layer is SEO competitor analysis automation. Claude Code monitors which pages competitors are adding, what keywords they're targeting in title tags and H1s, and whether they're building out programmatic landing pages at scale.
I built this feature specifically for clients in competitive local markets — Vancouver real estate, legal services, home services. In those verticals, winning on SEO often comes down to who builds location-based landing pages faster. If a competitor launches 50 new neighborhood pages overnight, I want to know about it the next day, not three months later when they're already ranking.
The system pulls sitemaps from competitor sites, diffs them against the previous week's version, and flags any new URLs. Then it scrapes the new pages and extracts the primary keyword target. The output is a simple table: "Competitor Z added 22 pages this week targeting [city name] + [service type] long-tail keywords."
This kind of tracking is especially useful for programmatic SEO strategies where speed matters. If you see a competitor scaling their landing page production, you know you need to match or beat their pace.
What I Still Do Manually
The automation handles the data collection. I handle the interpretation. Claude Code can tell me that a competitor changed their homepage H1 from "Marketing Automation for SMBs" to "AI-Powered Marketing for Growth Teams." It can't tell me whether that's a smart repositioning or a panic move.
Same with pricing changes. The system will flag a 20% price increase, but it won't tell me whether that increase is sustainable or whether they're about to lose half their customers. That kind of analysis still requires human judgment and context about the market.
I also manually review any major product launches or messaging overhauls before including them in client reports. The last thing I want is to alarm a client about a competitor move that turns out to be a test or a rollback.
How to Build This for Your Business
If you want to set up something similar, here's the sequence I'd recommend:
- List the 5–10 competitors you actually care about tracking — more than that and the signal-to-noise ratio breaks down
- Identify the 3–4 pages on each competitor site that matter most (homepage, pricing, about, and one product/service page)
- Build a basic scraper in Claude Code using Puppeteer or a similar tool — start with just extracting and saving the HTML
- Add diff detection so you can compare the current scrape to the last one and calculate a change percentage
- Set a threshold for what counts as a "meaningful" change and only flag those for review
- Schedule the script to run daily and pipe the output into Slack, email, or a shared doc
The first working version will be rough. That's fine. You'll refine it over time as you see what kinds of changes actually matter and which ones are noise. I've rebuilt parts of this system three times now as I've learned what my clients care about.
If you're already doing web scraping with Claude Code, this is a natural next step. And if you're not, this is a good first project because the ROI is immediate and measurable.
Real Results from Automated Competitor Tracking
Here's what this system has delivered in the six months it's been running:
- Caught 11 pricing changes across 8 competitors, three of which directly informed client pricing strategy
- Flagged 4 major product launches early enough that clients could prepare competitive responses
- Identified 2 competitors who quietly pivoted their messaging toward AI — both are now on our "watch closely" list
- Saved roughly 120 hours of manual research time (30 hours/month × 4 months, since I built it in February)
The time savings alone justify the build. But the strategic value — catching moves early and being able to advise clients proactively — is worth more than the hours.
For more on how I use Claude Code to automate the rest of my consulting workflow, check out my daily workflow breakdown. And if you're wondering whether this kind of system makes sense for your business, the FAQ page covers most of the common questions.
Competitor analysis doesn't have to be a manual grind. Automate the data collection, keep the human judgment, and spend your time on strategy instead of screenshots.