Most B2B businesses I work with in Vancouver have the same problem: they know exactly who their ideal customer is, but finding a clean list of those prospects takes weeks of manual research. Rented lead databases are expensive and full of junk. LinkedIn scrapers get accounts banned. Cold calling lists from brokers are stale before you even download them.
Web scraping for lead generation solves this differently. Instead of buying generic contact lists or manually copying data from directories, I build custom scrapers with Claude Code that pull exactly the leads a client needs — from public sources, legally, at scale. Here's how I do it and why it works better than the alternatives.
Why Web Scraping Beats Rented Lead Lists
The problem with purchased databases is they're built for everyone, which means they're optimized for no one. You get a CSV with 10,000 contacts, but only 200 match your actual criteria. The rest are noise. And because those lists are sold to dozens of other companies, your prospects are already getting hammered by competitors using the same data.
Web scraping for lead generation is different because it's custom. You define the exact filters — industry, company size, location, tech stack, job titles — and the scraper finds only those matches. No filler. No duplicates from last year's campaign. Just fresh, targeted data pulled directly from the source.
The other advantage: you own the process. A rented list is a one-time asset that decays the moment you download it. A custom scraper is a repeatable system. Run it monthly and you have an evergreen lead pipeline that updates itself.
What I Actually Scrape and Where
The best sources for B2B lead scraping are public business directories, industry association listings, and review sites. These are places where companies voluntarily publish their contact info because they want to be found. You're not bypassing any access controls or scraping personal data — you're just automating the research a sales rep would do manually.
Here are the three types of sources I use most often:
- Industry-specific directories — trade association member lists, niche marketplaces, certification databases
- Local business registries — chambers of commerce, Better Business Bureau listings, municipal business licenses
- Review and rating platforms — sites like Clutch, G2, Capterra, or Google Business Profiles where companies list their services
For a client selling to law firms, I scraped the BC Law Society's public directory. For a marketing agency targeting restaurants, I built a scraper that pulled every Vancouver restaurant with a Google Business Profile and at least 50 reviews. For a SaaS company going after e-commerce brands, I scraped Shopify's public app store to find stores using specific plugins.
In each case, the data was already public. The scraper just made it usable.
How I Build a Lead Scraper with Claude Code
The technical setup is straightforward if you know what you're doing, but there are a few patterns I follow to keep scrapers reliable and legal.
First, I always start with a site audit. I check the robots.txt file to see what's allowed, I look at the page structure to understand how data is organized, and I test a few manual requests to confirm the site doesn't block automated traffic. If a site explicitly prohibits scraping or requires login to access the data, I don't touch it.
Once I know the source is scrapeable, I build the extraction logic in Claude Code. The script does three things:
- Fetches the HTML from each target page (with rate limiting to avoid overloading the server)
- Parses the HTML to extract the relevant fields — company name, address, phone, email, website, industry tags
- Writes the cleaned data to a CSV or pushes it directly into the client's CRM
The key is making the scraper resilient. Websites change their layouts. Listings get removed. New fields get added. I build in error handling so the script logs issues instead of crashing, and I schedule it to run on a cadence so the client gets fresh data without having to think about it.
Most scrapers I build take 2–4 hours to develop and test. After that, they run autonomously. A client paid me $1,200 to build a scraper that now generates 150 qualified leads per month. That's $8 per lead, forever, with zero ongoing cost beyond server fees.
The Legal and Ethical Boundaries
This is the part people always ask about, and it's important to get right. Web scraping for lead generation is legal in Canada when done correctly, but there are clear lines you can't cross.
The rule I follow: only scrape data that's publicly accessible without authentication. If a site requires login, if the terms of service explicitly ban scraping, or if the data includes personal information that isn't business-related, I don't scrape it.
I also respect robots.txt directives and rate limits. Overloading a server with requests is both unethical and a quick way to get your IP banned. I throttle requests to one every few seconds and rotate user agents to mimic normal browsing behavior.
Finally, I'm careful about how scraped data is used. Under Canadian privacy law (PIPEDA), you can't just add someone to a marketing list without consent, even if their email is public. The scraped data should feed into manual outreach or targeted campaigns where the recipient has a reasonable expectation of being contacted because of their publicly listed business role.
If you want the full legal nuances, consult a lawyer. But the short version: scrape responsibly, use the data ethically, and don't be a jerk about it.
Real Results from Custom Lead Scrapers
I built a scraper for a Vancouver-based consulting firm that wanted to target mid-sized tech companies in Western Canada. They had been buying lists from a data broker for $3,000 per quarter and getting a 2% response rate on cold email because the contacts were outdated and over-targeted.
The custom scraper pulled from three sources: a tech industry directory, Crunchbase's public company profiles, and LinkedIn company pages (just the public info, not behind login walls). It filtered for companies with 50–200 employees, Series A or B funding, and headquarters in BC, Alberta, or Saskatchewan.
First run: 340 companies. All current. All within the exact target profile. The consulting firm ran a personalized outbound campaign and booked 11 discovery calls in the first month. Two of those turned into contracts worth a combined $48,000.
The scraper cost $1,500 to build and takes 20 minutes to re-run each month. Compare that to the $12,000 per year they were spending on rented lists that didn't convert.
When Web Scraping Isn't the Right Move
Not every business benefits from custom scrapers. If your target market is small and well-defined — say, the 30 largest hospital systems in Canada — you don't need automation. You can research that list manually in an afternoon.
Web scraping for lead generation makes sense when you need volume, when your ideal customer profile is specific but the market is large, and when public directories already contain the data you need. If you're targeting consumers instead of businesses, or if your prospects aren't listed in scrapeable sources, you're better off with a different lead gen strategy.
It's also not a replacement for good outreach. A scraper gives you the list. What you do with that list — the quality of your messaging, the relevance of your offer, the timing of your follow-up — still determines whether you actually close deals.
How to Get Started
If you're considering web scraping for lead generation, start by identifying where your ideal prospects are already listed publicly. Look for industry directories, review sites, or association databases where companies in your target market have profiles.
Once you have a source, test the feasibility. Can you access the data without logging in? Does the site's robots.txt allow scraping? Is the page structure consistent enough to parse reliably?
If the answers are yes, you can either build the scraper yourself (if you have the technical skills) or hire someone who knows Claude Code to build it for you. A well-built scraper pays for itself quickly — usually within the first batch of closed deals.
I've built lead scrapers for consulting firms, agencies, SaaS companies, and service providers across a dozen industries. If you want to explore whether this makes sense for your business, reach out and we can talk through your specific use case. And if you have questions about the legal or technical side, check the FAQ page — I've answered most of the common ones there.
Lead generation doesn't have to mean paying for stale lists or spending hours on manual research. The tools exist to automate this. The only question is whether you're going to use them before your competitors do.