Company Check Script — Plan

Purpose

Bulk company tech stack analyzer. Takes a CSV of companies/websites and produces a detailed report of their marketing, sales, and analytics technologies.

Input

File: company_check.csv
Columns: Company ID, Website, Clean Domain, Company Type (optional)

Processing Pipeline (per company, 4 in parallel)

1. Website Loading

DNS check first (fast fail for dead domains)
Selenium headless Chrome with fallback across 4 URL variants: https://, https://www., http://, http://www.
5-second wait for JS rendering

2. Tech Stack Detection (HTML signature matching)

Category	Examples
Chatbots	Drift, Intercom, Zendesk Chat, Tidio, Crisp, Tawk.to, Olark
Marketing Automation (MAP)	HubSpot, Marketo, Pardot, Mailchimp, Klaviyo, ActiveCampaign
ABM Tools	6sense, Demandbase, RollWorks, Terminus, Bombora
CDPs	Segment, Tealium, Treasure Data, mParticle, Rudderstack, Bloomreach
De-anonymization	Clearbit, ZoomInfo, Leadfeeder, RB2B, Warmly, Koala, Snitcher
Payment Processors	Stripe, PayPal, Square, Braintree, Adyen, Paddle, Recurly
Analytics	GA4, GTM, Mixpanel, Hotjar, FullStory, PostHog, Heap, Amplitude, Adobe Analytics

3. Business Classification (LLM)

Sends first 8000 chars of page text to OpenAI GPT
Classifies as: B2C, B2B - Solopreneurs, B2B - SMB, B2B - Enterprise, or Unclear
Skipped if Company Type already provided in input CSV

4. Sitemap Analysis

Discovers sitemaps via robots.txt and standard paths (/sitemap.xml, /sitemap_index.xml)
Falls back to Selenium browser if HTTP requests fail (handles Cloudflare/JS challenges)
Handles sitemap index files (recursive fetch of child sitemaps)
Counts total pages and buckets by last-modified date:
- 0–3 months
- 3–6 months
- 6–9 months
- 9–12 months
- 12+ months

5. Google `site:` Fallback

If no sitemap found, performs a site:domain Google search in a visible Chrome window
Extracts approximate result count from Google's "About X results"
Rate-limited to 5 queries/min, max 180 total
Includes CAPTCHA detection with manual-solve wait

Output

File: company_check_output_YYYYMMDD_HHMMSS.csv
Columns: Company ID, Website, Clean Domain, Company Type, Chatbot, MAP, ABM, CDP, DeAnon, TotalPages, 0_3mo, 3_6mo, 6_9mo, 9_12mo, 12mo_plus, Payment, Analytics

Key Design Details

Parallelism: 4 threads via ThreadPoolExecutor, results written in original row order
Error handling: DNS failures, timeouts, unreachable sites all tracked per-field
Rate limiting: Google searches throttled; sitemap fetches include retry with backoff for 429s
Browser reuse: Single visible Chrome instance for Google searches, auto-relaunches on crash
Test mode: python company_check.py --test domain.com tests sitemap parsing for a single domain

Next Step

Run company_check_linkedin.py using the output file.