Product & Market Web Scraper
Python/BeautifulSoup scraper for structured product data collection — the same techniques later uncovered exposed API keys and insecure API calls on a target site.
Overview
Built a web scraper to collect structured information about sun-shading products from competitor and supplier websites for market analysis.
Technical approach
- Stack: Python with
requestsandBeautifulSoup4 - Parsing: HTML content extracted and structured into typed data records
- Output: Clean datasets exported to CSV and JSON for downstream analysis
Unintended security finding
While applying the same scraping and inspection techniques to a company website, significant security issues were identified:
- Exposed API keys present in client-side JavaScript bundles
- Directly accessible API endpoints without authentication
- Sensitive internal identifiers visible in public network requests
These findings were reported internally through responsible disclosure. The incident highlighted how routine web scraping and browser developer tools can surface serious vulnerabilities that might otherwise go unnoticed.
Key takeaway
Security auditing and web scraping share a toolset. Understanding how a website delivers its content is the first step in both building integrations and identifying exposure.