- Terms of service.
- Watch only public pages that a normal visitor can see, and respect each site's terms and robots.txt.
- Do not monitor social accounts by scraping: X's Terms ban crawling or scraping without written consent, and LinkedIn's User Agreement bans scraping.
- For social posts, read them yourself or use the platform's official API under its terms.
- Keep request volume low.
You will need
- A list of competitors' public pages: pricing, changelog, blog or RSS feed
- Firecrawl (free plan: 1,000 credits a month) or the open-source changedetection.io (self-hosted)
- n8n for the weekly summary email (n8n Cloud is paid after a free trial; self-hosting is free)
- A model API key for the summary (billed per token)
You will end up with
- A weekly email listing only pages that changed, with a short summary and a link to each
List the pages to watch
For each competitor, list their pricing page, changelog and blog (an RSS feed is better than scraping if they have one). Leave out social media accounts (see the warning). Check each site's terms and robots.txt; robots.txt tells crawlers what they are asked not to fetch.
Sources: RFC 9309: Robots Exclusion Protocol · X Terms of Service · LinkedIn User Agreement
Track changes with Firecrawl
Scrape each page with Firecrawl's changeTracking format. It compares the page with the last time you scraped it and marks it new, same, changed or removed, with an optional git-style diff. Firecrawl's Monitoring feature can run these checks on a schedule for you. Prefer self-hosting? changedetection.io is an open-source alternative.
Sources: Firecrawl docs: Change Tracking · Firecrawl docs: Monitoring · dgtlmoon/changedetection.io on GitHub (Apache-2.0)
Run it weekly in n8n
Add a Schedule Trigger for once a week, then the Firecrawl node (or an HTTP Request node) for each URL. Keep only results whose status is changed or removed.
Sources: n8n docs: Schedule Trigger node · Firecrawl docs: n8n guide · n8n docs: HTTP Request node
Summarize only what changed
Pass the diffs to a Basic LLM Chain node.
Sources: n8n docs: Basic LLM Chain node
Prompt Here are the changes found this week on competitor pages, as diffs with URLs: {{changes}}. For each page, write one line on what changed (for example a new price, plan or feature) and include the URL. Use only what is in the diff. If a diff is only layout or wording, say 'minor wording change'.Email the digest and check it
Send the summary with the Send Email or Gmail node. Before acting on a change, open the page and confirm it yourself.
Sources: n8n docs: Send Email node
Human checkpoint. A person approves this step before the agent goes further.