Ever wished you could automatically track specific topics across your favorite websites without manually checking them every day? In this tutorial, I'll show you how to create a powerful automated workflow using Relay.app that combines web scraping with AI to track and organize website content effortlessly.
What we'll build
We'll create a workflow that automatically:
- Checks the TechCrunch homepage daily at 8am Pacific time
- Identifies articles containing "AI" in their titles
- Saves the articles' titles and URLs to a spreadsheet
Demo Video
Step-by-step guide
1. Setting up the trigger
First, we'll create a recurring schedule trigger:
- Add a new recurring schedule trigger
- Set it to run daily at 8am Pacific time
2. Implementing web scraping
Next, we'll add the web scraping functionality:
- Add the "Scrape Text from Website" utility
- Input the target URL (techcrunch.com in this example)
Note: This step will output the raw text content visible on the website.
3. Adding AI analysis
To process the scraped content:
- Add an AI custom prompt step
- Use the following prompt:
"Please analyze the content of this website, the TechCrunch homepage, and output a list of all of the articles that have AI in the title. For each article, include both the title and the URL."Important: Make sure to pass the web content from the previous step as additional context for the AI analysis.
4. Set up the Iterator and the "Add Row to Sheet" steps
To process each article individually:
- Add an iterator step
- Select the list of articles as your data source
- Configure the "Add Row to Sheet" step within the iterator
- Map the title and URL fields to your spreadsheet columns
Testing the workflow
When you run the workflow, it will:
- Scrape the TechCrunch homepage
- Use AI to analyze the content and identify relevant articles
- Add each matching article to your spreadsheet
Conclusion
This powerful combination of web scraping and AI automation can save hours of manual work and help you stay on top of important content in your field. The workflow we've built is just one example - you can adapt this approach to track any topic across various websites and store the data in your preferred format.
Ready to build your own automated web scraping workflow? Head over to Relay.app and try recreating this tutorial. Don't forget to experiment with different websites and search criteria to suit your specific needs.
