instal the new for mac Screaming Frog SEO Spider 19.0

Tick any additional APIs you need (PageSpeed Insights is required in our example): 1.2.3 Tab “Export” 1.2.2 “Start Options” TabĮnter your site URL or Choose the crawl mode “List”, then select the file with URLs to crawl and the configuration file.ĭo not forget the configuration file! Otherwise, the crawl uses a default configuration meaning we won’t have the necessary data. For example, you may want it to run every week.

You can set a date and time for the crawl to run, alongside how often this repeats. This is important for having a unique crawl export location later. Once we have the following we need to set up the scheduled crawl (File > Scheduling > Add). This will leave us with the following files: Extraction of HTTP headers (Configuration > Spider > Extraction) – this is required to gather a timestamp that Data Studio can useĭo not forget to save the configuration as a new configuration file (File > Configuration > Save As).Time to First Byte (Configuration > API Access > PageSpeed Insights) – this is my example’s metric, but this works with any data collected.Then I opened the SEO Spider and built a new configuration file with all crawling options deactivated except for the following: (You can gather these from an existing Spider mode crawl).

In my example, I took some URLs from our firm’s blog, created a file list.txt and saved URLs into it. The primary data source is an export of a scheduled SF crawl. Screaming Frog are planning to add this in the future, but until that time, follow this guide. To have it fully automated we also need this data to be appended (not overwritten) to a master Google Sheet.

In order to use crawl data within a time-series in Data Studio, we need a Google Sheet export with a date/time dimension added when the crawl was run. This lets you gather data natively from a crawl, by an API connector, or even scraped by custom extraction – all of which can be reported and compared over time within Data Studio automatically. Looking to integrate your crawl results with Google Data Studio? Here’s a quick guide on how to build fully automated time-series reports with Data Studio, based on scheduled Screaming Frog crawls. View our updated guide on How To Automate Crawl Reports In Data Studio. Update: With the release of version 16 of the SEO Spider, you can now schedule a bespoke export designed for Data Studio integration directly within the tool.

Instal the new for mac Screaming Frog SEO Spider 19.0