Infinite Scroll with Interactive Extractors
If a website requires you to scroll to load more data, you can build an interactive extractor with infinite scroll to extract from it.
An interactive extractor with infinite scroll allows you to record the steps required to scroll on a webpage, and the extractor will combine all screens to extract web data.
Your account will be charged per page paginated using this method. For example, running a paginated extractor with 1 input with 10 pages and screenshots enabled should consume 11 queries (10 pages + 1 screenshot).
Adding Infinite Scroll
To add infinite scroll to your interactive extractor, simply click the Scroll Action.
Recording Infinite Scroll
Once you've added pagination to your interactive extractor, record the interaction steps required to navigate to the next page. For the example below, we're extracting Tweets from https://twitter.com/google. We simply record ourselves scrolling to the bottom of the page to load more data.
For this example, we only want the most recent 6 pages of Tweets. By setting the screens variable to 5, the extractor will scroll to the bottom of the page 5 times and extract the from the entire webpage.
To modify what element is being scrolled to, you can modify the selector of the scroll action for the extractor to scroll to.
Once the pagination is done, you can train the extractor like you would normally. You can modify the number of screens to scroll by modifying the value in the settings for the extractor.
