F-droid.org APKs crawler built over scrapy

Crawler application

Crawls a URL following same-domain hyperlinks and saves path and page size in KB into a CSV file.

🚗 🚙 A crawler can get all cars' info from https://autos.yahoo.com.tw/

Web crawler for parsing data from http://140.112.166.97/power/fn3/index.aspx

Supermarket reviews crawler

A simple crawler for fetching pictures from http://gank.io, implemented in golang.

A simple crawler that collects windsurf boards offers from different sites.

Web crawling framework powered by Playwright

代理池

Use Jsoup crawler Japan Anime season information.

Crawler implementation for SearchSECO

Data analysis of Tsinghua Student Community Management and Service website (advice and feedback section)

Automates weekly web scraping using Scrapy, shell scripting, and cron job scheduling to extract relevant articles from The Economist, and update to MongoDB Cloud, with email notifications function.

Using CloudScraper, Requests, API, Thread, Async... for scrape the data

A basic python web crawler to crawl different endpoints from a provided URL.

Pricy is a product crawler and search engine. This crawler helps you to collect products data on the internet. It's easy to customize.

This project is designed to run crawlers and process the results based on the specified output format. It takes command-line arguments to select the crawler and output format.

web crawler