Grab - Site Scraping Framework¶
Grab is site scraping framework. Grab could be used for:
- website data mining
- work with network API
- automation of actions performed on websites i.e. creation of profile on some site
Grab consists of following parts:
- Grab interface for creating network requests and working with results of these requests. This interface is good for simple scripts where is no need in multithreading.
- Grab::Spider interface which allows to develop complex multithreaded asynchronous site scrapers. This interface has two main benefits:
- It restrict you spider to have clean structure
- It allows to perform multiple concurrent requests without big CPU/memory consumption
Grab User Guide¶
- Введение в Grab
- Установка библиотеки Grab
- Настройка Grab-объекта
- Отладка запросов
- Полный список настроек
- Настройка HTTP-заголовков
- Методы HTTP-запросов
- Прочие возможности
- Кодировка документа
- Работа с кукисами
- Обработка сетевых ошибок, таймауты
- Работа с прокси-серверами
- Работа с ответом
- Технические детали устройства Grab
- Работа с формами
- Работа с DOM-деревом
- Поиск в тексте документа
- Другие расширения
- Cетевые транспорты
- Полезные утилиты
Grab::Spider User Guide¶
TODO:
* Работа с прокси
* Утилиты:
* process_links
* process_next_page
* inc_count/add_item/save_list/render_stats/save_all_lists
* process_object_image
Grab API Reference¶
If you are looking for information on a specific function, class or method, this part of the documentation is for you.
Base Interface
Extensions:
Tools:
Grab is powerful python site scraping framework which allows to build both simple scripts and complex multithreaded website spiders.
Translations
Useful Links
This Page
