Scrapy - python spider

Scrapy 是一个 python 下面功能丰富、使用快捷方便的爬虫框架。

    $ pip install scrapy

     $ cat > myspider.py <<EOF
    import scrapy

    class BlogSpider(scrapy.Spider):
        name = 'blogspider'
        start_urls = ['https://blog.scrapinghub.com']

        def parse(self, response):
            for url in response.css('ul li a::attr("href")').re('.*/category/.*'):
                yield scrapy.Request(response.urljoin(url), self.parse_titles)

        def parse_titles(self, response):
            for post_title in response.css('div.entries > ul > li a::text').extract():
                yield {'title': post_title}
 EOF

$ scrapy runspider myspider.py