用scrapy爬虫安居客网站，spiders的程序不知道哪里不对，运行不成功。

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

class ajkSpider(CrawlSpider):
    name = "ajkk"
    allowed_domains = ["anjuke.com"]
    start_urls = ['http://hangzhou.anjuke.com/community']

    rules = [
        Rule(LinkExtractor(allow=["/community/p\d+"])),
        Rule(LinkExtractor(allow=["/community/view/\d+"]),callback='parse_item')
    ]

    def replace(self, x):
        x = re.sub("\n", "", x)
        x = re.sub(" ", "", x)
        return x.strip()

    def parse_item(self, response):
        ajkSpiderItem = AjkItem()
        ajkSpiderItem['city'] = ' '.join(response.xpath('//div[@class="select_icon"]/span[1]/text()').extract())
        ajkSpiderItem['name'] = ' '.join(response.xpath('//dl[@class="comm-l-detail float-l"]/dd[1]/text()').extract())
        ajkSpiderItem['average_price'] = ' '.join(response.xpath('//div[@class="comm-cont"]/p/em/text()').extract())


运行结果：


2017-06-08 23:08:32 [scrapy] INFO: Scrapy 1.0.3 started (bot: ajk)
2017-06-08 23:08:32 [scrapy] INFO: Optional features available: ssl, http11
2017-06-08 23:08:32 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'ajk.spiders', 'SPIDER_MODULES': ['ajk.spiders'], 'BOT_NAME': 'ajk'}
2017-06-08 23:08:32 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2017-06-08 23:08:33 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2017-06-08 23:08:33 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2017-06-08 23:08:33 [scrapy] INFO: Enabled item pipelines: AjkPipeline
2017-06-08 23:08:33 [scrapy] INFO: Spider opened
2017-06-08 23:08:33 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-06-08 23:08:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-06-08 23:08:33 [scrapy] DEBUG: Redirecting (301) to <GET http://hangzhou.anjuke.com/community/> from <GET http://hangzhou.anjuke.com/community>
2017-06-08 23:08:33 [scrapy] DEBUG: Redirecting (302) to <GET http://hangzhou.anjuke.com/404/?from=antispam> from <GET http://hangzhou.anjuke.com/community/>
2017-06-08 23:08:33 [scrapy] DEBUG: Crawled (404) <GET http://hangzhou.anjuke.com/404/?from=antispam> (referer: None)
2017-06-08 23:08:33 [scrapy] DEBUG: Ignoring response <404 http://hangzhou.anjuke.com/404/?from=antispam>: HTTP status code is not handled or not allowed
2017-06-08 23:08:33 [scrapy] INFO: Closing spider (finished)
2017-06-08 23:08:33 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 856,
'downloader/request_count': 3,
'downloader/request_method_count/GET': 3,
'downloader/response_bytes': 10391,
'downloader/response_count': 3,
'downloader/response_status_count/301': 1,
'downloader/response_status_count/302': 1,
'downloader/response_status_count/404': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 6, 8, 15, 8, 33, 939000),
'log_count/DEBUG': 5,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 3,
'scheduler/dequeued/memory': 3,
'scheduler/enqueued': 3,
'scheduler/enqueued/memory': 3,
'start_time': datetime.datetime(2017, 6, 8, 15, 8, 33, 17000)}
2017-06-08 23:08:33 [scrapy] INFO: Spider closed (finished)

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Spiders Spider scrapy CRAP ERS replace return 程序 网站