問題描述

再編寫爬蟲的時候，總是跑了一段時間（不會超過12個小時）就會被OOM掉。很是無奈！！！根據官方的文檔， 使用這個prefs()但是實在找不出問題的所在。

Live ReferencesHtmlResponse 42 oldest: 753s agoMySuteSpider1 oldest: 2964s agoRequest 32412 oldest: 2920s agoSelector 42 oldest: 751s agoTripItem 37 oldest: 751s ago

爬蟲的處理是獲取所有頁面的a標簽的鏈接：

#獲取域名的后綴def get_domain_suffix(domain): if ’com’ in tldextract.extract(domain).suffix:return True return False#拼接域名。只存主域名def save_domain(domain): domain_name = tldextract.extract(domain).domain suffix_name = tldextract.extract(domain).suffix return domain_name + ’.’ + suffix_name#獲取域名ipdef get_domain_ip(domain): try:ip = socket.gethostbyname(domain)return ip except:return ’114.114.114.114’# 獲取域名所在的國家def get_domain_ct_iso(ip): GEO = geoip2.database.Reader(’/var/test/geodb/GeoLite2-City.mmdb’) r = GEO.city(ip) return r.country.iso_codeclass MyDomainSpider(scrapy.Spider): name = ’my_domain’ start_urls = [’http://xxx.com ] def parse_items(self, response):item = TripItem()for url in response.xpath(’//a/@href’).extract(): if url.startswith(’http’): domain = urlparse.urlparse(url).netloc if get_domain_tw(domain) or get_domain_ct_iso(get_domain_ip(domain)) == ’US’:item[’domain’] = save_domain(domain)item[’ip’] = get_domain_ip(domain)item[’datetime’] = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')yield item def parse(self, response):for url in response.xpath(’//a/@href’).extract(): if url.startswith(’http’):domain = urlparse.urlparse(url).netlocif get_domain_tw(domain) or get_domain_ct_iso(get_domain_ip(domain)) == ’US’: yield scrapy.Request(url, callback=self.parse_items)

請指教一下謝謝

問題解答

yield item 是不是得落地，存文件或者db，不然一直存內存了