AR
Size: a a a
AR
S
ИБ
MМ
scrapy_1 | 2020-10-09 15:27:11 [scrapy.core.scraper] ERROR: Error downloading <GET >
scrapy_1 | Traceback (most recent call last):
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
scrapy_1 | result = g.send(result)
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
scrapy_1 | response = yield deferred_from_coro(method(request=request, spider=spider))
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/rotating_proxies/middlewares.py", line 128, in process_request
scrapy_1 | raise CloseSpider("no_proxies")
scrapy_1 | scrapy.exceptions.CloseSpider
scrapy_1 | 2020-10-09 15:27:11 [scrapy.core.scraper] ERROR: Error downloading <GET >
scrapy_1 | Traceback (most recent call last):
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
scrapy_1 | result = g.send(result)
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
scrapy_1 | response = yield deferred_from_coro(method(request=request, spider=spider))
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/rotating_proxies/middlewares.py", line 128, in process_request
scrapy_1 | raise CloseSpider("no_proxies")
scrapy_1 | scrapy.exceptions.CloseSpider
scrapy_1 | 2020-10-09 15:27:11 [scrapy.core.scraper] ERROR: Error downloading <GET >
scrapy_1 | Traceback (most recent call last):
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
scrapy_1 | result = g.send(result)
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
scrapy_1 | response = yield deferred_from_coro(method(request=request, spider=spider))
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/rotating_proxies/middlewares.py", line 128, in process_request
scrapy_1 | raise Clos^C
AR
МС
MМ
AR
MМ
ROTATING_PROXY_CLOSE_SPIDER = True
ROTATING_PROXY_BAN_POLICY = 'clinicalsynopsis.policy.MyPolicy'
DOWNLOADER_MIDDLEWARES = {
'rotating_proxies.middlewares.RotatingProxyMiddleware': 61,
'rotating_proxies.middlewares.BanDetectionMiddleware': 62,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400
}
МС
MМ
ROTATING_PROXY_LIST_PATH = 'proxies.txt'В списке все нормально. При старте он проверяет их
class MyPolicy(BanDetectionPolicy):
def response_is_ban(self, request, response):
if f:=response.xpath('body/pre'):
if f.get()[0:20] == '<pre>\nYour IP addres':
logging.warning("IP banned, rotating")
return True
return False
MМ
MМ
MМ
scrapy_1 | 2020-10-09 15:27:11 [scrapy.core.scraper] ERROR: Error downloading <GET >
scrapy_1 | Traceback (most recent call last):
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
scrapy_1 | result = g.send(result)
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
scrapy_1 | response = yield deferred_from_coro(method(request=request, spider=spider))
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/rotating_proxies/middlewares.py", line 128, in process_request
scrapy_1 | raise CloseSpider("no_proxies")
scrapy_1 | scrapy.exceptions.CloseSpider
scrapy_1 | 2020-10-09 15:27:11 [scrapy.core.scraper] ERROR: Error downloading <GET >
scrapy_1 | Traceback (most recent call last):
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
scrapy_1 | result = g.send(result)
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
scrapy_1 | response = yield deferred_from_coro(method(request=request, spider=spider))
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/rotating_proxies/middlewares.py", line 128, in process_request
scrapy_1 | raise CloseSpider("no_proxies")
scrapy_1 | scrapy.exceptions.CloseSpider
scrapy_1 | 2020-10-09 15:27:11 [scrapy.core.scraper] ERROR: Error downloading <GET >
scrapy_1 | Traceback (most recent call last):
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
scrapy_1 | result = g.send(result)
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 36, in process_request
scrapy_1 | response = yield deferred_from_coro(method(request=request, spider=spider))
scrapy_1 | File "/usr/local/lib/python3.8/site-packages/rotating_proxies/middlewares.py", line 128, in process_request
scrapy_1 | raise Clos^C
MМ
U
МС
MМ
MМ