I had question related to captcha solving while crawling a site with scrapy.

The site I'm trying to scrape have captcha before reaching the target ( an pdf link ).

We could use 2captcha or other services for solving captchas. I just don't know how to incorporate in scrapy crawler.

I feel the asynchronous nature of scrapy won't allow to wait for the captcha solution to be submitted from solving service. I could be wrong.

If someone have experienced same problem, please give insight. Thanks

If captcha is simple, you can use middleware to handle it. As I remember, it should be called after Httpmiddleware

Yes, there is no simple solution for this. You can try to scrape synchronously if your tasks allow you to scrape slowly.
In the opposite case, you have to split the flow of requests into separate flows, sessions, stop some requests, and retry them to wait for the solution of a captcha

Thanks for Middle ware hint.
It's cloud flair bot detection, followed by re capatch2. So two different requests I guess.

Thanks for the inputs.
Yeah. We could go with synchronous approach.

Also you have to write your own functions to interact with captcha solving services because most of them have libs based on the synchronous requests package

synchronous way is the easiest

Yeah. We'll need to use simple requests, form reuqest.

I checked for example etc on GitHub, so far no luck with it. I'll search scrapy issues if I find something.

Yeah :)

Then just put your requests in a chain(call every request from the callback of the previous request), check every response for a captcha. With this approach, you can even use requests from  within scrapy without any problem

Currently, I start navigation through home page to target, other information is scrapped in that journey. For i.e car name, car year, make etc.

captchas comes for all and last page where pdf download is

So If only I could chain last response with all information in meta tag, I could add pdf link to all that information from solved captchas.

Could you give hint on in general how we make it synchronous? The scrapy requests?

I've given

A chain of scrapy requests

i.e
def get_captcha(self, response):
 yield Request ( url = 2captchaEndpoint, callback = parse_something)

I found a reference link. Thanks sir :)

Sounds like you probably don't need scrapy at all

Actually, we have crawlers running in scrappinghub. So it'll be good if it could be achieved with scrapy

Otherwise it'll be big change to move to apify for puppeteer based automation. It may come to that too, yet in not recent future

Кто-то замерял если ли разница в скорости скрапинга если запускать его из допустим пайчарма или из wsl?

Пайчарм-то при чём