Thank you, I have been trying to download CSV files from a website which uses javascript. I basically need to choose from a list of drop-downs and click on a download button. I learnt that I need to use Splash for such tasks along with Scrapy. The problem is I can't figure out a way to click the download button once the drop-downs are selected. I couldn't find any examples for such tasks on the internet and using Splash documentation is pretty overwhelming. I'm kinda new to Scrapy itself, so far I only used it to scrape tabular data using xpaths. Can someone help me with it or share any open source git repos relating this kind of tasks(I don't want to use Selenium)
если уж брать одноразовые акции то, повторюсь, фри тир амазона, он год действует
Я пользовался пару раз гуглом (на двух аккаунтах). Там любые машины можно запускать и не 200 а 300$ и на год. Amazon free-tier кажется ограничен в том, что там можно запускать, только микро инстанс. На гугле в переделах кредита почти любые сервисы или вообще любые.
Thank you, I have been trying to download CSV files from a website which uses javascript. I basically need to choose from a list of drop-downs and click on a download button. I learnt that I need to use Splash for such tasks along with Scrapy. The problem is I can't figure out a way to click the download button once the drop-downs are selected. I couldn't find any examples for such tasks on the internet and using Splash documentation is pretty overwhelming. I'm kinda new to Scrapy itself, so far I only used it to scrape tabular data using xpaths. Can someone help me with it or share any open source git repos relating this kind of tasks(I don't want to use Selenium)
Thank you, I have been trying to download CSV files from a website which uses javascript. I basically need to choose from a list of drop-downs and click on a download button. I learnt that I need to use Splash for such tasks along with Scrapy. The problem is I can't figure out a way to click the download button once the drop-downs are selected. I couldn't find any examples for such tasks on the internet and using Splash documentation is pretty overwhelming. I'm kinda new to Scrapy itself, so far I only used it to scrape tabular data using xpaths. Can someone help me with it or share any open source git repos relating this kind of tasks(I don't want to use Selenium)
First at all, you should find xhr requests that fetch csv data on a site, and try to recreate them
Not many examples, I really don't get how to incorporate whatever the documentation says into a Spider class
Maybe a link to webpage you are trying to scrape would help. If it is public and easy to describe what you would like to accomplish. It can be much easier to explain the direction.
First at all, you should find xhr requests that fetch csv data on a site, and try to recreate them
I did that, there's not even a token, the other form details are easy to modify in the url itself, I thought I could loop through the choices using for loops if I managed to download the CSV file in the first place
Maybe a link to webpage you are trying to scrape would help. If it is public and easy to describe what you would like to accomplish. It can be much easier to explain the direction.
On Firefox the request disappears immediately once the file is download but I just realised on chrome I could see the request. It generates a request form with some moduleIds that I can't get before actually clicking the button I believe