Size: a a a

2020 July 21

YB

Yaswanth Bangaru in Scrapy
Can I manage in this group only using English?
источник

К

Кирилл in Scrapy
sure you can
источник

YB

Yaswanth Bangaru in Scrapy
Thank you, I have been trying to download CSV files from a website which uses javascript. I basically need to choose from a list of drop-downs and click on a download button. I learnt that I need to use Splash for such tasks along with Scrapy. The problem is I can't figure out a way to click the download button once the drop-downs are selected. I couldn't find any examples for such tasks on the internet and using Splash documentation is pretty overwhelming.  I'm kinda new to Scrapy itself, so far I only used it to scrape tabular data using xpaths. Can someone help me with it or share any open source git repos relating this kind of tasks(I don't want to use Selenium)
источник

ИБ

Иван Батурин... in Scrapy
Andrey Rahmatullin
если уж брать одноразовые акции то, повторюсь, фри тир амазона, он год действует
Я пользовался пару раз гуглом (на двух аккаунтах). Там любые машины можно запускать и не 200 а 300$ и на год. Amazon free-tier кажется ограничен в том, что там можно запускать, только микро инстанс.  На гугле в переделах кредита почти любые сервисы или вообще любые.
источник

AR

Andrey Rahmatullin in Scrapy
Yaswanth Bangaru
Thank you, I have been trying to download CSV files from a website which uses javascript. I basically need to choose from a list of drop-downs and click on a download button. I learnt that I need to use Splash for such tasks along with Scrapy. The problem is I can't figure out a way to click the download button once the drop-downs are selected. I couldn't find any examples for such tasks on the internet and using Splash documentation is pretty overwhelming.  I'm kinda new to Scrapy itself, so far I only used it to scrape tabular data using xpaths. Can someone help me with it or share any open source git repos relating this kind of tasks(I don't want to use Selenium)
источник

YB

Yaswanth Bangaru in Scrapy
Not many examples, I really don't get how to incorporate whatever the documentation says into a Spider class
источник

К

Кирилл in Scrapy
Yaswanth Bangaru
Thank you, I have been trying to download CSV files from a website which uses javascript. I basically need to choose from a list of drop-downs and click on a download button. I learnt that I need to use Splash for such tasks along with Scrapy. The problem is I can't figure out a way to click the download button once the drop-downs are selected. I couldn't find any examples for such tasks on the internet and using Splash documentation is pretty overwhelming.  I'm kinda new to Scrapy itself, so far I only used it to scrape tabular data using xpaths. Can someone help me with it or share any open source git repos relating this kind of tasks(I don't want to use Selenium)
First at all, you should find xhr requests that fetch csv data on a site, and try to recreate them
источник

AR

Andrey Rahmatullin in Scrapy
Yaswanth Bangaru
Not many examples, I really don't get how to incorporate whatever the documentation says into a Spider class
Spiders make requests, the doc explains which requests you need to make
источник

ИБ

Иван Батурин... in Scrapy
Yaswanth Bangaru
Not many examples, I really don't get how to incorporate whatever the documentation says into a Spider class
Maybe a link to webpage you are trying to scrape would help. If it is public and easy to describe what you would like to accomplish.  It can be much easier to explain the direction.
источник

YB

Yaswanth Bangaru in Scrapy
Кирилл
First at all, you should find xhr requests that fetch csv data on a site, and try to recreate them
I did that, there's not even a token, the other form details are easy to modify in the url itself, I thought I could loop through the choices using for loops if I managed to download the CSV file in the first place
источник

YB

Yaswanth Bangaru in Scrapy
Иван Батурин
Maybe a link to webpage you are trying to scrape would help. If it is public and easy to describe what you would like to accomplish.  It can be much easier to explain the direction.
Sure
источник

YB

Yaswanth Bangaru in Scrapy
This is the actual link https://www.smard.de/en/   --> Data Download

This is how the link changes with the filled in form data

https://www.smard.de/en/downloadcenter/download_market_data/5730#!?downloadAttributes={"selectedCategory":1,"selectedSubCategory":2,"selectedRegion":"DE","from":1590962400000,"to":1596232799999,"selectedFileType":"CSV"}
источник

AR

Andrey Rahmatullin in Scrapy
press the button. look at the request. replicate the request in Scrapy.
источник

AR

Andrey Rahmatullin in Scrapy
(that's also what the official doc tells you to do)
источник

YB

Yaswanth Bangaru in Scrapy
You make it sound so easy but I have been struggling with it. I'll go through the docs once again, thank you
источник

AR

Andrey Rahmatullin in Scrapy
have you pressed the button? have you looked at the request?
источник

YB

Yaswanth Bangaru in Scrapy
Yeah, I did
источник

YB

Yaswanth Bangaru in Scrapy
On Firefox the request disappears immediately once the file is download but I just realised on chrome I could see the request. It generates a request form with some moduleIds that I can't get before actually clicking the button I believe
источник

ИБ

Иван Батурин... in Scrapy
источник

ИБ

Иван Батурин... in Scrapy
{"request_form":[{"format":"CSV","moduleIds":[1001224,1004066,1004067,1004068,1001223,1004069,1004071,1004070,1001226,1001228,1001227],"region":"Amprion","timestamp_from":1594418400000,"timestamp_to":1595455199999,"type":"discrete","language":"en"}]}
источник