ИБ
It seems that you will need to know the list of regions, how to format timestamp and what is modulelds parameters and you can try to send post request and receive csv in response. Probably it can be done without scrapy.
Size: a a a
2020 July 21
ИБ
AR
Yaswanth Bangaru
On Firefox the request disappears immediately once the file is download but I just realised on chrome I could see the request. It generates a request form with some moduleIds that I can't get before actually clicking the button I believe
you need to recreate the moduleIds list based on the result of market_data_configuration.json request in the same way as the page JS does
YB
It seems that you will need to know the list of regions, how to format timestamp and what is modulelds parameters and you can try to send post request and receive csv in response. Probably it can be done without scrapy.
Yes, I collected all the timestamps I need, as they allow 2 year durations it wasn't though
ИБ
https://www.smard.de/app/chart_configuration/market_data_configuration.json
there are seems to be all necessary Id's for request
there are seems to be all necessary Id's for request
YB
It seems that you will need to know the list of regions, how to format timestamp and what is modulelds parameters and you can try to send post request and receive csv in response. Probably it can be done without scrapy.
Without scrapy--> but I wanted to schedule the Scrapy project to run regularly, so I thought Scrapy is a better choice
YB
https://www.smard.de/app/chart_configuration/market_data_configuration.json
there are seems to be all necessary Id's for request
there are seems to be all necessary Id's for request
How did you get this please?
ИБ
I mean there is no real parsing the data from page or clicking the buttons necessary (if there is no some kind of protection by session/cookies). Just make a list of request you need base on regions and id information from the above requests.
AR
Yaswanth Bangaru
How did you get this please?
looked at he XHR request list (also the JS code mentions market params)
ИБ
Yaswanth Bangaru
How did you get this please?
just network monitor (firefox) with persistance on and cache off. These requests done without any parameters. So they just represents all the info which is further used in JS scripts
YB
Thank you so much guys, I'll try it again. I hope I could get back to you if I can't figure out something
ИБ
https://www.smard.de/app/chart_configuration/date_range.json and https://www.smard.de/app/chart_configuration/date_configuration.json can also be useful to understand the limits of data available or something like this. They are made without any request parameters as well.
2020 July 22
MD
Hi everybody
OS
Mamadou DIA
Hi everybody
Hi!
MM
hello i have qustion. i need scrap some information from website, but when i run my code i have result: [] https://pastebin.com/f8KB4pnS
MM
what i do wrong
AR
there is no
a.module_products_list on that page 🤷♂️MM
there is no
a.module_products_list on that page 🤷♂️after my post, i tried replace it on "class="item main_item_wrapper" but result is that same
MM
or my mistake in 'a' sorry i ma noob )