[tlug] frameworks to automate login to a website/scraping of values: selenium

[Christian Horn <chorn@example.com>]

Hoi Tlug,

happy new spring!

Tepco provides https://www.app.kurashi.tepco.co.jp/ for login, one can
then confirm the consumed electrical power.
I had that working from scripts one or 2 years ago, until the website
got reworked.

Tepco offers APIs to confirm how much power they overall provide at the
moment, but the consumption of individual persons/contracts.

The scheme for login is quite simple:
one enters user ans password, clicks a button, and the data becomes
available.
I fiddled now for a while with https://www.selenium.dev , but to no 
avail.  I got python code together which uses selenium to open the
browser, open the website, enter user/password and click the button,
and then I get this:

Access Denied
You don't have permission to access "http://epauth.tepco.co.jp/u/login?"; on this server.
Reference #18.4f332d17.1677142161.8c0fcc2e 

- I have code for both Firefox and Opera, both show this behaviour.
  If someone wants to give it a try: [1].  The code also has details
  on the few steps to install selenium, and the Firefox-driver.
- I suspected that the selenium controlled browser would send a
  user-agent in the request which the website denies, but opening
  the firefox inspector (<strg>+<shift>+<i>), and then looking at
  the network tab and the single requests, I see that the normal
  Firefox user-agent referrer is used.
- I wondered if I need to extra 'allow' cookies in selenium, 
  but that does not apply: the firefox-inspector shows for the
  selenium controlled session the same cookies as for the 
  manual Firefox session - where the login works.
- I wondered about traffic sniffing.. but now with https one
  won't see much.  So I just looked around in Firefox inspector,
  but did not see anything obvious.

[1] https://fluxcoil.net/files/tmp/selenium_tepco_query_firefox.py

Has anybody further debug ideas?
Or good experiences with other frameworks for the same purpose?

cheers,
Christian

Attachment: signature.asc
Description: PGP signature