85 thoughts on “Python Requests – 2. Using Python Requests to Log Into a Website

  1. U best question.. I know Python has the module called requests that you can import. My question is always how do people actually find the correct url to post the request?

  2. What happens if the response name contains symbols that python recognizes as an expression? For example, the login form has two inputs, the email field which is called admin_user[email] and the password field is called admin_user[password]. This is not acceptable in putting it into the dictionary. Any help would be great!

    Cheers! 🙂 

  3. Hi Chris,

    this tutorial was awesome… exactly what I needed. So I am logged on successfully and now I want to make another request with a form on the page that triggers a file download. How do I get the response file saved? 

    If I run the download manually chrome shows me that the post request carries the following Response Headers:

    Cache-Control:private
    Connection:close
    Content-Disposition:attachment; filename=invxml.txt;
    Content-Type:text/xml;charset=UTF-8
    Date:Thu, 16 Apr 2015 23:01:25 GMT
    Server:Microsoft-IIS/7.5
    Set-Cookie:lngOpt=1; expires=Sat, 16-May-2015 23:01:24 GMT; path=/
    Set-Cookie:isCountryID=DE; path=/
    Set-Cookie:BLalertCnt=1184; domain=bricklink.com; path=/
    Set-Cookie:BLdiscussFlag=; domain=bricklink.com; path=/
    Set-Cookie:viewCurrencyID=2; path=/
    X-Powered-By:ASP.NET

    Can you please help me with getting the response file saved?

  4. Nice tutorial man! U are really good at explaining! But i have one question, the c.get(url) doesn't work with mine, do you know what the problem is? 
    (sorry for bad english)   🙂

  5. Hey, nice tutorial thought. I'm having a problem with the login button integrated on the website. Its name is "continue" which seems to be reserved by python itself. Any idea how i could fix that?

  6. I have the same code as you do but, it prints a django error page. Do you know why?
    ———————————
    import requests
    with requests.Session() as c:
        url = 'http://www.noobmovies.com/accounts/login/?next=/'
        USERNAME = 'bob'
        PASSWORD = 'booob'
        c.get(url)
        login_data = dict(username=USERNAME, password=PASSWORD, next='/')
        c.post(url, data=login_data, headers={"Referer": "http://www.noobmovies.com/"})
        page = c.get('http://www.noobmovies.com/user/profile/0/')
        print (page.content)
    ———————————–
    TypeError at /user/profile/0/
    int() argument must be a string or a number, not 'SimpleLazyObject'
    Request Method: GET
    Request URL: http://www.noobmovies.com/user/profile/0/
    Django Version: 1.7.7
    Exception Type: TypeError
    Exception Value:
    int() argument must be a string or a number, not 'SimpleLazyObject'
    Exception Location: /home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages/django/db/models/fields/__init__.py in get_prep_value, line 915
    Python Executable: /usr/bin/python
    Python Version: 2.7.6
    Python Path:
    ['/home/chrshawkes/webapps/movies/myproject',
     '/home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages',
     '/usr/lib/python2.7',
     '/usr/lib/python2.7/plat-x86_64-linux-gnu',
     '/usr/lib/python2.7/lib-tk',
     '/usr/lib/python2.7/lib-old',
     '/usr/lib/python2.7/lib-dynload',
     '/usr/local/lib/python2.7/dist-packages',
     '/usr/lib/python2.7/dist-packages',
     '/home/chrshawkes/webapps/myproject',
     '/home/chrshawkes/webapps/movies/myproject/myproject']
    Server time: Sun, 12 Jul 2015 19:16:36 -0400

  7. Thanks for sharing this. Your post is definitely a good addition to the requests website. I tried it on 'https://login.comcast.net/login' and it works. Note that, I did not have to use headers in the login_data dict. Also I used wakari.io's ipython notebook for testing which is really easy and has a lot of python libraries built-in.

  8. I just gave the data as my cookie and I simply omitted the username and password.. Yet am getting the post response as 200. How is this possible when I dont give the user name and password?

  9. Nice tutorial, I have a website from where we download reports (excel file) once a month. At the moment this process is manual and hence cumbersome. Is it possible using python to login and download this file every month automatically ? Can you direct me to right direction if it is possible ?

  10. i tried this on fb but it has to many parameters like isd , qsstamp etc..
    can anyone guide me for doing this on fb?

  11. Why are you using such a high resolution for a screencast? Your videos are generally unusable since I would need to view it in atleast 720p along with a high res monitor to catch the code (for it to be legible). I believe you're cutting out a sizeable enough demographic with lower specced computers. Not to mention that even if I had all that extra bandwidth to kill its sooooooooooo hard to read the code. It's a difference between downloading a 175mb webm or a ~600mb webm.

    Would it be too dificult to record your videos using a region select tool for your screencaster, or presentation mode (if your IDE supports it).

  12. ᄅ호ᅩᅥᅡᅢ노ᅮᅢᅧᄅ호ᅩᅥᅡᅢᄂ서ᅢᅡᄂᄒᄎᄂ ᄇᄃ퍼ᅢᅭᄉᄂᄐ츄ᅥᅮᅥᅡᅵᅢ says:

    thanks brother for your education videos 🙂

  13. Thank you!!! The part about the csrftoken was especially helpful, as I was trying to log into a website all day with requests and couldn't get it to work, but that's what I was missing (not csrftoken specifically, but another type of info that needed to be sent in the POST request).

  14. Hello Chris, I have been attempting to use this tutorial for a project, however I can't seem to get my script logging in to the site I am interested in. I would very much appreciate any advice or guidance you could give me. I do not have a lot of experience with programing, however I have gotten by with some other scripts in python before. Please let me know if you have a moment to help me out and perhaps we can touch bases through another medium.

  15. if the site im trying to login to doesnt have a csrfmiddlewaretoken, what do i need to do?

  16. could never get the cookie part. pretty ill explained on that part. would have liked it a little more detailed.
    thumbs up tho

  17. import requests as r

    with r.Session() as c:
    url = 'https://lagunita.stanford.edu/login'
    emailid=' '
    passw=' ''
    c.get(url)
    csrftoken=c.cookies['csrftoken']
    login_data=dict(csrfmiddlewaretoken=csrftoken,email=emailid, password=passw)
    c.post(url,data=login_data,headers={"Referer":"https://lagunita.stanford.edu/login"})
    page=c.get('https://lagunita.stanford.edu/dashboard')
    print(page.text)

    hey chris why the above code doesn't work
    please help me

  18. hi thank you for the time spent on recording these tutorials.
    I have a question here. Can I direct to another page in the website to scarp other data after logging in?

  19. I've been trying to do this, however the site I'm trying to acces does not seem to have a csrftoken, I tried with an authenticity_token (which I found with the inspector) but also gave me nothing, one that did not gave me an error was session_id but did not work anyways, any other ideas what I can try?

    Great video btw. Just starting with python a bit and thought it would be a good exercise to get some info out of a forum (which you have to access using a password) but so far no succes, I have the code doing what i want except for the log in 😛

  20. Hi chris.. im currently beating my head against a wall trying to log into an ASP.NET site. A video tutorial of doing that will be very helpful if you're up for it 🙂 thanks

  21. has anyone come across a website that doesn't have the csrmiddlewaretoken ? This site i'm using only uses SHA-1 encryption.

  22. Thank you. I want to login to m.vk.com , I try too much but failed. How can I login to that website? It has not any csrfmiddlewaretoken , but has some keys and values like that. Thank you so much.

  23. I think I'm close – but I get an error regarding my csrf tokens. Fairly stuck at this point, not seeing much in forums.

    Form Data
    username:MYEMAIL
    password:MYPASSWORD
    mfa_input:
    login_token:LT-xxxxxx-xxxxxxx
    service:
    csrf_token:xxxxxx-xxxxxxx
    form:
    csrf_token:xxxxxx-xxxxxxx

    Error:

    Traceback (most recent call last):
    File "C:.py", line 8, in <module>
    csrftoken = c.cookies['csrf_token']
    File "C:Python27libsite-packagesrequestscookies.py", line 329, in _getitem_
    return self._find_no_duplicates(name)
    File "C:Python27libsite-packagesrequestscookies.py", line 400, in _find_no_duplicates
    raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
    KeyError: "name='csrf_token', domain=None, path=None"

    Code:

    import requests

    c = requests.Session()
    url = 'https://site-I'm-Accessing'
    USERNAME = '[email protected]'
    PASSWORD = 'Super Secret Password'
    c.get(url)
    csrftoken = c.cookies['csrf_token']
    logintoken = c.cookies['login_token']
    login_data = dict(username=USERNAME, password=PASSWORD, login_token=logintoken, csrf_token=csrftoken, next='/')
    c.post(url, data=login_data, headers=dict(Referer=url))
    page = c.get('https://site-I'm-Accessing')
    print(page.content)

  24. Hello Chris. Excellent tutorial. I tried the same process but I am getting response 200 as a return. Need your help regarding this. Thank you.

  25. Great video!!
    My problem is the Form Data is different, and it doesnt look like it has a CSRF code, which is suspect is the _xfToken.

    Do you have any suggestions for the below form data:

    login:XXXX
    password:XXXX
    remember:1
    tos:Y
    cookie_check:1
    redirect:https://hotcopper.com.au/
    _xfToken:

  26. Hey Chris,
    on your website "HipsterCode" on the tutorial series of "web scraping with python" , it was mentioned that after the end of the video series there shall be a quiz. For those who pass that quiz shall receive a certificate. I have finished the series and neither found the quiz nor a way to earn the certificate. Can you please elaborate on that.
    Thank you very much for your most appreciated efforts to teach people programming.

  27. Thanks for the video, I'm getting the following error: NameError: name 'username' is not defined . Does anyone have a fix ?

  28. That was hilarious. 2 times – Let me see if I can make this any bigger.
    After 15 minutes of Googling, you come back "Yeah, not sure how to make that bigger"

    Anyway, excellent tutorial.

  29. Great Tutorial!
    Could you please help with

    1. I am using session object and able to login successfully to a site post authentication I need to do another POST that returns a 500 Error invalid cookies

    having the same issues with another site but I donot get any error messges. I guess cookie authentication is the problem. Any pointers how we can fix it?

    2. Also i need to set the user agent for all traffic to mozilla/chrome
    but when i do session.headers.update({'User-Agent':'my user agent string…'})
    only the next subsequesnt request is reflecting and then it reverts back to pytho/cpython user agent sting!

  30. I dont really know much about hacking after so many tries i met Cyberhacking lord who later help me find out my husband has been cheating on me and stealing from my bank account, he had this scheme going for 6 months. He gave me access to his mail,social media account,phone(could see deleted messages) and even track his location, still going to sue to him. Having doubts in your relationship? contact him ([email protected])

  31. lol why does every one make tutorials that are soooooo small???????? do u guys have laswers for eyes? command ++++++++++++++

  32. I really appreciate the effort you put into trying to explain this without giving up (and now i kinda now what to do) but man you need to organize more. Good vid 🙂

  33. Hey man, really like this video. I have a question tho, when I try to get the post login parameters from a website I throw me so fast into the next site and clears my network tab in the console and fill it with new stuff. Is there anyway to run it as a debugger or save it to a file??

  34. I tried the post method on login into Quora and Twitter, websites don't accept this method anymore I think, I received two HTTP responses one 405 and other 403, one suggests that the method is not allowed the other one suggests that request is forbidden..any workaround, except selenium?

  35. I cant see any explaine even in requests documentations. I can log in and log in authenticate user and i want to get that user authenticated values. Like username, email etc.

  36. Hello Chris I'm trying your but but i get an response saying couldn't not authenticate you… I'm hoping to access a json from zendesk

  37. Hi Chris, thank you very much for your tutorial. I can't get to login on this ssl2 website. https://manybooks.net/. I would like to download all the books to use them in Machine learning but I can't login!! Any advice?

  38. Great tutorial, quick question though:
    Couldn't you just print page.ok or page.status_code to verify it was successful?
    The ok method yielding TRUE if it was successful and the status_code method yielding 200 or 300 response if it was successful

Leave a Reply

Your email address will not be published. Required fields are marked *