U best question.. I know Python has the module called requests that you can import. My question is always how do people actually find the correct url to post the request?
What happens if the response name contains symbols that python recognizes as an expression? For example, the login form has two inputs, the email field which is called admin_user[email] and the password field is called admin_user[password]. This is not acceptable in putting it into the dictionary. Any help would be great!
this tutorial was awesome… exactly what I needed. So I am logged on successfully and now I want to make another request with a form on the page that triggers a file download. How do I get the response file saved?
If I run the download manually chrome shows me that the post request carries the following Response Headers:
Nice tutorial man! U are really good at explaining! But i have one question, the c.get(url) doesn't work with mine, do you know what the problem is? (sorry for bad english) 🙂
Hey, nice tutorial thought. I'm having a problem with the login button integrated on the website. Its name is "continue" which seems to be reserved by python itself. Any idea how i could fix that?
I have the same code as you do but, it prints a django error page. Do you know why? ——————————— import requests with requests.Session() as c: url = 'http://www.noobmovies.com/accounts/login/?next=/' USERNAME = 'bob' PASSWORD = 'booob' c.get(url) login_data = dict(username=USERNAME, password=PASSWORD, next='/') c.post(url, data=login_data, headers={"Referer": "http://www.noobmovies.com/"}) page = c.get('http://www.noobmovies.com/user/profile/0/') print (page.content) ———————————– TypeError at /user/profile/0/ int() argument must be a string or a number, not 'SimpleLazyObject' Request Method: GET Request URL: http://www.noobmovies.com/user/profile/0/ Django Version: 1.7.7 Exception Type: TypeError Exception Value: int() argument must be a string or a number, not 'SimpleLazyObject' Exception Location: /home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages/django/db/models/fields/__init__.py in get_prep_value, line 915 Python Executable: /usr/bin/python Python Version: 2.7.6 Python Path: ['/home/chrshawkes/webapps/movies/myproject', '/home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/home/chrshawkes/webapps/myproject', '/home/chrshawkes/webapps/movies/myproject/myproject'] Server time: Sun, 12 Jul 2015 19:16:36 -0400
Thanks for sharing this. Your post is definitely a good addition to the requests website. I tried it on 'https://login.comcast.net/login' and it works. Note that, I did not have to use headers in the login_data dict. Also I used wakari.io's ipython notebook for testing which is really easy and has a lot of python libraries built-in.
I just gave the data as my cookie and I simply omitted the username and password.. Yet am getting the post response as 200. How is this possible when I dont give the user name and password?
Nice tutorial, I have a website from where we download reports (excel file) once a month. At the moment this process is manual and hence cumbersome. Is it possible using python to login and download this file every month automatically ? Can you direct me to right direction if it is possible ?
Why are you using such a high resolution for a screencast? Your videos are generally unusable since I would need to view it in atleast 720p along with a high res monitor to catch the code (for it to be legible). I believe you're cutting out a sizeable enough demographic with lower specced computers. Not to mention that even if I had all that extra bandwidth to kill its sooooooooooo hard to read the code. It's a difference between downloading a 175mb webm or a ~600mb webm.
Would it be too dificult to record your videos using a region select tool for your screencaster, or presentation mode (if your IDE supports it).
Thank you!!! The part about the csrftoken was especially helpful, as I was trying to log into a website all day with requests and couldn't get it to work, but that's what I was missing (not csrftoken specifically, but another type of info that needed to be sent in the POST request).
Hello Chris, I have been attempting to use this tutorial for a project, however I can't seem to get my script logging in to the site I am interested in. I would very much appreciate any advice or guidance you could give me. I do not have a lot of experience with programing, however I have gotten by with some other scripts in python before. Please let me know if you have a moment to help me out and perhaps we can touch bases through another medium.
hi thank you for the time spent on recording these tutorials. I have a question here. Can I direct to another page in the website to scarp other data after logging in?
I've been trying to do this, however the site I'm trying to acces does not seem to have a csrftoken, I tried with an authenticity_token (which I found with the inspector) but also gave me nothing, one that did not gave me an error was session_id but did not work anyways, any other ideas what I can try?
Great video btw. Just starting with python a bit and thought it would be a good exercise to get some info out of a forum (which you have to access using a password) but so far no succes, I have the code doing what i want except for the log in 😛
Hi chris.. im currently beating my head against a wall trying to log into an ASP.NET site. A video tutorial of doing that will be very helpful if you're up for it 🙂 thanks
Thank you. I want to login to m.vk.com , I try too much but failed. How can I login to that website? It has not any csrfmiddlewaretoken , but has some keys and values like that. Thank you so much.
I think I'm close – but I get an error regarding my csrf tokens. Fairly stuck at this point, not seeing much in forums.
Form Data username:MYEMAIL password:MYPASSWORD mfa_input: login_token:LT-xxxxxx-xxxxxxx service: csrf_token:xxxxxx-xxxxxxx form: csrf_token:xxxxxx-xxxxxxx
Error:
Traceback (most recent call last): File "C:.py", line 8, in <module> csrftoken = c.cookies['csrf_token'] File "C:Python27libsite-packagesrequestscookies.py", line 329, in _getitem_ return self._find_no_duplicates(name) File "C:Python27libsite-packagesrequestscookies.py", line 400, in _find_no_duplicates raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path)) KeyError: "name='csrf_token', domain=None, path=None"
Hey Chris, on your website "HipsterCode" on the tutorial series of "web scraping with python" , it was mentioned that after the end of the video series there shall be a quiz. For those who pass that quiz shall receive a certificate. I have finished the series and neither found the quiz nor a way to earn the certificate. Can you please elaborate on that. Thank you very much for your most appreciated efforts to teach people programming.
That was hilarious. 2 times – Let me see if I can make this any bigger. After 15 minutes of Googling, you come back "Yeah, not sure how to make that bigger"
1. I am using session object and able to login successfully to a site post authentication I need to do another POST that returns a 500 Error invalid cookies
having the same issues with another site but I donot get any error messges. I guess cookie authentication is the problem. Any pointers how we can fix it?
2. Also i need to set the user agent for all traffic to mozilla/chrome but when i do session.headers.update({'User-Agent':'my user agent string…'}) only the next subsequesnt request is reflecting and then it reverts back to pytho/cpython user agent sting!
I dont really know much about hacking after so many tries i met Cyberhacking lord who later help me find out my husband has been cheating on me and stealing from my bank account, he had this scheme going for 6 months. He gave me access to his mail,social media account,phone(could see deleted messages) and even track his location, still going to sue to him. Having doubts in your relationship? contact him ([email protected])
I really appreciate the effort you put into trying to explain this without giving up (and now i kinda now what to do) but man you need to organize more. Good vid 🙂
Hey man, really like this video. I have a question tho, when I try to get the post login parameters from a website I throw me so fast into the next site and clears my network tab in the console and fill it with new stuff. Is there anyway to run it as a debugger or save it to a file??
I tried the post method on login into Quora and Twitter, websites don't accept this method anymore I think, I received two HTTP responses one 405 and other 403, one suggests that the method is not allowed the other one suggests that request is forbidden..any workaround, except selenium?
I cant see any explaine even in requests documentations. I can log in and log in authenticate user and i want to get that user authenticated values. Like username, email etc.
Hi Chris, thank you very much for your tutorial. I can't get to login on this ssl2 website. https://manybooks.net/. I would like to download all the books to use them in Machine learning but I can't login!! Any advice?
Great tutorial, quick question though: Couldn't you just print page.ok or page.status_code to verify it was successful? The ok method yielding TRUE if it was successful and the status_code method yielding 200 or 300 response if it was successful
great, it works, thanks
U best question.. I know Python has the module called requests that you can import. My question is always how do people actually find the correct url to post the request?
Unity3D and photoshop..ma nigga.
What happens if the response name contains symbols that python recognizes as an expression? For example, the login form has two inputs, the email field which is called admin_user[email] and the password field is called admin_user[password]. This is not acceptable in putting it into the dictionary. Any help would be great!
Cheers! 🙂
Hi Chris,
this tutorial was awesome… exactly what I needed. So I am logged on successfully and now I want to make another request with a form on the page that triggers a file download. How do I get the response file saved?
If I run the download manually chrome shows me that the post request carries the following Response Headers:
Cache-Control:private
Connection:close
Content-Disposition:attachment; filename=invxml.txt;
Content-Type:text/xml;charset=UTF-8
Date:Thu, 16 Apr 2015 23:01:25 GMT
Server:Microsoft-IIS/7.5
Set-Cookie:lngOpt=1; expires=Sat, 16-May-2015 23:01:24 GMT; path=/
Set-Cookie:isCountryID=DE; path=/
Set-Cookie:BLalertCnt=1184; domain=bricklink.com; path=/
Set-Cookie:BLdiscussFlag=; domain=bricklink.com; path=/
Set-Cookie:viewCurrencyID=2; path=/
X-Powered-By:ASP.NET
Can you please help me with getting the response file saved?
Nice tutorial man! U are really good at explaining! But i have one question, the c.get(url) doesn't work with mine, do you know what the problem is?
(sorry for bad english) 🙂
Hey, nice tutorial thought. I'm having a problem with the login button integrated on the website. Its name is "continue" which seems to be reserved by python itself. Any idea how i could fix that?
I have the same code as you do but, it prints a django error page. Do you know why?
———————————
import requests
with requests.Session() as c:
url = 'http://www.noobmovies.com/accounts/login/?next=/'
USERNAME = 'bob'
PASSWORD = 'booob'
c.get(url)
login_data = dict(username=USERNAME, password=PASSWORD, next='/')
c.post(url, data=login_data, headers={"Referer": "http://www.noobmovies.com/"})
page = c.get('http://www.noobmovies.com/user/profile/0/')
print (page.content)
———————————–
TypeError at /user/profile/0/
int() argument must be a string or a number, not 'SimpleLazyObject'
Request Method: GET
Request URL: http://www.noobmovies.com/user/profile/0/
Django Version: 1.7.7
Exception Type: TypeError
Exception Value:
int() argument must be a string or a number, not 'SimpleLazyObject'
Exception Location: /home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages/django/db/models/fields/__init__.py in get_prep_value, line 915
Python Executable: /usr/bin/python
Python Version: 2.7.6
Python Path:
['/home/chrshawkes/webapps/movies/myproject',
'/home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/home/chrshawkes/webapps/myproject',
'/home/chrshawkes/webapps/movies/myproject/myproject']
Server time: Sun, 12 Jul 2015 19:16:36 -0400
Thanks for sharing this. Your post is definitely a good addition to the requests website. I tried it on 'https://login.comcast.net/login' and it works. Note that, I did not have to use headers in the login_data dict. Also I used wakari.io's ipython notebook for testing which is really easy and has a lot of python libraries built-in.
How can I get Requests to open certain classes on my page.
Excellent tutorial brother
I just gave the data as my cookie and I simply omitted the username and password.. Yet am getting the post response as 200. How is this possible when I dont give the user name and password?
!THANK YOU! that was very practical and helpfull
anyway to open that session on your default browser?
ur great..i really wanna learn this programming language
is this intro to hacking using python? lol
Thanks a lot
Nice tutorial, I have a website from where we download reports (excel file) once a month. At the moment this process is manual and hence cumbersome. Is it possible using python to login and download this file every month automatically ? Can you direct me to right direction if it is possible ?
This was very useful… many Thanks
Thank you.
i tried this on fb but it has to many parameters like isd , qsstamp etc..
can anyone guide me for doing this on fb?
how do i change the value of the input on a web page then click a button?
I want to do the same thing on a portal of web server. And its not working..
Can you help??
Why are you using such a high resolution for a screencast? Your videos are generally unusable since I would need to view it in atleast 720p along with a high res monitor to catch the code (for it to be legible). I believe you're cutting out a sizeable enough demographic with lower specced computers. Not to mention that even if I had all that extra bandwidth to kill its sooooooooooo hard to read the code. It's a difference between downloading a 175mb webm or a ~600mb webm.
Would it be too dificult to record your videos using a region select tool for your screencaster, or presentation mode (if your IDE supports it).
what should i do if the website have verification code？
thanks brother for your education videos 🙂
Thank you!!! The part about the csrftoken was especially helpful, as I was trying to log into a website all day with requests and couldn't get it to work, but that's what I was missing (not csrftoken specifically, but another type of info that needed to be sent in the POST request).
Hello Chris, I have been attempting to use this tutorial for a project, however I can't seem to get my script logging in to the site I am interested in. I would very much appreciate any advice or guidance you could give me. I do not have a lot of experience with programing, however I have gotten by with some other scripts in python before. Please let me know if you have a moment to help me out and perhaps we can touch bases through another medium.
Thanks, You the man!
if the site im trying to login to doesnt have a csrfmiddlewaretoken, what do i need to do?
Thanks bro, very useful.
Great video… I use linux. I prefer it over windows because it already has python installed.
could never get the cookie part. pretty ill explained on that part. would have liked it a little more detailed.
thumbs up tho
guy ninja sheet pokemon GO fack your selph
import requests as r
with r.Session() as c:
url = 'https://lagunita.stanford.edu/login'
emailid=' '
passw=' ''
c.get(url)
csrftoken=c.cookies['csrftoken']
login_data=dict(csrfmiddlewaretoken=csrftoken,email=emailid, password=passw)
c.post(url,data=login_data,headers={"Referer":"https://lagunita.stanford.edu/login"})
page=c.get('https://lagunita.stanford.edu/dashboard')
print(page.text)
hey chris why the above code doesn't work
please help me
hi thank you for the time spent on recording these tutorials.
I have a question here. Can I direct to another page in the website to scarp other data after logging in?
thanks for your video!
noob's? It's noobs
I've been trying to do this, however the site I'm trying to acces does not seem to have a csrftoken, I tried with an authenticity_token (which I found with the inspector) but also gave me nothing, one that did not gave me an error was session_id but did not work anyways, any other ideas what I can try?
Great video btw. Just starting with python a bit and thought it would be a good exercise to get some info out of a forum (which you have to access using a password) but so far no succes, I have the code doing what i want except for the log in 😛
Thanks Chris,wow it really help, now i can login on any website that i want.
Hi chris.. im currently beating my head against a wall trying to log into an ASP.NET site. A video tutorial of doing that will be very helpful if you're up for it 🙂 thanks
has anyone come across a website that doesn't have the csrmiddlewaretoken ? This site i'm using only uses SHA-1 encryption.
thank you~
Thank you. I want to login to m.vk.com , I try too much but failed. How can I login to that website? It has not any csrfmiddlewaretoken , but has some keys and values like that. Thank you so much.
stop moaning
Great!
петушня какаято
I think I'm close – but I get an error regarding my csrf tokens. Fairly stuck at this point, not seeing much in forums.
Form Data
username:MYEMAIL
password:MYPASSWORD
mfa_input:
login_token:LT-xxxxxx-xxxxxxx
service:
csrf_token:xxxxxx-xxxxxxx
form:
csrf_token:xxxxxx-xxxxxxx
Error:
Traceback (most recent call last):
File "C:.py", line 8, in <module>
csrftoken = c.cookies['csrf_token']
File "C:Python27libsite-packagesrequestscookies.py", line 329, in _getitem_
return self._find_no_duplicates(name)
File "C:Python27libsite-packagesrequestscookies.py", line 400, in _find_no_duplicates
raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
KeyError: "name='csrf_token', domain=None, path=None"
Code:
import requests
c = requests.Session()
url = 'https://site-I'm-Accessing'
USERNAME = '[email protected]'
PASSWORD = 'Super Secret Password'
c.get(url)
csrftoken = c.cookies['csrf_token']
logintoken = c.cookies['login_token']
login_data = dict(username=USERNAME, password=PASSWORD, login_token=logintoken, csrf_token=csrftoken, next='/')
c.post(url, data=login_data, headers=dict(Referer=url))
page = c.get('https://site-I'm-Accessing')
print(page.content)
KeyError: "name='csrftoken', domain=None, path=None"
Hello Chris. Excellent tutorial. I tried the same process but I am getting response 200 as a return. Need your help regarding this. Thank you.
Great video!!
My problem is the Form Data is different, and it doesnt look like it has a CSRF code, which is suspect is the _xfToken.
Do you have any suggestions for the below form data:
login:XXXX
password:XXXX
remember:1
tos:Y
cookie_check:1
redirect:https://hotcopper.com.au/
_xfToken:
do you can do a 2017 version with python 3.6?
Hey Chris,
on your website "HipsterCode" on the tutorial series of "web scraping with python" , it was mentioned that after the end of the video series there shall be a quiz. For those who pass that quiz shall receive a certificate. I have finished the series and neither found the quiz nor a way to earn the certificate. Can you please elaborate on that.
Thank you very much for your most appreciated efforts to teach people programming.
any idea how to make this work with google oauth?
Thanks for the video, I'm getting the following error: NameError: name 'username' is not defined . Does anyone have a fix ?
What do you do if your usename has a dollar sign in it, like 'zzz$m_password'
Gosh you sound tired as fuck, lol, get some rest dude.
still very helpful, thanks!
Exactly what I need! You deserve more likes!!
Thank you.
That was hilarious. 2 times – Let me see if I can make this any bigger.
After 15 minutes of Googling, you come back "Yeah, not sure how to make that bigger"
Anyway, excellent tutorial.
Great Tutorial!
Could you please help with
1. I am using session object and able to login successfully to a site post authentication I need to do another POST that returns a 500 Error invalid cookies
having the same issues with another site but I donot get any error messges. I guess cookie authentication is the problem. Any pointers how we can fix it?
2. Also i need to set the user agent for all traffic to mozilla/chrome
but when i do session.headers.update({'User-Agent':'my user agent string…'})
only the next subsequesnt request is reflecting and then it reverts back to pytho/cpython user agent sting!
I dont really know much about hacking after so many tries i met Cyberhacking lord who later help me find out my husband has been cheating on me and stealing from my bank account, he had this scheme going for 6 months. He gave me access to his mail,social media account,phone(could see deleted messages) and even track his location, still going to sue to him. Having doubts in your relationship? contact him ([email protected])
lol why does every one make tutorials that are soooooo small???????? do u guys have laswers for eyes? command ++++++++++++++
可不可以说中文，亲
I really appreciate the effort you put into trying to explain this without giving up (and now i kinda now what to do) but man you need to organize more. Good vid 🙂
Hey man, really like this video. I have a question tho, when I try to get the post login parameters from a website I throw me so fast into the next site and clears my network tab in the console and fill it with new stuff. Is there anyway to run it as a debugger or save it to a file??
I tried the post method on login into Quora and Twitter, websites don't accept this method anymore I think, I received two HTTP responses one 405 and other 403, one suggests that the method is not allowed the other one suggests that request is forbidden..any workaround, except selenium?
'Lemme see how to make this bigger' 1 day later 'Alright so I dont know how to make this bigger.' 😂😂
I cant see any explaine even in requests documentations. I can log in and log in authenticate user and i want to get that user authenticated values. Like username, email etc.
Nice tutorial I learned a lot 👍
tutorial starts at 2:57
What should you do if when you do inspect the element, there is no token under form data?
Hello Chris I'm trying your but but i get an response saying couldn't not authenticate you… I'm hoping to access a json from zendesk
Hey is there any way I can email you or DM you for you can help me with a quick question with requests?
from like import .
Hi Chris, thank you very much for your tutorial. I can't get to login on this ssl2 website. https://manybooks.net/. I would like to download all the books to use them in Machine learning but I can't login!! Any advice?
Great tutorial, congratulations!!!!!!!!!!!!
Thank you
Thank you! Have spent two days with no result until I found your video!
Great tutorial, quick question though:
Couldn't you just print page.ok or page.status_code to verify it was successful?
The ok method yielding TRUE if it was successful and the status_code method yielding 200 or 300 response if it was successful
t
thanks this helped was having issues with cookies
just curious ……. was your machine lagging OR my video lagging ???
Chris, Thank You for this video!!!! You had saved my sanity. Keep up the great work!!!!