![]() ![]() In order to make your requests from web scrapers look as if they came from a real browser: They have a huge database of the combination of headers that are sent by specific versions of a browser on different operating systems and websites. Let’s add these missing headers and make the request look like it came from a real Chrome browser If you are making many requests for web scraping a website, it is a good idea to randomize. Any website could tell that this came from PythonRequests, and may already have measures in place to block such user agents.Īs before lets ignore the headers that start with X- as they are generated by Amazon Load Balancer used by Hatpin, and not from what we sent to the server Although we had set an user agent, the other headers that we sent are different from what the real Chrome browser would have sent. Ignore the X-Amzn-Trace-Id as it is not sent by PythonRequests, instead generated by Amazon Load Balancer used by Hatpin.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |