#PYTHON DOWNLOAD WEBPAGE CODE#

If you print out resp you should see the message Response 200, which means the connection to the webpage was successful (otherwise youвЂ™ll get a different message). This gets stored in a response variable, resp. Similar to the requests package, we can use a session object to get the webpage we need. # Use the object above to connect to needed webpage

#PYTHON DOWNLOAD WEBPAGE CODE#

Now, letвЂ™s use requests_html to run the JavaScript code in order to render the HTML weвЂ™re looking for. This means if we try just scraping the HTML, the JavaScript wonвЂ™t be executed, and thus, we wonвЂ™t see the tags containing the expiration dates. it modifies the HTML of the page dynamically to allow a user to select one of the possible expiration dates. Why the disconnect? The reason why we see option tags when looking at the source code in a browser is that the browser is executing JavaScript code that renders that HTML i.e. However, if we look at the source via a web browser, we can see that there are, indeed, option tags: This is because there are no option tags found in the HTML we scrapped from the webpage above. Running the above code shows us that option_tags is an empty list. To demonstrate, letвЂ™s try doing that to see what happens. We can try using requests with BeautifulSoup, but that wonвЂ™t work quite the way we want. What if we want to get all the possible choices вЂ“ i.e. On this webpage thereвЂ™s a drop-down box allowing us to view data by other expiration dates. If we go to the below site, we can see the option chain information for the earliest upcoming options expiration date for Netflix:

As an example, letвЂ™s look at Netflix (since itвЂ™s well known).

LetвЂ™s say we want to scrape options data for a particular stock.

Python download webpage