[GitHub]Python Web Scraping - Second Edition

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Python Web Scraping - Second Edition

本帖隐藏的内容

Python-Web-Scraping-Second-Edition-master.zip (61.33 KB)

2017-7-7 21:54:57 上传

This is the code repository for Python Web Scraping - Second Edition, published by Packt. It contains all the supporting project files necessary to work through the book from start to finish.

About the Book

The internet contains the most useful set of data ever assembled, largely publicly accessible for free. However, this data is not easily reusable. It is embedded within the structure and style of websites and needs to be carefully extracted. Web scraping is becoming increasingly useful as a means to gather and make sense of the wealth of information available online.

This book is the ultimate guide to using the latest features of Python 3.x to scrape data from websites. In the early chapters, you’ll see how to extract data from static web pages. You’ll learn to use caching with databases and files to save time and manage the load on servers. After covering the basics, you’ll get hands-on practice in building a more sophisticated crawler using browsers, crawlers, and concurrent scrapers.

Instructions and Navigation

All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter02.

The code will look like the following:

from urllib.request import urlopenfrom urllib.error import URLErrorurl = 'http://example.webscraping.com'try: html = urlopen(url).read()except urllib2.URLError as e: html = None

To help illustrate the crawling examples we have created a sample website at http://example.webscraping.com. The source code used to generate this website is available at http://bitbucket.org/WebScrapingWithPython/website, which includes instructions how to host the website yourself if you prefer. We decided to build a custom website for the examples instead of scraping live websites so we have full control over the environment. This provides us stability - live websites are updated more often than books and by the time you try a scraping example it may no longer work. Also a custom website allows us to craft examples that illustrate specific skills and avoid distractions. Finally a live website might not appreciate us using them to learn about web scraping and might then block our scrapers. Using our own custom website avoids these risks, however the skills learnt in these examples can certainly still be applied to live websites.

Related Products

• Web Scraping with Python

• Flask: Building Python Web Services

• Building RESTful Python Web Services

  
  

Suggestions and Feedback

Click here if you have any feedback or suggestions.

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Edition python dition editio Second

1. import re
   
2. import urllib.request
   
3. from urllib import robotparser
   
4. from urllib.parse import urljoin
   
5. from urllib.error import URLError, HTTPError, ContentTooShortError
   
6. from chp1.throttle import Throttle
   
7. 
   
8. 
   
9. def download(url, user_agent='wswp', num_retries=2, charset='utf-8', proxy=None):
   
10.     """ Download a given URL and return the page content
    
11.         args:
    
12.             url (str): URL
    
13.         kwargs:
    
14.             user_agent (str): user agent (default: wswp)
    
15.             charset (str): charset if website does not include one in headers
    
16.             proxy (str): proxy url, ex 'http://IP' (default: None)
    
17.             num_retries (int): number of retries if a 5xx error is seen (default: 2)
    
18.     """
    
19.     print('Downloading:', url)
    
20.     request = urllib.request.Request(url)
    
21.     request.add_header('User-agent', user_agent)
    
22.     try:
    
23.         if proxy:
    
24.             proxy_support = urllib.request.ProxyHandler({'http': proxy})
    
25.             opener = urllib.request.build_opener(proxy_support)
    
26.             urllib.request.install_opener(opener)
    
27.         resp = urllib.request.urlopen(request)
    
28.         cs = resp.headers.get_content_charset()
    
29.         if not cs:
    
30.             cs = charset
    
31.         html = resp.read().decode(cs)
    
32.     except (URLError, HTTPError, ContentTooShortError) as e:
    
33.         print('Download error:', e.reason)
    
34.         html = None
    
35.         if num_retries > 0:
    
36.             if hasattr(e, 'code') and 500 <= e.code < 600:
    
37.                 # recursively retry 5xx HTTP errors
    
38.                 return download(url, num_retries - 1)
    
39.     return html
    
40. 
    
41. 
    
42. def get_robots_parser(robots_url):
    
43.     " Return the robots parser object using the robots_url "
    
44.     rp = robotparser.RobotFileParser()
    
45.     rp.set_url(robots_url)
    
46.     rp.read()
    
47.     return rp
    
48. 
    
49. 
    
50. def get_links(html):
    
51.     " Return a list of links (using simple regex matching) from the html content "
    
52.     # a regular expression to extract all links from the webpage
    
53.     webpage_regex = re.compile("""<a[^>]+href=["'](.*?)["']""", re.IGNORECASE)
    
54.     # list of all links from the webpage
    
55.     return webpage_regex.findall(html)
    
56. 
    
57. 
    
58. def link_crawler(start_url, link_regex, robots_url=None, user_agent='wswp',
    
59.                  proxy=None, delay=3, max_depth=4):
    
60.     """ Crawl from the given start URL following links matched by link_regex. In the current
    
61.         implementation, we do not actually scrapy any information.
    
62. 
    
63.         args:
    
64.             start_url (str): web site to start crawl
    
65.             link_regex (str): regex to match for links
    
66.         kwargs:
    
67.             robots_url (str): url of the site's robots.txt (default: start_url + /robots.txt)
    
68.             user_agent (str): user agent (default: wswp)
    
69.             proxy (str): proxy url, ex 'http://IP' (default: None)
    
70.             delay (int): seconds to throttle between requests to one domain (default: 3)
    
71.             max_depth (int): maximum crawl depth (to avoid traps) (default: 4)
    
72.     """
    
73.     crawl_queue = [start_url]
    
74.     # keep track which URL's have seen before
    
75.     seen = {}
    
76.     if not robots_url:
    
77.         robots_url = '{}/robots.txt'.format(start_url)
    
78.     rp = get_robots_parser(robots_url)
    
79.     throttle = Throttle(delay)
    
80.     while crawl_queue:
    
81.         url = crawl_queue.pop()
    
82.         # check url passes robots.txt restrictions
    
83.         if rp.can_fetch(user_agent, url):
    
84.             depth = seen.get(url, 0)
    
85.             if depth == max_depth:
    
86.                 print('Skipping %s due to depth' % url)
    
87.                 continue
    
88.             throttle.wait(url)
    
89.             html = download(url, user_agent=user_agent, proxy=proxy)
    
90.             if not html:
    
91.                 continue
    
92.             # TODO: add actual data scraping here
    
93.             # filter for links matching our regular expression
    
94.             for link in get_links(html):
    
95.                 if re.match(link_regex, link):
    
96.                     abs_link = urljoin(start_url, link)
    
97.                     if abs_link not in seen:
    
98.                         seen[abs_link] = depth + 1
    
99.                         crawl_queue.append(abs_link)
    
100.         else:
     
101.             print('Blocked by robots.txt:', url)

复制代码

1. from bs4 import BeautifulSoup
   
2. from chp1.advanced_link_crawler import download
   
3. 
   
4. url = 'http://example.webscraping.com/view/UnitedKingdom-239'
   
5. html = download(url)
   
6. soup = BeautifulSoup(html, 'html5lib')
   
7. 
   
8. # locate the area row
   
9. tr = soup.find(attrs={'id': 'places_area__row'})
   
10. td = tr.find(attrs={'class': 'w2p_fw'})  # locate the data
    
11. area = td.text  # extract the data
    
12. print(area)

复制代码

1. from bs4 import BeautifulSoup
   
2. from chp1.advanced_link_crawler import download
   
3. 
   
4. broken_html = '<ul class=country><li>Area<li>Population</ul>'
   
5. 
   
6. soup = BeautifulSoup(broken_html, 'html.parser')
   
7. fixed_html = soup.prettify()
   
8. print(fixed_html)
   
9. 
   
10. # still broken, so try a different parser
    
11. 
    
12. soup = BeautifulSoup(broken_html, 'html5lib')
    
13. fixed_html = soup.prettify()
    
14. print(fixed_html)
    
15. 
    
16. # now we can try and extract the data from the html
    
17. 
    
18. ul = soup.find('ul', attrs={'class': 'country'})
    
19. print(ul.find('li'))  # returns just the first match
    
20. print(ul.find_all('li'))  # returns all matches

复制代码

1. from lxml.html import fromstring
   
2. from chp1.advanced_link_crawler import download
   
3. 
   
4. url = 'http://example.webscraping.com/view/UnitedKingdom-239'
   
5. html = download(url)
   
6. 
   
7. tree = fromstring(html)
   
8. td = tree.cssselect('tr#places_area__row > td.w2p_fw')[0]
   
9. area = td.text_content()
   
10. print(area)

复制代码

1. from lxml.html import fromstring
   
2. from chp1.advanced_link_crawler import download
   
3. 
   
4. url = 'http://example.webscraping.com/view/UnitedKingdom-239'
   
5. html = download(url)
   
6. 
   
7. tree = fromstring(html)
   
8. area = tree.xpath('//tr[@id="places_area__row"]/td[@class="w2p_fw"]/text()')[0]
   
9. print(area)

复制代码

谢谢楼主分享

谢谢分享

谢谢楼主分享！