Introduction to Web and Data Scraping Tutorial

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

本帖隐藏的内容

https://github.com/kjam/python-web-scraping-tutorial

A tutorial-based introduction to web scraping with Python.

Virtual Env

If you'd like to use virtual environments, please follow the following instructions. It is not required for the tutorial but may be helpful.

For more details on virtual environments

If you don't have virtual env wrapper and/or pip:

$ easy_install pip$ pip install virtualenvwrapper

and read the additional instructions here

$ mkvirtualenv scraper_tutorial$ pip install -r requirements.txtLXML and Selenium

You will need both LXML and Selenium to follow this tutorial in it's entirety.

If you are using a Mac, I would highly recommend using Homebrew. It will help make pip install very easy for you to use.

• Homebrew and LXML Installation
• More help on Installing LXML on Mac
• And additional suggestions for LXML on Mac
  

If you are using Windows, it might be worth it to run this within a Linux Virtual Machine. If you are a Windows + Python guru, please follow these installation instructions. I can help as needed but I have not programmed on Windows in more than 5 years.

• Installing Selenium on Windows
• Installing LXML on Windows
  

Please reach out to me if you have any questions on getting the initial requirements set up. Thanks!

Firefox Web Browser

Firefox comes as the default web driver for Selenium. To use Selenium easily, please download and install Firefox.

Using PIP

If you have never used PIP before you will need to sudo easy_install pip or brew install pip. PIP is a python package manager and it's really super so I highly advise using it!

Questions?

/msg kjam on freenode or @kjam on twitter

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：introduction troduction Tutorial intro tutor

1. from bs4 import BeautifulSoup
   
2. 
   
3. # Let's grab the simple page source.
   
4. simple_page = open('data/simple.html').read()
   
5. 
   
6. # Let's open it with BS so we can iterate over the family tree.
   
7. simple_soup = BeautifulSoup(simple_page)
   
8. 
   
9. # Let's highlight our current element and return it so we can play around!
   
10. current_elem = simple_soup.findAll('div', {'class': 'navblock'})[1]

复制代码

1. import json
   
2. from urllib import urlopen
   
3. 
   
4. ip_info = urlopen('http://freegeoip.net/json/').read()
   
5. 
   
6. my_ip = json.loads(ip_info)
   
7. 
   
8. print "I think you're at: %f lat, %f long and in %s" % (
   
9.     my_ip.get('latitude'), my_ip.get('longitude'), my_ip.get('city'))

复制代码

1. from selenium import webdriver
   
2. from selenium.webdriver.common.keys import Keys
   
3. from selenium.webdriver.common.action_chains import ActionChains
   
4. from selenium.webdriver.common.by import By
   
5. from selenium.webdriver.support.ui import WebDriverWait
   
6. from selenium.webdriver.support import expected_conditions as EC
   
7. from time import sleep
   
8. 
   
9. MY_EMAIL = ''
   
10. MY_PASSWORD = ''
    
11. MY_PROFILE_NAME = ''
    
12. 
    
13. browser = webdriver.Firefox()
    
14. browser.get('http://netflix.com')
    
15. browser.find_element_by_link_text('Sign In').click()
    
16. email = browser.find_element_by_css_selector('input#email')
    
17. email.send_keys(MY_EMAIL)
    
18. pw = browser.find_element_by_css_selector('input#password')
    
19. pw.send_keys(MY_PASSWORD, Keys.RETURN)
    
20. browser.implicitly_wait(10)  # seconds
    
21. browser.find_element_by_link_text(MY_PROFILE_NAME).click()
    
22. browser.maximize_window()
    
23. rows = browser.find_elements_by_css_selector('div.mrow')
    
24. 
    
25. for r in rows:
    
26.     if 'Top Picks' in r.text:
    
27.         top_pix = r
    
28.         break
    
29. 
    
30. movie_recs = top_pix.find_elements_by_css_selector(
    
31.     'div.agMovieSet div.agMovie')
    
32. 
    
33. 
    
34. first_movie = movie_recs[0].location
    
35. scroll_down = ActionChains(browser).move_by_offset(
    
36.     10, first_movie.get('y') - 10)
    
37. scroll_down.perform()
    
38. 
    
39. movie_dict = {}
    
40. 
    
41. for movie in movie_recs:
    
42.     movie_link = movie.find_element_by_css_selector('a.bobbable')
    
43.     try:
    
44.         arrow = top_pix.find_element_by_css_selector('div.next.sliderButton')
    
45.         if arrow.location.get('x') - movie_link.location.get('x') < 80:
    
46.             hover = ActionChains(browser).move_to_element(arrow)
    
47.             hover.perform()
    
48.             sleep(4)
    
49.         hover = ActionChains(browser).move_to_element(movie_link)
    
50.         hover.perform()
    
51.     except Exception, e:
    
52.         print e
    
53.         hover = ActionChains(browser).move_to_element(arrow)
    
54.         hover.perform()
    
55.         sleep(5)
    
56.         move_off_arrow = ActionChains(browser).move_by_offset(-450, -130)
    
57.         move_off_arrow.perform()
    
58.         hover = ActionChains(browser).move_to_element(movie_link)
    
59.         hover.perform()
    
60.     try:
    
61.         movie_info = WebDriverWait(browser, 10).until(
    
62.             EC.element_to_be_clickable((By.ID, 'BobMovie')))
    
63.         title = movie_info.find_element_by_class_name('title').text
    
64.         link = movie_info.find_element_by_class_name(
    
65.             'mdpLink').get_attribute('href')
    
66.         desc = movie_info.find_element_by_class_name(
    
67.             'bobMovieContent').text.split('\n')[0]
    
68.         cast = movie_info.find_element_by_tag_name('dd').text
    
69.         movie_dict[title] = {'link': link, 'title': title,
    
70.                              'desc': desc, 'cast': cast}
    
71.     except:
    
72.         print "taking too long!"
    
73.     scroll_off = ActionChains(browser).move_by_offset(30, -130)
    
74.     scroll_off.perform()
    
75.     sleep(2)

复制代码

Introduction to Web and Data Scraping Tutorial

谢谢分享

kankan

谢谢楼主分享！