数据解析概念

将爬取到的数据中指定的数据进行单独提取。

数据解析通用原理:

在一张页面中,想要解析的数据是存在于相关的html的标签中。

可以先将指定的标签进行定位,然后可以将该标签中展示的数据进行提取。

python 中可以实现数据解析的技术:

bs4

环境安装

pip install bs4
pip install lxml # 或者 conda install -c conda-forge lxml

bs4数据解析的流程

具体解析的操作

在当前目录下新建一个test.html文件,然后将下述内容拷贝到该文件中

<html lang="en">
<head>
      <meta charset="UTF-8" />
      <title>测试bs4</title>
</head>
<body>
      <div>
              <p>百里守约</p>
      </div>
      <div class="song">
              <p>李清照</p>
              <p>王安石</p>
              <p>苏轼</p>
              <p>柳宗元</p>
              <a href="http://www.song.com/" title="赵匡胤" target="_self">
                      <span>this is span</span>
              宋朝是最强大的王朝,不是军队的强大,而是经济很强大,国民都很有钱</a>
              <a href="" class="du">总为浮云能蔽日,长安不见使人愁</a>
              <img src="http://www.baidu.com/meinv.jpg" alt="" />
      </div>
      <div class="tang">
              <ul>
                      <li><a href="http://www.baidu.com" title="qing">清明时节雨纷纷,路上行人欲断魂,借问酒家何处有,牧童遥指杏花村</a></li>
                      <li><a href="http://www.163.com" title="qin">秦时明月汉时关,万里长征人未还,但使龙城飞将在,不教胡马度阴山</a></li>
                      <li><a href="http://www.126.com" alt="qi">岐王宅里寻常见,崔九堂前几度闻,正是江南好风景,落花时节又逢君</a></li>
                      <li><a href="http://www.sina.com" class="du">杜甫</a></li>
                      <li><a href="http://www.dudu.com" class="du">杜牧</a></li>
                      <li><b>杜小月</b></li>
                      <li><i>度蜜月</i></li>
                      <li><a href="http://www.haha.com" id="feng">凤凰台上凤凰游,凤去台空江自流,吴宫花草埋幽径,晋代衣冠成古丘</a></li>
              </ul>
      </div>
</body>
</html>

有了test.html文件后,在练习如下操作

  1. 创建一个BeautifulSoup的工具对象,然后把即将被解析的页面源码数据加载到该对象中

参数1:被解析的页面源码数据
参数2:固定形式的lxml(一种解析器)

from bs4 import BeautifulSoup
fp = open('test.html','r',encoding="UTF-8")
soup = BeautifulSoup(fp,'lxml')
print(soup)
  1. 可以调用BeautifulSoup对象的相关函数和属性进行标签定位和数据提取常用的选择器:class选择器(.class属性值) id选择器(#id的属性值)

    1. 标签定位-方式1:soup.tagName(只可以定位到第一次出现的该标签)
    title_tag = soup.title
    p_tag = soup.p
    print(title_tag)
    print(p_tag)
    
    1. 标签定位-方式2(属性定位):soup.find(tagName,attrName='value')
      注意:find只可以定位满足要求的第一个标签,如果使用class属性值的话,find参数class_
    div_tag = soup.find('div',class_='song')
    print(div_tag)
    
    <div class="song">
    <p>李清照</p>
    <p>王安石</p>
    <p>苏轼</p>
    <p>柳宗元</p>
    <a href="http://www.song.com/" target="_self" title="赵匡胤">
    <span>this is span</span>
                  宋朝是最强大的王朝,不是军队的强大,而是经济很强大,国民都很有钱</a>
    <a class="du" href="">总为浮云能蔽日,长安不见使人愁</a>
    <img alt="" src="http://www.baidu.com/meinv.jpg"/>
    </div>
    
    a_tag = soup.find('a',class_='du')
    print(a_tag)
    
    <a class="du" href="">总为浮云能蔽日,长安不见使人愁</a>
    
    a_tag = soup.find('a',id='feng')
    print(a_tag)
    
    <a href="http://www.haha.com" id="feng">凤凰台上凤凰游,凤去台空江自流,吴宫花草埋幽径,晋代衣冠成古丘</a>
    
    1. 标签定位-方式3(属性定位):soup.find_all(tagName,attrName='value')
    tags = soup.find_all('a',class_='du')
    print(tags)
    
    <bs4.element.ResultSet object at 0x7ffa9290af60>
    
    1. 标签定位-方式4(select 选择器定位):
    tags = soup.select('#feng') #定位到id的属性值为feng对应的所有标签
    tags = soup.select('.du') #定位到class属性值为du对应的所有标签
    
    tags = soup.select('.tang > ul > li > a')
    tags = soup.select('.tang a')
    print(tags)
    

补充:定位到标签内部数据的提取

方式1:提取标签内的文本数据
tag.string:只可以将标签直系的文本内容取出
tag.text:可以将标签内部所有的文本内容取出

tag = soup.find('a',id='feng')
content = tag.string

div_tag = soup.find('div',class_='tang')
content = div_tag.text

方式2:提取标签的属性值 tag['attrName']

img_tag = soup.find('img')
img_src = img_tag['src']
print(img_src)

案例应用:碧血剑文本爬取

import requests
from bs4 import BeautifulSoup
import os

headers = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
}

url = 'https://bixuejian.5000yan.com/'
response = requests.get(url = url, headers = headers)
response.encoding = 'utf-8'
page_text = response.text
# print(page_text)

# with open('bixuejian.html','w',encoding="UTF-8") as fp:
#     fp.write(page_text)

file_path = "D:/demo/bixuejian/"
if not os.path.exists(file_path):
    os.mkdir(file_path)

soup = BeautifulSoup(page_text,'lxml')
a_list = soup.select('.p-2 a')

for a in a_list:
    title = a.string
    detail_url = a['href']
    detail_response = requests.get(url = detail_url, headers=headers)
    detail_response.encoding = 'utf-8'
    detail_response_text = detail_response.text
    
    detail_soup = BeautifulSoup(detail_response_text, 'lxml')
    div_tag = detail_soup.find('div',class_='grap')
    content = div_tag.text
    
    file_name = file_path + title + '.txt'

    with open(file_name, 'w', encoding='utf-8') as fp:
        fp.write(title + '\n' + content)
    print(f"{title} 保存成功!")

Xpath(重点)

环境安装

pip install lxml

xpath解析的编码流程

具体解析的操作

from lxml import etree

# 1.创建一个etree的工具对象,然后把即将被解析的页面源码数据加载到该对象中
tree = etree.parse('test.html')

# 2.调用etree对象的xpath函数然后结合着不用形式的xpath表达式进行标签定位和数据提取
# xpath函数返回的是列表,列表中存储的是满足定位要求的所有标签
# /html/head/title 定位到 html 下面的 head 下面的 title 标签
title_tag = tree.xpath('/html/head/title') # 输出 [<Element title at 0x7f7cc77f7c00>]
# title在页面源码中定位到所有的title标签
title_tag = tree.xpath('//title') # 输出 [<Element title at 0x7f7cc77f7c00>]

# 属性定位
# 定位到所有的div标签
div_tags = tree.xpath('//div') # 输出 [<Element div at 0x7f7cc77f7c40>, <Element div at 0x7f7cc77f7c80>, <Element div at 0x7f7cc77f7940>]

# 定位到class属性值为song的div标签 //tagName[@attrName='value']
div_tag = tree.xpath('//div[@class="song"]')
# 索引定位://tag[index]
# 注意:索引是从1开始的
div_tag = tree.xpath('//div[1]')

# 层级定位
# /表示一个层级  //表示多个层级
a_list = tree.xpath('//div[@class="tang"]/ul/li/a')
a_list = tree.xpath('//div[@class="tang"]//a')

# 数据提取
# 1.提取标签中的文本内容:/text()取直系文本  //text()取所有文本
a_content = tree.xpath('//a[@id="feng"]/text()')[0]
div_content = tree.xpath('//div[@class="song"]//text()')
# 2.提取标签的属性值://tag/@attrName
img_src = tree.xpath('//img/@src')[0]
print(img_src)

案例应用:图片数据爬取

http://pic.netbian.com/4kmeinv/

将爬取到的图片存储到指定的文件夹中

爬取第一页

from lxml import etree
import requests
import os
#新建一个文件夹
dirName = 'girls'
if not os.path.exists(dirName):#如果文件夹不存在,则新建,否则不新建
  os.mkdir(dirName)

headers = {
  'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'
}

url = 'https://pic.netbian.com/4kmeinv/index.html'
response = requests.get(url=url,headers=headers)
response.encoding = 'gbk'
page_text = response.text

#数据解析:图片地址+图片名称
tree = etree.HTML(page_text)#HTML()专门用来解析网络请求到的页面源码数据
#该列表中存储的是每一个li标签
li_list = tree.xpath('//div[@class="slist"]/ul/li')
for li in li_list:
  #局部解析:将li标签中指定的内容解析出来
  img_title = li.xpath('./a/b/text()')[0]+'.jpg'# 左侧./表示xpath的调用者对应的标签
  img_src = 'https://pic.netbian.com'+li.xpath('./a/img/@src')[0]

  #对图片发起请求,存储图片数据
  img_data = requests.get(url=img_src,headers=headers).content
  # girls/123.jpg
  img_path = dirName + '/' + img_title
  with open(img_path,'wb') as fp:
      fp.write(img_data)
  print(img_title,'下载保存成功!')

爬取多页

from lxml import etree
import requests
import os
#新建一个文件夹
dirName = 'girls'
if not os.path.exists(dirName):#如果文件夹不存在,则新建,否则不新建
  os.mkdir(dirName)

headers = {
  'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'
}
#创建一个通用的url:除了第一页其他页码的通用url
url = 'https://pic.netbian.com/4kmeinv/index_%d.html'
for page in range(1,6):
  if page == 1:
      new_url = 'https://pic.netbian.com/4kmeinv/index.html'
  else:
      new_url = format(url%page)
  print('----------正在请求下载第%d页的图片数据----------'%page)
  response = requests.get(url=new_url,headers=headers)
  response.encoding =2 'gbk'
  page_text = response.text

  #数据解析:图片地址+图片名称
  tree = etree.HTML(page_text)#HTML()专门用来解析网络请求到的页面源码数据
  #该列表中存储的是每一个li标签
  li_list = tree.xpath('//div[@class="slist"]/ul/li')
  for li in li_list:
      #局部解析:将li标签中指定的内容解析出来
      img_title = li.xpath('./a/b/text()')[0]+'.jpg'# 左侧./表示xpath的调用者对应的标签
      img_src = 'https://pic.netbian.com'+li.xpath('./a/img/@src')[0]

      #对图片发起请求,存储图片数据
      img_data = requests.get(url=img_src,headers=headers).content
      # girls/123.jpg
      img_path = dirName + '/' + img_title
      with open(img_path,'wb') as fp:
          fp.write(img_data)
      print(img_title,'下载保存成功!')

pandas数据表格(重点)

环境安装

pip install openpyxl
pip install pandas

基础操作

import pandas as pd
#pandas会帮我们创建一个数据表格,可以对数据表格进行相关操作,最后将该数据表格同步到excel

#1.手动的创建一个pandas的数据表格
table = pd.DataFrame(columns=['name','salary','job_title'])
#2.给数据表格插入数据
table.loc[0] = ['Jay',40000,'singer']
table.loc[1] = ['Tom',10000,'sale']
print(table)
#3.将数据表格转换成excel
table.to_excel('test.xlsx',sheet_name='haha')

爬取数据存储到excel

from lxml import etree
import requests
import pandas as pd
table = pd.DataFrame(columns=['img_title','img_src'])

headers = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'
}

url = 'https://pic.netbian.com/4kmeinv/index.html'
response = requests.get(url=url,headers=headers)
response.encoding = 'gbk'
page_text = response.text

#数据解析:图片地址+图片名称
tree = etree.HTML(page_text)#HTML()专门用来解析网络请求到的页面源码数据
#该列表中存储的是每一个li标签
li_list = tree.xpath('//div[@class="slist"]/ul/li')
index = 0
for li in li_list:
    #局部解析:将li标签中指定的内容解析出来
    img_title = li.xpath('./a/b/text()')[0]+'.jpg'# 左侧./表示xpath的调用者对应的标签
    img_src = 'https://pic.netbian.com'+li.xpath('./a/img/@src')[0]

    table.loc[index] = [img_title,img_src]
    index += 1

    #对图片发起请求,存储图片数据
    img_data = requests.get(url=img_src,headers=headers).content
    # girls/123.jpg
    img_path = 'girls' + '/' + img_title
    with open(img_path,'wb') as fp:
        fp.write(img_data)
    print(img_title,'下载保存成功!')

table.to_excel('gilr.xlsx')

案例:

简历模版下载:https://sc.chinaz.com/jianli/free.html

图片下载:url:https://sc.chinaz.com/tupian/meinvtupian.html

from lxml import etree
import requests

headers = {
    'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'
}
url = 'https://sc.chinaz.com/jianli/free.html'
response = requests.get(url=url,headers=headers)
response.encoding = 'utf-8'
page_text = response.text
#数据解析:简历名称+详情页的url
tree = etree.HTML(page_text)
div_list = tree.xpath('//*[@id="container"]/div')
for div in div_list:
    title = div.xpath('./p/a/text()')[0]+'.rar'
    detail_url = 'https:'+div.xpath('./p/a/@href')[0]
    # print(title,detail_url)
    #对详情页的url发起请求
    detail_page_text = requests.get(url=detail_url,headers=headers).text
    #数据解析:下载地址
    tree = etree.HTML(detail_page_text)
    download_url = tree.xpath('//*[@id="down"]/div[2]/ul/li[1]/a/@href')[0]
    #在下载请求建立模板
    data = requests.get(url=download_url,headers=headers).content
    with open(title,'wb') as fp:
        fp.write(data)
    print(title,'保存下载成功!')