欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

多线程爬虫批量下载pcgame图片url 保存为xml的实现代码

程序员文章站 2022-05-08 17:51:19
复制代码 代码如下:#coding=gbkfrom xml.dom import minidom,nodeimport urllib2,re,osdef readsrc(s...

复制代码 代码如下:

#coding=gbk
from xml.dom import minidom,node
import urllib2,re,os
def readsrc(src):
    try:
        url = urllib2.urlopen(src)
        content = url.read()#.decode('utf-8')
        return content
    except:
        print 'error'
        return none
def pictype(content):
    '''
    通过抓取网站导航栏,获得网站的图片类型
    返回列表,每个列表元素为一个字典,addr代表图片类型对于的链接,name代表图片类型的名称
    错误会返回none
    '''
    p = re.compile(r'<ul>(.*)</ul>',re.s)
    r=p.search(content)
    if r:
        content=r.group()
    else:
        print none
    p = re.compile(r'<li\s*.*?>\s*<a href *= *"(?p<addr>.*?)">(?p<name>.*?)\s*</a>\s*</li>')

    l = [i.groupdict() for i in p.finditer(content)]
    l=l[1:]
    if len(l):return l
    else:return none
def pageinfo(src):
    '''
    获取一个页面的详细信息
    返回对于的字典列表
    name:图片的名字
    cutaddr:缩小的浏览图
    picaddr:实际图片的地址
    '''
    d=os.path.split(src)[0]
    try:
        url = urllib2.urlopen(src)
        content = url.read()#.decode('utf-8')
    except:
        print 'error'
        return none
    #find all the pictures info in a page
    p = re.compile(r'<ul.*?>(.*?)</ul>',re.s)
    r = p.findall(content)
    if not r: return none
    r = r[1]
    p = re.compile(r'<li><a href="(?p<picaddr>.*?)".*?><img.*?alt="(?p<name>.*?)" *src="(?p<cutaddr>.*?)" */></a>.*?</li>')
    l = [ i.groupdict() for i in p.finditer(r)]
    for i in l:
        i['picaddr']=d+'/'+i['picaddr']
    if len(l): return l
    else: return none

def nextpageaddr(src):
    '''
    从页面的html源码中获取下一个页面地址的名称,最后一页返回none
    '''
    content=readsrc(src)
    p = re.compile(r'<a class="next" href="(.*?)">.*?</a>')
    r = p.search(content)
    if r:
        return os.path.dirname(src)+"/"+r.group(1)
    else:
        return none
def picinfoaddr(src):
    '''
    参数相册图集的html代码
    返回全部图片的相对地址
    '''
    content=readsrc(src)
    p = re.compile(r'<div class="picinfo">.*?<a href="(?p<addr>.*?)".*?>.*?</div>',re.s)
    r = p.search(content)
    if r:
        return os.path.dirname(src)+"/"+r.group(1)
    else:
        return none
def parseinfo(content):
    '''
    读取全部图片html代码,获得一个相册的详细信息
    kw:关键字
    title:标题
    type:类型
    pic:各个图片的地址列表,末尾加上_220x165,_medium,_small 可以得到不同大小的图片
    '''
    info={}
    temp=str()

    #title
    temp=''
    r=re.search('<h1>(.*?)</h1>',content)#get the pic title
    if r:
        temp = r.group(1)
    info['title']=temp

    #keyword
    temp=''
    r=re.search('<meta name="keywords" content="(.*?)" />',content)
    if r:
        temp = r.group(1)
    info['kw']=temp

    #type
    r=re.findall('<i><a.*?>(.*?)</a></i>.*?&gt',content)
    if r:
        info['type']=':'.join(r)
    else:
        info['type']=''
    r=re.search('<ul class=".*?">(.*?)</ul>',content,re.s)
    if not r:return none
    content=r.group(1)#filter content
#    print content
    r=re.findall('<a href=".*?<img.*?src="(.*?)".*?</a>',content)

    for index,i in enumerate(r):
        r[index]=i[0:i.rfind('_')]
#        print r[index]
    info['pic']=r
    return info
import threading
class mthread(threading.thread):
    def __init__(self,tp,addr,lock):
        threading.thread.__init__(self)
#        self.doc = minidom.document()
        self.doc=minidom.document()
        self.tp=tp
        self.lock=lock
        self.addr=addr
        self.thread_stop=false
        self.picdoc=none
    def run(self):
        self.picdoc = self.doc.createelement('urlclass')
#        print self.tp
        self.picdoc.setattribute('type',self.tp)
#        self.doc.appendchild(self.picdoc)
        m=pageinfo(self.addr)
        while self.addr:
            for i in m:
#                print i['picaddr']
                picaddr=picinfoaddr(i['picaddr'])
#                print picaddr
                info=parseinfo(readsrc(picaddr))
                name=info['title']


                picture=doc.createelement('picture')

                title = doc.createelement('title')
                title.appendchild(doc.createtextnode(info['title']))
                picture.appendchild(title)

                keyword = doc.createelement('keywords')
                keyword.appendchild(doc.createtextnode(info['kw']))
                picture.appendchild(keyword)

                tp = doc.createelement('pictype')
                tp.appendchild(doc.createtextnode(info['type']))
                picture.appendchild(tp)

                cuturl = doc.createelement('piccut')
                cuturl.appendchild(doc.createtextnode(i['cutaddr']))
                picture.appendchild(cuturl)

                urls = doc.createelement('urls')
                self.lock.acquire()
                print 'downloading ',name
                self.lock.release()
                for picurl in info['pic']:
                    singleurl=doc.createelement('url')
                    singleurl.appendchild(doc.createtextnode(picurl+'.jpg'))
                    urls.appendchild(singleurl)

                picture.appendchild(urls)
                self.picdoc.appendchild(picture)
            m=pageinfo(self.addr)
            self.addr=nextpageaddr(self.addr)
#        f = open('c:\\'+self.tp+'.xml','w')
#        f.write(doc.toprettyxml(indent = ''))
#        f.close()
    def stop(self):
        self.thread_stop=true


path='c:\\pict\\'#下载的路径
#import sys
sys.exit(12)
content=readsrc('http://photos.pcgames.com.cn/cate/3/1.html')
r=pictype(content)
lt=[]
doc = minidom.document()
root=doc.createelement('url_resource')
root.setattribute('type','url')
root.setattribute('urltype','image')
root.setattribute('imgfmt','jpg')
doc.appendchild(root)
lock=threading.rlock()
for iaddr in r:
    print 'downloading type: ',iaddr['name']
    addr=iaddr['addr']
    th=mthread(iaddr['name'],addr,lock)
    lt.append(th)
    th.start()
for t in lt:
    t.join()
    root.appendchild(t.picdoc)

print 'write'
f = open('c:\\'+'urls'+'.xml','w')
f.write(doc.toprettyxml(indent = ''))
f.close()
print doc.toprettyxml()
print 'end'

上一篇: 我女儿用得着你管

下一篇: PHP笔记