通过asyncio+aiohttp实现异步请求

asyncio+aiohttp示例

通过python自带的包asyncio可以实现异步io, 但是由于asyncio只支持TCP请求, 不支持HTTP请求, 所以我们需要对请求进行封装. 调用第三方包aiohttp可以很方便的实现封装.

直接上代码:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2019/1/6 18:16
# @Author  : Vince
# @Site    : https://zws910.github.io


import aiohttp
import asyncio


@asyncio.coroutine
def fetch_aync(url):
    print(url)
    # response = yield from aiohttp.request('GET', url)
    response = yield from aiohttp.ClientSession().get(url)
    print(url, response)
    response.close()


tasks = [
    fetch_aync('http://www.baidu.com'),
    fetch_aync('http://www.google.com'),
]

event_loop = asyncio.get_event_loop()
results = event_loop.run_until_complete(asyncio.gather(*tasks))
event_loop.close()

书上写的是通过response = yield from aiohttp.request('GET', url)进行请求, 但是实际运行会报如下错误:

1	TypeError: '_SessionRequestContextManager' object is not iterable

网上查了半天, 发现应该用aiohttp.ClientSession().get进行下载. 这个函数的作用是保证相关的TCP资源能够得到释放，比如TCP链接.

asyncio+requests示例

通过常用的requests模块也能对请求进行封装, 代码如下

import asyncio
import requests

def task(func, *args):
    loop = asyncio.get_event_loop()
    future = loop.run_in_executor(None, func, *args)
    response = yield from future
    print(response.url, response.content)


tasks = [
    task(requests.get, 'http://www.baidu.com'),
    task(requests.get, 'https://www.cnblogs.com'),
]

loop = asyncio.get_event_loop()
results = loop.run_until_complete(asyncio.gather(*tasks))
loop.close()