之前做分布式爬蟲的時候,都是從push url來拿到爬蟲消費的鏈接,這里提出一個問題,假如這個請求是post請求的呢,我觀察了scrapy-redis的源碼,其中spider.py的代碼是這樣寫的

1.scrapy-redis源碼分析

    def make_request_from_data(self, data):"""Returns a `Request` instance for data coming from Redis.Overriding this function to support the `json` requested `data` that contains`url` ,`meta` and other optional parameters. `meta` is a nested json which contains sub-data.Along with:After accessing the data, sending the FormRequest with `url`, `meta` and addition `formdata`, `method`For example:.. code:: json{"url": "https://example.com","meta": {"job-id":"123xsd","start-date":"dd/mm/yy",},"url_cookie_key":"fertxsas","method":"POST",}If `url` is empty, return `[]`. So you should verify the `url` in the data.If `method` is empty, the request object will set method to 'GET', optional.If `meta` is empty, the request object will set `meta` to an empty dictionary, optional.This json supported data can be accessed from 'scrapy.spider' through response.'request.url', 'request.meta', 'request.cookies', 'request.method'Parameters----------data : bytesMessage from redis."""formatted_data = bytes_to_str(data, self.redis_encoding)if is_dict(formatted_data):parameter = json.loads(formatted_data)else:self.logger.warning(f"{TextColor.WARNING}WARNING: String request is deprecated, please use JSON data format. "f"Detail information, please check https://github.com/rmax/scrapy-redis#features{TextColor.ENDC}")return FormRequest(formatted_data, dont_filter=True)if parameter.get("url", None) is None:self.logger.warning(f"{TextColor.WARNING}The data from Redis has no url key in push data{TextColor.ENDC}")return []url = parameter.pop("url")method = parameter.pop("method").upper() if "method" in parameter else "GET"metadata = parameter.pop("meta") if "meta" in parameter else {}return FormRequest(url, dont_filter=True, method=method, formdata=parameter, meta=metadata)

源碼地址:https://github.com/rmax/scrapy-redis
可以看到這里是可以處理post請求的

2.scrapy-rabbitmq-schrduler源碼分析

地址:

https://github.com/aox-lei/scrapy-rabbitmq-scheduler

class RabbitSpider(scrapy.Spider):def _make_request(self, mframe, hframe, body):try:request = request_from_dict(pickle.loads(body), self)except Exception as e:body = body.decode()request = scrapy.Request(body, callback=self.parse, dont_filter=True)return request

可以看到RabbitSpider繼承了spider的嘞,改寫了request,當我們發(fā)我post請求的時候?request_from_dict(pickle.loads(body), self)會報錯

builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

pick.loads 在嘗試反序列化字節(jié)數(shù)據(jù)時遇到無法解碼的字節(jié)序列造成的。具體來說，UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte 說明傳入的數(shù)據(jù)包含非 UTF-8 編碼的字節(jié)，可能是二進制數(shù)據(jù)或其他編碼格式的數(shù)據(jù)。

def _make_request(self, mframe, hframe, body):try:# 反序列化 body 數(shù)據(jù)data = pickle.loads(body)# 獲取請求的 URL 和其他參數(shù)url = data.get('url')method = data.get('method', 'GET').upper()  # 默認 GET，如果是 POST 需要設置為 'POST'headers = data.get('headers', {})cookies = data.get('cookies', {})body_data = data.get('body')  # 可能是 POST 請求的表單數(shù)據(jù)callback_str = data.get('callback')  # 回調(diào)函數(shù)名稱（字符串）errback_str = data.get('errback')  # 錯誤回調(diào)函數(shù)名稱（字符串）meta = data.get('meta', {})# 嘗試從全局字典中獲取回調(diào)函數(shù)# 使用爬蟲實例的 `getattr` 方法獲取回調(diào)函數(shù)callback = getattr(self, callback_str, None) if callback_str else Noneerrback = getattr(self, errback_str, None) if errback_str else None# # 確?；卣{(diào)函數(shù)存在# if callback is None:#     self.logger.error(f"Callback function '{callback_str}' not found.")# if errback is None:#     self.logger.error(f"Errback function '{errback_str}' not found.")# 判斷請求方法，如果是 POST，則使用 FormRequestif callback:if method == 'POST':# FormRequest 適用于帶有表單數(shù)據(jù)的 POST 請求request = scrapy.FormRequest(url=url,method='POST',headers=headers,cookies=cookies,body=body_data,  # 請求的主體callback=callback,errback=errback,meta=meta,dont_filter=True)else:# 默認處理 GET 請求request = scrapy.Request(url=url,headers=headers,cookies=cookies,callback=callback,errback=errback,meta=meta,dont_filter=True)else: passexcept Exception as e:body = body.decode()request = scrapy.Request(body, callback=self.parse, dont_filter=True)return request

直接獲取callback是個字符串而不是函數(shù),要在spider中獲取到對應的函數(shù)

注:由于scrapy-rabbitmq-scheduler無人更新維護,目前新的scrapy已經(jīng)不支持,上述最新的代碼已推github:https://github.com/tieyongjie/scrapy-rabbitmq-task

安裝直接安裝

pip install scrapy-rabbitmq-task

?