这是关于腾讯新闻网爬取军事分类新闻的一个可行的代码 需要注意的是腾讯新闻解析文章详情的代码是通用的,这里没有给出(使用tencent_parser.py即可) 注意这里需要使用到动态加载(继承DynamicCrawler,并且无需重写_fetch_page()) ```python import requests from bs4 import BeautifulSoup URL = "https://news.qq.com/ch/milite" headers = { "User-Agent": ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/120.0.0.0 Safari/537.36" ) } resp = requests.get(URL,headers=headers,timeout=10) resp.raise_for_status() resp.encoding = "utf-8" # print(resp.text) # with open("example/example-13.html","r",encoding="utf-8") as f: # html = f.read() soup = BeautifulSoup(resp.text,"lxml") # soup = BeautifulSoup(html,"lxml") div_list = soup.select("div[id='channel-feed-area'] div.channel-feed-list div.channel-feed-item[dt-params*='article_type=0']") for div in div_list: href = div.select_one("a.article-title").get("href") print(href) ```