@@ -1,5 +1,14 @@
<?xml version="1.0" encoding="utf-8"?>
<search >
<entry >
<title > page</title>
<url > /posts/1441.html</url>
<content > <![CDATA[<ul>
<li > <a href= "./deploy" > deploy</a> </li>
<li > </li>
</ul>
]]></content>
</entry>
<entry >
<title > Hello World</title>
<url > /posts/16107.html</url>
@@ -39,22 +48,48 @@
</tags>
</entry>
<entry >
<title > page </title>
<url > /posts/1441 .html</url>
<content > <![CDATA[<ul>
<li > <a href= "./deploy" > deploy </a> </li >
<li > </li >
<title > 决策树算法 </title>
<url > /posts/95 .html</url>
<content > <![CDATA[<h3 id="C4-5"> <a href= "#C4-5" class= "headerlink" title= "C4.5" > </a> C4.5 </h3> <p > C4.5 是一种用于生成决策树的算法,不再使用信息增益,而是使用信息增益比,来避免偏向于选择取值较多的特征。信息增益比是信息增益与特征的熵的比值。 </p>
<h3 id= "ID3" > <a href= "#ID3" class= "headerlink" title= "ID3" > </a> ID3 </h3> <p > D3 是一种基于信息增益(Information Gain)的决策树算法 </p >
<h3 id= "Cart" > <a href= "#Cart" class= "headerlink" title= "Cart" > </a> Cart </h3> <p > CART(分类与回归树)一种决策树算法,,CART 使用 <strong > 二叉树结构 </strong> ,即每个节点只能有两个子节点。 </p >
<h3 id= "cart剪枝" > <a href= "#cart剪枝" class= "headerlink" title= "cart剪枝" > </a> cart剪枝</h3> <p > CART 决策树的剪枝方法分为 <strong > 预剪枝</strong> ( Pre-pruning)和 <strong > 后剪枝</strong> ( Post-pruning)两种:</p>
<h4 id= "预剪枝:" > <a href= "#预剪枝:" class= "headerlink" title= "预剪枝:" > </a> <strong > 预剪枝</strong> : </h4> <p > 预剪枝是在构建决策树时就决定是否停止进一步划分某个节点。主要通过以下标准来控制:</p>
<ul >
<li > 当某个节点的样本数小于某个阈值时,不再继续划分。</li>
<li > 当某个节点的 Gini 不纯度小于某个阈值时,不再继续划分。<br > 预剪枝的优点是能够减少计算量,但缺点是可能会导致模型不够复杂,从而产生欠拟合。</li>
</ul>
]]> </content >
</entry >
<entry >
<title > script </title >
<url > /posts/34849.html </url >
<content > <![CDATA[<h3 id="查看CPU、内存使用率"> <a href= "#查看CPU、内存使用率" class= "headerlink" title= "查看CPU、内存使用率" > </a> 查看CPU、内存使用率 </h3> <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "meta" > #!/bin/bash </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 定义颜色 </span> </span> <br > <span class= "line" > RED= <span class= "string" > ' \033[0;31m ' </span> </span> <br > <span class= "line" > GREEN= <span class= "string" > ' \033[0;32m ' </span> </span> <br > <span class= "line" > YELLOW= <span class= "string" > ' \033[0;33m ' </span> </span> <br > <span class= "line" > BLUE= <span class= "string" > ' \033[0;34m ' </span> </span> <br > <span class= "line" > NC= <span class= "string" > ' \033[0m ' </span> <span class= "comment" > # 无颜色 </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "keyword" > while </span> <span class= "literal" > true </span> ; <span class= "keyword" > do </span> </span> <br > <span class= "line" > <span class= "comment" > # 获取所有进程的CPU使用率和内存使用率 </span> </span> <br > <span class= "line" > cpu_usage=$(ps aux | awk <span class= "string" > ' { sum_cpu += $3 } END { print sum_cpu } ' </span> ) </span> <br > <span class= "line" > mem_usage=$(ps aux | awk <span class= "string" > ' { sum_mem += $4 } END { print sum_mem } ' </span> ) </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 打印结果,带有时间戳、分隔线和颜色高亮 </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { BLUE } </span> ============================== <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { YELLOW } </span> Timestamp: <span class= "subst" > $(date) </span> <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { BLUE } </span> ============================== <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { GREEN } </span> Total CPU usage: <span class= "variable" > $ { RED } </span> <span class= "variable" > $cpu_usage </span> % <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { GREEN } </span> Total Memory usage: <span class= "variable" > $ { RED } </span> <span class= "variable" > $mem_usage </span> % <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { BLUE } </span> ============================== <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 等待5秒后再次执行 </span> </span> <br > <span class= "line" > <span class= "built_in" > sleep </span> 5 </span> <br > <span class= "line" > <span class= "keyword" > done </span> </span> <br > <span class= "line" > </span> <br > </pre> </td> </tr> </table> </figure >
<p > <strong > 保存脚本到 / usr / local / bin目录下 </strong> </p >
<figure class= "highlight bash " > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_ in" > mv </span> usage.sh /usr/local/bin/usage </span> <br > </pre> </td> </tr> </table> </figure>
<h4 id= "后剪枝:" > <a href= "#后剪枝:" class= "headerlink" title= "后剪枝:" > </a> <strong > 后剪枝 </strong> : </h4> <p > 后剪枝是在决策树完全构建出来之后,对树进行修剪。具体过程如下: </p >
<ul >
<li > 构建完整的决策树。 </li >
<li > 从叶子节点开始,逐渐向上遍历树的每个节点。 </li >
<li > 对每个节点进行判断,是否合适剪去该节点及其子树。如果剪去该子树后,模型的性能没有显著下降,就可以剪枝。 <br > 后剪枝通过避免过度拟合来提高模型的泛化能力,但其计算开销较大。 </li >
</ul >
<h3 id= "特征工程-特征提取" > <a href= "#特征工程-特征提取" class= "headerlink" title= "特征工程(特征提取)" > </a> 特征工程(特征提取) </h3> <ul >
<li > <strong > 字典特征提取 </strong> <br > 主要用于处理包含键值对(key-value pairs)的数据结构 <br > <figure class= "highlight python " > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from </span> sklearn.feature_extraction <span class= "keyword" > import </span> DictVectorizer </span> <br > <span class= "l ine "> <span class= "comment" > # 字典特征提取 </span> </span> <br > <span class= "line" > data = [</span> <br > <span class= "line" > { <span class= "string" > ' city ' </span> : <span class= "string" > ' beijing ' </span> , <span class= "string" > ' temperature ' </span> : <span class= "number" > 100 </span> } , </span> <br > <span class= "line" > { <span class= "string" > ' city ' </span> : <span class= "string" > ' shanghai ' </span> , <span class= "string" > ' temperature ' </span> : <span class= "number" > 95 </span> } , </span> <br > <span class= "line" > { <span class= "string" > ' city ' </span> : <span class= "string" > ' guangzhou ' </span> , <span class= "string" > ' temperature ' </span> : <span class= "number" > 98 </span> } </span> <br > <span class= "line" > ]</span> <br > <span class= "line" > transfer = DictVectorizer(sparse=<span class= "literal" > False </span> ) </span> <br > <span class= "line" > new_data = transfer.fit_transform(data)</span> <br > <span class= "line" > <span class= "built_in" > print </span> (transfer.feature_names_) </span> <br > <span class= "line" > <span class= "built_in" > print </span> (new_data) </span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <strong > 文本特征提取</strong> <br > 主要用于将文本数据(如句子、段落、文章等)转换成数值型特征。这对于文本分类、信息检索等任务非常重要。<br > <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction.text <span class= "keyword" > import</span> CountVectorizer</span> <br > <span class= "line" > <span class= "comment" > # 示例文本数据</span> </span> <br > <span class= "line" > data = [</span> <br > <span class= "line" > <span class= "string" > " I love programming" </span> ,</span> <br > <span class= "line" > <span class= "string" > " Python is great" </span> ,</span> <br > <span class= "line" > <span class= "string" > " I love machine learning" </span> </span> <br > <span class= "line" > ]</span> <br > <span class= "line" > <span class= "comment" > # 创建 CountVectorizer 对象</span> </span> <br > <span class= "line" > transfer = CountVectorizer()</span> <br > <span class= "line" > <span class= "comment" > # 将文本数据转换为特征向量</span> </span> <br > <span class= "line" > new_data = transfer.fit_transform(data)</span> <br > <span class= "line" > <span class= "comment" > # 输出特征名称</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Feature Names:" </span> , transfer.get_feature_names_out())</span> <br > <span class= "line" > <span class= "comment" > # 输出转换后的特征矩阵</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Transformed Data:" </span> , new_data.toarray())</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > 文本特征提取(中文文本) <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction.text <span class= "keyword" > import</span> CountVectorizer</span> <br > <span class= "line" > <span class= "keyword" > import</span> jieba</span> <br > <span class= "line" > <span class= "comment" > # 中文文本数据(大于20个字)</span> </span> <br > <span class= "line" > data = [</span> <br > <span class= "line" > <span class= "string" > " 我热爱编程,学习编程语言是一件非常有趣的事情,它能够提升我们解决问题的能力,编程让我变得更加有创意。" </span> ,</span> <br > <span class= "line" > <span class= "string" > " Python语言是一门非常强大的编程语言,具有简洁的语法和丰富的库,可以帮助开发者更高效地完成任务。" </span> ,</span> <br > <span class= "line" > <span class= "string" > " 机器学习是一项非常有前途的技术,它能够让计算机从数据中自动学习,逐步提高模型的精确度,解决实际问题。" </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > ]</span> <br > <span class= "line" > <span class= "comment" > # 使用jieba分词</span> </span> <br > <span class= "line" > text_list = []</span> <br > <span class= "line" > <span class= "keyword" > for</span> line <span class= "keyword" > in</span> data:</span> <br > <span class= "line" > text_list.append(<span class= "string" > " " </span> .join(<span class= "built_in" > list</span> (jieba.cut(line))))</span> <br > <span class= "line" > <span class= "comment" > # 创建 CountVectorizer 对象</span> </span> <br > <span class= "line" > transfer = CountVectorizer()</span> <br > <span class= "line" > <span class= "comment" > # 将文本数据转换为特征向量</span> </span> <br > <span class= "line" > new_data = transfer.fit_transform(text_list)</span> <br > <span class= "line" > <span class= "comment" > # 输出特征名称</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Feature Names:" </span> , transfer.get_feature_names_out())</span> <br > <span class= "line" > <span class= "comment" > # 输出转换后的特征矩阵</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Transformed Data:" </span> , new_data.toarray())</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<h3 id= "tf-idf" > <a href= "#tf-idf" class= "headerlink" title= "tf-idf" > </a> tf-idf</h3> <blockquote >
<p > 词频 * 逆文档频率</p>
</blockquote>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # tfi-df</span> </span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction.text <span class= "keyword" > import</span> TfidfVectorizer</span> <br > <span class= "line" > <span class= "keyword" > import</span> jieba</span> <br > <span class= "line" > data=[<span class= "string" > " 一种还是一种今天很残酷,明天更残酷,后天很美好,但绝对大部分是死在明天晚上,所以每个人不要放弃今天。" </span> ,</span> <br > <span class= "line" > <span class= "string" > " 我们看到的从很远星系来的光是在几百万年之前发出的,这样当我们看到宇宙时,我们是在看它的过去" </span> ,</span> <br > <span class= "line" > <span class= "string" > " 如果只用一种方式了解某样事物,你就不会真正了解它。了解事物真正含义的秘密取决于如何将其与我们所了解的事物相联系" </span> ]</span> <br > <span class= "line" > <span class= "built_in" > list</span> = []</span> <br > <span class= "line" > <span class= "keyword" > for</span> item <span class= "keyword" > in</span> data:</span> <br > <span class= "line" > <span class= "built_in" > list</span> .append(<span class= "string" > " " </span> .join(jieba.cut(item)))</span> <br > <span class= "line" > transfer = TfidfVectorizer()</span> <br > <span class= "line" > new_data = transfer.fit_transform(<span class= "built_in" > list</span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 特征名字:\n<span class= "subst" > { transfer.get_feature_names_out()} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 转换后的特征矩阵:\n<span class= "subst" > { new_data.toarray()} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 转换后的数据:\n<span class= "subst" > { new_data} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
<h3 id= "Shell脚本编写的基本信息" > <a href= "#Shell脚本编写的基本信息" class= "headerlink" title= "Shell脚本编写的基本信息" > </a> Shell脚本编写的基本信息</h3> <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "meta" > #! /bin/bash</span> </span> <br > <span class= "line" > <span class= "comment" > # -------------------------------------------------</span> </span> <br > <span class= "line" > <span class= "comment" > # Filename: test.sh</span> </span> <br > <span class= "line" > <span class= "comment" > # Version: 1.0</span> </span> <br > <span class= "line" > <span class= "comment" > # Date: 2024/05/02</span> </span> <br > <span class= "line" > <span class= "comment" > # Author: shenjianZ</span> </span> <br > <span class= "line" > <span class= "comment" > # Email: shenjianZLT@gmail.com</span> </span> <br > <span class= "line" > <span class= "comment" > # Website: https://blog.shenjianl.cn</span> </span> <br > <span class= "line" > <span class= "comment" > # Description: this is a test shell</span> </span> <br > <span class= "line" > <span class= "comment" > # CopyRight: 2024 All rights reserved shenjianZ</span> </span> <br > <span class= "line" > <span class= "comment" > # License GPL</span> </span> <br > <span class= "line" > <span class= "comment" > # ------------------------------------------------</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # Your script logic goes here</span> </span> <br > </pre> </td> </tr> </table> </figure> ]]></content>
<h3 id= "回归决策树" > <a href= "#回归决策树" class= "headerlink" title= "回归决策树" > </a> 回归决策树</h3> <h4 id= "决策树算法的应用-(泰坦尼克号沉船幸存者预测)" > <a href= "#决策树算法的应用-(泰坦尼克号沉船幸存者预测)" class= "headerlink" title= "决策树算法的应用 (泰坦尼克号沉船幸存者预测)" > </a> 决策树算法的应用 (泰坦尼克号沉船幸存者预测)</h4> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> seaborn <span class= "keyword" > as</span> sns</span> <br > <span class= "line" > <span class= "keyword" > import</span> pandas <span class= "keyword" > as</span> pd</span> <br > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction <span class= "keyword" > import</span> DictVectorizer</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.tree <span class= "keyword" > import</span> DecisionTreeClassifier,export_graphviz</span> <br > <span class= "line" > <span class= "comment" > # 1.获取数据集 - 加载 Titanic 数据集</span> </span> <br > <span class= "line" > titanic = sns.load_dataset(<span class= "string" > ' titanic' </span> )</span> <br > <span class= "line" > missing_age_count = titanic[<span class= "string" > ' age' </span> ].isna().<span class= "built_in" > sum</span> ()</span> <br > <span class= "line" > <span class= "comment" > # print(f" 缺失的 age 数量: { missing_age_count} " )</span> </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据基本处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 确认特征值、目标值</span> </span> <br > <span class= "line" > X = titanic[[<span class= "string" > ' pclass' </span> ,<span class= "string" > ' age' </span> ,<span class= "string" > ' sex' </span> ]]</span> <br > <span class= "line" > y = titanic[<span class= "string" > ' survived' </span> ]</span> <br > <span class= "line" > <span class= "comment" > # 2.2 缺失值处理</span> </span> <br > <span class= "line" > X.loc[:, <span class= "string" > ' age' </span> ] = X[<span class= "string" > ' age' </span> ].fillna(value=X[<span class= "string" > ' age' </span> ].mean()) <span class= "comment" > # 使用 .loc 进行修改</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.3 划分数据集</span> </span> <br > <span class= "line" > X_train,X_test,y_train,y_test = train_test_split(X,y,random_state=<span class= "number" > 22</span> )</span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程(字典特征提取)</span> </span> <br > <span class= "line" > X_train = X_train.to_dict(orient=<span class= "string" > " records" </span> )</span> <br > <span class= "line" > X_test= X_test.to_dict(orient=<span class= "string" > " records" </span> )</span> <br > <span class= "line" > transfer = DictVectorizer()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test)</span> <br > <span class= "line" > <span class= "comment" > # 4. 机器学习 决策树算法</span> </span> <br > <span class= "line" > estimator = DecisionTreeClassifier(criterion=<span class= "string" > " gini" </span> )</span> <br > <span class= "line" > estimator.fit(X_train,y_train)</span> <br > <span class= "line" > y_pred = estimator.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的测试集的预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > ret = estimator.score(X_test,y_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的评分:<span class= "subst" > { ret} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (X_test.toarray())</span> <br > </pre> </td> </tr> </table> </figure>
<p > 生成对应的图</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.tree <span class= "keyword" > import</span> export_graphviz</span> <br > <span class= "line" > <span class= "keyword" > import</span> graphviz <span class= "comment" > # 用于渲染图像</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 导出决策树的 Graphviz 表示</span> </span> <br > <span class= "line" > export_graphviz(estimator, out_file=<span class= "string" > ' ./data/tree.dot' </span> , </span> <br > <span class= "line" > feature_names=transfer.get_feature_names_out()) <span class= "comment" > # 特征名称</span> </span> <br > <span class= "line" > <span class= "comment" > # 使用 graphviz 渲染 .dot 文件</span> </span> <br > <span class= "line" > <span class= "keyword" > with</span> <span class= "built_in" > open</span> (<span class= "string" > ' ./data/tree.dot' </span> , <span class= "string" > ' r' </span> ) <span class= "keyword" > as</span> f:</span> <br > <span class= "line" > dot_graph = f.read()</span> <br > <span class= "line" > <span class= "comment" > # 渲染决策树</span> </span> <br > <span class= "line" > graph = graphviz.Source(dot_graph)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 设置保存路径</span> </span> <br > <span class= "line" > output_path = <span class= "string" > ' ./data/decision_tree' </span> <span class= "comment" > # 自定义保存路径</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 保存图像到指定路径,格式可以是 .png, .pdf, .jpg 等</span> </span> <br > <span class= "line" > <span class= "comment" > # graph.render(output_path, format=' png' ) # 保存为 .png 文件</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 显示图像</span> </span> <br > <span class= "line" > graph.view(output_path) <span class= "comment" > # 打开图像,path为保存路径,不需要加后缀</span> </span> <br > <span class= "line" > </span> <br > </pre> </td> </tr> </table> </figure>
<p > <a href= "http://webgraphviz.com/" > Webgraphviz</a> ,这个网站可以将<code > tree.dot</code> 文件的内容生成对应的可视化树</p>
<h4 id= "回归决策树与线性回归的对比" > <a href= "#回归决策树与线性回归的对比" class= "headerlink" title= "回归决策树与线性回归的对比" > </a> 回归决策树与线性回归的对比</h4> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > import</span> matplotlib.pyplot <span class= "keyword" > as</span> plt</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LinearRegression</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.tree <span class= "keyword" > import</span> DecisionTreeRegressor</span> <br > <span class= "line" > <span class= "keyword" > from</span> matplotlib <span class= "keyword" > import</span> rcParams</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 设置matplotlib使用的字体为SimHei(黑体)</span> </span> <br > <span class= "line" > rcParams[<span class= "string" > ' font.sans-serif' </span> ] = [<span class= "string" > ' SimHei' </span> ] <span class= "comment" > # 也可以使用 ' Microsoft YaHei' </span> </span> <br > <span class= "line" > rcParams[<span class= "string" > ' axes.unicode_minus' </span> ] = <span class= "literal" > False</span> <span class= "comment" > # 正常显示负号</span> </span> <br > <span class= "line" > x = np.array(<span class= "built_in" > list</span> (<span class= "built_in" > range</span> (<span class= "number" > 1</span> ,<span class= "number" > 11</span> ))).reshape(-<span class= "number" > 1</span> ,<span class= "number" > 1</span> )</span> <br > <span class= "line" > y = ([<span class= "number" > 5.56</span> ,<span class= "number" > 5.70</span> ,<span class= "number" > 5.91</span> ,<span class= "number" > 6.40</span> ,<span class= "number" > 6.80</span> ,<span class= "number" > 7.05</span> ,<span class= "number" > 8.90</span> ,<span class= "number" > 8.70</span> ,<span class= "number" > 9.00</span> ,<span class= "number" > 9.05</span> ])</span> <br > <span class= "line" > </span> <br > <span class= "line" > m1 = DecisionTreeRegressor(max_depth=<span class= "number" > 1</span> )</span> <br > <span class= "line" > m2 = DecisionTreeRegressor(max_depth=<span class= "number" > 3</span> )</span> <br > <span class= "line" > m3 = DecisionTreeRegressor()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 模型训练</span> </span> <br > <span class= "line" > m1.fit(x,y)</span> <br > <span class= "line" > m2.fit(x,y)</span> <br > <span class= "line" > m3.fit(x,y)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 模型预测</span> </span> <br > <span class= "line" > x_test = np.arange(<span class= "number" > 0</span> ,<span class= "number" > 10</span> ,<span class= "number" > 0.01</span> ).reshape(-<span class= "number" > 1</span> ,<span class= "number" > 1</span> )</span> <br > <span class= "line" > y_1 = m1.predict(x_test)</span> <br > <span class= "line" > y_2 = m2.predict(x_test)</span> <br > <span class= "line" > y_3 = m3.predict(x_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 结果展示</span> </span> <br > <span class= "line" > plt.figure(figsize=(<span class= "number" > 10</span> ,<span class= "number" > 6</span> ),dpi=<span class= "number" > 100</span> )</span> <br > <span class= "line" > plt.scatter(x,y ,label = <span class= "string" > " data" </span> )</span> <br > <span class= "line" > plt.plot(x_test,y_1,label = <span class= "string" > " max_depth=1" </span> )</span> <br > <span class= "line" > plt.plot(x_test,y_2,label = <span class= "string" > " max_depth=3" </span> )</span> <br > <span class= "line" > plt.plot(x_test,y_3,label = <span class= "string" > " linearregression" </span> )</span> <br > <span class= "line" > plt.xlabel(<span class= "string" > " 数据" </span> )</span> <br > <span class= "line" > plt.ylabel(<span class= "string" > " 预测值" </span> )</span> <br > <span class= "line" > plt.legend()</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
]]></content>
<categories >
<category > machinelearning</category>
</categories>
<tags >
<tag > decisiontree</tag>
</tags>
</entry>
<entry >
<title > 集成学习</title>
@@ -74,6 +109,49 @@
<tag > ensemble-learning</tag>
</tags>
</entry>
<entry >
<title > 线性回归</title>
<url > /posts/52662.html</url>
<content > <![CDATA[<h3 id="线性回归简介"> <a href= "#线性回归简介" class= "headerlink" title= "线性回归简介" > </a> 线性回归简介</h3> <blockquote >
<p > 用于预测一个连续的目标变量(因变量),与一个或多个特征(自变量)之间存在线性关系。</p>
</blockquote>
<p > 假设函数:<br > $$y = w_1x_1 + w_2x_2 + \cdot\cdot\cdot+w_nx_n$$</p>
<ul >
<li > $y$ 是目标变量(因变量),即我们希望预测的值。</li>
<li > $x1 ,x2,…,xn$ 是特征变量(自变量),即输入的值。</li>
</ul>
<h3 id= "损失函数" > <a href= "#损失函数" class= "headerlink" title= "损失函数" > </a> 损失函数</h3> <p > 为了找到最佳的线性模型,我们需要通过最小化损失函数来优化模型参数。在线性回归中,常用的损失函数是 <strong > 均方误差(MSE) </strong> : <br > $$J(\theta) = \frac{1}{2N} \sum_{i= 1}^{N} (y_i - f_\theta(x_i))^2$$</p>
<ul >
<li > N 是样本的数量。</li>
<li > $y_i$ 是第 i 个样本的真实值。</li>
<li > $f_\theta(x_i)$ 是模型预测的第 i 个样本的值。</li>
</ul>
<h3 id= "线性回归优化" > <a href= "#线性回归优化" class= "headerlink" title= "线性回归优化" > </a> 线性回归优化</h3> <ul >
<li > <p > 梯度下降法</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> SGDRegressor</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4.机器学习- 梯度下降法</span> </span> <br > <span class= "line" > estimater = SGDRegressor(max_iter=<span class= "number" > 1000</span> , eta0=<span class= "number" > 0.01</span> )</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
</li>
<li > <p > 正规方程</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LinearRegression</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.fit_transform(X_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4.机器学习- 线性回归</span> </span> <br > <span class= "line" > estimater = LinearRegression()</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
</li>
<li > <p > 岭回归</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> Ridge, RidgeCV</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4.机器学习- 岭回归 使用了Ridge的alpha的搜索</span> </span> <br > <span class= "line" > <span class= "comment" > # estimater = Ridge(alpha=1.0)</span> </span> <br > <span class= "line" > estimater = RidgeCV(alphas=[<span class= "number" > 0.001</span> , <span class= "number" > 0.01</span> , <span class= "number" > 0.1</span> , <span class= "number" > 1</span> , <span class= "number" > 10</span> , <span class= "number" > 100</span> ])</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 查看最佳 alpha</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 最佳 alpha 值是:<span class= "subst" > { estimater.alpha_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<p > 这样每个代码块的缩进保持一致,便于阅读和理解。如果有其他优化需求,随时告诉我!</p>
<p > <img src= "/img/machinelearning/linear.png" > </p>
<p > <img src= "/img/machinelearning/fitting.png" > </p>
<h3 id= "模型保存和加载" > <a href= "#模型保存和加载" class= "headerlink" title= "模型保存和加载" > </a> 模型保存和加载</h3> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> Ridge, RidgeCV</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > <span class= "keyword" > import</span> joblib</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "keyword" > def</span> <span class= "title function_" > save_model</span> ():</span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > <span class= "comment" > # 4. 机器学习 - 岭回归 使用了Ridge的alpha的搜索</span> </span> <br > <span class= "line" > estimater = RidgeCV(alphas=[<span class= "number" > 0.001</span> , <span class= "number" > 0.01</span> , <span class= "number" > 0.1</span> , <span class= "number" > 1</span> , <span class= "number" > 10</span> , <span class= "number" > 100</span> ])</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 保存模型</span> </span> <br > <span class= "line" > joblib.dump(estimater, <span class= "string" > ' ridge_model.pkl' </span> )</span> <br > <span class= "line" > <span class= "comment" > # 查看最佳 alpha</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 最佳 alpha 值是:<span class= "subst" > { estimater.alpha_} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "keyword" > def</span> <span class= "title function_" > load_model</span> ():</span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > <span class= "comment" > # 加载模型</span> </span> <br > <span class= "line" > estimater = joblib.load(<span class= "string" > ' ridge_model.pkl' </span> )</span> <br > <span class= "line Line truncated
<categories >
<category > machinelearning</category>
</categories>
<tags >
<tag > linear-regression</tag>
</tags>
</entry>
<entry >
<title > 网络相关</title>
<url > /posts/41168.html</url>
<content > <![CDATA[]]> </content>
</entry>
<entry >
<title > k近邻算法(K-Nearest Neighbors) KNN</title>
<url > /posts/29139.html</url>
@@ -140,148 +218,13 @@
</tags>
</entry>
<entry >
<title > 决策树算法 </title>
<url > /posts/95 .html</url>
<content > <![CDATA[<h3 id="C4-5 "> <a href= "#C4-5" class= "header link " title= "C4.5" > </a> C4.5 </h3> <p > C4.5 是一种用于生成决策树的算法,不再使用信息增益,而是使用信息增益比,来避免偏向于选择取值较多的特征。信息增益比是信息增益与特征的熵的比值。 </p >
<h3 id= "ID3" > <a href= "#ID3" class= "headerlink" title= "ID3" > </a> ID3 </h3> <p > D3 是一种基于信息增益(Information Gain)的决策树算法 </p>
<h3 id= "Cart" > <a href= "#Cart" class= "header link " title= "Cart "> </a> Cart </h3 ><p > CART(分类与回归树)一种决策树算法,,CART 使用 <strong > 二叉树结构 </s trong> ,即每个节点只能有两个子节点。 </p >
<h3 id= "cart剪枝" > <a href= "#cart剪枝" class= "headerlink" title= "cart剪枝" > </a> cart剪枝</h3> <p > CART 决策树的剪枝方法分为 <strong > 预剪枝</strong> ( Pre-pruning)和 <strong > 后剪枝</strong> ( Post-pruning)两种:</p>
<h4 id= "预剪枝:" > <a href= "#预剪枝:" class= "headerlink" title= "预剪枝:" > </a> <strong > 预剪枝</strong> : </h4> <p > 预剪枝是在构建决策树时就决定是否停止进一步划分某个节点。主要通过以下标准来控制:</p>
<ul >
<li > 当某个节点的样本数小于某个阈值时,不再继续划分。</li>
<li > 当某个节点的 Gini 不纯度小于某个阈值时,不再继续划分。<br > 预剪枝的优点是能够减少计算量,但缺点是可能会导致模型不够复杂,从而产生欠拟合。</li>
</ul>
<h4 id= "后剪枝:" > <a href= "#后剪枝:" class= "headerlink" title= "后剪枝:" > </a> <strong > 后剪枝</strong> : </h4> <p > 后剪枝是在决策树完全构建出来之后,对树进行修剪。具体过程如下:</p>
<ul >
<li > 构建完整的决策树。</li>
<li > 从叶子节点开始,逐渐向上遍历树的每个节点。</li>
<li > 对每个节点进行判断,是否合适剪去该节点及其子树。如果剪去该子树后,模型的性能没有显著下降,就可以剪枝。<br > 后剪枝通过避免过度拟合来提高模型的泛化能力,但其计算开销较大。</li>
</ul>
<h3 id= "特征工程-特征提取" > <a href= "#特征工程-特征提取" class= "headerlink" title= "特征工程(特征提取)" > </a> 特征工程(特征提取)</h3> <ul >
<li > <strong > 字典特征提取</strong> <br > 主要用于处理包含键值对(key-value pairs)的数据结构<br > <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction <span class= "keyword" > import</span> DictVectorizer</span> <br > <span class= "line" > <span class= "comment" > # 字典特征提取</span> </span> <br > <span class= "line" > data = [</span> <br > <span class= "line" > { <span class= "string" > ' city' </span> : <span class= "string" > ' beijing' </span> , <span class= "string" > ' temperature' </span> : <span class= "number" > 100</span> } ,</span> <br > <span class= "line" > { <span class= "string" > ' city' </span> : <span class= "string" > ' shanghai' </span> , <span class= "string" > ' temperature' </span> : <span class= "number" > 95</span> } ,</span> <br > <span class= "line" > { <span class= "string" > ' city' </span> : <span class= "string" > ' guangzhou' </span> , <span class= "string" > ' temperature' </span> : <span class= "number" > 98</span> } </span> <br > <span class= "line" > ]</span> <br > <span class= "line" > transfer = DictVectorizer(sparse=<span class= "literal" > False</span> )</span> <br > <span class= "line" > new_data = transfer.fit_transform(data)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (transfer.feature_names_)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (new_data)</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <strong > 文本特征提取</strong> <br > 主要用于将文本数据(如句子、段落、文章等)转换成数值型特征。这对于文本分类、信息检索等任务非常重要。<br > <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction.text <span class= "keyword" > import</span> CountVectorizer</span> <br > <span class= "line" > <span class= "comment" > # 示例文本数据</span> </span> <br > <span class= "line" > data = [</span> <br > <span class= "line" > <span class= "string" > " I love programming" </span> ,</span> <br > <span class= "line" > <span class= "string" > " Python is great" </span> ,</span> <br > <span class= "line" > <span class= "string" > " I love machine learning" </span> </span> <br > <span class= "line" > ]</span> <br > <span class= "line" > <span class= "comment" > # 创建 CountVectorizer 对象</span> </span> <br > <span class= "line" > transfer = CountVectorizer()</span> <br > <span class= "line" > <span class= "comment" > # 将文本数据转换为特征向量</span> </span> <br > <span class= "line" > new_data = transfer.fit_transform(data)</span> <br > <span class= "line" > <span class= "comment" > # 输出特征名称</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Feature Names:" </span> , transfer.get_feature_names_out())</span> <br > <span class= "line" > <span class= "comment" > # 输出转换后的特征矩阵</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Transformed Data:" </span> , new_data.toarray())</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > 文本特征提取(中文文本) <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction.text <span class= "keyword" > import</span> CountVectorizer</span> <br > <span class= "line" > <span class= "keyword" > import</span> jieba</span> <br > <span class= "line" > <span class= "comment" > # 中文文本数据(大于20个字)</span> </span> <br > <span class= "line" > data = [</span> <br > <span class= "line" > <span class= "string" > " 我热爱编程,学习编程语言是一件非常有趣的事情,它能够提升我们解决问题的能力,编程让我变得更加有创意。" </span> ,</span> <br > <span class= "line" > <span class= "string" > " Python语言是一门非常强大的编程语言,具有简洁的语法和丰富的库,可以帮助开发者更高效地完成任务。" </span> ,</span> <br > <span class= "line" > <span class= "string" > " 机器学习是一项非常有前途的技术,它能够让计算机从数据中自动学习,逐步提高模型的精确度,解决实际问题。" </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > ]</span> <br > <span class= "line" > <span class= "comment" > # 使用jieba分词</span> </span> <br > <span class= "line" > text_list = []</span> <br > <span class= "line" > <span class= "keyword" > for</span> line <span class= "keyword" > in</span> data:</span> <br > <span class= "line" > text_list.append(<span class= "string" > " " </span> .join(<span class= "built_in" > list</span> (jieba.cut(line))))</span> <br > <span class= "line" > <span class= "comment" > # 创建 CountVectorizer 对象</span> </span> <br > <span class= "line" > transfer = CountVectorizer()</span> <br > <span class= "line" > <span class= "comment" > # 将文本数据转换为特征向量</span> </span> <br > <span class= "line" > new_data = transfer.fit_transform(text_list)</span> <br > <span class= "line" > <span class= "comment" > # 输出特征名称</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Feature Names:" </span> , transfer.get_feature_names_out())</span> <br > <span class= "line" > <span class= "comment" > # 输出转换后的特征矩阵</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > " Transformed Data:" </span> , new_data.toarray())</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<h3 id= "tf-idf" > <a href= "#tf-idf" class= "headerlink" title= "tf-idf" > </a> tf-idf</h3> <blockquote >
<p > 词频 * 逆文档频率</p>
</blockquote>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # tfi-df</span> </span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction.text <span class= "keyword" > import</span> TfidfVectorizer</span> <br > <span class= "line" > <span class= "keyword" > import</span> jieba</span> <br > <span class= "line" > data=[<span class= "string" > " 一种还是一种今天很残酷,明天更残酷,后天很美好,但绝对大部分是死在明天晚上,所以每个人不要放弃今天。" </span> ,</span> <br > <span class= "line" > <span class= "string" > " 我们看到的从很远星系来的光是在几百万年之前发出的,这样当我们看到宇宙时,我们是在看它的过去" </span> ,</span> <br > <span class= "line" > <span class= "string" > " 如果只用一种方式了解某样事物,你就不会真正了解它。了解事物真正含义的秘密取决于如何将其与我们所了解的事物相联系" </span> ]</span> <br > <span class= "line" > <span class= "built_in" > list</span> = []</span> <br > <span class= "line" > <span class= "keyword" > for</span> item <span class= "keyword" > in</span> data:</span> <br > <span class= "line" > <span class= "built_in" > list</span> .append(<span class= "string" > " " </span> .join(jieba.cut(item)))</span> <br > <span class= "line" > transfer = TfidfVectorizer()</span> <br > <span class= "line" > new_data = transfer.fit_transform(<span class= "built_in" > list</span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 特征名字:\n<span class= "subst" > { transfer.get_feature_names_out()} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 转换后的特征矩阵:\n<span class= "subst" > { new_data.toarray()} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 转换后的数据:\n<span class= "subst" > { new_data} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
<title > script </title>
<url > /posts/3484 9.html</url>
<content > <![CDATA[<h3 id="查看CPU、内存使用率 "> <a href= "#查看CPU、内存使用率" class= "headerlink" title= "查看CPU、内存使用率" > </a> 查看CPU、内存使用率 </h3> <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "meta" > #!/bin/bash </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 定义颜色 </span> </span> <br > <span class= "line" > RED= <span class= "string" > ' \033[0;31m ' </span> </span> <br > <span class= "line" > GREEN= <span class= "string" > ' \033[0;32m ' </span> </span> <br > <span class= "line" > YELLOW= <span class= "string" > ' \033[0;33m ' </span> </span> <br > <span class= "line" > BLUE= <span class= "string" > ' \033[0;34m ' </span> </span> <br > <span class= "line" > NC= <span class= "string" > ' \033[0m ' </span> <span class= "comment" > # 无颜色 </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "keyword" > while </span> <span class= "literal" > true </span> ; <span class= "keyword" > do </span> </span> <br > <span class= "line" > <span class= "comment" > # 获取所有进程的CPU使用率和内存使用率 </span> </span> <br > <span class= "line" > cpu_usage=$(ps aux | awk <span class= "string" > ' { sum_cpu += $3 } END { print sum_cpu } ' </span> ) </span> <br > <span class= "line" > mem_usage=$(ps aux | awk <span class= "string" > ' { sum_mem += $4 } END { print sum_mem } ' </span> ) </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 打印结果,带有时间戳、分隔线和颜色高亮 </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { BLUE } </span> ============================== <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { YELLOW } </span> Timestamp: <span class= "subst" > $(date) </span> <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= " line "> <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { BLUE } </span> ============================== <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { GREEN } </span> Total CPU usage: <span class= "variable" > $ { RED } </span> <span class= "variable" > $cpu_usage </span> % <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { GREEN } </span> Total Memory usage: <span class= "variable" > $ { RED } </span> <span class= "variable" > $mem_usage </span> % <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > <span class= "built_in" > echo </span> -e <span class= "string" > " <span class= "variable" > $ { BLUE } </span> ============================== <span class= "variable" > $ { NC } </span> " </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 等待5秒后再次执行 </span> </span> <br > <span class= "line" > <span class= "built_in" > sleep </span> 5 </span> <br > <span class= "line" > <span class= "keyword" > done </span> </span> <br > <span class= "line" > </span> <br > </pre> </td> </tr> </table> </figure >
<p > <strong > 保存脚本到 / usr / local / bin目录下 </strong> </p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line " > <span class= "built_in "> mv </span> usage.sh /usr/local/bin/usage </span ><br > </pre> </td> </tr> </table> </figure >
<h3 id= "回归决策树" > <a href= "#回归决策树" class= "headerlink" title= "回归决策树" > </a> 回归决策树</h3> <h4 id= "决策树算法的应用-(泰坦尼克号沉船幸存者预测)" > <a href= "#决策树算法的应用-(泰坦尼克号沉船幸存者预测)" class= "headerlink" title= "决策树算法的应用 (泰坦尼克号沉船幸存者预测)" > </a> 决策树算法的应用 (泰坦尼克号沉船幸存者预测)</h4> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> seaborn <span class= "keyword" > as</span> sns</span> <br > <span class= "line" > <span class= "keyword" > import</span> pandas <span class= "keyword" > as</span> pd</span> <br > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.feature_extraction <span class= "keyword" > import</span> DictVectorizer</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.tree <span class= "keyword" > import</span> DecisionTreeClassifier,export_graphviz</span> <br > <span class= "line" > <span class= "comment" > # 1.获取数据集 - 加载 Titanic 数据集</span> </span> <br > <span class= "line" > titanic = sns.load_dataset(<span class= "string" > ' titanic' </span> )</span> <br > <span class= "line" > missing_age_count = titanic[<span class= "string" > ' age' </span> ].isna().<span class= "built_in" > sum</span> ()</span> <br > <span class= "line" > <span class= "comment" > # print(f" 缺失的 age 数量: { missing_age_count} " )</span> </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据基本处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 确认特征值、目标值</span> </span> <br > <span class= "line" > X = titanic[[<span class= "string" > ' pclass' </span> ,<span class= "string" > ' age' </span> ,<span class= "string" > ' sex' </span> ]]</span> <br > <span class= "line" > y = titanic[<span class= "string" > ' survived' </span> ]</span> <br > <span class= "line" > <span class= "comment" > # 2.2 缺失值处理</span> </span> <br > <span class= "line" > X.loc[:, <span class= "string" > ' age' </span> ] = X[<span class= "string" > ' age' </span> ].fillna(value=X[<span class= "string" > ' age' </span> ].mean()) <span class= "comment" > # 使用 .loc 进行修改</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.3 划分数据集</span> </span> <br > <span class= "line" > X_train,X_test,y_train,y_test = train_test_split(X,y,random_state=<span class= "number" > 22</span> )</span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程(字典特征提取)</span> </span> <br > <span class= "line" > X_train = X_train.to_dict(orient=<span class= "string" > " records" </span> )</span> <br > <span class= "line" > X_test= X_test.to_dict(orient=<span class= "string" > " records" </span> )</span> <br > <span class= "line" > transfer = DictVectorizer()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test)</span> <br > <span class= "line" > <span class= "comment" > # 4. 机器学习 决策树算法</span> </span> <br > <span class= "line" > estimator = DecisionTreeClassifier(criterion=<span class= "string" > " gini" </span> )</span> <br > <span class= "line" > estimator.fit(X_train,y_train)</span> <br > <span class= "line" > y_pred = estimator.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的测试集的预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > ret = estimator.score(X_test,y_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的评分:<span class= "subst" > { ret} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (X_test.toarray())</span> <br > </pre> </td> </tr> </table> </figure>
<p > 生成对应的图</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.tree <span class= "keyword" > import</span> export_graphviz</span> <br > <span class= "line" > <span class= "keyword" > import</span> graphviz <span class= "comment" > # 用于渲染图像</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 导出决策树的 Graphviz 表示</span> </span> <br > <span class= "line" > export_graphviz(estimator, out_file=<span class= "string" > ' ./data/tree.dot' </span> , </span> <br > <span class= "line" > feature_names=transfer.get_feature_names_out()) <span class= "comment" > # 特征名称</span> </span> <br > <span class= "line" > <span class= "comment" > # 使用 graphviz 渲染 .dot 文件</span> </span> <br > <span class= "line" > <span class= "keyword" > with</span> <span class= "built_in" > open</span> (<span class= "string" > ' ./data/tree.dot' </span> , <span class= "string" > ' r' </span> ) <span class= "keyword" > as</span> f:</span> <br > <span class= "line" > dot_graph = f.read()</span> <br > <span class= "line" > <span class= "comment" > # 渲染决策树</span> </span> <br > <span class= "line" > graph = graphviz.Source(dot_graph)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 设置保存路径</span> </span> <br > <span class= "line" > output_path = <span class= "string" > ' ./data/decision_tree' </span> <span class= "comment" > # 自定义保存路径</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 保存图像到指定路径,格式可以是 .png, .pdf, .jpg 等</span> </span> <br > <span class= "line" > <span class= "comment" > # graph.render(output_path, format=' png' ) # 保存为 .png 文件</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 显示图像</span> </span> <br > <span class= "line" > graph.view(output_path) <span class= "comment" > # 打开图像,path为保存路径,不需要加后缀</span> </span> <br > <span class= "line" > </span> <br > </pre> </td> </tr> </table> </figure>
<p > <a href= "http://webgraphviz.com/" > Webgraphviz</a> ,这个网站可以将<code > tree.dot</code> 文件的内容生成对应的可视化树</p>
<h4 id= "回归决策树与线性回归的对比" > <a href= "#回归决策树与线性回归的对比" class= "headerlink" title= "回归决策树与线性回归的对比" > </a> 回归决策树与线性回归的对比</h4> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > import</span> matplotlib.pyplot <span class= "keyword" > as</span> plt</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LinearRegression</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.tree <span class= "keyword" > import</span> DecisionTreeRegressor</span> <br > <span class= "line" > <span class= "keyword" > from</span> matplotlib <span class= "keyword" > import</span> rcParams</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 设置matplotlib使用的字体为SimHei(黑体)</span> </span> <br > <span class= "line" > rcParams[<span class= "string" > ' font.sans-serif' </span> ] = [<span class= "string" > ' SimHei' </span> ] <span class= "comment" > # 也可以使用 ' Microsoft YaHei' </span> </span> <br > <span class= "line" > rcParams[<span class= "string" > ' axes.unicode_minus' </span> ] = <span class= "literal" > False</span> <span class= "comment" > # 正常显示负号</span> </span> <br > <span class= "line" > x = np.array(<span class= "built_in" > list</span> (<span class= "built_in" > range</span> (<span class= "number" > 1</span> ,<span class= "number" > 11</span> ))).reshape(-<span class= "number" > 1</span> ,<span class= "number" > 1</span> )</span> <br > <span class= "line" > y = ([<span class= "number" > 5.56</span> ,<span class= "number" > 5.70</span> ,<span class= "number" > 5.91</span> ,<span class= "number" > 6.40</span> ,<span class= "number" > 6.80</span> ,<span class= "number" > 7.05</span> ,<span class= "number" > 8.90</span> ,<span class= "number" > 8.70</span> ,<span class= "number" > 9.00</span> ,<span class= "number" > 9.05</span> ])</span> <br > <span class= "line" > </span> <br > <span class= "line" > m1 = DecisionTreeRegressor(max_depth=<span class= "number" > 1</span> )</span> <br > <span class= "line" > m2 = DecisionTreeRegressor(max_depth=<span class= "number" > 3</span> )</span> <br > <span class= "line" > m3 = DecisionTreeRegressor()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 模型训练</span> </span> <br > <span class= "line" > m1.fit(x,y)</span> <br > <span class= "line" > m2.fit(x,y)</span> <br > <span class= "line" > m3.fit(x,y)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 模型预测</span> </span> <br > <span class= "line" > x_test = np.arange(<span class= "number" > 0</span> ,<span class= "number" > 10</span> ,<span class= "number" > 0.01</span> ).reshape(-<span class= "number" > 1</span> ,<span class= "number" > 1</span> )</span> <br > <span class= "line" > y_1 = m1.predict(x_test)</span> <br > <span class= "line" > y_2 = m2.predict(x_test)</span> <br > <span class= "line" > y_3 = m3.predict(x_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 结果展示</span> </span> <br > <span class= "line" > plt.figure(figsize=(<span class= "number" > 10</span> ,<span class= "number" > 6</span> ),dpi=<span class= "number" > 100</span> )</span> <br > <span class= "line" > plt.scatter(x,y ,label = <span class= "string" > " data" </span> )</span> <br > <span class= "line" > plt.plot(x_test,y_1,label = <span class= "string" > " max_depth=1" </span> )</span> <br > <span class= "line" > plt.plot(x_test,y_2,label = <span class= "string" > " max_depth=3" </span> )</span> <br > <span class= "line" > plt.plot(x_test,y_3,label = <span class= "string" > " linearregression" </span> )</span> <br > <span class= "line" > plt.xlabel(<span class= "string" > " 数据" </span> )</span> <br > <span class= "line" > plt.ylabel(<span class= "string" > " 预测值" </span> )</span> <br > <span class= "line" > plt.legend()</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
]]></content>
<categories >
<category > machinelearning</category>
</categories>
<tags >
<tag > decisiontree</tag>
</tags>
</entry>
<entry >
<title > 逻辑回归</title>
<url > /posts/60504.html</url>
<content > <![CDATA[<h3 id="logistic-regression-code"> <a href= "#logistic-regression-code" class= "headerlink" title= "logistic regression code" > </a> logistic regression code</h3> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> pandas <span class= "keyword" > as</span> pd</span> <br > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> load_breast_cancer</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LogisticRegression</span> <br > <span class= "line" > <span class= "comment" > # 1. 加载乳腺癌数据集</span> </span> <br > <span class= "line" > data = load_breast_cancer()</span> <br > <span class= "line" > <span class= "comment" > # 2.1 数据集基本处理</span> </span> <br > <span class= "line" > df = pd.DataFrame(data.data, columns=data.feature_names)</span> <br > <span class= "line" > df[<span class= "string" > ' target' </span> ] = data.target</span> <br > <span class= "line" > <span class= "keyword" > for</span> i <span class= "keyword" > in</span> df.columns:</span> <br > <span class= "line" > <span class= "comment" > # 检查列是否有缺失值</span> </span> <br > <span class= "line" > <span class= "keyword" > if</span> np.<span class= "built_in" > any</span> (pd.isnull(df[i])):</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Filling missing values in column: <span class= "subst" > { i} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > #2.2 确认特征值、目标值</span> </span> <br > <span class= "line" > X = df.iloc[:,<span class= "number" > 0</span> :df.shape[<span class= "number" > 1</span> ] - <span class= "number" > 1</span> ]</span> <br > <span class= "line" > y = df.loc[:,<span class= "string" > " target" </span> ]</span> <br > <span class= "line" > <span class= "comment" > # 2.3 分割数据</span> </span> <br > <span class= "line" > X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=<span class= "number" > 0.3</span> )</span> <br > <span class= "line" > <span class= "comment" > # 显示前几行数据</span> </span> <br > <span class= "line" > df.head(<span class= "number" > 1</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4 机器学习 逻辑回归</span> </span> <br > <span class= "line" > estimator = LogisticRegression()</span> <br > <span class= "line" > estimator.fit(X_train,y_train)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型准确率:<span class= "subst" > { estimator.score(X_test,y_test)} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型预测值为:\n<span class= "subst" > { estimator.predict(X_test)} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
<h3 id= "分类评估的参数" > <a href= "#分类评估的参数" class= "headerlink" title= "分类评估的参数" > </a> 分类评估的参数</h3> <ul >
<li > <p > 准确率<br > 准确率是所有预测正确的样本占总样本的比例<br > $$Accuracy = \frac{TP+TN}{TP+FN+FP+TN}$$</p>
</li>
<li > <p > 精准率<br > 精准率(又称查准率)是指所有被预测为正类的样本中,真正为正类的比例<br > $$Precision = \frac{TP}{TP+FP}$$</p>
</li>
<li > <p > 召回率<br > 召回率(又称查全率)是指所有实际为正类的样本中,被正确预测为正类的比例<br > $$Recall = \frac{TP}{TP+FN}$$</p>
</li>
<li > <p > F1-score<br > F1 值(F1 Score)是精准率和召回率的调和平均数,综合考虑了精准率和召回率的影响。<br > $$ F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} $$</p>
</li>
<li > <p > roc曲线<br > tpr、fpr来衡量不平衡的二分类问题</p>
</li>
</ul>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> pandas <span class= "keyword" > as</span> pd</span> <br > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> load_breast_cancer</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LogisticRegression</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> classification_report, roc_auc_score</span> <br > <span class= "line" > <span class= "comment" > # 1. 加载乳腺癌数据集</span> </span> <br > <span class= "line" > data = load_breast_cancer()</span> <br > <span class= "line" > <span class= "comment" > # 2.1 数据集基本处理</span> </span> <br > <span class= "line" > df = pd.DataFrame(data.data, columns=data.feature_names)</span> <br > <span class= "line" > df[<span class= "string" > ' target' </span> ] = data.target</span> <br > <span class= "line" > <span class= "keyword" > for</span> i <span class= "keyword" > in</span> df.columns:</span> <br > <span class= "line" > <span class= "comment" > # 检查列是否有缺失值</span> </span> <br > <span class= "line" > <span class= "keyword" > if</span> np.<span class= "built_in" > any</span> (pd.isnull(df[i])):</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Filling missing values in column: <span class= "subst" > { i} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 2.2 确认特征值、目标值</span> </span> <br > <span class= "line" > X = df.iloc[:, <span class= "number" > 0</span> :df.shape[<span class= "number" > 1</span> ] - <span class= "number" > 1</span> ]</span> <br > <span class= "line" > y = df.loc[:, <span class= "string" > " target" </span> ]</span> <br > <span class= "line" > <span class= "comment" > # 2.3 分割数据</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span class= "number" > 0.3</span> )</span> <br > <span class= "line" > <span class= "comment" > # 显示前几行数据</span> </span> <br > <span class= "line" > df.head(<span class= "number" > 1</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4 机器学习 逻辑回归</span> </span> <br > <span class= "line" > estimator = LogisticRegression()</span> <br > <span class= "line" > estimator.fit(X_train, y_train)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型准确率:<span class= "subst" > { estimator.score(X_test, y_test)} </span> " </span> )</span> <br > <span class= "line" > y_pred = estimator.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型预测值为:\n<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 5.1 精确率、召回率</span> </span> <br > <span class= "line" > ret = classification_report(y_test, y_pred, labels=[<span class= "number" > 1</span> , <span class= "number" > 0</span> ], target_names=[<span class= "string" > " 良性" </span> , <span class= "string" > " 恶性" </span> ])</span> <br > <span class= "line" > roc_score = roc_auc_score(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 准确率、召回率:<span class= "subst" > { ret} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" roc_score:<span class= "subst" > { roc_score} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
<h3 id= "类别不平衡的处理" > <a href= "#类别不平衡的处理" class= "headerlink" title= "类别不平衡的处理" > </a> 类别不平衡的处理</h3> <p > 先准备类别不平衡的数据</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> imblearn.over_sampling <span class= "keyword" > import</span> RandomOverSampler,SMOTE</span> <br > <span class= "line" > <span class= "keyword" > from</span> imblearn.under_sampling <span class= "keyword" > import</span> RandomUnderSampler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> make_classification</span> <br > <span class= "line" > <span class= "keyword" > import</span> matplotlib.pyplot <span class= "keyword" > as</span> plt</span> <br > <span class= "line" > <span class= "keyword" > from</span> collections <span class= "keyword" > import</span> Counter</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1.准备类别不平衡的数据</span> </span> <br > <span class= "line" > X, y = make_classification(</span> <br > <span class= "line" > n_samples=<span class= "number" > 5000</span> ,</span> <br > <span class= "line" > n_features=<span class= "number" > 2</span> ,</span> <br > <span class= "line" > n_informative=<span class= "number" > 2</span> ,</span> <br > <span class= "line" > n_redundant=<span class= "number" > 0</span> ,</span> <br > <span class= "line" > n_repeated=<span class= "number" > 0</span> ,</span> <br > <span class= "line" > n_classes=<span class= "number" > 3</span> ,</span> <br > <span class= "line" > n_clusters_per_class=<span class= "number" > 1</span> ,</span> <br > <span class= "line" > weights=[<span class= "number" > 0.01</span> , <span class= "number" > 0.05</span> , <span class= "number" > 0.94</span> ],</span> <br > <span class= "line" > random_state=<span class= "number" > 0</span> ,</span> <br > <span class= "line" > )</span> <br > <span class= "line" > counter = Counter(y)</span> <br > <span class= "line" > plt.scatter(X[:,<span class= "number" > 0</span> ],X[:,<span class= "number" > 1</span> ],c=y)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<ul >
<li > 过采样<br > 增加训练集的少数的类别的样本,使得正反例样本数据接近 <ul >
<li > 随机过采样(RandomOverSampler)<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > ros = RandomOverSampler()</span> <br > <span class= "line" > X_resampled,y_resampled = ros.fit_resample(X,y)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (Counter(y_resampled))</span> <br > <span class= "line" > plt.scatter(X_resampled[:,<span class= "number" > 0</span> ],X_resampled[:,<span class= "number" > 1</span> ],c=y_resampled)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<img src= "/img/machinelearning/over_random_sampling.png" > </li>
<li > <code > SMOTE</code> 过采样(SMOTE) <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > smote = SMOTE()</span> <br > <span class= "line" > X_resampled,y_resampled = smote.fit_resample(X,y)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (Counter(y_resampled))</span> <br > <span class= "line" > plt.scatter(X_resampled[:,<span class= "number" > 0</span> ],X_resampled[:,<span class= "number" > 1</span> ],c=y_resampled)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<img src= "/img/machinelearning/over_smote_sampling.png" > </li>
</ul>
</li>
<li > 欠采样<br > 减少训练集的多数的类别的样本,使得正反例样本数据接近<ul >
<li > 随机欠采样(RandomUnderSampler) <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > rus = RandomUnderSampler(random_state=<span class= "number" > 0</span> )</span> <br > <span class= "line" > X_resampled,y_resampled = rus.fit_resample(X,y)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (Counter(y_resampled))</span> <br > <span class= "line" > plt.scatter(X_resampled[:,<span class= "number" > 0</span> ],X_resampled[:,<span class= "number" > 1</span> ],c=y_resampled)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<img src= "/img/machinelearning/under_sampling.png" > </li>
</ul>
</li>
</ul>
]]></content>
<categories >
<category > machinelearning</category>
</categories>
<tags >
<tag > logistic-regression</tag>
</tags>
</entry>
<entry >
<title > 网络相关</title>
<url > /posts/41168.html</url>
<content > <![CDATA[]]> </content>
</entry>
<entry >
<title > 线性回归</title>
<url > /posts/52662.html</url>
<content > <![CDATA[<h3 id="线性回归简介"> <a href= "#线性回归简介" class= "headerlink" title= "线性回归简介" > </a> 线性回归简介</h3> <blockquote >
<p > 用于预测一个连续的目标变量(因变量),与一个或多个特征(自变量)之间存在线性关系。</p>
</blockquote>
<p > 假设函数:<br > $$y = w_1x_1 + w_2x_2 + \cdot\cdot\cdot+w_nx_n$$</p>
<ul >
<li > $y$ 是目标变量(因变量),即我们希望预测的值。</li>
<li > $x1 ,x2,…,xn$ 是特征变量(自变量),即输入的值。</li>
</ul>
<h3 id= "损失函数" > <a href= "#损失函数" class= "headerlink" title= "损失函数" > </a> 损失函数</h3> <p > 为了找到最佳的线性模型,我们需要通过最小化损失函数来优化模型参数。在线性回归中,常用的损失函数是 <strong > 均方误差(MSE) </strong> : <br > $$J(\theta) = \frac{1}{2N} \sum_{i= 1}^{N} (y_i - f_\theta(x_i))^2$$</p>
<ul >
<li > N 是样本的数量。</li>
<li > $y_i$ 是第 i 个样本的真实值。</li>
<li > $f_\theta(x_i)$ 是模型预测的第 i 个样本的值。</li>
</ul>
<h3 id= "线性回归优化" > <a href= "#线性回归优化" class= "headerlink" title= "线性回归优化" > </a> 线性回归优化</h3> <ul >
<li > <p > 梯度下降法</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> SGDRegressor</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4.机器学习- 梯度下降法</span> </span> <br > <span class= "line" > estimater = SGDRegressor(max_iter=<span class= "number" > 1000</span> , eta0=<span class= "number" > 0.01</span> )</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" SGD模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
</li>
<li > <p > 正规方程</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LinearRegression</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.fit_transform(X_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4.机器学习- 线性回归</span> </span> <br > <span class= "line" > estimater = LinearRegression()</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
</li>
<li > <p > 岭回归</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> Ridge, RidgeCV</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4.机器学习- 岭回归 使用了Ridge的alpha的搜索</span> </span> <br > <span class= "line" > <span class= "comment" > # estimater = Ridge(alpha=1.0)</span> </span> <br > <span class= "line" > estimater = RidgeCV(alphas=[<span class= "number" > 0.001</span> , <span class= "number" > 0.01</span> , <span class= "number" > 0.1</span> , <span class= "number" > 1</span> , <span class= "number" > 10</span> , <span class= "number" > 100</span> ])</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 查看最佳 alpha</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 最佳 alpha 值是:<span class= "subst" > { estimater.alpha_} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型预测值:<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<p > 这样每个代码块的缩进保持一致,便于阅读和理解。如果有其他优化需求,随时告诉我!</p>
<p > <img src= "/img/machinelearning/linear.png" > </p>
<p > <img src= "/img/machinelearning/fitting.png" > </p>
<h3 id= "模型保存和加载" > <a href= "#模型保存和加载" class= "headerlink" title= "模型保存和加载" > </a> 模型保存和加载</h3> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> fetch_california_housing</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> Ridge, RidgeCV</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> mean_squared_error</span> <br > <span class= "line" > <span class= "keyword" > import</span> joblib</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "keyword" > def</span> <span class= "title function_" > save_model</span> ():</span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > <span class= "comment" > # 4. 机器学习 - 岭回归 使用了Ridge的alpha的搜索</span> </span> <br > <span class= "line" > estimater = RidgeCV(alphas=[<span class= "number" > 0.001</span> , <span class= "number" > 0.01</span> , <span class= "number" > 0.1</span> , <span class= "number" > 1</span> , <span class= "number" > 10</span> , <span class= "number" > 100</span> ])</span> <br > <span class= "line" > estimater.fit(X_train, y_train)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的偏置是:<span class= "subst" > { estimater.intercept_} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型的系数是:<span class= "subst" > { estimater.coef_} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 保存模型</span> </span> <br > <span class= "line" > joblib.dump(estimater, <span class= "string" > ' ridge_model.pkl' </span> )</span> <br > <span class= "line" > <span class= "comment" > # 查看最佳 alpha</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 最佳 alpha 值是:<span class= "subst" > { estimater.alpha_} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > y_pred = estimater.predict(X_test)</span> <br > <span class= "line" > mse = mean_squared_error(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Ridge模型均方误差:<span class= "subst" > { mse} </span> " </span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "keyword" > def</span> <span class= "title function_" > load_model</span> ():</span> <br > <span class= "line" > <span class= "comment" > # 1. 获取数据集</span> </span> <br > <span class= "line" > housing = fetch_california_housing()</span> <br > <span class= "line" > <span class= "comment" > # 2. 数据集处理</span> </span> <br > <span class= "line" > <span class= "comment" > # 2.1 分割数据集</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class= "number" > 0.25</span> )</span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程</span> </span> <br > <span class= "line" > <span class= "comment" > # 3.1 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test) <span class= "comment" > # 使用 transform() 而不是 fit_transform()</span> </span> <br > <span class= "line" > <span class= "comment" > # 加载模型</span> </span> <br > <span class= "line" > estimater = joblib.load(<span class= "string" > ' ridge_model.pkl' </span> )</span> <br > <span class= "line Line truncated
<categories >
<category > machinelearning</category>
</categories>
<tags >
<tag > linear-regression</tag>
</tags>
</entry>
<entry >
<title > 郑伯克段于鄢</title>
<url > /posts/58638.html</url>
<content > <![CDATA[<p> 原文如下:</p>
<p > 初,郑武公娶于申【申国】,曰武姜【武为武公谥号,姜为其宗族之性】。生庄公及共叔段【共表示其曾出逃到共,叔为老三,段为名】。庄公寤生【出生时头先出,难产】,惊姜氏,故名曰“寤生”, 遂恶之,爱【喜爱】共叔段,欲立【立为储君】之,亟(qì)【多次】请于武公,及庄公即位,为之【共叔段】请制【一个叫制的封地,虎牢关所在】。公曰:“制,岩邑【险要的城邑】也,虢叔死焉,佗【通“他”,其他】邑唯命(是听)。”请京,使居之,谓之“京城大叔”。</p>
]]></content>
<categories >
<category > 古文观止</category>
</categories>
<tags >
<tag > 古文观止</tag>
</tags>
<h3 id= "Shell脚本编写的基本信息" > <a href= "#Shell脚本编写的基本信息" class= "headerlink" title= "Shell脚本编写的基本信息" > </a> Shell脚本编写的基本信息</h3> <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "meta" > #! /bin/bash</span> </span> <br > <span class= "line" > <span class= "comment" > # -------------------------------------------------</span> </span> <br > <span class= "line" > <span class= "comment" > # Filename: test.sh</span> </span> <br > <span class= "line" > <span class= "comment" > # Version: 1.0</span> </span> <br > <span class= "line" > <span class= "comment" > # Date: 2024/05/02</span> </span> <br > <span class= "line" > <span class= "comment" > # Author: shenjianZ</span> </span> <br > <span class= "line" > <span class= "comment" > # Email: shenjianZLT@gmail.com</span> </span> <br > <span class= "line" > <span class= "comment" > # Website: https://blog.shenjianl.cn</span> </span> <br > <span class= "line" > <span class= "comment" > # Description: this is a test shell</span> </span> <br > <span class= "line" > <span class= "comment" > # CopyRight: 2024 All rights reserved shenjianZ</span> </span> <br > <span class= "line" > <span class= "comment" > # License GPL</span> </span> <br > <span class= "line" > <span class= "comment" > # ------------------------------------------------</span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # Your script logic goes here</span> </span> <br > </pre> </td> </tr> </table> </figure> ]]></content>
</entry>
<entry >
<title > Hadoop集群搭建基础环境</title>
@@ -343,6 +286,63 @@
</ul>
]]></content>
</entry>
<entry >
<title > 逻辑回归</title>
<url > /posts/60504.html</url>
<content > <![CDATA[<h3 id="logistic-regression-code"> <a href= "#logistic-regression-code" class= "headerlink" title= "logistic regression code" > </a> logistic regression code</h3> <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> pandas <span class= "keyword" > as</span> pd</span> <br > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> load_breast_cancer</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LogisticRegression</span> <br > <span class= "line" > <span class= "comment" > # 1. 加载乳腺癌数据集</span> </span> <br > <span class= "line" > data = load_breast_cancer()</span> <br > <span class= "line" > <span class= "comment" > # 2.1 数据集基本处理</span> </span> <br > <span class= "line" > df = pd.DataFrame(data.data, columns=data.feature_names)</span> <br > <span class= "line" > df[<span class= "string" > ' target' </span> ] = data.target</span> <br > <span class= "line" > <span class= "keyword" > for</span> i <span class= "keyword" > in</span> df.columns:</span> <br > <span class= "line" > <span class= "comment" > # 检查列是否有缺失值</span> </span> <br > <span class= "line" > <span class= "keyword" > if</span> np.<span class= "built_in" > any</span> (pd.isnull(df[i])):</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Filling missing values in column: <span class= "subst" > { i} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > #2.2 确认特征值、目标值</span> </span> <br > <span class= "line" > X = df.iloc[:,<span class= "number" > 0</span> :df.shape[<span class= "number" > 1</span> ] - <span class= "number" > 1</span> ]</span> <br > <span class= "line" > y = df.loc[:,<span class= "string" > " target" </span> ]</span> <br > <span class= "line" > <span class= "comment" > # 2.3 分割数据</span> </span> <br > <span class= "line" > X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=<span class= "number" > 0.3</span> )</span> <br > <span class= "line" > <span class= "comment" > # 显示前几行数据</span> </span> <br > <span class= "line" > df.head(<span class= "number" > 1</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4 机器学习 逻辑回归</span> </span> <br > <span class= "line" > estimator = LogisticRegression()</span> <br > <span class= "line" > estimator.fit(X_train,y_train)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型准确率:<span class= "subst" > { estimator.score(X_test,y_test)} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型预测值为:\n<span class= "subst" > { estimator.predict(X_test)} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
<h3 id= "分类评估的参数" > <a href= "#分类评估的参数" class= "headerlink" title= "分类评估的参数" > </a> 分类评估的参数</h3> <ul >
<li > <p > 准确率<br > 准确率是所有预测正确的样本占总样本的比例<br > $$Accuracy = \frac{TP+TN}{TP+FN+FP+TN}$$</p>
</li>
<li > <p > 精准率<br > 精准率(又称查准率)是指所有被预测为正类的样本中,真正为正类的比例<br > $$Precision = \frac{TP}{TP+FP}$$</p>
</li>
<li > <p > 召回率<br > 召回率(又称查全率)是指所有实际为正类的样本中,被正确预测为正类的比例<br > $$Recall = \frac{TP}{TP+FN}$$</p>
</li>
<li > <p > F1-score<br > F1 值(F1 Score)是精准率和召回率的调和平均数,综合考虑了精准率和召回率的影响。<br > $$ F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} $$</p>
</li>
<li > <p > roc曲线<br > tpr、fpr来衡量不平衡的二分类问题</p>
</li>
</ul>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > import</span> pandas <span class= "keyword" > as</span> pd</span> <br > <span class= "line" > <span class= "keyword" > import</span> numpy <span class= "keyword" > as</span> np</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> load_breast_cancer</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.model_selection <span class= "keyword" > import</span> train_test_split</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.preprocessing <span class= "keyword" > import</span> StandardScaler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.linear_model <span class= "keyword" > import</span> LogisticRegression</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.metrics <span class= "keyword" > import</span> classification_report, roc_auc_score</span> <br > <span class= "line" > <span class= "comment" > # 1. 加载乳腺癌数据集</span> </span> <br > <span class= "line" > data = load_breast_cancer()</span> <br > <span class= "line" > <span class= "comment" > # 2.1 数据集基本处理</span> </span> <br > <span class= "line" > df = pd.DataFrame(data.data, columns=data.feature_names)</span> <br > <span class= "line" > df[<span class= "string" > ' target' </span> ] = data.target</span> <br > <span class= "line" > <span class= "keyword" > for</span> i <span class= "keyword" > in</span> df.columns:</span> <br > <span class= "line" > <span class= "comment" > # 检查列是否有缺失值</span> </span> <br > <span class= "line" > <span class= "keyword" > if</span> np.<span class= "built_in" > any</span> (pd.isnull(df[i])):</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" Filling missing values in column: <span class= "subst" > { i} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 2.2 确认特征值、目标值</span> </span> <br > <span class= "line" > X = df.iloc[:, <span class= "number" > 0</span> :df.shape[<span class= "number" > 1</span> ] - <span class= "number" > 1</span> ]</span> <br > <span class= "line" > y = df.loc[:, <span class= "string" > " target" </span> ]</span> <br > <span class= "line" > <span class= "comment" > # 2.3 分割数据</span> </span> <br > <span class= "line" > X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span class= "number" > 0.3</span> )</span> <br > <span class= "line" > <span class= "comment" > # 显示前几行数据</span> </span> <br > <span class= "line" > df.head(<span class= "number" > 1</span> )</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 3. 特征工程 标准化</span> </span> <br > <span class= "line" > transfer = StandardScaler()</span> <br > <span class= "line" > X_train = transfer.fit_transform(X_train)</span> <br > <span class= "line" > X_test = transfer.transform(X_test)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 4 机器学习 逻辑回归</span> </span> <br > <span class= "line" > estimator = LogisticRegression()</span> <br > <span class= "line" > estimator.fit(X_train, y_train)</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 5. 模型评估</span> </span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型准确率:<span class= "subst" > { estimator.score(X_test, y_test)} </span> " </span> )</span> <br > <span class= "line" > y_pred = estimator.predict(X_test)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 模型预测值为:\n<span class= "subst" > { y_pred} </span> " </span> )</span> <br > <span class= "line" > <span class= "comment" > # 5.1 精确率、召回率</span> </span> <br > <span class= "line" > ret = classification_report(y_test, y_pred, labels=[<span class= "number" > 1</span> , <span class= "number" > 0</span> ], target_names=[<span class= "string" > " 良性" </span> , <span class= "string" > " 恶性" </span> ])</span> <br > <span class= "line" > roc_score = roc_auc_score(y_test, y_pred)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" 准确率、召回率:<span class= "subst" > { ret} </span> " </span> )</span> <br > <span class= "line" > <span class= "built_in" > print</span> (<span class= "string" > f" roc_score:<span class= "subst" > { roc_score} </span> " </span> )</span> <br > </pre> </td> </tr> </table> </figure>
<h3 id= "类别不平衡的处理" > <a href= "#类别不平衡的处理" class= "headerlink" title= "类别不平衡的处理" > </a> 类别不平衡的处理</h3> <p > 先准备类别不平衡的数据</p>
<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "keyword" > from</span> imblearn.over_sampling <span class= "keyword" > import</span> RandomOverSampler,SMOTE</span> <br > <span class= "line" > <span class= "keyword" > from</span> imblearn.under_sampling <span class= "keyword" > import</span> RandomUnderSampler</span> <br > <span class= "line" > <span class= "keyword" > from</span> sklearn.datasets <span class= "keyword" > import</span> make_classification</span> <br > <span class= "line" > <span class= "keyword" > import</span> matplotlib.pyplot <span class= "keyword" > as</span> plt</span> <br > <span class= "line" > <span class= "keyword" > from</span> collections <span class= "keyword" > import</span> Counter</span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "comment" > # 1.准备类别不平衡的数据</span> </span> <br > <span class= "line" > X, y = make_classification(</span> <br > <span class= "line" > n_samples=<span class= "number" > 5000</span> ,</span> <br > <span class= "line" > n_features=<span class= "number" > 2</span> ,</span> <br > <span class= "line" > n_informative=<span class= "number" > 2</span> ,</span> <br > <span class= "line" > n_redundant=<span class= "number" > 0</span> ,</span> <br > <span class= "line" > n_repeated=<span class= "number" > 0</span> ,</span> <br > <span class= "line" > n_classes=<span class= "number" > 3</span> ,</span> <br > <span class= "line" > n_clusters_per_class=<span class= "number" > 1</span> ,</span> <br > <span class= "line" > weights=[<span class= "number" > 0.01</span> , <span class= "number" > 0.05</span> , <span class= "number" > 0.94</span> ],</span> <br > <span class= "line" > random_state=<span class= "number" > 0</span> ,</span> <br > <span class= "line" > )</span> <br > <span class= "line" > counter = Counter(y)</span> <br > <span class= "line" > plt.scatter(X[:,<span class= "number" > 0</span> ],X[:,<span class= "number" > 1</span> ],c=y)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<ul >
<li > 过采样<br > 增加训练集的少数的类别的样本,使得正反例样本数据接近 <ul >
<li > 随机过采样(RandomOverSampler)<figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > ros = RandomOverSampler()</span> <br > <span class= "line" > X_resampled,y_resampled = ros.fit_resample(X,y)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (Counter(y_resampled))</span> <br > <span class= "line" > plt.scatter(X_resampled[:,<span class= "number" > 0</span> ],X_resampled[:,<span class= "number" > 1</span> ],c=y_resampled)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<img src= "/img/machinelearning/over_random_sampling.png" > </li>
<li > <code > SMOTE</code> 过采样(SMOTE) <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > smote = SMOTE()</span> <br > <span class= "line" > X_resampled,y_resampled = smote.fit_resample(X,y)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (Counter(y_resampled))</span> <br > <span class= "line" > plt.scatter(X_resampled[:,<span class= "number" > 0</span> ],X_resampled[:,<span class= "number" > 1</span> ],c=y_resampled)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<img src= "/img/machinelearning/over_smote_sampling.png" > </li>
</ul>
</li>
<li > 欠采样<br > 减少训练集的多数的类别的样本,使得正反例样本数据接近<ul >
<li > 随机欠采样(RandomUnderSampler) <figure class= "highlight python" > <table > <tr > <td class= "code" > <pre > <span class= "line" > rus = RandomUnderSampler(random_state=<span class= "number" > 0</span> )</span> <br > <span class= "line" > X_resampled,y_resampled = rus.fit_resample(X,y)</span> <br > <span class= "line" > <span class= "built_in" > print</span> (Counter(y_resampled))</span> <br > <span class= "line" > plt.scatter(X_resampled[:,<span class= "number" > 0</span> ],X_resampled[:,<span class= "number" > 1</span> ],c=y_resampled)</span> <br > <span class= "line" > plt.show()</span> <br > </pre> </td> </tr> </table> </figure>
<img src= "/img/machinelearning/under_sampling.png" > </li>
</ul>
</li>
</ul>
]]></content>
<categories >
<category > machinelearning</category>
</categories>
<tags >
<tag > logistic-regression</tag>
</tags>
</entry>
<entry >
<title > 郑伯克段于鄢</title>
<url > /posts/58638.html</url>
<content > <![CDATA[<p> 原文如下:</p>
<p > 初,郑武公娶于申【申国】,曰武姜【武为武公谥号,姜为其宗族之性】。生庄公及共叔段【共表示其曾出逃到共,叔为老三,段为名】。庄公寤生【出生时头先出,难产】,惊姜氏,故名曰“寤生”, 遂恶之,爱【喜爱】共叔段,欲立【立为储君】之,亟(qì)【多次】请于武公,及庄公即位,为之【共叔段】请制【一个叫制的封地,虎牢关所在】。公曰:“制,岩邑【险要的城邑】也,虢叔死焉,佗【通“他”,其他】邑唯命(是听)。”请京,使居之,谓之“京城大叔”。</p>
]]></content>
<categories >
<category > 古文观止</category>
</categories>
<tags >
<tag > 古文观止</tag>
</tags>
</entry>
<entry >
<title > Hadoop集群HDFS配置</title>
<url > /posts/61252.html</url>
@@ -390,6 +390,65 @@
</ul>
<h3 id= "集群启动" > <a href= "#集群启动" class= "headerlink" title= "集群启动" > </a> 集群启动</h3> <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > start-dfs.sh</span> <br > </pre> </td> </tr> </table> </figure> ]]></content>
</entry>
<entry >
<title > Hadoop集群Zookeeper配置</title>
<url > /posts/61251.html</url>
<content > <![CDATA[<h3 id="Zookeeper脚本配置"> <a href= "#Zookeeper脚本配置" class= "headerlink" title= "Zookeeper脚本配置" > </a> Zookeeper脚本配置</h3> <ul >
<li > 拷贝ips<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > cp</span> ips ips_zookeeper</span> <br > </pre> </td> </tr> </table> </figure>
修改为<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > nn1</span> <br > <span class= "line" > nn2</span> <br > <span class= "line" > nn3</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > 拷贝三个脚本<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > cp</span> scp_all.sh scp_all_zookeeper.sh </span> <br > <span class= "line" > <span class= "built_in" > cp</span> ssh_all.sh ssh_all_zookeeper.sh </span> <br > <span class= "line" > <span class= "built_in" > cp</span> ssh_root.sh ssh_root_zookeeper.sh</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > 修改脚本<figure class= "highlight shell" > <table > <tr > <td class= "code" > <pre > <span class= "line" > vim scp_all_zookeeper.sh </span> <br > <span class= "line" > vim ssh_all_zookeeper.sh </span> <br > <span class= "line" > vim ssh_root_zookeeper.sh </span> <br > </pre> </td> </tr> </table> </figure>
将三个脚本中的ips改为ips_zookeeper</li>
</ul>
<h3 id= "Zookeeper安装" > <a href= "#Zookeeper安装" class= "headerlink" title= "Zookeeper安装" > </a> Zookeeper安装</h3> <ul >
<li > 上传到<code > /tmp</code> 目录下,解压 <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > sudo</span> tar -zxvf /tmp/zookeeper-3.4.8.tar.gz -C /usr/local/</span> <br > <span class= "line" > scp -r /usr/local/zookeeper-3.4.8/ root@nn2:/usr/local/</span> <br > <span class= "line" > scp -r /usr/local/zookeeper-3.4.8/ root@nn3:/usr/local/</span> <br > <span class= "line" > ssh_root_zookeeper.sh <span class= "built_in" > chown</span> -R hadoop:hadoop /usr/local/zookeeper-3.4.8</span> <br > <span class= "line" > ssh_root_zookeeper.s <span class= "built_in" > ln</span> -s /usr/local/zookeeper-3.4.8/ /usr/local/zookeeper</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<h3 id= "Zookeeper配置" > <a href= "#Zookeeper配置" class= "headerlink" title= "Zookeeper配置" > </a> Zookeeper配置</h3> <ul >
<li > <p > zoo.cfg配置</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > cd</span> /usr/local/zookeeper/conf/</span> <br > <span class= "line" > <span class= "built_in" > cp</span> zoo_sample.cfg zoo.cfg</span> <br > </pre> </td> </tr> </table> </figure>
<p > 然后<code > vim zoo.cfg</code> ,修改如下:</p>
<figure class= "highlight properties" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # 修改dataDir</span> </span> <br > <span class= "line" > <span class= "attr" > dataDir</span> =<span class= "string" > /data/zookeeper</span> </span> <br > <span class= "line" > <span class= "comment" > # 添加一下内容</span> </span> <br > <span class= "line" > <span class= "attr" > server.1</span> =<span class= "string" > nn1:2888:3888 </span> </span> <br > <span class= "line" > <span class= "attr" > server.2</span> =<span class= "string" > nn2:2888:3888 </span> </span> <br > <span class= "line" > <span class= "attr" > server.3</span> =<span class= "string" > nn3:2888:3888</span> </span> <br > </pre> </td> </tr> </table> </figure>
<p > 分发给nn2、nn3<br > <code > scp_all_zookeeper.sh /usr/local/zookeeper/conf/zoo.cfg /usr/local/zookeeper/conf/</code> </p>
</li>
<li > <p > <code > zkEnv.sh</code> 配置<br > <code > vim /usr/local/zookeeper/bin/zkEnv.sh</code> <br > <img src= "/../../../img/000001.png" alt= "000001.png" > <br > 分发到nn2、nn3</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > scp_all_zookeeper.sh /usr/local/zookeeper/bin/zkEnv.sh /usr/local/zookeeper/bin/</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <p > 创建zookeeper数据目录</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > ssh_root_zookeeper.sh <span class= "built_in" > mkdir</span> -p /data/zookeeper</span> <br > <span class= "line" > ssh_root_zookeeper.sh <span class= "built_in" > chown</span> -R hadoop:hadoop /data</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <p > 创建myid文件</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > ssh nn1 <span class= "string" > ' echo " 1" > /data/zookeeper/myid' </span> </span> <br > <span class= "line" > ssh nn2 <span class= "string" > ' echo " 2" > /data/zookeeper/myid' </span> </span> <br > <span class= "line" > ssh nn3 <span class= "string" > ' echo " 3" > /data/zookeeper/myid' </span> </span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <p > 配置Zookeeper环境变量</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # 在其他所有主机也执行</span> </span> <br > <span class= "line" > <span class= "built_in" > sudo</span> <span class= "built_in" > chown</span> -R hadoop:hadoop /etc/profile.d/myEnv.sh</span> <br > </pre> </td> </tr> </table> </figure>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > echo</span> <span class= "string" > ' export ZOOKEEPER_HOME=/usr/local/zookeeper' </span> > > /etc/profile.d/myEnv.sh</span> <br > <span class= "line" > <span class= "built_in" > echo</span> <span class= "string" > ' export PATH=$PATH:$ZOOKEEPER_HOME/bin' </span> > > /etc/profile.d/myEnv.sh</span> <br > </pre> </td> </tr> </table> </figure>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # 分发到nn2、nn3</span> </span> <br > <span class= "line" > scp_all_zookeeper.sh /etc/profile.d/myEnv.sh /etc/profile.d/</span> <br > <span class= "line" > <span class= "comment" > # source 环境变量</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh <span class= "built_in" > source</span> /etc/profile</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<h3 id= "Zookeeper的命令" > <a href= "#Zookeeper的命令" class= "headerlink" title= "Zookeeper的命令" > </a> Zookeeper的命令</h3> <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > #启动zk服务</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh start</span> <br > <span class= "line" > <span class= "comment" > #查看每个机器ZK运行的状态</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh status</span> <br > <span class= "line" > <span class= "comment" > #整体停止服务</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh stop </span> <br > <span class= "line" > <span class= "comment" > #重启zk服务</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh restart</span> <br > </pre> </td> </tr> </table> </figure>
<figure class= "highlight shell" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "meta prompt_" > #</span> <span class= "language-bash" > 启动zookeeper客户端,并连接zookeeper集群</span> </span> <br > <span class= "line" > /usr/local/zookeeper/bin/zkCli.sh -server nn1:2181</span> <br > <span class= "line" > <span class= "meta prompt_" > # </span> <span class= "language-bash" > 可以简化为:</span> </span> <br > <span class= "line" > zkCli.sh</span> <br > </pre> </td> </tr> </table> </figure>
]]></content>
</entry>
<entry >
<title > 组件使用</title>
<url > /posts/33957.html</url>
<content > <![CDATA[<h3 id="组件自动导入"> <a href= "#组件自动导入" class= "headerlink" title= "组件自动导入" > </a> 组件自动导入</h3> <figure class= "highlight json" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "attr" > " easycom" </span> <span class= "punctuation" > :</span> <span class= "punctuation" > { </span> </span> <br > <span class= "line" > <span class= "attr" > " autoscan" </span> <span class= "punctuation" > :</span> <span class= "literal" > <span class= "keyword" > true</span> </span> <span class= "punctuation" > ,</span> </span> <br > <span class= "line" > <span class= "attr" > " custom" </span> <span class= "punctuation" > :</span> <span class= "punctuation" > { </span> </span> <br > <span class= "line" > <span class= "attr" > " ^tui-(.*)" </span> <span class= "punctuation" > :</span> <span class= "string" > " @/components/thorui/tui-$1/tui-$1.vue" </span> <span class= "comment" > // 匹配components目录内的vue文件</span> </span> <br > <span class= "line" > <span class= "punctuation" > } </span> </span> <br > <span class= "line" > <span class= "punctuation" > } </span> </span> <br > </pre> </td> </tr> </table> </figure>
<h3 id= "tui-sticky-吸顶容器" > <a href= "#tui-sticky-吸顶容器" class= "headerlink" title= "tui-sticky 吸顶容器" > </a> <code > tui-sticky 吸顶容器</code> </h3> <blockquote >
<p > 包含 以下 <code > tui</code> 组件 :</p>
<ul >
<li > tui-sticky</li>
<li > tui-list-view</li>
<li > tui-list-cell</li>
</ul>
</blockquote>
<figure class= "highlight html" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "tag" > < <span class= "name" > tui-sticky</span> <span class= "attr" > :scrollTop</span> =<span class= "string" > " scrollTop" </span> <span class= "attr" > stickyHeight</span> =<span class= "string" > " 104rpx" </span> <span class= "attr" > container</span> > </span> </span> <br > <span class= "line" > <span class= "comment" > < !-- header start --> </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > template</span> <span class= "attr" > v-slot:header</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > view</span> <span class= "attr" > class</span> =<span class= "string" > " sticky-item" </span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > view</span> <span class= "attr" > class</span> =<span class= "string" > " setting" </span> > </span> 设置<span class= "tag" > < /<span class= "name" > view</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > view</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > template</span> > </span> </span> <br > <span class= "line" > <span class= "comment" > < !-- header end --> </span> </span> <br > <span class= "line" > <span class= "comment" > < !--内容 start--> </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > template</span> <span class= "attr" > v-slot:content</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > tui-list-view</span> <span class= "attr" > class</span> =<span class= "string" > " content" </span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > tui-list-cell</span> <span class= "attr" > :arrow</span> =<span class= "string" > " false" </span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > switch</span> <span class= "attr" > class</span> =<span class= "string" > ' switch' </span> <span class= "attr" > checked</span> <span class= "attr" > color</span> =<span class= "string" > " #FFCC33" </span> /> </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > tui-list-cell</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > tui-list-view</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > template</span> > </span> </span> <br > <span class= "line" > <span class= "comment" > < !--内容 end--> </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > tui-sticky</span> > </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > script</span> <span class= "attr" > setup</span> > </span> <span class= "language-javascript" > </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "keyword" > import</span> { ref } <span class= "keyword" > from</span> <span class= "string" > ' vue' </span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "keyword" > import</span> { onPageScroll } <span class= "keyword" > from</span> <span class= "string" > ' @dcloudio/uni-app' </span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "comment" > // 定义 scrollTop 响应式变量</span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "keyword" > const</span> scrollTop = <span class= "title function_" > ref</span> (<span class= "number" > 0</span> )</span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "comment" > // 监听页面滚动事件</span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "title function_" > onPageScroll</span> (<span class= "function" > (<span class= "params" > e</span> ) => </span> { </span> </span> <br > <span class= "line" > <span class= "language-javascript" > scrollTop.<span class= "property" > value</span> = e.<span class= "property" > scrollTop</span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > } )</span> </span> <br > <span class= "line" > <span class= "language-javascript" > </span> <span class= "tag" > < /<span class= "name" > script</span> > </span> </span> <br > </pre> </td> </tr> </table> </figure>
]]></content>
<tags >
<tag > uniapp</tag>
</tags>
</entry>
<entry >
<title > Docker被墙,如何继续使用?</title>
<url > /posts/47807.html</url>
@@ -432,65 +491,6 @@
</ol>
</li>
</ol>
]]></content>
</entry>
<entry >
<title > 组件使用</title>
<url > /posts/33957.html</url>
<content > <![CDATA[<h3 id="组件自动导入"> <a href= "#组件自动导入" class= "headerlink" title= "组件自动导入" > </a> 组件自动导入</h3> <figure class= "highlight json" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "attr" > " easycom" </span> <span class= "punctuation" > :</span> <span class= "punctuation" > { </span> </span> <br > <span class= "line" > <span class= "attr" > " autoscan" </span> <span class= "punctuation" > :</span> <span class= "literal" > <span class= "keyword" > true</span> </span> <span class= "punctuation" > ,</span> </span> <br > <span class= "line" > <span class= "attr" > " custom" </span> <span class= "punctuation" > :</span> <span class= "punctuation" > { </span> </span> <br > <span class= "line" > <span class= "attr" > " ^tui-(.*)" </span> <span class= "punctuation" > :</span> <span class= "string" > " @/components/thorui/tui-$1/tui-$1.vue" </span> <span class= "comment" > // 匹配components目录内的vue文件</span> </span> <br > <span class= "line" > <span class= "punctuation" > } </span> </span> <br > <span class= "line" > <span class= "punctuation" > } </span> </span> <br > </pre> </td> </tr> </table> </figure>
<h3 id= "tui-sticky-吸顶容器" > <a href= "#tui-sticky-吸顶容器" class= "headerlink" title= "tui-sticky 吸顶容器" > </a> <code > tui-sticky 吸顶容器</code> </h3> <blockquote >
<p > 包含 以下 <code > tui</code> 组件 :</p>
<ul >
<li > tui-sticky</li>
<li > tui-list-view</li>
<li > tui-list-cell</li>
</ul>
</blockquote>
<figure class= "highlight html" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "tag" > < <span class= "name" > tui-sticky</span> <span class= "attr" > :scrollTop</span> =<span class= "string" > " scrollTop" </span> <span class= "attr" > stickyHeight</span> =<span class= "string" > " 104rpx" </span> <span class= "attr" > container</span> > </span> </span> <br > <span class= "line" > <span class= "comment" > < !-- header start --> </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > template</span> <span class= "attr" > v-slot:header</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > view</span> <span class= "attr" > class</span> =<span class= "string" > " sticky-item" </span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > view</span> <span class= "attr" > class</span> =<span class= "string" > " setting" </span> > </span> 设置<span class= "tag" > < /<span class= "name" > view</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > view</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > template</span> > </span> </span> <br > <span class= "line" > <span class= "comment" > < !-- header end --> </span> </span> <br > <span class= "line" > <span class= "comment" > < !--内容 start--> </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > template</span> <span class= "attr" > v-slot:content</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > tui-list-view</span> <span class= "attr" > class</span> =<span class= "string" > " content" </span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > tui-list-cell</span> <span class= "attr" > :arrow</span> =<span class= "string" > " false" </span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > switch</span> <span class= "attr" > class</span> =<span class= "string" > ' switch' </span> <span class= "attr" > checked</span> <span class= "attr" > color</span> =<span class= "string" > " #FFCC33" </span> /> </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > tui-list-cell</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > tui-list-view</span> > </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > template</span> > </span> </span> <br > <span class= "line" > <span class= "comment" > < !--内容 end--> </span> </span> <br > <span class= "line" > <span class= "tag" > < /<span class= "name" > tui-sticky</span> > </span> </span> <br > <span class= "line" > </span> <br > <span class= "line" > <span class= "tag" > < <span class= "name" > script</span> <span class= "attr" > setup</span> > </span> <span class= "language-javascript" > </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "keyword" > import</span> { ref } <span class= "keyword" > from</span> <span class= "string" > ' vue' </span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "keyword" > import</span> { onPageScroll } <span class= "keyword" > from</span> <span class= "string" > ' @dcloudio/uni-app' </span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "comment" > // 定义 scrollTop 响应式变量</span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "keyword" > const</span> scrollTop = <span class= "title function_" > ref</span> (<span class= "number" > 0</span> )</span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "comment" > // 监听页面滚动事件</span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > <span class= "title function_" > onPageScroll</span> (<span class= "function" > (<span class= "params" > e</span> ) => </span> { </span> </span> <br > <span class= "line" > <span class= "language-javascript" > scrollTop.<span class= "property" > value</span> = e.<span class= "property" > scrollTop</span> </span> </span> <br > <span class= "line" > <span class= "language-javascript" > } )</span> </span> <br > <span class= "line" > <span class= "language-javascript" > </span> <span class= "tag" > < /<span class= "name" > script</span> > </span> </span> <br > </pre> </td> </tr> </table> </figure>
]]></content>
<tags >
<tag > uniapp</tag>
</tags>
</entry>
<entry >
<title > Hadoop集群Zookeeper配置</title>
<url > /posts/61251.html</url>
<content > <![CDATA[<h3 id="Zookeeper脚本配置"> <a href= "#Zookeeper脚本配置" class= "headerlink" title= "Zookeeper脚本配置" > </a> Zookeeper脚本配置</h3> <ul >
<li > 拷贝ips<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > cp</span> ips ips_zookeeper</span> <br > </pre> </td> </tr> </table> </figure>
修改为<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > nn1</span> <br > <span class= "line" > nn2</span> <br > <span class= "line" > nn3</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > 拷贝三个脚本<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > cp</span> scp_all.sh scp_all_zookeeper.sh </span> <br > <span class= "line" > <span class= "built_in" > cp</span> ssh_all.sh ssh_all_zookeeper.sh </span> <br > <span class= "line" > <span class= "built_in" > cp</span> ssh_root.sh ssh_root_zookeeper.sh</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > 修改脚本<figure class= "highlight shell" > <table > <tr > <td class= "code" > <pre > <span class= "line" > vim scp_all_zookeeper.sh </span> <br > <span class= "line" > vim ssh_all_zookeeper.sh </span> <br > <span class= "line" > vim ssh_root_zookeeper.sh </span> <br > </pre> </td> </tr> </table> </figure>
将三个脚本中的ips改为ips_zookeeper</li>
</ul>
<h3 id= "Zookeeper安装" > <a href= "#Zookeeper安装" class= "headerlink" title= "Zookeeper安装" > </a> Zookeeper安装</h3> <ul >
<li > 上传到<code > /tmp</code> 目录下,解压 <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > sudo</span> tar -zxvf /tmp/zookeeper-3.4.8.tar.gz -C /usr/local/</span> <br > <span class= "line" > scp -r /usr/local/zookeeper-3.4.8/ root@nn2:/usr/local/</span> <br > <span class= "line" > scp -r /usr/local/zookeeper-3.4.8/ root@nn3:/usr/local/</span> <br > <span class= "line" > ssh_root_zookeeper.sh <span class= "built_in" > chown</span> -R hadoop:hadoop /usr/local/zookeeper-3.4.8</span> <br > <span class= "line" > ssh_root_zookeeper.s <span class= "built_in" > ln</span> -s /usr/local/zookeeper-3.4.8/ /usr/local/zookeeper</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<h3 id= "Zookeeper配置" > <a href= "#Zookeeper配置" class= "headerlink" title= "Zookeeper配置" > </a> Zookeeper配置</h3> <ul >
<li > <p > zoo.cfg配置</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > cd</span> /usr/local/zookeeper/conf/</span> <br > <span class= "line" > <span class= "built_in" > cp</span> zoo_sample.cfg zoo.cfg</span> <br > </pre> </td> </tr> </table> </figure>
<p > 然后<code > vim zoo.cfg</code> ,修改如下:</p>
<figure class= "highlight properties" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # 修改dataDir</span> </span> <br > <span class= "line" > <span class= "attr" > dataDir</span> =<span class= "string" > /data/zookeeper</span> </span> <br > <span class= "line" > <span class= "comment" > # 添加一下内容</span> </span> <br > <span class= "line" > <span class= "attr" > server.1</span> =<span class= "string" > nn1:2888:3888 </span> </span> <br > <span class= "line" > <span class= "attr" > server.2</span> =<span class= "string" > nn2:2888:3888 </span> </span> <br > <span class= "line" > <span class= "attr" > server.3</span> =<span class= "string" > nn3:2888:3888</span> </span> <br > </pre> </td> </tr> </table> </figure>
<p > 分发给nn2、nn3<br > <code > scp_all_zookeeper.sh /usr/local/zookeeper/conf/zoo.cfg /usr/local/zookeeper/conf/</code> </p>
</li>
<li > <p > <code > zkEnv.sh</code> 配置<br > <code > vim /usr/local/zookeeper/bin/zkEnv.sh</code> <br > <img src= "/../../../img/000001.png" alt= "000001.png" > <br > 分发到nn2、nn3</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > scp_all_zookeeper.sh /usr/local/zookeeper/bin/zkEnv.sh /usr/local/zookeeper/bin/</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <p > 创建zookeeper数据目录</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > ssh_root_zookeeper.sh <span class= "built_in" > mkdir</span> -p /data/zookeeper</span> <br > <span class= "line" > ssh_root_zookeeper.sh <span class= "built_in" > chown</span> -R hadoop:hadoop /data</span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <p > 创建myid文件</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > ssh nn1 <span class= "string" > ' echo " 1" > /data/zookeeper/myid' </span> </span> <br > <span class= "line" > ssh nn2 <span class= "string" > ' echo " 2" > /data/zookeeper/myid' </span> </span> <br > <span class= "line" > ssh nn3 <span class= "string" > ' echo " 3" > /data/zookeeper/myid' </span> </span> <br > </pre> </td> </tr> </table> </figure> </li>
<li > <p > 配置Zookeeper环境变量</p>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # 在其他所有主机也执行</span> </span> <br > <span class= "line" > <span class= "built_in" > sudo</span> <span class= "built_in" > chown</span> -R hadoop:hadoop /etc/profile.d/myEnv.sh</span> <br > </pre> </td> </tr> </table> </figure>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "built_in" > echo</span> <span class= "string" > ' export ZOOKEEPER_HOME=/usr/local/zookeeper' </span> > > /etc/profile.d/myEnv.sh</span> <br > <span class= "line" > <span class= "built_in" > echo</span> <span class= "string" > ' export PATH=$PATH:$ZOOKEEPER_HOME/bin' </span> > > /etc/profile.d/myEnv.sh</span> <br > </pre> </td> </tr> </table> </figure>
<figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > # 分发到nn2、nn3</span> </span> <br > <span class= "line" > scp_all_zookeeper.sh /etc/profile.d/myEnv.sh /etc/profile.d/</span> <br > <span class= "line" > <span class= "comment" > # source 环境变量</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh <span class= "built_in" > source</span> /etc/profile</span> <br > </pre> </td> </tr> </table> </figure> </li>
</ul>
<h3 id= "Zookeeper的命令" > <a href= "#Zookeeper的命令" class= "headerlink" title= "Zookeeper的命令" > </a> Zookeeper的命令</h3> <figure class= "highlight bash" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "comment" > #启动zk服务</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh start</span> <br > <span class= "line" > <span class= "comment" > #查看每个机器ZK运行的状态</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh status</span> <br > <span class= "line" > <span class= "comment" > #整体停止服务</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh stop </span> <br > <span class= "line" > <span class= "comment" > #重启zk服务</span> </span> <br > <span class= "line" > ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh restart</span> <br > </pre> </td> </tr> </table> </figure>
<figure class= "highlight shell" > <table > <tr > <td class= "code" > <pre > <span class= "line" > <span class= "meta prompt_" > #</span> <span class= "language-bash" > 启动zookeeper客户端,并连接zookeeper集群</span> </span> <br > <span class= "line" > /usr/local/zookeeper/bin/zkCli.sh -server nn1:2181</span> <br > <span class= "line" > <span class= "meta prompt_" > # </span> <span class= "language-bash" > 可以简化为:</span> </span> <br > <span class= "line" > zkCli.sh</span> <br > </pre> </td> </tr> </table> </figure>
]]></content>
</entry>
<entry >
@@ -536,7 +536,14 @@
</ul>
</li>
<li > <p > <code > 赔钱机场</code> </p>
<p > <img src= "/img/peiqian.png" > <br > <code > 赔钱机场 </code> 的订阅共有9种方案,这里我仅显示自己正在使用的,个人认为十分优惠: </p>
<p > <img src= "/img/peiqian.png" > <br > 可以看到 </p>
<ul >
<li > <code > 18元/年</code> ,每月100GB的可用额度,允许最多10个设备同时在线,下个月重置流量额度</li>
<li > <code > 34.99元/年</code> ,每月有500GB的可用额度,允许最多15个设备同时在线,下个月重置流量额度</li>
<li > <code > 68.99元/年</code> ,每个月1000GB的可用额度,允许最多20个设备同时在线,下个月重置流量额度</li>
<li > 其余可以自行查看</li>
</ul>
<p > <code > 赔钱机场</code> 的订阅共有5种方案(按周期付费),这里我仅显示自己正在使用的,个人认为十分优惠:</p>
<ul >
<li > <code > 34.99元/年</code> ,每月500GB的可用额度,根据我观察和使用,这个订阅方案比<code > 一元机场</code> 的性价比更高,且流量使用额度也不用担心</li>
</ul>