558 lines
166 KiB
XML
558 lines
166 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
||
<search>
|
||
<entry>
|
||
<title>Hello World</title>
|
||
<url>/posts/16107.html</url>
|
||
<content><![CDATA[<p>Welcome to <a href="https://hexo.io/">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues">GitHub</a>.</p>
|
||
<h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo new <span class="string">"My New Post"</span></span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/writing.html">Writing</a></p>
|
||
<h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo server</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/server.html">Server</a></p>
|
||
<h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo generate</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/generating.html">Generating</a></p>
|
||
<h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo deploy</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/one-command-deployment.html">Deployment</a></p>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>C lang</title>
|
||
<url>/posts/12462.html</url>
|
||
<content><![CDATA[<h3 id="c-lang在windows下的开发（VS-code）"><a href="#c-lang在windows下的开发（VS-code）" class="headerlink" title="c lang在windows下的开发（VS code）"></a>c lang在windows下的开发（VS code）</h3><p><a href="https://winlibs.com/#download-release">WinLibs - GCC+MinGW-w64 compiler for Windows</a>下载你需要的版本<br>解压到<code>D:\ProgramModule</code>，并将 <code>bin\</code>加入环境变量<code>PATH</code><br>打开新的<code>Terminal</code>输入<code>gcc -v</code>，查看<code>gcc</code>是否安装成功<br>在<code>VS code</code> 的插件管理下载<code>Code Runner</code>、<code>C\C++</code>这两个插件<br>在<code>*.c</code>源文件的内容区，右键点击<code>Run Code</code> ，即可运行成功<br><img src="/img/language/c-env-conf.png"></p>
|
||
<h3 id="数据类型"><a href="#数据类型" class="headerlink" title="数据类型"></a>数据类型</h3><ul>
|
||
<li>整数类型<figure class="highlight c"><table><tr><td class="code"><pre><span class="line"> <span class="type">short</span> a = <span class="number">12</span>;</span><br><span class="line"> <span class="type">int</span> b = <span class="number">100</span>;</span><br><span class="line"> <span class="type">long</span> c = <span class="number">1000L</span>;</span><br><span class="line"> <span class="type">long</span> <span class="type">long</span> d = <span class="number">1000000LL</span>;</span><br><span class="line"> <span class="type">unsigned</span> <span class="type">int</span> e = <span class="number">10</span>;</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"a: %hd\n"</span>,a);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"b: %d\n"</span>,b);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"c: %ld\n"</span>,c);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"d: %lld\n"</span>,d);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"e: %u\n"</span>,e);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"f: %.3f\n"</span>,f);</span><br></pre></td></tr></table></figure></li>
|
||
<li>小数类型<figure class="highlight c"><table><tr><td class="code"><pre><span class="line"><span class="type">float</span> f = <span class="number">3.14F</span>;</span><br><span class="line"><span class="built_in">printf</span>(<span class="string">"f: %.3f\n"</span>,f);</span><br><span class="line"><span class="type">double</span> g = <span class="number">5.65</span>;</span><br><span class="line"><span class="built_in">printf</span>(<span class="string">"g: %.2lf\n"</span>,g);</span><br></pre></td></tr></table></figure></li>
|
||
<li>字符类型<figure class="highlight c"><table><tr><td class="code"><pre><span class="line"><span class="type">char</span> h = <span class="string">'x'</span>;</span><br><span class="line"><span class="built_in">printf</span>(<span class="string">"x: %c\n"</span>,x);</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="类型转换"><a href="#类型转换" class="headerlink" title="类型转换"></a>类型转换</h3><ul>
|
||
<li>隐式转换</li>
|
||
<li>强制转换<figure class="highlight c"><table><tr><td class="code"><pre><span class="line"><span class="type">int</span> b = <span class="number">23</span>;</span><br><span class="line"><span class="type">short</span> c = (<span class="type">short</span>) b;</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="数组"><a href="#数组" class="headerlink" title="数组"></a>数组</h3><figure class="highlight c"><table><tr><td class="code"><pre><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string"><stdio.h></span></span></span><br><span class="line"></span><br><span class="line"><span class="type">int</span> <span class="title function_">main</span><span class="params">()</span>{</span><br><span class="line"> <span class="type">int</span> arr [<span class="number">10</span>] = {<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>,<span class="number">5</span>,<span class="number">6</span>,<span class="number">7</span>,<span class="number">8</span>,<span class="number">9</span>,<span class="number">10</span>,<span class="number">11</span>};</span><br><span class="line"> arr[<span class="number">0</span>] = <span class="number">1525</span>;</span><br><span class="line"> *(arr+<span class="number">1</span>) = <span class="number">25</span>;</span><br><span class="line"> <span class="type">int</span> len = <span class="keyword">sizeof</span>(arr)/<span class="keyword">sizeof</span>(arr[<span class="number">0</span>]);</span><br><span class="line"> <span class="type">void</span> <span class="title function_">printArr</span><span class="params">(<span class="type">int</span> arr[], <span class="type">int</span> len)</span>{</span><br><span class="line"> <span class="keyword">for</span> (<span class="type">int</span> i = <span class="number">0</span>; i < len;i++){</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"%d\t"</span>,arr[i]);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> printArr(arr,len);</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
|
||
<h3 id="指针"><a href="#指针" class="headerlink" title="指针"></a>指针</h3><figure class="highlight c"><table><tr><td class="code"><pre><span class="line"><span class="comment">// swap the value of a and b</span></span><br><span class="line"> <span class="type">void</span> <span class="title function_">swap</span><span class="params">(<span class="type">int</span>* x, <span class="type">int</span>* y)</span>{</span><br><span class="line"> <span class="type">int</span> temp = *x;</span><br><span class="line"> *x = *y;</span><br><span class="line"> *y = temp;</span><br><span class="line"></span><br><span class="line"> }</span><br><span class="line"> <span class="type">int</span> a = <span class="number">5</span>;</span><br><span class="line"> <span class="type">int</span> b = <span class="number">10</span>;</span><br><span class="line"> swap(&a, &b);</span><br><span class="line"> <span class="built_in">printf</span>(<span class="string">"a = %d b = %d\n"</span>, a, b);</span><br></pre></td></tr></table></figure>
|
||
]]></content>
|
||
<tags>
|
||
<tag>C C++</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>page</title>
|
||
<url>/posts/1441.html</url>
|
||
<content><![CDATA[<ul>
|
||
<li><a href="./deploy">deploy</a></li>
|
||
<li></li>
|
||
</ul>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>script</title>
|
||
<url>/posts/34849.html</url>
|
||
<content><![CDATA[<h3 id="查看CPU、内存使用率"><a href="#查看CPU、内存使用率" class="headerlink" title="查看CPU、内存使用率"></a>查看CPU、内存使用率</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 定义颜色</span></span><br><span class="line">RED=<span class="string">'\033[0;31m'</span></span><br><span class="line">GREEN=<span class="string">'\033[0;32m'</span></span><br><span class="line">YELLOW=<span class="string">'\033[0;33m'</span></span><br><span class="line">BLUE=<span class="string">'\033[0;34m'</span></span><br><span class="line">NC=<span class="string">'\033[0m'</span> <span class="comment"># 无颜色</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span> <span class="literal">true</span>; <span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 获取所有进程的CPU使用率和内存使用率</span></span><br><span class="line"> cpu_usage=$(ps aux | awk <span class="string">'{sum_cpu += $3} END {print sum_cpu}'</span>)</span><br><span class="line"> mem_usage=$(ps aux | awk <span class="string">'{sum_mem += $4} END {print sum_mem}'</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 打印结果，带有时间戳、分隔线和颜色高亮</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${BLUE}</span>==============================<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${YELLOW}</span>Timestamp: <span class="subst">$(date)</span><span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${BLUE}</span>==============================<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${GREEN}</span>Total CPU usage: <span class="variable">${RED}</span><span class="variable">$cpu_usage</span>%<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${GREEN}</span>Total Memory usage: <span class="variable">${RED}</span><span class="variable">$mem_usage</span>%<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${BLUE}</span>==============================<span class="variable">${NC}</span>"</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 等待5秒后再次执行</span></span><br><span class="line"> <span class="built_in">sleep</span> 5</span><br><span class="line"><span class="keyword">done</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
|
||
<p><strong>保存脚本到/usr/local/bin目录下</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">mv</span> usage.sh /usr/local/bin/usage</span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="Shell脚本编写的基本信息"><a href="#Shell脚本编写的基本信息" class="headerlink" title="Shell脚本编写的基本信息"></a>Shell脚本编写的基本信息</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># -------------------------------------------------</span></span><br><span class="line"><span class="comment"># Filename: test.sh</span></span><br><span class="line"><span class="comment"># Version: 1.0</span></span><br><span class="line"><span class="comment"># Date: 2024/05/02</span></span><br><span class="line"><span class="comment"># Author: shenjianZ</span></span><br><span class="line"><span class="comment"># Email: shenjianZLT@gmail.com</span></span><br><span class="line"><span class="comment"># Website: https://blog.shenjianl.cn</span></span><br><span class="line"><span class="comment"># Description: this is a test shell</span></span><br><span class="line"><span class="comment"># CopyRight: 2024 All rights reserved shenjianZ</span></span><br><span class="line"><span class="comment"># License GPL</span></span><br><span class="line"><span class="comment"># ------------------------------------------------</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># Your script logic goes here</span></span><br></pre></td></tr></table></figure>]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>集成学习</title>
|
||
<url>/posts/8816.html</url>
|
||
<content><![CDATA[<h3 id="Bagging"><a href="#Bagging" class="headerlink" title="Bagging"></a>Bagging</h3><h3 id="随机森林"><a href="#随机森林" class="headerlink" title="随机森林"></a>随机森林</h3><blockquote>
|
||
<p><code>Random-Forest</code> 就是<code>Bagging + Decisiontree</code></p>
|
||
</blockquote>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> seaborn <span class="keyword">as</span> sns</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split,GridSearchCV</span><br><span class="line"><span class="keyword">from</span> sklearn.feature_extraction <span class="keyword">import</span> DictVectorizer</span><br><span class="line"><span class="keyword">from</span> sklearn.ensemble <span class="keyword">import</span> RandomForestClassifier</span><br><span class="line"><span class="comment"># 1.获取数据集 - 加载 Titanic 数据集</span></span><br><span class="line">titanic = sns.load_dataset(<span class="string">'titanic'</span>)</span><br><span class="line">missing_age_count = titanic[<span class="string">'age'</span>].isna().<span class="built_in">sum</span>()</span><br><span class="line"><span class="comment"># print(f"缺失的 age 数量: {missing_age_count}")</span></span><br><span class="line"><span class="comment"># 2. 数据基本处理</span></span><br><span class="line"><span class="comment"># 2.1 确认特征值、目标值</span></span><br><span class="line">X = titanic[[<span class="string">'pclass'</span>,<span class="string">'age'</span>,<span class="string">'sex'</span>]]</span><br><span class="line">y = titanic[<span class="string">'survived'</span>]</span><br><span class="line"><span class="comment"># 2.2 缺失值处理</span></span><br><span class="line">X.loc[:, <span class="string">'age'</span>] = X[<span class="string">'age'</span>].fillna(value=X[<span class="string">'age'</span>].mean()) <span class="comment"># 使用 .loc 进行修改</span></span><br><span class="line"><span class="comment"># 2.3 划分数据集</span></span><br><span class="line">X_train,X_test,y_train,y_test = train_test_split(X,y,random_state=<span class="number">22</span>)</span><br><span class="line"><span class="comment"># 3. 特征工程（字典特征提取）</span></span><br><span class="line">X_train = X_train.to_dict(orient=<span class="string">"records"</span>)</span><br><span class="line">X_test= X_test.to_dict(orient=<span class="string">"records"</span>)</span><br><span class="line">transfer = DictVectorizer()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"><span class="comment"># 4. 机器学习 随机森林</span></span><br><span class="line">rf = RandomForestClassifier()</span><br><span class="line">gc = GridSearchCV(estimator=rf ,param_grid={<span class="string">"n_estimators"</span>:[<span class="number">100</span>,<span class="number">120</span>,<span class="number">300</span>],<span class="string">"max_depth"</span>:[<span class="number">3</span>,<span class="number">7</span>,<span class="number">11</span>]},cv=<span class="number">3</span>)</span><br><span class="line">gc.fit(X_train,y_train)</span><br><span class="line">y_pred = gc.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型的测试集的预测值：<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line">ret = gc.score(X_test,y_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最佳模型在测试集上的评分:<span class="subst">{ret}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最佳模型的参数：<span class="subst">{gc.best_estimator_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最佳模型在训练集上的评分：<span class="subst">{gc.best_score_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(X_test.toarray())</span><br></pre></td></tr></table></figure>
|
||
<p><img src="/img/machinelearning/random-forest.png"></p>
|
||
<h3 id="ott案例"><a href="#ott案例" class="headerlink" title="ott案例"></a>ott案例</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="keyword">from</span> imblearn.under_sampling <span class="keyword">import</span> RandomUnderSampler</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> LabelEncoder</span><br><span class="line"><span class="keyword">from</span> sklearn.ensemble <span class="keyword">import</span> RandomForestClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> log_loss</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> OneHotEncoder</span><br><span class="line"><span class="comment"># 1. 获取数据集</span></span><br><span class="line">data = pd.read_csv(<span class="string">'./data/train.csv'</span>)</span><br><span class="line"><span class="comment"># 查看目标值分类</span></span><br><span class="line"><span class="keyword">import</span> seaborn <span class="keyword">as</span> sns</span><br><span class="line">sns.countplot(data=data, x=<span class="string">'target'</span>, hue=<span class="string">'target'</span>, palette=<span class="string">"Set2"</span>, legend=<span class="literal">False</span>) <span class="comment"># 使用 hue='target' 替代 palette</span></span><br><span class="line">plt.show()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2. 数据集的基本处理</span></span><br><span class="line"><span class="comment"># 2.1 确定特征值、目标值</span></span><br><span class="line">x = data.drop([<span class="string">"id"</span>, <span class="string">"target"</span>], axis=<span class="number">1</span>)</span><br><span class="line">y = data[<span class="string">'target'</span>]</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2.2 使用随机欠采样进行平衡</span></span><br><span class="line">undersampler = RandomUnderSampler(sampling_strategy=<span class="string">'auto'</span>, random_state=<span class="number">0</span>)</span><br><span class="line">x_resampled, y_resampled = undersampler.fit_resample(x, y)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看欠采样后的类别分布</span></span><br><span class="line"><span class="comment"># print(f"欠采样后训练集中的类别分布：\n{y_train_resampled.value_counts()}")</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 2.3. 将标签转换为数字</span></span><br><span class="line">le = LabelEncoder()</span><br><span class="line">y_resampled = le.fit_transform(y_resampled)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2.4. 划分训练集和测试集</span></span><br><span class="line">x_train, x_test, y_train, y_test = train_test_split(x_resampled, y_resampled, test_size=<span class="number">0.2</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 3. 机器学习</span></span><br><span class="line">rf = RandomForestClassifier(oob_score = <span class="literal">True</span>)</span><br><span class="line">rf.fit(x_train,y_train)</span><br><span class="line">y_pred = rf.predict(x_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"预测值:<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"评分：<span class="subst">{rf.score(x_test,y_test)}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># # 4. 模型评估 （解决二分类预测问题）</span></span><br><span class="line"><span class="comment"># import numpy as np</span></span><br><span class="line"><span class="comment"># from sklearn.metrics import log_loss</span></span><br><span class="line"><span class="comment"># # 假设 y_pred_prob 是通过 predict_proba 得到的预测概率</span></span><br><span class="line"><span class="comment"># # 对预测概率进行裁剪，将其限制在 [eps, 1-eps] 范围内</span></span><br><span class="line"><span class="comment"># eps = 1e-15 # 设置一个小的eps值，避免极端值</span></span><br><span class="line"><span class="comment"># y_pred_prob = rf.predict_proba(x_test)</span></span><br><span class="line"><span class="comment"># y_pred_prob = np.clip(y_pred_prob, eps, 1 - eps)</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># # 计算 log_loss</span></span><br><span class="line"><span class="comment"># loss = log_loss(y_test, y_pred_prob, normalize=True)</span></span><br><span class="line"><span class="comment"># print(f"Log Loss: {loss}")</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 4. 模型评估 （解决多分类预测问题）</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 获取预测的概率</span></span><br><span class="line">y_pred_prob = rf.predict_proba(x_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 使用 OneHotEncoder 对 y_test 进行 One-Hot 编码</span></span><br><span class="line">encoder = OneHotEncoder(sparse_output=<span class="literal">False</span>) <span class="comment"># 确保返回的是密集矩阵</span></span><br><span class="line">y_test_one_hot = encoder.fit_transform(y_test.reshape(-<span class="number">1</span>, <span class="number">1</span>))</span><br><span class="line"></span><br><span class="line"><span class="comment"># 对预测概率进行裁剪，将其限制在 [eps, 1-eps] 范围内</span></span><br><span class="line">eps = <span class="number">1e-15</span></span><br><span class="line">y_pred_prob = np.clip(y_pred_prob, eps, <span class="number">1</span> - eps)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 计算 log_loss</span></span><br><span class="line">loss = log_loss(y_test_one_hot, y_pred_prob, normalize=<span class="literal">True</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"Log Loss: <span class="subst">{loss}</span>"</span>)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
|
||
<p><img src="/img/machinelearning/ott.png"></p>
|
||
]]></content>
|
||
<categories>
|
||
<category>machinelearning</category>
|
||
</categories>
|
||
<tags>
|
||
<tag>ensemble-learning</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>k近邻算法（K-Nearest Neighbors）KNN</title>
|
||
<url>/posts/29139.html</url>
|
||
<content><![CDATA[<h2 id="k近邻算法（K-Nearest-Neighbors）KNN"><a href="#k近邻算法（K-Nearest-Neighbors）KNN" class="headerlink" title="k近邻算法（K-Nearest Neighbors）KNN"></a><strong>k近邻算法（K-Nearest Neighbors）KNN</strong></h2><p>将当前样本的类别归类于距离最近的<strong>k</strong>个样本的类别</p>
|
||
<h4 id="距离公式-2维"><a href="#距离公式-2维" class="headerlink" title="距离公式(2维)"></a><strong>距离公式(2维)</strong></h4><ul>
|
||
<li>欧式距离<br>$$<br>d = \sqrt{(x_1-y_1)^2 + (x_2 - y_2)^2}<br>$$</li>
|
||
<li>曼哈顿距离<br>$$<br>d = |x_1 - x_2| + |y_1 - y_2|<br>$$</li>
|
||
<li>切比雪夫距离<br>$$<br>d = \max\left(|x_1 - x_2|, |y_1 - y_2|\right)<br>$$</li>
|
||
</ul>
|
||
<h4 id="k值选择问题"><a href="#k值选择问题" class="headerlink" title="k值选择问题"></a>k值选择问题</h4><table>
|
||
<thead>
|
||
<tr>
|
||
<th>k值</th>
|
||
<th>影响</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody><tr>
|
||
<td>越大</td>
|
||
<td>模型过拟合，准确率波动较大</td>
|
||
</tr>
|
||
<tr>
|
||
<td>越小</td>
|
||
<td>模型欠拟合，准确率趋于稳定但可能较低</td>
|
||
</tr>
|
||
</tbody></table>
|
||
<h3 id="特征预处理"><a href="#特征预处理" class="headerlink" title="特征预处理"></a>特征预处理</h3><blockquote>
|
||
<p>通过一些转换函数将特征数据转换成更加适合算法模型的特征数据过程 </p>
|
||
</blockquote>
|
||
<ul>
|
||
<li><p>归一化<br>将数据变换到指定区间（默认是[0,1]）<br>$$ x’ = \frac{x- x_{\text {min}}}{x_{\text{max}} - x_{\text{min}}} $$<br>若需要缩放到任意区间 ([a, b])，公式为： $$ x’ = a + \frac{(x - x_{\text{min}}) \cdot (b - a)}{x_{\text{max}} - x_{\text{min}}} $$<br>其中：( [a, b] )：目标区间的范围<br>归一化受到数据集的异常值的影响，需要进行标准化处理（更加合理）</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> MinMaxScaler <span class="comment"># 归一化</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p>标准化<br>将数据调整为均值为 0，标准差为 1 的标准正态分布<br>$$ z = \frac{x - \mu}{\sigma} $$<br>( z )：标准化后的值 ( x )：原始数据值 ( $\mu$ )：数据的均值 ( $\sigma$)：数据的标准差</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler <span class="comment"># 标准化</span></span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="KNN代码实现"><a href="#KNN代码实现" class="headerlink" title="KNN代码实现"></a>KNN代码实现</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> seaborn <span class="keyword">as</span> sns</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt </span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> load_iris</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> MinMaxScaler,StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.neighbors <span class="keyword">import</span> KNeighborsClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> accuracy_score</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1 数据集获取</span></span><br><span class="line">iris = load_iris()</span><br><span class="line"><span class="comment"># print(iris.feature_names)</span></span><br><span class="line">iris_data = pd.DataFrame(iris.data,columns=[<span class="string">'Sepal_Length'</span>, <span class="string">'Sepal_Width'</span>, <span class="string">'Petal_Length'</span>, <span class="string">'Petal_Width'</span>])</span><br><span class="line">iris_data[<span class="string">'target'</span>] = iris.target</span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">iris_plot</span>(<span class="params">data,col1,col2</span>):</span><br><span class="line"> sns.lmplot(x=col1,y=col2,data=data,hue=<span class="string">"target"</span>,fit_reg=<span class="literal">False</span>)</span><br><span class="line"> plt.show()</span><br><span class="line"><span class="comment"># 2 数据集可视化</span></span><br><span class="line"><span class="comment"># iris_plot(iris_data, 'Sepal_Width', 'Petal_Length')</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 3 数据集的划分</span></span><br><span class="line">X_train,X_test,y_train,y_test = train_test_split(iris.data,iris.target,test_size=<span class="number">0.2</span>,random_state=<span class="number">44</span>)</span><br><span class="line"><span class="comment"># print("训练集的特征值：\n",X_train)</span></span><br><span class="line"><span class="comment"># print("训练集的目标值：\n",y_train)</span></span><br><span class="line"><span class="comment"># print("测试集的特征值：\n",X_test)</span></span><br><span class="line"><span class="comment"># print("测试集的特征值：\n",y_test)</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 4 归一化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"><span class="comment"># print("归一化的,X_train：\n",X_train)</span></span><br><span class="line"><span class="comment"># print("归一化的X_test：\n",X_test)</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 5 机器学习 KNN</span></span><br><span class="line"><span class="comment"># 5.1 实例化估计器</span></span><br><span class="line">estimator = KNeighborsClassifier(n_neighbors=<span class="number">9</span>)</span><br><span class="line"><span class="comment"># 5.2 进行训练</span></span><br><span class="line">estimator.fit(X_train,y_train)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 6 模型评估</span></span><br><span class="line">y_pred = estimator.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值：\n"</span>,y_pre)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值与真实值是否相等：\n"</span>,y_pred==y_test)</span><br><span class="line">accuracy = accuracy_score(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"\nKNN 模型的准确率: <span class="subst">{accuracy:<span class="number">.4</span>f}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
|
||
<p><img src="/img/machinelearning/knn-01.png"></p>
|
||
<h3 id="交叉验证与网格搜索"><a href="#交叉验证与网格搜索" class="headerlink" title="交叉验证与网格搜索"></a>交叉验证与网格搜索</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> seaborn <span class="keyword">as</span> sns</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt </span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split,GridSearchCV</span><br><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> load_iris</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> MinMaxScaler,StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.neighbors <span class="keyword">import</span> KNeighborsClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> accuracy_score</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1 数据集获取</span></span><br><span class="line">iris = load_iris()</span><br><span class="line">iris_data = pd.DataFrame(iris.data,columns=[<span class="string">'Sepal_Length'</span>, <span class="string">'Sepal_Width'</span>, <span class="string">'Petal_Length'</span>, <span class="string">'Petal_Width'</span>])</span><br><span class="line">iris_data[<span class="string">'target'</span>] = iris.target</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3 数据集的划分</span></span><br><span class="line">X_train,X_test,y_train,y_test = train_test_split(iris.data,iris.target,test_size=<span class="number">0.2</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 4 归一化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 5 机器学习 KNN</span></span><br><span class="line"><span class="comment"># 5.1 实例化估计器</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment">#不指定 <code> n_neighbors </code> ,使用网格搜索进行循环训练</span></span><br><span class="line">estimator = KNeighborsClassifier()</span><br><span class="line"><span class="comment"># 5.2 模型调优 -- 交叉验证，网格搜素</span></span><br><span class="line">estimator = GridSearchCV(estimator,param_grid={<span class="string">"n_neighbors"</span>:[<span class="number">1</span>,<span class="number">3</span>,<span class="number">5</span>,<span class="number">7</span>]},cv=<span class="number">5</span>) <span class="comment"># 5 折</span></span><br><span class="line"><span class="comment"># 5.2 进行训练</span></span><br><span class="line">estimator.fit(X_train,y_train)</span><br><span class="line"> </span><br><span class="line"><span class="comment"># 6 模型评估</span></span><br><span class="line">y_pred = estimator.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值：\n"</span>,y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值与真实值是否相等：\n"</span>,y_pred==y_test)</span><br><span class="line">accuracy = accuracy_score(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"\nKNN 模型的准确率: <span class="subst">{accuracy:<span class="number">.4</span>f}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 交叉验证的相关参数</span></span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最好结果：<span class="subst">{estimator.best_score_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最好模型：<span class="subst">{estimator.best_estimator_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最好模型结果：<span class="subst">{estimator.cv_results_}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
<p><img src="/img/machinelearning/cros-valid.png"></p>
|
||
<h3 id="机器学习的基本步骤"><a href="#机器学习的基本步骤" class="headerlink" title="机器学习的基本步骤"></a>机器学习的基本步骤</h3><ul>
|
||
<li>获取数据集</li>
|
||
<li>数据集基本处理<ul>
|
||
<li>去重去空、填充等操作 </li>
|
||
<li>确定特征值和目标值</li>
|
||
<li>分割数据集</li>
|
||
</ul>
|
||
</li>
|
||
<li>特征工程（特征预处理 标准化等）</li>
|
||
<li>机器学习</li>
|
||
<li>模型评估</li>
|
||
</ul>
|
||
<h3 id="数据分割的方法"><a href="#数据分割的方法" class="headerlink" title="数据分割的方法"></a>数据分割的方法</h3><ul>
|
||
<li>留出法<br>训练/测试集的划分要尽可能保持数据分布的一致性，避免因数据划分过程引入额外的偏差而对最终结果产生影响。<br>单次使用留出法得到的估计结果往往不够稳定可靠，在使用留出法时，一般要采用若干次随机划分、重复进行实验评估后取平均值作为留出法的评估结果。<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> KFold,StratifiedKFold</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line">X = np.array([</span><br><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>],</span><br><span class="line">[<span class="number">11</span>,<span class="number">12</span>,<span class="number">13</span>,<span class="number">14</span>],</span><br><span class="line">[<span class="number">21</span>,<span class="number">22</span>,<span class="number">23</span>,<span class="number">24</span>],</span><br><span class="line">[<span class="number">31</span>,<span class="number">32</span>,<span class="number">33</span>,<span class="number">34</span>],</span><br><span class="line">[<span class="number">41</span>,<span class="number">42</span>,<span class="number">43</span>,<span class="number">44</span>],</span><br><span class="line">[<span class="number">51</span>,<span class="number">52</span>,<span class="number">53</span>,<span class="number">54</span>],</span><br><span class="line">[<span class="number">61</span>,<span class="number">62</span>,<span class="number">63</span>,<span class="number">64</span>],</span><br><span class="line">[<span class="number">71</span>,<span class="number">72</span>,<span class="number">73</span>,<span class="number">74</span>]</span><br><span class="line">])</span><br><span class="line">y=np.array([<span class="number">1</span>,<span class="number">1</span>,<span class="number">0</span>,<span class="number">0</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">0</span>,<span class="number">0</span>])</span><br><span class="line">folder = KFold(n_splits=<span class="number">4</span>)</span><br><span class="line">sfloder = StratifiedKFold(n_splits=<span class="number">4</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"KFOLD:"</span>)</span><br><span class="line"><span class="keyword">for</span> train,test <span class="keyword">in</span> folder.split(X,y):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"train:<span class="subst">{train}</span>,test:<span class="subst">{test}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"SKFOLD:"</span>)</span><br><span class="line"><span class="keyword">for</span> train,test <span class="keyword">in</span> sfloder.split(X,y):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"train:<span class="subst">{train}</span>,test:<span class="subst">{test}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
<img src="/img/machinelearning/kfold-skfold.png"></li>
|
||
<li>自助法</li>
|
||
<li>交叉验证法</li>
|
||
</ul>
|
||
]]></content>
|
||
<categories>
|
||
<category>machinelearning</category>
|
||
</categories>
|
||
<tags>
|
||
<tag>KNN</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>决策树算法</title>
|
||
<url>/posts/95.html</url>
|
||
<content><![CDATA[<h3 id="C4-5"><a href="#C4-5" class="headerlink" title="C4.5"></a>C4.5</h3><p>C4.5 是一种用于生成决策树的算法，不再使用信息增益，而是使用信息增益比，来避免偏向于选择取值较多的特征。信息增益比是信息增益与特征的熵的比值。</p>
|
||
<h3 id="ID3"><a href="#ID3" class="headerlink" title="ID3"></a>ID3</h3><p>D3 是一种基于信息增益（Information Gain）的决策树算法</p>
|
||
<h3 id="Cart"><a href="#Cart" class="headerlink" title="Cart"></a>Cart</h3><p>CART（分类与回归树）一种决策树算法，，CART 使用 <strong>二叉树结构</strong>，即每个节点只能有两个子节点。</p>
|
||
<h3 id="cart剪枝"><a href="#cart剪枝" class="headerlink" title="cart剪枝"></a>cart剪枝</h3><p>CART 决策树的剪枝方法分为 <strong>预剪枝</strong>（Pre-pruning）和 <strong>后剪枝</strong>（Post-pruning）两种：</p>
|
||
<h4 id="预剪枝："><a href="#预剪枝：" class="headerlink" title="预剪枝："></a><strong>预剪枝</strong>：</h4><p>预剪枝是在构建决策树时就决定是否停止进一步划分某个节点。主要通过以下标准来控制：</p>
|
||
<ul>
|
||
<li>当某个节点的样本数小于某个阈值时，不再继续划分。</li>
|
||
<li>当某个节点的 Gini 不纯度小于某个阈值时，不再继续划分。<br>预剪枝的优点是能够减少计算量，但缺点是可能会导致模型不够复杂，从而产生欠拟合。</li>
|
||
</ul>
|
||
<h4 id="后剪枝："><a href="#后剪枝：" class="headerlink" title="后剪枝："></a><strong>后剪枝</strong>：</h4><p>后剪枝是在决策树完全构建出来之后，对树进行修剪。具体过程如下：</p>
|
||
<ul>
|
||
<li>构建完整的决策树。</li>
|
||
<li>从叶子节点开始，逐渐向上遍历树的每个节点。</li>
|
||
<li>对每个节点进行判断，是否合适剪去该节点及其子树。如果剪去该子树后，模型的性能没有显著下降，就可以剪枝。<br>后剪枝通过避免过度拟合来提高模型的泛化能力，但其计算开销较大。</li>
|
||
</ul>
|
||
<h3 id="特征工程-特征提取"><a href="#特征工程-特征提取" class="headerlink" title="特征工程(特征提取)"></a>特征工程(特征提取)</h3><ul>
|
||
<li><strong>字典特征提取</strong><br>主要用于处理包含键值对（key-value pairs）的数据结构<br> <figure class="highlight python"><table><tr><td class="code"><pre><span class="line"> <span class="keyword">from</span> sklearn.feature_extraction <span class="keyword">import</span> DictVectorizer</span><br><span class="line"> <span class="comment"># 字典特征提取</span></span><br><span class="line"> data = [</span><br><span class="line"> {<span class="string">'city'</span>: <span class="string">'beijing'</span>, <span class="string">'temperature'</span>: <span class="number">100</span>},</span><br><span class="line"> {<span class="string">'city'</span>: <span class="string">'shanghai'</span>, <span class="string">'temperature'</span>: <span class="number">95</span>},</span><br><span class="line"> {<span class="string">'city'</span>: <span class="string">'guangzhou'</span>, <span class="string">'temperature'</span>: <span class="number">98</span>}</span><br><span class="line"> ]</span><br><span class="line"> transfer = DictVectorizer(sparse=<span class="literal">False</span>)</span><br><span class="line"> new_data = transfer.fit_transform(data)</span><br><span class="line"> <span class="built_in">print</span>(transfer.feature_names_)</span><br><span class="line"> <span class="built_in">print</span>(new_data)</span><br></pre></td></tr></table></figure></li>
|
||
<li><strong>文本特征提取</strong><br>主要用于将文本数据（如句子、段落、文章等）转换成数值型特征。这对于文本分类、信息检索等任务非常重要。<br> <figure class="highlight python"><table><tr><td class="code"><pre><span class="line"> <span class="keyword">from</span> sklearn.feature_extraction.text <span class="keyword">import</span> CountVectorizer</span><br><span class="line"> <span class="comment"># 示例文本数据</span></span><br><span class="line"> data = [</span><br><span class="line"> <span class="string">"I love programming"</span>,</span><br><span class="line"> <span class="string">"Python is great"</span>,</span><br><span class="line"> <span class="string">"I love machine learning"</span></span><br><span class="line"> ]</span><br><span class="line"> <span class="comment"># 创建 CountVectorizer 对象</span></span><br><span class="line"> transfer = CountVectorizer()</span><br><span class="line"> <span class="comment"># 将文本数据转换为特征向量</span></span><br><span class="line"> new_data = transfer.fit_transform(data)</span><br><span class="line"> <span class="comment"># 输出特征名称</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"Feature Names:"</span>, transfer.get_feature_names_out())</span><br><span class="line"> <span class="comment"># 输出转换后的特征矩阵</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"Transformed Data:"</span>, new_data.toarray())</span><br></pre></td></tr></table></figure></li>
|
||
<li>文本特征提取（中文文本) <figure class="highlight python"><table><tr><td class="code"><pre><span class="line"> <span class="keyword">from</span> sklearn.feature_extraction.text <span class="keyword">import</span> CountVectorizer</span><br><span class="line"> <span class="keyword">import</span> jieba</span><br><span class="line"> <span class="comment"># 中文文本数据（大于20个字）</span></span><br><span class="line"> data = [</span><br><span class="line"> <span class="string">"我热爱编程，学习编程语言是一件非常有趣的事情，它能够提升我们解决问题的能力，编程让我变得更加有创意。"</span>,</span><br><span class="line"> <span class="string">"Python语言是一门非常强大的编程语言，具有简洁的语法和丰富的库，可以帮助开发者更高效地完成任务。"</span>,</span><br><span class="line"> <span class="string">"机器学习是一项非常有前途的技术，它能够让计算机从数据中自动学习，逐步提高模型的精确度，解决实际问题。"</span></span><br><span class="line"></span><br><span class="line"> ]</span><br><span class="line"> <span class="comment"># 使用jieba分词</span></span><br><span class="line"> text_list = []</span><br><span class="line"> <span class="keyword">for</span> line <span class="keyword">in</span> data:</span><br><span class="line"> text_list.append(<span class="string">" "</span>.join(<span class="built_in">list</span>(jieba.cut(line))))</span><br><span class="line"> <span class="comment"># 创建 CountVectorizer 对象</span></span><br><span class="line"> transfer = CountVectorizer()</span><br><span class="line"> <span class="comment"># 将文本数据转换为特征向量</span></span><br><span class="line"> new_data = transfer.fit_transform(text_list)</span><br><span class="line"> <span class="comment"># 输出特征名称</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"Feature Names:"</span>, transfer.get_feature_names_out())</span><br><span class="line"> <span class="comment"># 输出转换后的特征矩阵</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"Transformed Data:"</span>, new_data.toarray())</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="tf-idf"><a href="#tf-idf" class="headerlink" title="tf-idf"></a>tf-idf</h3><blockquote>
|
||
<p>词频 * 逆文档频率</p>
|
||
</blockquote>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># tfi-df</span></span><br><span class="line"><span class="keyword">from</span> sklearn.feature_extraction.text <span class="keyword">import</span> TfidfVectorizer</span><br><span class="line"><span class="keyword">import</span> jieba</span><br><span class="line">data=[<span class="string">"一种还是一种今天很残酷，明天更残酷，后天很美好，但绝对大部分是死在明天晚上，所以每个人不要放弃今天。"</span>,</span><br><span class="line"> <span class="string">"我们看到的从很远星系来的光是在几百万年之前发出的，这样当我们看到宇宙时，我们是在看它的过去"</span>,</span><br><span class="line"> <span class="string">"如果只用一种方式了解某样事物，你就不会真正了解它。了解事物真正含义的秘密取决于如何将其与我们所了解的事物相联系"</span>]</span><br><span class="line"><span class="built_in">list</span> = []</span><br><span class="line"><span class="keyword">for</span> item <span class="keyword">in</span> data:</span><br><span class="line"> <span class="built_in">list</span>.append(<span class="string">" "</span>.join(jieba.cut(item)))</span><br><span class="line">transfer = TfidfVectorizer()</span><br><span class="line">new_data = transfer.fit_transform(<span class="built_in">list</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"特征名字：\n<span class="subst">{transfer.get_feature_names_out()}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"转换后的特征矩阵:\n<span class="subst">{ new_data.toarray()}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"转换后的数据:\n<span class="subst">{new_data}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
|
||
|
||
<h3 id="回归决策树"><a href="#回归决策树" class="headerlink" title="回归决策树"></a>回归决策树</h3><h4 id="决策树算法的应用-（泰坦尼克号沉船幸存者预测）"><a href="#决策树算法的应用-（泰坦尼克号沉船幸存者预测）" class="headerlink" title="决策树算法的应用 （泰坦尼克号沉船幸存者预测）"></a>决策树算法的应用 （泰坦尼克号沉船幸存者预测）</h4><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> seaborn <span class="keyword">as</span> sns</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.feature_extraction <span class="keyword">import</span> DictVectorizer</span><br><span class="line"><span class="keyword">from</span> sklearn.tree <span class="keyword">import</span> DecisionTreeClassifier,export_graphviz</span><br><span class="line"><span class="comment"># 1.获取数据集 - 加载 Titanic 数据集</span></span><br><span class="line">titanic = sns.load_dataset(<span class="string">'titanic'</span>)</span><br><span class="line">missing_age_count = titanic[<span class="string">'age'</span>].isna().<span class="built_in">sum</span>()</span><br><span class="line"><span class="comment"># print(f"缺失的 age 数量: {missing_age_count}")</span></span><br><span class="line"><span class="comment"># 2. 数据基本处理</span></span><br><span class="line"><span class="comment"># 2.1 确认特征值、目标值</span></span><br><span class="line">X = titanic[[<span class="string">'pclass'</span>,<span class="string">'age'</span>,<span class="string">'sex'</span>]]</span><br><span class="line">y = titanic[<span class="string">'survived'</span>]</span><br><span class="line"><span class="comment"># 2.2 缺失值处理</span></span><br><span class="line">X.loc[:, <span class="string">'age'</span>] = X[<span class="string">'age'</span>].fillna(value=X[<span class="string">'age'</span>].mean()) <span class="comment"># 使用 .loc 进行修改</span></span><br><span class="line"><span class="comment"># 2.3 划分数据集</span></span><br><span class="line">X_train,X_test,y_train,y_test = train_test_split(X,y,random_state=<span class="number">22</span>)</span><br><span class="line"><span class="comment"># 3. 特征工程（字典特征提取）</span></span><br><span class="line">X_train = X_train.to_dict(orient=<span class="string">"records"</span>)</span><br><span class="line">X_test= X_test.to_dict(orient=<span class="string">"records"</span>)</span><br><span class="line">transfer = DictVectorizer()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"><span class="comment"># 4. 机器学习 决策树算法</span></span><br><span class="line">estimator = DecisionTreeClassifier(criterion=<span class="string">"gini"</span>)</span><br><span class="line">estimator.fit(X_train,y_train)</span><br><span class="line">y_pred = estimator.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型的测试集的预测值：<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line">ret = estimator.score(X_test,y_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型的评分:<span class="subst">{ret}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(X_test.toarray())</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>生成对应的图</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.tree <span class="keyword">import</span> export_graphviz</span><br><span class="line"><span class="keyword">import</span> graphviz <span class="comment"># 用于渲染图像</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 导出决策树的 Graphviz 表示</span></span><br><span class="line">export_graphviz(estimator, out_file=<span class="string">'./data/tree.dot'</span>, </span><br><span class="line"> feature_names=transfer.get_feature_names_out()) <span class="comment"># 特征名称</span></span><br><span class="line"><span class="comment"># 使用 graphviz 渲染 .dot 文件</span></span><br><span class="line"><span class="keyword">with</span> <span class="built_in">open</span>(<span class="string">'./data/tree.dot'</span>, <span class="string">'r'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> dot_graph = f.read()</span><br><span class="line"><span class="comment"># 渲染决策树</span></span><br><span class="line">graph = graphviz.Source(dot_graph)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 设置保存路径</span></span><br><span class="line">output_path = <span class="string">'./data/decision_tree'</span> <span class="comment"># 自定义保存路径</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 保存图像到指定路径，格式可以是 .png, .pdf, .jpg 等</span></span><br><span class="line"><span class="comment"># graph.render(output_path, format='png') # 保存为 .png 文件</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 显示图像</span></span><br><span class="line">graph.view(output_path) <span class="comment"># 打开图像，path为保存路径，不需要加后缀</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
|
||
|
||
<p><a href="http://webgraphviz.com/">Webgraphviz</a>，这个网站可以将<code>tree.dot</code>文件的内容生成对应的可视化树</p>
|
||
<h4 id="回归决策树与线性回归的对比"><a href="#回归决策树与线性回归的对比" class="headerlink" title="回归决策树与线性回归的对比"></a>回归决策树与线性回归的对比</h4><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> LinearRegression</span><br><span class="line"><span class="keyword">from</span> sklearn.tree <span class="keyword">import</span> DecisionTreeRegressor</span><br><span class="line"><span class="keyword">from</span> matplotlib <span class="keyword">import</span> rcParams</span><br><span class="line"></span><br><span class="line"><span class="comment"># 设置matplotlib使用的字体为SimHei（黑体）</span></span><br><span class="line">rcParams[<span class="string">'font.sans-serif'</span>] = [<span class="string">'SimHei'</span>] <span class="comment"># 也可以使用 'Microsoft YaHei'</span></span><br><span class="line">rcParams[<span class="string">'axes.unicode_minus'</span>] = <span class="literal">False</span> <span class="comment"># 正常显示负号</span></span><br><span class="line">x = np.array(<span class="built_in">list</span>(<span class="built_in">range</span>(<span class="number">1</span>,<span class="number">11</span>))).reshape(-<span class="number">1</span>,<span class="number">1</span>)</span><br><span class="line">y = ([<span class="number">5.56</span>,<span class="number">5.70</span>,<span class="number">5.91</span>,<span class="number">6.40</span>,<span class="number">6.80</span>,<span class="number">7.05</span>,<span class="number">8.90</span>,<span class="number">8.70</span>,<span class="number">9.00</span>,<span class="number">9.05</span>])</span><br><span class="line"></span><br><span class="line">m1 = DecisionTreeRegressor(max_depth=<span class="number">1</span>)</span><br><span class="line">m2 = DecisionTreeRegressor(max_depth=<span class="number">3</span>)</span><br><span class="line">m3 = DecisionTreeRegressor()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 模型训练</span></span><br><span class="line">m1.fit(x,y)</span><br><span class="line">m2.fit(x,y)</span><br><span class="line">m3.fit(x,y)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 模型预测</span></span><br><span class="line">x_test = np.arange(<span class="number">0</span>,<span class="number">10</span>,<span class="number">0.01</span>).reshape(-<span class="number">1</span>,<span class="number">1</span>)</span><br><span class="line">y_1 = m1.predict(x_test)</span><br><span class="line">y_2 = m2.predict(x_test)</span><br><span class="line">y_3 = m3.predict(x_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 结果展示</span></span><br><span class="line">plt.figure(figsize=(<span class="number">10</span>,<span class="number">6</span>),dpi=<span class="number">100</span>)</span><br><span class="line">plt.scatter(x,y ,label = <span class="string">"data"</span>)</span><br><span class="line">plt.plot(x_test,y_1,label = <span class="string">"max_depth=1"</span>)</span><br><span class="line">plt.plot(x_test,y_2,label = <span class="string">"max_depth=3"</span>)</span><br><span class="line">plt.plot(x_test,y_3,label = <span class="string">"linearregression"</span>)</span><br><span class="line">plt.xlabel(<span class="string">"数据"</span>)</span><br><span class="line">plt.ylabel(<span class="string">"预测值"</span>)</span><br><span class="line">plt.legend()</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure>
|
||
]]></content>
|
||
<categories>
|
||
<category>machinelearning</category>
|
||
</categories>
|
||
<tags>
|
||
<tag>decisiontree</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>逻辑回归</title>
|
||
<url>/posts/60504.html</url>
|
||
<content><![CDATA[<h3 id="logistic-regression-code"><a href="#logistic-regression-code" class="headerlink" title="logistic regression code"></a>logistic regression code</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> load_breast_cancer</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> LogisticRegression</span><br><span class="line"><span class="comment"># 1. 加载乳腺癌数据集</span></span><br><span class="line">data = load_breast_cancer()</span><br><span class="line"><span class="comment"># 2.1 数据集基本处理</span></span><br><span class="line">df = pd.DataFrame(data.data, columns=data.feature_names)</span><br><span class="line">df[<span class="string">'target'</span>] = data.target</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> df.columns:</span><br><span class="line"> <span class="comment"># 检查列是否有缺失值</span></span><br><span class="line"> <span class="keyword">if</span> np.<span class="built_in">any</span>(pd.isnull(df[i])):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Filling missing values in column: <span class="subst">{i}</span>"</span>)</span><br><span class="line"><span class="comment">#2.2 确认特征值、目标值</span></span><br><span class="line">X = df.iloc[:,<span class="number">0</span>:df.shape[<span class="number">1</span>] - <span class="number">1</span>]</span><br><span class="line">y = df.loc[:,<span class="string">"target"</span>]</span><br><span class="line"><span class="comment"># 2.3 分割数据</span></span><br><span class="line">X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=<span class="number">0.3</span>)</span><br><span class="line"><span class="comment"># 显示前几行数据</span></span><br><span class="line">df.head(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3. 特征工程 标准化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 4 机器学习 逻辑回归</span></span><br><span class="line">estimator = LogisticRegression()</span><br><span class="line">estimator.fit(X_train,y_train)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 5. 模型评估</span></span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型准确率：<span class="subst">{estimator.score(X_test,y_test)}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型预测值为:\n<span class="subst">{estimator.predict(X_test)}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="分类评估的参数"><a href="#分类评估的参数" class="headerlink" title="分类评估的参数"></a>分类评估的参数</h3><ul>
|
||
<li><p>准确率<br>准确率是所有预测正确的样本占总样本的比例<br>$$Accuracy = \frac{TP+TN}{TP+FN+FP+TN}$$</p>
|
||
</li>
|
||
<li><p>精准率<br>精准率（又称查准率）是指所有被预测为正类的样本中，真正为正类的比例<br>$$Precision = \frac{TP}{TP+FP}$$</p>
|
||
</li>
|
||
<li><p>召回率<br>召回率（又称查全率）是指所有实际为正类的样本中，被正确预测为正类的比例<br>$$Recall = \frac{TP}{TP+FN}$$</p>
|
||
</li>
|
||
<li><p>F1-score<br>F1 值（F1 Score）是精准率和召回率的调和平均数，综合考虑了精准率和召回率的影响。<br>$$ F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} $$</p>
|
||
</li>
|
||
<li><p>roc曲线<br>tpr、fpr来衡量不平衡的二分类问题</p>
|
||
</li>
|
||
</ul>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> load_breast_cancer</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> LogisticRegression</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> classification_report, roc_auc_score</span><br><span class="line"><span class="comment"># 1. 加载乳腺癌数据集</span></span><br><span class="line">data = load_breast_cancer()</span><br><span class="line"><span class="comment"># 2.1 数据集基本处理</span></span><br><span class="line">df = pd.DataFrame(data.data, columns=data.feature_names)</span><br><span class="line">df[<span class="string">'target'</span>] = data.target</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> df.columns:</span><br><span class="line"> <span class="comment"># 检查列是否有缺失值</span></span><br><span class="line"> <span class="keyword">if</span> np.<span class="built_in">any</span>(pd.isnull(df[i])):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Filling missing values in column: <span class="subst">{i}</span>"</span>)</span><br><span class="line"><span class="comment"># 2.2 确认特征值、目标值</span></span><br><span class="line">X = df.iloc[:, <span class="number">0</span>:df.shape[<span class="number">1</span>] - <span class="number">1</span>]</span><br><span class="line">y = df.loc[:, <span class="string">"target"</span>]</span><br><span class="line"><span class="comment"># 2.3 分割数据</span></span><br><span class="line">X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span class="number">0.3</span>)</span><br><span class="line"><span class="comment"># 显示前几行数据</span></span><br><span class="line">df.head(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3. 特征工程 标准化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 4 机器学习 逻辑回归</span></span><br><span class="line">estimator = LogisticRegression()</span><br><span class="line">estimator.fit(X_train, y_train)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 5. 模型评估</span></span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型准确率：<span class="subst">{estimator.score(X_test, y_test)}</span>"</span>)</span><br><span class="line">y_pred = estimator.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型预测值为:\n<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line"><span class="comment"># 5.1 精确率、召回率</span></span><br><span class="line">ret = classification_report(y_test, y_pred, labels=[<span class="number">1</span>, <span class="number">0</span>], target_names=[<span class="string">"良性"</span>, <span class="string">"恶性"</span>])</span><br><span class="line">roc_score = roc_auc_score(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"准确率、召回率：<span class="subst">{ret}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"roc_score:<span class="subst">{roc_score}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="类别不平衡的处理"><a href="#类别不平衡的处理" class="headerlink" title="类别不平衡的处理"></a>类别不平衡的处理</h3><p>先准备类别不平衡的数据</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> imblearn.over_sampling <span class="keyword">import</span> RandomOverSampler,SMOTE</span><br><span class="line"><span class="keyword">from</span> imblearn.under_sampling <span class="keyword">import</span> RandomUnderSampler</span><br><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> make_classification</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt</span><br><span class="line"><span class="keyword">from</span> collections <span class="keyword">import</span> Counter</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1.准备类别不平衡的数据</span></span><br><span class="line">X, y = make_classification(</span><br><span class="line"> n_samples=<span class="number">5000</span>,</span><br><span class="line"> n_features=<span class="number">2</span>,</span><br><span class="line"> n_informative=<span class="number">2</span>,</span><br><span class="line"> n_redundant=<span class="number">0</span>,</span><br><span class="line"> n_repeated=<span class="number">0</span>,</span><br><span class="line"> n_classes=<span class="number">3</span>,</span><br><span class="line"> n_clusters_per_class=<span class="number">1</span>,</span><br><span class="line"> weights=[<span class="number">0.01</span>, <span class="number">0.05</span>, <span class="number">0.94</span>],</span><br><span class="line"> random_state=<span class="number">0</span>,</span><br><span class="line">)</span><br><span class="line">counter = Counter(y)</span><br><span class="line">plt.scatter(X[:,<span class="number">0</span>],X[:,<span class="number">1</span>],c=y)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure>
|
||
|
||
<ul>
|
||
<li>过采样<br>增加训练集的少数的类别的样本，使得正反例样本数据接近 <ul>
|
||
<li>随机过采样（RandomOverSampler)<figure class="highlight python"><table><tr><td class="code"><pre><span class="line">ros = RandomOverSampler()</span><br><span class="line">X_resampled,y_resampled = ros.fit_resample(X,y)</span><br><span class="line"><span class="built_in">print</span>(Counter(y_resampled))</span><br><span class="line">plt.scatter(X_resampled[:,<span class="number">0</span>],X_resampled[:,<span class="number">1</span>],c=y_resampled)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure>
|
||
<img src="/img/machinelearning/over_random_sampling.png"></li>
|
||
<li><code>SMOTE</code>过采样（SMOTE）<figure class="highlight python"><table><tr><td class="code"><pre><span class="line">smote = SMOTE()</span><br><span class="line">X_resampled,y_resampled = smote.fit_resample(X,y)</span><br><span class="line"><span class="built_in">print</span>(Counter(y_resampled))</span><br><span class="line">plt.scatter(X_resampled[:,<span class="number">0</span>],X_resampled[:,<span class="number">1</span>],c=y_resampled)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure>
|
||
<img src="/img/machinelearning/over_smote_sampling.png"></li>
|
||
</ul>
|
||
</li>
|
||
<li>欠采样<br>减少训练集的多数的类别的样本，使得正反例样本数据接近<ul>
|
||
<li>随机欠采样（RandomUnderSampler）<figure class="highlight python"><table><tr><td class="code"><pre><span class="line">rus = RandomUnderSampler(random_state=<span class="number">0</span>)</span><br><span class="line">X_resampled,y_resampled = rus.fit_resample(X,y)</span><br><span class="line"><span class="built_in">print</span>(Counter(y_resampled))</span><br><span class="line">plt.scatter(X_resampled[:,<span class="number">0</span>],X_resampled[:,<span class="number">1</span>],c=y_resampled)</span><br><span class="line">plt.show()</span><br></pre></td></tr></table></figure>
|
||
<img src="/img/machinelearning/under_sampling.png"></li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
]]></content>
|
||
<categories>
|
||
<category>machinelearning</category>
|
||
</categories>
|
||
<tags>
|
||
<tag>logistic-regression</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>网络相关</title>
|
||
<url>/posts/41168.html</url>
|
||
<content><![CDATA[]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>线性回归</title>
|
||
<url>/posts/52662.html</url>
|
||
<content><![CDATA[<h3 id="线性回归简介"><a href="#线性回归简介" class="headerlink" title="线性回归简介"></a>线性回归简介</h3><blockquote>
|
||
<p>用于预测一个连续的目标变量（因变量），与一个或多个特征（自变量）之间存在线性关系。</p>
|
||
</blockquote>
|
||
<p>假设函数：<br>$$y = w_1x_1 + w_2x_2 + \cdot\cdot\cdot+w_nx_n$$</p>
|
||
<ul>
|
||
<li>$y$ 是目标变量（因变量），即我们希望预测的值。</li>
|
||
<li>$x1,x2,…,xn$ 是特征变量（自变量），即输入的值。</li>
|
||
</ul>
|
||
<h3 id="损失函数"><a href="#损失函数" class="headerlink" title="损失函数"></a>损失函数</h3><p>为了找到最佳的线性模型，我们需要通过最小化损失函数来优化模型参数。在线性回归中，常用的损失函数是 <strong>均方误差（MSE）</strong>：<br>$$J(\theta) = \frac{1}{2N} \sum_{i=1}^{N} (y_i - f_\theta(x_i))^2$$</p>
|
||
<ul>
|
||
<li>N 是样本的数量。</li>
|
||
<li>$y_i$ 是第 i 个样本的真实值。</li>
|
||
<li>$f_\theta(x_i)$ 是模型预测的第 i 个样本的值。</li>
|
||
</ul>
|
||
<h3 id="线性回归优化"><a href="#线性回归优化" class="headerlink" title="线性回归优化"></a>线性回归优化</h3><ul>
|
||
<li><p>梯度下降法</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> fetch_california_housing</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> SGDRegressor</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> mean_squared_error</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1. 获取数据集</span></span><br><span class="line">housing = fetch_california_housing()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2. 数据集处理</span></span><br><span class="line"><span class="comment"># 2.1 分割数据集</span></span><br><span class="line">X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class="number">0.25</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3. 特征工程</span></span><br><span class="line"><span class="comment"># 3.1 标准化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test) <span class="comment"># 使用 transform() 而不是 fit_transform()</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 4.机器学习- 梯度下降法</span></span><br><span class="line">estimater = SGDRegressor(max_iter=<span class="number">1000</span>, eta0=<span class="number">0.01</span>)</span><br><span class="line">estimater.fit(X_train, y_train)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"SGD模型的偏置是：<span class="subst">{estimater.intercept_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"SGD模型的系数是：<span class="subst">{estimater.coef_}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 5. 模型评估</span></span><br><span class="line">y_pred = estimater.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"SGD模型预测值：<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line">mse = mean_squared_error(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"SGD模型均方误差:<span class="subst">{mse}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>正规方程</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> fetch_california_housing</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> LinearRegression</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> mean_squared_error</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1. 获取数据集</span></span><br><span class="line">housing = fetch_california_housing()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2. 数据集处理</span></span><br><span class="line"><span class="comment"># 2.1 分割数据集</span></span><br><span class="line">X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class="number">0.25</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3. 特征工程</span></span><br><span class="line"><span class="comment"># 3.1 标准化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.fit_transform(X_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 4.机器学习- 线性回归</span></span><br><span class="line">estimater = LinearRegression()</span><br><span class="line">estimater.fit(X_train, y_train)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型的偏置是：<span class="subst">{estimater.intercept_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型的系数是：<span class="subst">{estimater.coef_}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 5. 模型评估</span></span><br><span class="line">y_pred = estimater.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型预测值：<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line">mse = mean_squared_error(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"模型均方误差:<span class="subst">{mse}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>岭回归</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> fetch_california_housing</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> Ridge, RidgeCV</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> mean_squared_error</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1. 获取数据集</span></span><br><span class="line">housing = fetch_california_housing()</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2. 数据集处理</span></span><br><span class="line"><span class="comment"># 2.1 分割数据集</span></span><br><span class="line">X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class="number">0.25</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3. 特征工程</span></span><br><span class="line"><span class="comment"># 3.1 标准化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test) <span class="comment"># 使用 transform() 而不是 fit_transform()</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 4.机器学习- 岭回归 使用了Ridge的alpha的搜索</span></span><br><span class="line"><span class="comment"># estimater = Ridge(alpha=1.0)</span></span><br><span class="line">estimater = RidgeCV(alphas=[<span class="number">0.001</span>, <span class="number">0.01</span>, <span class="number">0.1</span>, <span class="number">1</span>, <span class="number">10</span>, <span class="number">100</span>])</span><br><span class="line">estimater.fit(X_train, y_train)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"Ridge模型的偏置是：<span class="subst">{estimater.intercept_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"Ridge模型的系数是：<span class="subst">{estimater.coef_}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看最佳 alpha</span></span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最佳 alpha 值是：<span class="subst">{estimater.alpha_}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 5. 模型评估</span></span><br><span class="line">y_pred = estimater.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"Ridge模型预测值：<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line">mse = mean_squared_error(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"Ridge模型均方误差:<span class="subst">{mse}</span>"</span>)</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<p>这样每个代码块的缩进保持一致，便于阅读和理解。如果有其他优化需求，随时告诉我！</p>
|
||
<p><img src="/img/machinelearning/linear.png"></p>
|
||
<p><img src="/img/machinelearning/fitting.png"></p>
|
||
<h3 id="模型保存和加载"><a href="#模型保存和加载" class="headerlink" title="模型保存和加载"></a>模型保存和加载</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> fetch_california_housing</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.linear_model <span class="keyword">import</span> Ridge, RidgeCV</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> mean_squared_error</span><br><span class="line"><span class="keyword">import</span> joblib</span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">save_model</span>():</span><br><span class="line"> <span class="comment"># 1. 获取数据集</span></span><br><span class="line"> housing = fetch_california_housing()</span><br><span class="line"> <span class="comment"># 2. 数据集处理</span></span><br><span class="line"> <span class="comment"># 2.1 分割数据集</span></span><br><span class="line"> X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class="number">0.25</span>)</span><br><span class="line"> <span class="comment"># 3. 特征工程</span></span><br><span class="line"> <span class="comment"># 3.1 标准化</span></span><br><span class="line"> transfer = StandardScaler()</span><br><span class="line"> X_train = transfer.fit_transform(X_train)</span><br><span class="line"> X_test = transfer.transform(X_test) <span class="comment"># 使用 transform() 而不是 fit_transform()</span></span><br><span class="line"> <span class="comment"># 4. 机器学习 - 岭回归 使用了Ridge的alpha的搜索</span></span><br><span class="line"> estimater = RidgeCV(alphas=[<span class="number">0.001</span>, <span class="number">0.01</span>, <span class="number">0.1</span>, <span class="number">1</span>, <span class="number">10</span>, <span class="number">100</span>])</span><br><span class="line"> estimater.fit(X_train, y_train)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Ridge模型的偏置是：<span class="subst">{estimater.intercept_}</span>"</span>)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Ridge模型的系数是：<span class="subst">{estimater.coef_}</span>"</span>)</span><br><span class="line"> <span class="comment"># 保存模型</span></span><br><span class="line"> joblib.dump(estimater, <span class="string">'ridge_model.pkl'</span>)</span><br><span class="line"> <span class="comment"># 查看最佳 alpha</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"最佳 alpha 值是：<span class="subst">{estimater.alpha_}</span>"</span>)</span><br><span class="line"> <span class="comment"># 5. 模型评估</span></span><br><span class="line"> y_pred = estimater.predict(X_test)</span><br><span class="line"> mse = mean_squared_error(y_test, y_pred)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Ridge模型均方误差:<span class="subst">{mse}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">load_model</span>():</span><br><span class="line"> <span class="comment"># 1. 获取数据集</span></span><br><span class="line"> housing = fetch_california_housing()</span><br><span class="line"> <span class="comment"># 2. 数据集处理</span></span><br><span class="line"> <span class="comment"># 2.1 分割数据集</span></span><br><span class="line"> X_train, X_test, y_train, y_test = train_test_split(housing.data, housing.target, test_size=<span class="number">0.25</span>)</span><br><span class="line"> <span class="comment"># 3. 特征工程</span></span><br><span class="line"> <span class="comment"># 3.1 标准化</span></span><br><span class="line"> transfer = StandardScaler()</span><br><span class="line"> X_train = transfer.fit_transform(X_train)</span><br><span class="line"> X_test = transfer.transform(X_test) <span class="comment"># 使用 transform() 而不是 fit_transform()</span></span><br><span class="line"> <span class="comment"># 加载模型</span></span><br><span class="line"> estimater = joblib.load(<span class="string">'ridge_model.pkl'</span>)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Ridge模型的偏置是：<span class="subst">{estimater.intercept_}</span>"</span>)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Ridge模型的系数是：<span class="subst">{estimater.coef_}</span>"</span>)</span><br><span class="line"> <span class="comment"># 查看最佳 alpha</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"最佳 alpha 值是：<span class="subst">{estimater.alpha_}</span>"</span>)</span><br><span class="line"> <span class="comment"># 5. 模型评估</span></span><br><span class="line"> y_pred = estimater.predict(X_test)</span><br><span class="line"> mse = mean_squared_error(y_test, y_pred)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Ridge模型预测值：<span class="subst">{y_pred}</span>"</span>)</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"Ridge模型均方误差:<span class="subst">{mse}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="built_in">print</span>(<span class="string">"训练并保存模型："</span>)</span><br><span class="line">save_model()</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"加载模型"</span>)</span><br><span class="line">load_model()</span><br></pre></td></tr></table></figure>]]></content>
|
||
<categories>
|
||
<category>machinelearning</category>
|
||
</categories>
|
||
<tags>
|
||
<tag>linear-regression</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>郑伯克段于鄢</title>
|
||
<url>/posts/58638.html</url>
|
||
<content><![CDATA[<p>原文如下：</p>
|
||
<p> 初，郑武公娶于申【申国】，曰武姜【武为武公谥号，姜为其宗族之性】。生庄公及共叔段【共表示其曾出逃到共，叔为老三，段为名】。庄公寤生【出生时头先出，难产】，惊姜氏，故名曰“寤生”， 遂恶之，爱【喜爱】共叔段，欲立【立为储君】之，亟（qì）【多次】请于武公，及庄公即位，为之【共叔段】请制【一个叫制的封地，虎牢关所在】。公曰：“制，岩邑【险要的城邑】也，虢叔死焉，佗【通“他”，其他】邑唯命（是听）。”请京，使居之，谓之“京城大叔”。</p>
|
||
]]></content>
|
||
<categories>
|
||
<category>古文观止</category>
|
||
</categories>
|
||
<tags>
|
||
<tag>古文观止</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>Hadoop集群搭建基础环境</title>
|
||
<url>/posts/61253.html</url>
|
||
<content><![CDATA[<h3 id="防火墙关闭"><a href="#防火墙关闭" class="headerlink" title="防火墙关闭"></a>防火墙关闭</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 在 6 台主机执行</span></span><br><span class="line">systemctl stop firewalld</span><br><span class="line">systemctl <span class="built_in">disable</span> firewalld</span><br></pre></td></tr></table></figure>
|
||
<h3 id="配置yum源"><a href="#配置yum源" class="headerlink" title="配置yum源"></a>配置yum源</h3><ul>
|
||
<li>下载 repo 文件：<br><a href="http://mirrors.aliyun.com/repo/Centos-7.repo">Centos-7.repo</a><br>并上传到<code>/tmp</code>，进入到<code>/tmp</code></li>
|
||
<li>备份并且替换系统的repo文件<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"> <span class="built_in">cp</span> Centos-7.repo /etc/yum.repos.d/ </span><br><span class="line"><span class="built_in">cd</span> /etc/yum.repos.d/ </span><br><span class="line"><span class="built_in">mv</span> CentOS-Base.repo CentOS-Base.repo.bak </span><br><span class="line"><span class="built_in">mv</span> Centos-7.repo CentOS-Base.repo</span><br></pre></td></tr></table></figure></li>
|
||
<li>将<code>nn1</code>上的<code>CentOS-Base.repo</code>拷贝到其他主机<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@nn2:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@nn3:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@s1:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@s2:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@s3:/etc/yum.repos.d</span><br></pre></td></tr></table></figure></li>
|
||
<li>执行yum源更新命令<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">yum clean all</span><br><span class="line">yum makecache </span><br><span class="line">yum update -y </span><br></pre></td></tr></table></figure></li>
|
||
<li>安装常用软件 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl wget openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd ntp unrar bzip2</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="JDK安装"><a href="#JDK安装" class="headerlink" title="JDK安装"></a>JDK安装</h3><blockquote>
|
||
<p>注意需要在六台机器依次执行</p>
|
||
</blockquote>
|
||
<ul>
|
||
<li>上传到<code>/tmp</code>目录下，安装 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cd</span> /tmp</span><br><span class="line">rpm -ivh jdk-8u144-linux-x64.rpm</span><br></pre></td></tr></table></figure></li>
|
||
<li>配置环境变量 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">ln</span> -s /usr/java/jdk1.8.0_144/ /usr/java/jdk1.8</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export JAVA_HOME=/usr/java/jdk1.8'</span> >> /etc/profile.d/myEnv.sh </span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$JAVA_HOME/bin'</span> >> /etc/profile.d/myEnv.sh </span><br><span class="line"><span class="built_in">source</span> /etc/profile </span><br><span class="line">java -version</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="修改主机名和主机名映射"><a href="#修改主机名和主机名映射" class="headerlink" title="修改主机名和主机名映射"></a>修改主机名和主机名映射</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /etc/hostname</span><br></pre></td></tr></table></figure>
|
||
<p>6台机器分别为nn1、nn2、nn3、s1、s2、s3</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /etc/hosts</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>修改为</p>
|
||
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">192.168.1.30 nn1</span><br><span class="line">192.168.1.31 nn2</span><br><span class="line">192.168.1.32 nn3</span><br><span class="line">192.168.1.33 s1</span><br><span class="line">192.168.1.34 s2</span><br><span class="line">192.168.1.35 s3</span><br></pre></td></tr></table></figure>
|
||
<h3 id="创建hadoop用户"><a href="#创建hadoop用户" class="headerlink" title="创建hadoop用户"></a>创建hadoop用户</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#创建hadoop用户 </span></span><br><span class="line">useradd hadoop </span><br><span class="line"><span class="comment">#给hadoop用户设置密码: 12345678 </span></span><br><span class="line">passwd hadoop</span><br></pre></td></tr></table></figure>
|
||
<h3 id="禁止非-wheel-组用户切换到root，配置免密切换root"><a href="#禁止非-wheel-组用户切换到root，配置免密切换root" class="headerlink" title="禁止非 wheel 组用户切换到root，配置免密切换root"></a>禁止非 wheel 组用户切换到root，配置免密切换root</h3><ul>
|
||
<li>修改/etc/pam.d/su配置 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">sed -i <span class="string">'s/#auth\t\trequired\tpam_wheel.so/auth\t\trequired\tpam_wheel.so/g'</span> <span class="string">'/etc/pam.d/su'</span> </span><br><span class="line">sed -i <span class="string">'s/#auth\t\tsufficient\tpam_wheel.so/auth\t\tsufficient\tpam_wheel.so/g'</span> <span class="string">'/etc/pam.d/su'</span></span><br></pre></td></tr></table></figure></li>
|
||
<li>修改/etc/login.defs文件 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">"SU_WHEEL_ONLY yes"</span> >> /etc/login.defs</span><br></pre></td></tr></table></figure></li>
|
||
<li>添加用户到管理员，禁止普通用户su 到 root <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#把hadoop用户加到wheel组里</span></span><br><span class="line">gpasswd -a hadoop wheel</span><br><span class="line"><span class="comment">#查看wheel组里是否有hadoop用户</span></span><br><span class="line"><span class="built_in">cat</span> /etc/group | grep wheel</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="给hadoop用户，配置SSH密钥"><a href="#给hadoop用户，配置SSH密钥" class="headerlink" title="给hadoop用户，配置SSH密钥"></a>给hadoop用户，配置SSH密钥</h3><h4 id="配置hadoop用户ssh免密码登录到hadoop"><a href="#配置hadoop用户ssh免密码登录到hadoop" class="headerlink" title="配置hadoop用户ssh免密码登录到hadoop"></a>配置hadoop用户ssh免密码登录到hadoop</h4><ul>
|
||
<li>仅在<code>nn1</code>执行这段脚本命令即可<br>但是 <code>su - hadoop </code> ,<code> mkdir ~/.ssh</code> 需要在其他主机执行一下 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#切换到hadoop用户 </span></span><br><span class="line">su - hadoop</span><br><span class="line"><span class="comment">#生成ssh公私钥 </span></span><br><span class="line">ssh-keygen -t rsa -f ~/.ssh/id_rsa -P <span class="string">''</span></span><br><span class="line">ssh-copy-id nn1</span><br><span class="line">ssh-copy-id nn2</span><br><span class="line">ssh-copy-id nn3</span><br><span class="line">ssh-copy-id s1</span><br><span class="line">ssh-copy-id s2</span><br><span class="line">ssh-copy-id s3</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@nn2:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@nn3:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@s1:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@s2:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@s3:/home/hadoop/.ssh</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h4 id="配置hadoop用户ssh免密码登录到root"><a href="#配置hadoop用户ssh免密码登录到root" class="headerlink" title="配置hadoop用户ssh免密码登录到root"></a>配置hadoop用户ssh免密码登录到root</h4><ul>
|
||
<li>同上 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh-copy-id root@nn1</span><br><span class="line">ssh-copy-id root@ nn2</span><br><span class="line">ssh-copy-id root@nn3</span><br><span class="line">ssh-copy-id root@s1</span><br><span class="line">ssh-copy-id root@s2</span><br><span class="line">ssh-copy-id root@s3</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@nn2:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@nn3:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@s1:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@s2:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@s3:/root/.ssh</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="脚本配置"><a href="#脚本配置" class="headerlink" title="脚本配置"></a>脚本配置</h3><ul>
|
||
<li><p><strong>ips</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /home/hadoop/bin/ips</span><br></pre></td></tr></table></figure>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">nn1 </span><br><span class="line">nn2</span><br><span class="line">nn3</span><br><span class="line">s1 </span><br><span class="line">s2 </span><br><span class="line">s3</span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>ssh_all.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /home/hadoop/bin/ssh_all.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># 进入到当前脚本所在目录</span></span><br><span class="line"><span class="built_in">cd</span> `<span class="built_in">dirname</span> <span class="variable">$0</span>`</span><br><span class="line"><span class="comment"># 获取当前脚本所在目录</span></span><br><span class="line">dir_path=`<span class="built_in">pwd</span>`</span><br><span class="line"><span class="comment">#echo $dir_path</span></span><br><span class="line"><span class="comment"># 读ips文件得到数组(里面是一堆主机名)</span></span><br><span class="line">ip_arr=(`<span class="built_in">cat</span> <span class="variable">$dir_path</span>/ips`)</span><br><span class="line"><span class="comment"># 遍历数组里的主机名</span></span><br><span class="line"><span class="keyword">for</span> ip <span class="keyword">in</span> <span class="variable">${ip_arr[*]}</span></span><br><span class="line"><span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 拼接ssh命令: ssh hadoop@nn1.hadoop ls</span></span><br><span class="line"> cmd_=<span class="string">"ssh hadoop@<span class="variable">${ip}</span> \"$*\" "</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="variable">$cmd_</span></span><br><span class="line"> <span class="comment"># 通过eval命令 执行 拼接的ssh 命令</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">eval</span> <span class="variable">${cmd_}</span> ; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"OK"</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"FAIL"</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"><span class="keyword">done</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>ssh_root.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># 进入到当前脚本所在目录</span></span><br><span class="line"><span class="built_in">cd</span> `<span class="built_in">dirname</span> <span class="variable">$0</span>`</span><br><span class="line"><span class="comment"># 获取当前脚本所在目录</span></span><br><span class="line">dir_path=`<span class="built_in">pwd</span>`</span><br><span class="line"><span class="comment">#echo $dir_path</span></span><br><span class="line"><span class="comment"># 读ips文件得到数组(里面是一堆主机名)</span></span><br><span class="line">ip_arr=(`<span class="built_in">cat</span> <span class="variable">$dir_path</span>/ips`)</span><br><span class="line"><span class="comment"># 遍历数组里的主机名</span></span><br><span class="line"><span class="keyword">for</span> ip <span class="keyword">in</span> <span class="variable">${ip_arr[*]}</span></span><br><span class="line"><span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 拼接ssh命令: ssh hadoop@nn1.hadoop ls</span></span><br><span class="line"> cmd_=<span class="string">"ssh hadoop@<span class="variable">${ip}</span> ~/bin/exe.sh \"$*\""</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="variable">$cmd_</span></span><br><span class="line"> <span class="comment"># 通过eval命令 执行 拼接的ssh 命令</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">eval</span> <span class="variable">${cmd_}</span> ; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"OK"</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"FAIL"</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"><span class="keyword">done</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>scp_all.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># 进入到当前脚本所在目录</span></span><br><span class="line"><span class="built_in">cd</span> `<span class="built_in">dirname</span> <span class="variable">$0</span>`</span><br><span class="line"><span class="comment"># 获取当前脚本所在目录</span></span><br><span class="line">dir_path=`<span class="built_in">pwd</span>`</span><br><span class="line"><span class="comment">#echo $dir_path</span></span><br><span class="line"><span class="comment"># 读ips文件得到数组(里面是一堆主机名)</span></span><br><span class="line">ip_arr=(`<span class="built_in">cat</span> <span class="variable">$dir_path</span>/ips`)</span><br><span class="line"><span class="comment"># 源</span></span><br><span class="line">source_=<span class="variable">$1</span></span><br><span class="line"><span class="comment"># 目标</span></span><br><span class="line">target=<span class="variable">$2</span></span><br><span class="line"><span class="comment"># 遍历数组里的主机名</span></span><br><span class="line"><span class="keyword">for</span> ip <span class="keyword">in</span> <span class="variable">${ip_arr[*]}</span></span><br><span class="line"><span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 拼接scp命令: scp 源 hadoop@nn1.hadoop:目标</span></span><br><span class="line"> cmd_=<span class="string">"scp -r <span class="variable">${source_}</span> hadoop@<span class="variable">${ip}</span>:<span class="variable">${target}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="variable">$cmd_</span></span><br><span class="line"> <span class="comment"># 通过eval命令 执行 拼接的scp 命令</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">eval</span> <span class="variable">${cmd_}</span> ; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"OK"</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"FAIL"</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"><span class="keyword">done</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>exe.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#切换到root用户执行cmd命令</span></span><br><span class="line">cmd=$*</span><br><span class="line">su - << <span class="string">EOF</span></span><br><span class="line"><span class="string">$cmd</span></span><br><span class="line"><span class="string">EOF</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p>赋予执行权限</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">chmod</span> +x ssh_all.sh </span><br><span class="line"><span class="built_in">chmod</span> +x scp_all.sh</span><br><span class="line"><span class="built_in">chmod</span> +x ssh_root.sh</span><br><span class="line"><span class="built_in">chmod</span> +x exe.sh</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>分发到其他主机</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">./ssh_all.sh <span class="built_in">mkdir</span> /home/hadoop/bin</span><br><span class="line">./scp_all.sh /home/hadoop/bin/ips /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/exe.sh /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/ssh_all.sh /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/scp_all.sh /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/ssh_root.sh /home/hadoop/bin/</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>将 <code>/home/hadoop/bin</code>添加到hadoop用户的环境变量，需要切换到<code>hadoop</code>用户</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:/home/hadoop/bin'</span> >> ~/.bashrc && <span class="built_in">source</span> ~/.bashrc</span><br><span class="line">scp_all.sh /home/hadoop/.bashrc /home/hadoop/</span><br><span class="line">ssh_all.sh <span class="built_in">source</span> ~/.bashrc </span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>Hadoop集群HDFS配置</title>
|
||
<url>/posts/61252.html</url>
|
||
<content><![CDATA[<h3 id="上传hadoop-3-1-4-tar-gz到-tmp，解压"><a href="#上传hadoop-3-1-4-tar-gz到-tmp，解压" class="headerlink" title="上传hadoop-3.1.4.tar.gz到/tmp，解压"></a>上传<code>hadoop-3.1.4.tar.gz</code>到<code>/tmp</code>，解压</h3><blockquote>
|
||
<p>注意在六台机器均上传到<code>/tmp</code></p>
|
||
</blockquote>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 在6台机器执行</span></span><br><span class="line"><span class="built_in">sudo</span> tar -zxvf /tmp/hadoop-3.1.4.tar.gz -C /usr/local/</span><br><span class="line"><span class="comment"># 分发到其他主机</span></span><br><span class="line">ssh_root.sh <span class="built_in">chown</span> -R hadoop:hadoop /usr/local/hadoop-3.1.4</span><br><span class="line">ssh_root.sh <span class="built_in">ln</span> -s /usr/local/hadoop-3.1.4/ /usr/local/hadoop</span><br></pre></td></tr></table></figure>
|
||
<h3 id="配置环境变量"><a href="#配置环境变量" class="headerlink" title="配置环境变量"></a>配置环境变量</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">'export HADOOP_HOME=/usr/local/hadoop'</span> >> /etc/profile.d/myEnv.sh</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$HADOOP_HOME/bin'</span> >> /etc/profile.d/myEnv.sh</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$HADOOP_HOME/sbin'</span> >> /etc/profile.d/myEnv.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 分发到nn2、nn3、s1、s2、s3</span></span><br><span class="line">scp_all.sh /etc/profile.d/myEnv.sh /etc/profile.d/</span><br><span class="line"><span class="comment"># source 环境变量</span></span><br><span class="line">ssh_root.sh <span class="built_in">source</span> /etc/profile</span><br></pre></td></tr></table></figure>
|
||
<blockquote>
|
||
<p>还需要创建 <code>/data</code>这个目录，由于nn1、nn2、nn3已经创建<code>/data</code>，其他三台需要创建一下</p>
|
||
</blockquote>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">### 在s1、s2、s3执行</span></span><br><span class="line"><span class="built_in">sudo</span> <span class="built_in">mkdir</span> /data</span><br><span class="line"><span class="built_in">sudo</span> <span class="built_in">chown</span> -R hadoop:hadoop /data</span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="修改core-site-xml"><a href="#修改core-site-xml" class="headerlink" title="修改core-site.xml"></a>修改core-site.xml</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/core-site.xml </span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>fs.defaultFS<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hdfs://ns1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>默认文件服务的协议和NS逻辑名称，和hdfs-site.xml里的对应此配置替代了1.0里的fs.default.name<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.tmp.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/tmp<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>数据存储目录<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.proxyuser.root.groups<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hadoop<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span></span><br><span class="line"> hdfs dfsadmin –refreshSuperUserGroupsConfiguration,</span><br><span class="line"> yarn rmadmin –refreshSuperUserGroupsConfiguration</span><br><span class="line"> 使用这两个命令不用重启就能刷新</span><br><span class="line"> <span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.proxyuser.root.hosts<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>localhost<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>本地代理<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"><!-- zkfc的配置 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>ha.zookeeper.quorum<span class="tag"></<span class="name">name</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1:2181,nn2:2181,nn3:2181<span class="tag"></<span class="name">value</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>HA使用的zookeeper地址<span class="tag"></<span class="name">description</span>></span> </span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure>
|
||
<h3 id="修改hdfs-site-xml"><a href="#修改hdfs-site-xml" class="headerlink" title="修改hdfs-site.xml"></a>修改<code>hdfs-site.xml</code></h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml </span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.name.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/namenode<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>namenode本地文件存放地址<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.nameservices<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>ns1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>提供服务的NS逻辑名称，与core-site.xml里的对应<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"><!-- namenode的配置 --></span></span><br><span class="line"> <span class="comment"><!-- 主要的 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.namenodes.ns1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1,nn2,nn3<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>列出该逻辑名称下的NameNode逻辑名称<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.rpc-address.ns1.nn1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1:9000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的RPC位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.http-address.ns1.nn1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1:50070<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的Web Server位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.rpc-address.ns1.nn2<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn2:9000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的RPC位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.http-address.ns1.nn2<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn2:50070<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的Web Server位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.rpc-address.ns1.nn3<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn3:9000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的RPC位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.http-address.ns1.nn3<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn3:50070<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的Web Server位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.handler.count<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>77<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>namenode的工作线程数<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- journaldata配置,使得其他两个namenode同步第一个namenode数据 --></span> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.shared.edits.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>qjournal://nn1:8485;nn2:8485;nn3:8485/ns1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定用于HA存放edits的共享存储，通常是namenode的所在机器<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.journalnode.edits.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/journaldata/<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>journaldata服务存放文件的地址<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>ipc.client.connect.max.retries<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>10<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>namenode和journalnode的链接重试次数10次<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>ipc.client.connect.retry.interval<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>10000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>重试的间隔时间10s<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- zkfc的配置 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.fencing.methods<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>sshfence<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定HA做隔离的方法，缺省是ssh，可设为shell，稍后详述<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.fencing.ssh.private-key-files<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/home/hadoop/.ssh/id_rsa<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>杀死命令脚本的免密配置秘钥<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.client.failover.proxy.provider.ns1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定客户端用于HA切换的代理类，不同的NS可以用不同的代理类以上示例为Hadoop 2.0自带的缺省代理类<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.client.failover.proxy.provider.auto-ha<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span> </span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.automatic-failover.enabled<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>true<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="comment"><!-- datanode配置 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.datanode.data.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/datanode<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>datanode本地文件存放地址<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.replication<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>3<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>文件复本数<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.datanode.registration.ip-hostname-check<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>false<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.client.use.datanode.hostname<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>true<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.datanode.use.datanode.hostname<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>true<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure>
|
||
<h3 id="修改hadoop-env-sh"><a href="#修改hadoop-env-sh" class="headerlink" title="修改hadoop-env.sh"></a>修改<code>hadoop-env.sh</code></h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 添加这两行</span></span><br><span class="line"><span class="built_in">source</span> /etc/profile </span><br><span class="line"><span class="built_in">export</span> HADOOP_HEAPSIZE_MAX=512</span><br></pre></td></tr></table></figure>
|
||
<h3 id="分发这些配置文件"><a href="#分发这些配置文件" class="headerlink" title="分发这些配置文件"></a>分发这些配置文件</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/core-site.xml /usr/local/hadoop/etc/hadoop/</span><br><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/hdfs-site.xml /usr/local/hadoop/etc/hadoop/</span><br><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/hadoop-env.sh /usr/local/hadoop/etc/hadoop/</span><br></pre></td></tr></table></figure>
|
||
<h3 id="集群初始化"><a href="#集群初始化" class="headerlink" title="集群初始化"></a>集群初始化</h3><ul>
|
||
<li><p>需要先启动zookeeper集群</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh start</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 第一次启动先启动journalnode，便于3个namenode的元数据同步</span></span><br><span class="line">ssh_all_zookeeper.sh hadoop-daemon.sh start journalnode</span><br></pre></td></tr></table></figure></li>
|
||
<li><p><code>zkfc</code>搭建</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#在nn1节点执行 </span></span><br><span class="line">hdfs zkfc -formatZK</span><br><span class="line"><span class="comment">#nn1 nn2 nn3启动zkfc </span></span><br><span class="line">hadoop-daemon.sh start zkfc</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>初始化nn1的namenode，在nn1执行</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">hdfs namenode -format </span><br><span class="line">hadoop-daemon.sh start namenode</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>格式化第二台和第三台namenode，并且启动namenode,在nn2、nn3执行</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">hdfs namenode -bootstrapStandby </span><br><span class="line">hadoop-daemon.sh start namenode</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>修改<strong>workers</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/workers</span><br></pre></td></tr></table></figure>
|
||
<p> 修改为</p>
|
||
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">s1</span><br><span class="line">s2</span><br><span class="line">s3</span><br></pre></td></tr></table></figure>
|
||
<p> 分发给其他机器</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/workers /usr/local/hadoop/etc/hadoop</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>启动datanode节点，在s1、s2、s3执行</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#启动各个节点的datanode</span></span><br><span class="line">hadoop-daemons.sh start datanode</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="集群启动"><a href="#集群启动" class="headerlink" title="集群启动"></a>集群启动</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">start-dfs.sh</span><br></pre></td></tr></table></figure>]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>Docker被墙，如何继续使用？</title>
|
||
<url>/posts/47807.html</url>
|
||
<content><![CDATA[<h2 id="Docker-Download"><a href="#Docker-Download" class="headerlink" title="Docker Download"></a>Docker Download</h2><blockquote>
|
||
<p>自从docker官方仓库在中国大陆被墙后，docker的部署方式也发生了改变。<br>解决docker安装问题：<a href="https://github.com/shenjianZ/docker_installer">https://github.com/shenjianZ/docker_installer</a></p>
|
||
</blockquote>
|
||
<ol>
|
||
<li><p>安装docker </p>
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">sudo curl -fsSL https://gitee.com/tech-shrimp/docker_installer/releases/download/latest/linux.sh| bash -s docker --mirror Aliyun</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>启动docker</p>
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">sudo systemctl start docker</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>设置开机自启</p>
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">sudo systemctl enable docker</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>Docker pull images</p>
|
||
<blockquote>
|
||
<p>将image下载到阿里云镜像仓库中<br>解决docker pull 镜像问题:<a href="https://github.com/shenjianZ/docker_image_pusher">https://github.com/shenjianZ/docker_image_pusher</a></p>
|
||
</blockquote>
|
||
<ol>
|
||
<li><p><strong>登录阿里云镜像服务</strong> <a href="https://cr.console.aliyun.com/">https://cr.console.aliyun.com/</a>,</p>
|
||
<p>启用个人实例，创建一个命名空间（<code>ALIYUN_NAME_SPACE</code>）</p>
|
||
</li>
|
||
<li><p>在<strong>访问凭证</strong> 可以看到账号 用户名（<code>ALIYUN_REGISTRY_USER</code>)</p>
|
||
<p>密码（<code>ALIYUN_REGISTRY_PASSWORD</code>)</p>
|
||
<p>仓库地址（<code>ALIYUN_REGISTRY</code>）</p>
|
||
</li>
|
||
<li><p><strong>启动Action</strong><br>进入您自己的项目，点击<code>Action</code>，启用<code>Github Action</code>功能</p>
|
||
</li>
|
||
<li><p><strong>配置环境变量</strong><br>进入Settings->Secret and variables->Actions->New Repository secret<br> 将上一步的四个值<br> <code>ALIYUN_NAME_SPACE</code>,</p>
|
||
<p> <code> ALIYUN_REGISTRY_USER</code>，</p>
|
||
<p> <code>ALIYUN_REGISTRY_PASSWORD</code>，</p>
|
||
<p> <code>ALIYUN_REGISTRY</code><br> 配置成环境变量</p>
|
||
</li>
|
||
<li><p><strong>添加镜像</strong><br> 打开<code>images.txt</code>文件，添加你想要的镜像 可以加tag</p>
|
||
</li>
|
||
<li><p>使用镜像<br>回到阿里云，镜像仓库，点击任意镜像，可查看镜像状态。(可以改成公开，拉取镜像免登录)</p>
|
||
</li>
|
||
</ol>
|
||
</li>
|
||
</ol>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>组件使用</title>
|
||
<url>/posts/33957.html</url>
|
||
<content><![CDATA[<h3 id="组件自动导入"><a href="#组件自动导入" class="headerlink" title="组件自动导入"></a>组件自动导入</h3><figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="attr">"easycom"</span><span class="punctuation">:</span><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"autoscan"</span><span class="punctuation">:</span> <span class="literal"><span class="keyword">true</span></span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"custom"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"^tui-(.*)"</span><span class="punctuation">:</span> <span class="string">"@/components/thorui/tui-$1/tui-$1.vue"</span> <span class="comment">// 匹配components目录内的vue文件</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="tui-sticky-吸顶容器"><a href="#tui-sticky-吸顶容器" class="headerlink" title="tui-sticky 吸顶容器"></a><code>tui-sticky 吸顶容器</code></h3><blockquote>
|
||
<p>包含 以下 <code>tui</code> 组件 :</p>
|
||
<ul>
|
||
<li>tui-sticky</li>
|
||
<li>tui-list-view</li>
|
||
<li>tui-list-cell</li>
|
||
</ul>
|
||
</blockquote>
|
||
<figure class="highlight html"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">tui-sticky</span> <span class="attr">:scrollTop</span>=<span class="string">"scrollTop"</span> <span class="attr">stickyHeight</span>=<span class="string">"104rpx"</span> <span class="attr">container</span>></span></span><br><span class="line"> <span class="comment"><!-- header start --></span></span><br><span class="line"> <span class="tag"><<span class="name">template</span> <span class="attr">v-slot:header</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">view</span> <span class="attr">class</span>=<span class="string">"sticky-item"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">view</span> <span class="attr">class</span>=<span class="string">"setting"</span>></span>设置<span class="tag"></<span class="name">view</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">view</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">template</span>></span></span><br><span class="line"> <span class="comment"><!-- header end --></span></span><br><span class="line"> <span class="comment"><!--内容 start--></span></span><br><span class="line"> <span class="tag"><<span class="name">template</span> <span class="attr">v-slot:content</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">tui-list-view</span> <span class="attr">class</span>=<span class="string">"content"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">tui-list-cell</span> <span class="attr">:arrow</span>=<span class="string">"false"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">switch</span> <span class="attr">class</span>=<span class="string">'switch'</span> <span class="attr">checked</span> <span class="attr">color</span>=<span class="string">"#FFCC33"</span> /></span></span><br><span class="line"> <span class="tag"></<span class="name">tui-list-cell</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">tui-list-view</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">template</span>></span></span><br><span class="line"> <span class="comment"><!--内容 end--></span></span><br><span class="line"><span class="tag"></<span class="name">tui-sticky</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"><<span class="name">script</span> <span class="attr">setup</span>></span><span class="language-javascript"></span></span><br><span class="line"><span class="language-javascript"> <span class="keyword">import</span> { ref } <span class="keyword">from</span> <span class="string">'vue'</span></span></span><br><span class="line"><span class="language-javascript"> <span class="keyword">import</span> { onPageScroll } <span class="keyword">from</span> <span class="string">'@dcloudio/uni-app'</span></span></span><br><span class="line"><span class="language-javascript"></span></span><br><span class="line"><span class="language-javascript"> <span class="comment">// 定义 scrollTop 响应式变量</span></span></span><br><span class="line"><span class="language-javascript"> <span class="keyword">const</span> scrollTop = <span class="title function_">ref</span>(<span class="number">0</span>)</span></span><br><span class="line"><span class="language-javascript"> <span class="comment">// 监听页面滚动事件</span></span></span><br><span class="line"><span class="language-javascript"> <span class="title function_">onPageScroll</span>(<span class="function">(<span class="params">e</span>) =></span> {</span></span><br><span class="line"><span class="language-javascript"> scrollTop.<span class="property">value</span> = e.<span class="property">scrollTop</span></span></span><br><span class="line"><span class="language-javascript"> })</span></span><br><span class="line"><span class="language-javascript"></span><span class="tag"></<span class="name">script</span>></span></span><br></pre></td></tr></table></figure>
|
||
]]></content>
|
||
<tags>
|
||
<tag>uniapp</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>Hadoop集群Zookeeper配置</title>
|
||
<url>/posts/61251.html</url>
|
||
<content><![CDATA[<h3 id="Zookeeper脚本配置"><a href="#Zookeeper脚本配置" class="headerlink" title="Zookeeper脚本配置"></a>Zookeeper脚本配置</h3><ul>
|
||
<li>拷贝ips<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cp</span> ips ips_zookeeper</span><br></pre></td></tr></table></figure>
|
||
修改为<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">nn1</span><br><span class="line">nn2</span><br><span class="line">nn3</span><br></pre></td></tr></table></figure></li>
|
||
<li>拷贝三个脚本<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cp</span> scp_all.sh scp_all_zookeeper.sh </span><br><span class="line"><span class="built_in">cp</span> ssh_all.sh ssh_all_zookeeper.sh </span><br><span class="line"><span class="built_in">cp</span> ssh_root.sh ssh_root_zookeeper.sh</span><br></pre></td></tr></table></figure></li>
|
||
<li>修改脚本<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">vim scp_all_zookeeper.sh </span><br><span class="line">vim ssh_all_zookeeper.sh </span><br><span class="line">vim ssh_root_zookeeper.sh </span><br></pre></td></tr></table></figure>
|
||
|
||
将三个脚本中的ips改为ips_zookeeper</li>
|
||
</ul>
|
||
<h3 id="Zookeeper安装"><a href="#Zookeeper安装" class="headerlink" title="Zookeeper安装"></a>Zookeeper安装</h3><ul>
|
||
<li>上传到<code>/tmp</code>目录下，解压 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">sudo</span> tar -zxvf /tmp/zookeeper-3.4.8.tar.gz -C /usr/local/</span><br><span class="line">scp -r /usr/local/zookeeper-3.4.8/ root@nn2:/usr/local/</span><br><span class="line">scp -r /usr/local/zookeeper-3.4.8/ root@nn3:/usr/local/</span><br><span class="line">ssh_root_zookeeper.sh <span class="built_in">chown</span> -R hadoop:hadoop /usr/local/zookeeper-3.4.8</span><br><span class="line">ssh_root_zookeeper.s <span class="built_in">ln</span> -s /usr/local/zookeeper-3.4.8/ /usr/local/zookeeper</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="Zookeeper配置"><a href="#Zookeeper配置" class="headerlink" title="Zookeeper配置"></a>Zookeeper配置</h3><ul>
|
||
<li><p>zoo.cfg配置</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cd</span> /usr/local/zookeeper/conf/</span><br><span class="line"><span class="built_in">cp</span> zoo_sample.cfg zoo.cfg</span><br></pre></td></tr></table></figure>
|
||
<p>然后<code>vim zoo.cfg</code>,修改如下：</p>
|
||
<figure class="highlight properties"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 修改dataDir</span></span><br><span class="line"><span class="attr">dataDir</span>=<span class="string">/data/zookeeper</span></span><br><span class="line"><span class="comment"># 添加一下内容</span></span><br><span class="line"><span class="attr">server.1</span>=<span class="string">nn1:2888:3888 </span></span><br><span class="line"><span class="attr">server.2</span>=<span class="string">nn2:2888:3888 </span></span><br><span class="line"><span class="attr">server.3</span>=<span class="string">nn3:2888:3888</span></span><br></pre></td></tr></table></figure>
|
||
<p>分发给nn2、nn3<br> <code>scp_all_zookeeper.sh /usr/local/zookeeper/conf/zoo.cfg /usr/local/zookeeper/conf/</code></p>
|
||
</li>
|
||
<li><p><code>zkEnv.sh</code>配置<br><code>vim /usr/local/zookeeper/bin/zkEnv.sh</code><br><img src="/../../../img/000001.png" alt="000001.png"><br>分发到nn2、nn3</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp_all_zookeeper.sh /usr/local/zookeeper/bin/zkEnv.sh /usr/local/zookeeper/bin/</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>创建zookeeper数据目录</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh_root_zookeeper.sh <span class="built_in">mkdir</span> -p /data/zookeeper</span><br><span class="line">ssh_root_zookeeper.sh <span class="built_in">chown</span> -R hadoop:hadoop /data</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>创建myid文件</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh nn1 <span class="string">'echo "1" > /data/zookeeper/myid'</span></span><br><span class="line">ssh nn2 <span class="string">'echo "2" > /data/zookeeper/myid'</span></span><br><span class="line">ssh nn3 <span class="string">'echo "3" > /data/zookeeper/myid'</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p>配置Zookeeper环境变量</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 在其他所有主机也执行</span></span><br><span class="line"><span class="built_in">sudo</span> <span class="built_in">chown</span> -R hadoop:hadoop /etc/profile.d/myEnv.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">'export ZOOKEEPER_HOME=/usr/local/zookeeper'</span> >> /etc/profile.d/myEnv.sh</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$ZOOKEEPER_HOME/bin'</span> >> /etc/profile.d/myEnv.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 分发到nn2、nn3</span></span><br><span class="line">scp_all_zookeeper.sh /etc/profile.d/myEnv.sh /etc/profile.d/</span><br><span class="line"><span class="comment"># source 环境变量</span></span><br><span class="line">ssh_all_zookeeper.sh <span class="built_in">source</span> /etc/profile</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="Zookeeper的命令"><a href="#Zookeeper的命令" class="headerlink" title="Zookeeper的命令"></a>Zookeeper的命令</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#启动zk服务</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh start</span><br><span class="line"><span class="comment">#查看每个机器ZK运行的状态</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh status</span><br><span class="line"><span class="comment">#整体停止服务</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh stop </span><br><span class="line"><span class="comment">#重启zk服务</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh restart</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line"><span class="meta prompt_">#</span><span class="language-bash">启动zookeeper客户端，并连接zookeeper集群</span></span><br><span class="line">/usr/local/zookeeper/bin/zkCli.sh -server nn1:2181</span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">可以简化为:</span></span><br><span class="line">zkCli.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>uniapp 开发</title>
|
||
<url>/posts/58817.html</url>
|
||
<content><![CDATA[<ul>
|
||
<li><a href="../component1">uniapp component</a></li>
|
||
</ul>
|
||
]]></content>
|
||
<tags>
|
||
<tag>uniapp</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>无法访问外网？需要订阅代理服务？</title>
|
||
<url>/posts/14011.html</url>
|
||
<content><![CDATA[<div class="note info flat"><p><strong>由于中国大陆的GFW（防火墙）限制，无法访问外网网络，因此需要访问像GitHub、YouTube这样的<br>的网站将被屏蔽拦截，接下来我将给出一种使用<code>VPN</code>服务的可行的方案来保证服务的可靠性。</strong></p>
|
||
</div>
|
||
|
||
<h3 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h3><blockquote>
|
||
<p>根据测试，许多提供服务的提供商所在的网站需要使用<code>外部网络</code>才能打开，仅有少部分的网站(<strong>比较贵</strong>)可以直接使用<br>国内网络环境打开直接购买订阅服务。</p>
|
||
</blockquote>
|
||
<p>那么你现在可以有两个选择：</p>
|
||
<ol>
|
||
<li><strong>方案一</strong>：使用无需<code>外部网络</code>便能开通订阅服务的VPN，费用高，如果你选择此方案，那么你可自行搜索解决，此处仅仅讨论方案二。</li>
|
||
<li><strong>方案二</strong>：如果使用此方案，详见下方。</li>
|
||
</ol>
|
||
<h3 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h3><blockquote>
|
||
<p>采用<strong>方案二</strong>方式</p>
|
||
<p>这是一些订阅服务推广的链接： <a href="https://9.234456.xyz/abc.html?t=638586217737356738">https://9.234456.xyz/abc.html?t=638586217737356738</a> (此链接打开无需使用VPN，但进入对应的机场页面却仍无法打开)</p>
|
||
<p>此教程中我们使用的机场是 </p>
|
||
<ol>
|
||
<li><code>一元机场</code>: <a href="https://一元机场.com/">https://xn--4gq62f52gdss.com/</a></li>
|
||
<li><code>赔钱机场</code>：<a href="https://赔钱机场.com/">https://xn--mes358aby2apfg.com/</a></li>
|
||
</ol>
|
||
</blockquote>
|
||
<h3 id="机场选择的建议："><a href="#机场选择的建议：" class="headerlink" title="机场选择的建议："></a>机场选择的建议：</h3><ul>
|
||
<li><p><code>一元机场</code><br><img src="/img/yiyuan.png"><br>可以看到</p>
|
||
<ul>
|
||
<li><code>12元/年</code>,每月50GB的可用额度，下个月重置流量额度</li>
|
||
<li><code>15元/季</code>，即为<code>60元/年</code>，每月有4000GB的可用额度，下个月重置流量额度</li>
|
||
<li><code>7元/月</code>，即为<code>84元/年</code>，每个月8000GB的可用额度，下个月重置流量额度<br>根据我个人的使用情况，大多数情况下我每月的流量使用未超过50GB，如果没有频繁的流量使用，<br>建议选择<code>12元/年</code>，否则可以选择<code>15元/季</code>，这意味着每月将有4000GB的可用额度</li>
|
||
</ul>
|
||
</li>
|
||
<li><p><code>赔钱机场</code></p>
|
||
<p><img src="/img/peiqian.png"><br><code>赔钱机场</code>的订阅共有9种方案，这里我仅显示自己正在使用的，个人认为十分优惠：</p>
|
||
<ul>
|
||
<li><code>34.99元/年</code>,每月500GB的可用额度，根据我观察和使用，这个订阅方案比<code>一元机场</code>的性价比更高，且流量使用额度也不用担心</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h3 id="如何订阅？"><a href="#如何订阅？" class="headerlink" title="如何订阅？"></a>如何订阅？</h3><div class="note success flat"><p>由于需要外部网络才能完成订阅服务的购买，你可以向我的邮箱<code>15202078626@163.com</code>发送你的订阅计划方案，<br>扫描付款二维码，我将为你开通订阅（您只需要付款对应的订阅金额即可）</p>
|
||
</div>
|
||
<img src="/img/dingyue.png" width='150px'>
|
||
|
||
<h3 id="完成订阅后如何使用？"><a href="#完成订阅后如何使用？" class="headerlink" title="完成订阅后如何使用？"></a>完成订阅后如何使用？</h3><blockquote>
|
||
<p>你可以在 <code>Windows</code>、<code>Mac</code>、<code>Android</code>等平台使用此服务<br>使用订阅的对应链接： <a href="https://flowus.cn/shenjian/22f76d4f-e7b3-4b8a-8a89-561566f6eb60">https://flowus.cn/shenjian/22f76d4f-e7b3-4b8a-8a89-561566f6eb60</a></p>
|
||
</blockquote>
|
||
]]></content>
|
||
<tags>
|
||
<tag>网络代理</tag>
|
||
</tags>
|
||
</entry>
|
||
</search>
|