391 lines
96 KiB
XML
391 lines
96 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
||
<search>
|
||
<entry>
|
||
<title>script</title>
|
||
<url>/posts/34849.html</url>
|
||
<content><![CDATA[<h3 id="查看CPU、内存使用率"><a href="#查看CPU、内存使用率" class="headerlink" title="查看CPU、内存使用率"></a>查看CPU、内存使用率</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 定义颜色</span></span><br><span class="line">RED=<span class="string">'\033[0;31m'</span></span><br><span class="line">GREEN=<span class="string">'\033[0;32m'</span></span><br><span class="line">YELLOW=<span class="string">'\033[0;33m'</span></span><br><span class="line">BLUE=<span class="string">'\033[0;34m'</span></span><br><span class="line">NC=<span class="string">'\033[0m'</span> <span class="comment"># 无颜色</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span> <span class="literal">true</span>; <span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 获取所有进程的CPU使用率和内存使用率</span></span><br><span class="line"> cpu_usage=$(ps aux | awk <span class="string">'{sum_cpu += $3} END {print sum_cpu}'</span>)</span><br><span class="line"> mem_usage=$(ps aux | awk <span class="string">'{sum_mem += $4} END {print sum_mem}'</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 打印结果，带有时间戳、分隔线和颜色高亮</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${BLUE}</span>==============================<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${YELLOW}</span>Timestamp: <span class="subst">$(date)</span><span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${BLUE}</span>==============================<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${GREEN}</span>Total CPU usage: <span class="variable">${RED}</span><span class="variable">$cpu_usage</span>%<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${GREEN}</span>Total Memory usage: <span class="variable">${RED}</span><span class="variable">$mem_usage</span>%<span class="variable">${NC}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> -e <span class="string">"<span class="variable">${BLUE}</span>==============================<span class="variable">${NC}</span>"</span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"># 等待5秒后再次执行</span></span><br><span class="line"> <span class="built_in">sleep</span> 5</span><br><span class="line"><span class="keyword">done</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
|
||
<p><strong>保存脚本到/usr/local/bin目录下</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">mv</span> usage.sh /usr/local/bin/usage</span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="Shell脚本编写的基本信息"><a href="#Shell脚本编写的基本信息" class="headerlink" title="Shell脚本编写的基本信息"></a>Shell脚本编写的基本信息</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># -------------------------------------------------</span></span><br><span class="line"><span class="comment"># Filename: test.sh</span></span><br><span class="line"><span class="comment"># Version: 1.0</span></span><br><span class="line"><span class="comment"># Date: 2024/05/02</span></span><br><span class="line"><span class="comment"># Author: shenjianZ</span></span><br><span class="line"><span class="comment"># Email: shenjianZLT@gmail.com</span></span><br><span class="line"><span class="comment"># Website: https://blog.shenjianl.cn</span></span><br><span class="line"><span class="comment"># Description: this is a test shell</span></span><br><span class="line"><span class="comment"># CopyRight: 2024 All rights reserved shenjianZ</span></span><br><span class="line"><span class="comment"># License GPL</span></span><br><span class="line"><span class="comment"># ------------------------------------------------</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># Your script logic goes here</span></span><br></pre></td></tr></table></figure>]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>Hello World</title>
|
||
<url>/posts/16107.html</url>
|
||
<content><![CDATA[<p>Welcome to <a href="https://hexo.io/">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues">GitHub</a>.</p>
|
||
<h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo new <span class="string">"My New Post"</span></span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/writing.html">Writing</a></p>
|
||
<h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo server</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/server.html">Server</a></p>
|
||
<h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo generate</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/generating.html">Generating</a></p>
|
||
<h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ hexo deploy</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>More info: <a href="https://hexo.io/docs/one-command-deployment.html">Deployment</a></p>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>k近邻算法（K-Nearest Neighbors）KNN</title>
|
||
<url>/posts/29139.html</url>
|
||
<content><![CDATA[<h2 id="k近邻算法（K-Nearest-Neighbors）KNN"><a href="#k近邻算法（K-Nearest-Neighbors）KNN" class="headerlink" title="k近邻算法（K-Nearest Neighbors）KNN"></a><strong>k近邻算法（K-Nearest Neighbors）KNN</strong></h2><p>将当前样本的类别归类于距离最近的<strong>k</strong>个样本的类别</p>
|
||
<h4 id="距离公式-2维"><a href="#距离公式-2维" class="headerlink" title="距离公式(2维)"></a><strong>距离公式(2维)</strong></h4><ul>
|
||
<li>欧式距离<br>$$<br>d = \sqrt{(x_1-y_1)^2 + (x_2 - y_2)^2}<br>$$</li>
|
||
<li>曼哈顿距离<br>$$<br>d = |x_1 - x_2| + |y_1 - y_2|<br>$$</li>
|
||
<li>切比雪夫距离<br>$$<br>d = \max\left(|x_1 - x_2|, |y_1 - y_2|\right)<br>$$</li>
|
||
</ul>
|
||
<h4 id="k值选择问题"><a href="#k值选择问题" class="headerlink" title="k值选择问题"></a>k值选择问题</h4><table>
|
||
<thead>
|
||
<tr>
|
||
<th>k值</th>
|
||
<th>影响</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody><tr>
|
||
<td>越大</td>
|
||
<td>模型过拟合，准确率波动较大</td>
|
||
</tr>
|
||
<tr>
|
||
<td>越小</td>
|
||
<td>模型欠拟合，准确率趋于稳定但可能较低</td>
|
||
</tr>
|
||
</tbody></table>
|
||
<h3 id="特征预处理"><a href="#特征预处理" class="headerlink" title="特征预处理"></a>特征预处理</h3><blockquote>
|
||
<p>通过一些转换函数将特征数据转换成更加适合算法模型的特征数据过程 </p>
|
||
</blockquote>
|
||
<ul>
|
||
<li><p>归一化<br>将数据变换到指定区间（默认是[0,1]）<br>$$ x’ = \frac{x- x_{\text {min}}}{x_{\text{max}} - x_{\text{min}}} $$<br>若需要缩放到任意区间 ([a, b])，公式为： $$ x’ = a + \frac{(x - x_{\text{min}}) \cdot (b - a)}{x_{\text{max}} - x_{\text{min}}} $$<br>其中：( [a, b] )：目标区间的范围<br>归一化受到数据集的异常值的影响，需要进行标准化处理（更加合理）</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> MinMaxScaler <span class="comment"># 归一化</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p>标准化<br>将数据调整为均值为 0，标准差为 1 的标准正态分布<br>$$ z = \frac{x - \mu}{\sigma} $$<br>( z )：标准化后的值 ( x )：原始数据值 ( $\mu$ )：数据的均值 ( $\sigma$)：数据的标准差</p>
|
||
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> StandardScaler <span class="comment"># 标准化</span></span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="KNN代码实现"><a href="#KNN代码实现" class="headerlink" title="KNN代码实现"></a>KNN代码实现</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> seaborn <span class="keyword">as</span> sns</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt </span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split</span><br><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> load_iris</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> MinMaxScaler,StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.neighbors <span class="keyword">import</span> KNeighborsClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> accuracy_score</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1 数据集获取</span></span><br><span class="line">iris = load_iris()</span><br><span class="line"><span class="comment"># print(iris.feature_names)</span></span><br><span class="line">iris_data = pd.DataFrame(iris.data,columns=[<span class="string">'Sepal_Length'</span>, <span class="string">'Sepal_Width'</span>, <span class="string">'Petal_Length'</span>, <span class="string">'Petal_Width'</span>])</span><br><span class="line">iris_data[<span class="string">'target'</span>] = iris.target</span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">iris_plot</span>(<span class="params">data,col1,col2</span>):</span><br><span class="line"> sns.lmplot(x=col1,y=col2,data=data,hue=<span class="string">"target"</span>,fit_reg=<span class="literal">False</span>)</span><br><span class="line"> plt.show()</span><br><span class="line"><span class="comment"># 2 数据集可视化</span></span><br><span class="line"><span class="comment"># iris_plot(iris_data, 'Sepal_Width', 'Petal_Length')</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 3 数据集的划分</span></span><br><span class="line">X_train,X_test,y_train,y_test = train_test_split(iris.data,iris.target,test_size=<span class="number">0.2</span>,random_state=<span class="number">44</span>)</span><br><span class="line"><span class="comment"># print("训练集的特征值：\n",X_train)</span></span><br><span class="line"><span class="comment"># print("训练集的目标值：\n",y_train)</span></span><br><span class="line"><span class="comment"># print("测试集的特征值：\n",X_test)</span></span><br><span class="line"><span class="comment"># print("测试集的特征值：\n",y_test)</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 4 归一化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"><span class="comment"># print("归一化的,X_train：\n",X_train)</span></span><br><span class="line"><span class="comment"># print("归一化的X_test：\n",X_test)</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 5 机器学习 KNN</span></span><br><span class="line"><span class="comment"># 5.1 实例化估计器</span></span><br><span class="line">estimator = KNeighborsClassifier(n_neighbors=<span class="number">9</span>)</span><br><span class="line"><span class="comment"># 5.2 进行训练</span></span><br><span class="line">estimator.fit(X_train,y_train)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 6 模型评估</span></span><br><span class="line">y_pred = estimator.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值：\n"</span>,y_pre)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值与真实值是否相等：\n"</span>,y_pred==y_test)</span><br><span class="line">accuracy = accuracy_score(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"\nKNN 模型的准确率: <span class="subst">{accuracy:<span class="number">.4</span>f}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
|
||
<p><img src="/img/machinelearning/knn-01.png"></p>
|
||
<h3 id="交叉验证与网格搜索"><a href="#交叉验证与网格搜索" class="headerlink" title="交叉验证与网格搜索"></a>交叉验证与网格搜索</h3><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> seaborn <span class="keyword">as</span> sns</span><br><span class="line"><span class="keyword">import</span> matplotlib.pyplot <span class="keyword">as</span> plt </span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> train_test_split,GridSearchCV</span><br><span class="line"><span class="keyword">from</span> sklearn.datasets <span class="keyword">import</span> load_iris</span><br><span class="line"><span class="keyword">from</span> sklearn.preprocessing <span class="keyword">import</span> MinMaxScaler,StandardScaler</span><br><span class="line"><span class="keyword">from</span> sklearn.neighbors <span class="keyword">import</span> KNeighborsClassifier</span><br><span class="line"><span class="keyword">from</span> sklearn.metrics <span class="keyword">import</span> accuracy_score</span><br><span class="line"></span><br><span class="line"><span class="comment"># 1 数据集获取</span></span><br><span class="line">iris = load_iris()</span><br><span class="line">iris_data = pd.DataFrame(iris.data,columns=[<span class="string">'Sepal_Length'</span>, <span class="string">'Sepal_Width'</span>, <span class="string">'Petal_Length'</span>, <span class="string">'Petal_Width'</span>])</span><br><span class="line">iris_data[<span class="string">'target'</span>] = iris.target</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3 数据集的划分</span></span><br><span class="line">X_train,X_test,y_train,y_test = train_test_split(iris.data,iris.target,test_size=<span class="number">0.2</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 4 归一化</span></span><br><span class="line">transfer = StandardScaler()</span><br><span class="line">X_train = transfer.fit_transform(X_train)</span><br><span class="line">X_test = transfer.transform(X_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 5 机器学习 KNN</span></span><br><span class="line"><span class="comment"># 5.1 实例化估计器</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment">#不指定 <code> n_neighbors </code> ,使用网格搜索进行循环训练</span></span><br><span class="line">estimator = KNeighborsClassifier()</span><br><span class="line"><span class="comment"># 5.2 模型调优 -- 交叉验证，网格搜素</span></span><br><span class="line">estimator = GridSearchCV(estimator,param_grid={<span class="string">"n_neighbors"</span>:[<span class="number">1</span>,<span class="number">3</span>,<span class="number">5</span>,<span class="number">7</span>]},cv=<span class="number">5</span>) <span class="comment"># 5 折</span></span><br><span class="line"><span class="comment"># 5.2 进行训练</span></span><br><span class="line">estimator.fit(X_train,y_train)</span><br><span class="line"> </span><br><span class="line"><span class="comment"># 6 模型评估</span></span><br><span class="line">y_pred = estimator.predict(X_test)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值：\n"</span>,y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"预测值与真实值是否相等：\n"</span>,y_pred==y_test)</span><br><span class="line">accuracy = accuracy_score(y_test, y_pred)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"\nKNN 模型的准确率: <span class="subst">{accuracy:<span class="number">.4</span>f}</span>"</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 交叉验证的相关参数</span></span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最好结果：<span class="subst">{estimator.best_score_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最好模型：<span class="subst">{estimator.best_estimator_}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">f"最好模型结果：<span class="subst">{estimator.cv_results_}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
<p><img src="/img/machinelearning/cros-valid.png"></p>
|
||
<h3 id="机器学习的基本步骤"><a href="#机器学习的基本步骤" class="headerlink" title="机器学习的基本步骤"></a>机器学习的基本步骤</h3><ul>
|
||
<li>获取数据集</li>
|
||
<li>数据集基本处理<ul>
|
||
<li>去重去空、填充等操作 </li>
|
||
<li>确定特征值和目标值</li>
|
||
<li>分割数据集</li>
|
||
</ul>
|
||
</li>
|
||
<li>特征工程（特征预处理 标准化等）</li>
|
||
<li>机器学习</li>
|
||
<li>模型评估</li>
|
||
</ul>
|
||
<h3 id="数据分割的方法"><a href="#数据分割的方法" class="headerlink" title="数据分割的方法"></a>数据分割的方法</h3><ul>
|
||
<li>留出法<br>训练/测试集的划分要尽可能保持数据分布的一致性，避免因数据划分过程引入额外的偏差而对最终结果产生影响。<br>单次使用留出法得到的估计结果往往不够稳定可靠，在使用留出法时，一般要采用若干次随机划分、重复进行实验评估后取平均值作为留出法的评估结果。<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sklearn.model_selection <span class="keyword">import</span> KFold,StratifiedKFold</span><br><span class="line"><span class="keyword">import</span> pandas <span class="keyword">as</span> pd</span><br><span class="line">X = np.array([</span><br><span class="line">[<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>],</span><br><span class="line">[<span class="number">11</span>,<span class="number">12</span>,<span class="number">13</span>,<span class="number">14</span>],</span><br><span class="line">[<span class="number">21</span>,<span class="number">22</span>,<span class="number">23</span>,<span class="number">24</span>],</span><br><span class="line">[<span class="number">31</span>,<span class="number">32</span>,<span class="number">33</span>,<span class="number">34</span>],</span><br><span class="line">[<span class="number">41</span>,<span class="number">42</span>,<span class="number">43</span>,<span class="number">44</span>],</span><br><span class="line">[<span class="number">51</span>,<span class="number">52</span>,<span class="number">53</span>,<span class="number">54</span>],</span><br><span class="line">[<span class="number">61</span>,<span class="number">62</span>,<span class="number">63</span>,<span class="number">64</span>],</span><br><span class="line">[<span class="number">71</span>,<span class="number">72</span>,<span class="number">73</span>,<span class="number">74</span>]</span><br><span class="line">])</span><br><span class="line">y=np.array([<span class="number">1</span>,<span class="number">1</span>,<span class="number">0</span>,<span class="number">0</span>,<span class="number">1</span>,<span class="number">1</span>,<span class="number">0</span>,<span class="number">0</span>])</span><br><span class="line">folder = KFold(n_splits=<span class="number">4</span>)</span><br><span class="line">sfloder = StratifiedKFold(n_splits=<span class="number">4</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"KFOLD:"</span>)</span><br><span class="line"><span class="keyword">for</span> train,test <span class="keyword">in</span> folder.split(X,y):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"train:<span class="subst">{train}</span>,test:<span class="subst">{test}</span>"</span>)</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"SKFOLD:"</span>)</span><br><span class="line"><span class="keyword">for</span> train,test <span class="keyword">in</span> sfloder.split(X,y):</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"train:<span class="subst">{train}</span>,test:<span class="subst">{test}</span>"</span>)</span><br></pre></td></tr></table></figure>
|
||
<img src="/img/machinelearning/kfold-skfold.png"></li>
|
||
<li>自助法</li>
|
||
<li>交叉验证法</li>
|
||
</ul>
|
||
]]></content>
|
||
<tags>
|
||
<tag>machinelearning</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>page</title>
|
||
<url>/posts/1441.html</url>
|
||
<content><![CDATA[<ul>
|
||
<li><a href="./deploy">deploy</a></li>
|
||
<li></li>
|
||
</ul>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>网络相关</title>
|
||
<url>/posts/41168.html</url>
|
||
<content><![CDATA[]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>uniapp 开发</title>
|
||
<url>/posts/58817.html</url>
|
||
<content><![CDATA[<ul>
|
||
<li><a href="../component1">uniapp component</a></li>
|
||
</ul>
|
||
]]></content>
|
||
<tags>
|
||
<tag>uniapp</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>郑伯克段于鄢</title>
|
||
<url>/posts/58638.html</url>
|
||
<content><![CDATA[<p>原文如下：</p>
|
||
<p> 初，郑武公娶于申【申国】，曰武姜【武为武公谥号，姜为其宗族之性】。生庄公及共叔段【共表示其曾出逃到共，叔为老三，段为名】。庄公寤生【出生时头先出，难产】，惊姜氏，故名曰“寤生”， 遂恶之，爱【喜爱】共叔段，欲立【立为储君】之，亟（qì）【多次】请于武公，及庄公即位，为之【共叔段】请制【一个叫制的封地，虎牢关所在】。公曰：“制，岩邑【险要的城邑】也，虢叔死焉，佗【通“他”，其他】邑唯命（是听）。”请京，使居之，谓之“京城大叔”。</p>
|
||
]]></content>
|
||
<categories>
|
||
<category>古文观止</category>
|
||
</categories>
|
||
<tags>
|
||
<tag>古文观止</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>组件使用</title>
|
||
<url>/posts/33957.html</url>
|
||
<content><![CDATA[<h3 id="组件自动导入"><a href="#组件自动导入" class="headerlink" title="组件自动导入"></a>组件自动导入</h3><figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="attr">"easycom"</span><span class="punctuation">:</span><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"autoscan"</span><span class="punctuation">:</span> <span class="literal"><span class="keyword">true</span></span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"custom"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"^tui-(.*)"</span><span class="punctuation">:</span> <span class="string">"@/components/thorui/tui-$1/tui-$1.vue"</span> <span class="comment">// 匹配components目录内的vue文件</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="tui-sticky-吸顶容器"><a href="#tui-sticky-吸顶容器" class="headerlink" title="tui-sticky 吸顶容器"></a><code>tui-sticky 吸顶容器</code></h3><blockquote>
|
||
<p>包含 以下 <code>tui</code> 组件 :</p>
|
||
<ul>
|
||
<li>tui-sticky</li>
|
||
<li>tui-list-view</li>
|
||
<li>tui-list-cell</li>
|
||
</ul>
|
||
</blockquote>
|
||
<figure class="highlight html"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">tui-sticky</span> <span class="attr">:scrollTop</span>=<span class="string">"scrollTop"</span> <span class="attr">stickyHeight</span>=<span class="string">"104rpx"</span> <span class="attr">container</span>></span></span><br><span class="line"> <span class="comment"><!-- header start --></span></span><br><span class="line"> <span class="tag"><<span class="name">template</span> <span class="attr">v-slot:header</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">view</span> <span class="attr">class</span>=<span class="string">"sticky-item"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">view</span> <span class="attr">class</span>=<span class="string">"setting"</span>></span>设置<span class="tag"></<span class="name">view</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">view</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">template</span>></span></span><br><span class="line"> <span class="comment"><!-- header end --></span></span><br><span class="line"> <span class="comment"><!--内容 start--></span></span><br><span class="line"> <span class="tag"><<span class="name">template</span> <span class="attr">v-slot:content</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">tui-list-view</span> <span class="attr">class</span>=<span class="string">"content"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">tui-list-cell</span> <span class="attr">:arrow</span>=<span class="string">"false"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">switch</span> <span class="attr">class</span>=<span class="string">'switch'</span> <span class="attr">checked</span> <span class="attr">color</span>=<span class="string">"#FFCC33"</span> /></span></span><br><span class="line"> <span class="tag"></<span class="name">tui-list-cell</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">tui-list-view</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">template</span>></span></span><br><span class="line"> <span class="comment"><!--内容 end--></span></span><br><span class="line"><span class="tag"></<span class="name">tui-sticky</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"><<span class="name">script</span> <span class="attr">setup</span>></span><span class="language-javascript"></span></span><br><span class="line"><span class="language-javascript"> <span class="keyword">import</span> { ref } <span class="keyword">from</span> <span class="string">'vue'</span></span></span><br><span class="line"><span class="language-javascript"> <span class="keyword">import</span> { onPageScroll } <span class="keyword">from</span> <span class="string">'@dcloudio/uni-app'</span></span></span><br><span class="line"><span class="language-javascript"></span></span><br><span class="line"><span class="language-javascript"> <span class="comment">// 定义 scrollTop 响应式变量</span></span></span><br><span class="line"><span class="language-javascript"> <span class="keyword">const</span> scrollTop = <span class="title function_">ref</span>(<span class="number">0</span>)</span></span><br><span class="line"><span class="language-javascript"> <span class="comment">// 监听页面滚动事件</span></span></span><br><span class="line"><span class="language-javascript"> <span class="title function_">onPageScroll</span>(<span class="function">(<span class="params">e</span>) =></span> {</span></span><br><span class="line"><span class="language-javascript"> scrollTop.<span class="property">value</span> = e.<span class="property">scrollTop</span></span></span><br><span class="line"><span class="language-javascript"> })</span></span><br><span class="line"><span class="language-javascript"></span><span class="tag"></<span class="name">script</span>></span></span><br></pre></td></tr></table></figure>
|
||
]]></content>
|
||
<tags>
|
||
<tag>uniapp</tag>
|
||
</tags>
|
||
</entry>
|
||
<entry>
|
||
<title>Docker被墙，如何继续使用？</title>
|
||
<url>/posts/47807.html</url>
|
||
<content><![CDATA[<h2 id="Docker-Download"><a href="#Docker-Download" class="headerlink" title="Docker Download"></a>Docker Download</h2><blockquote>
|
||
<p>自从docker官方仓库在中国大陆被墙后，docker的部署方式也发生了改变。<br>解决docker安装问题：<a href="https://github.com/shenjianZ/docker_installer">https://github.com/shenjianZ/docker_installer</a></p>
|
||
</blockquote>
|
||
<ol>
|
||
<li><p>安装docker </p>
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">sudo curl -fsSL https://gitee.com/tech-shrimp/docker_installer/releases/download/latest/linux.sh| bash -s docker --mirror Aliyun</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>启动docker</p>
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">sudo systemctl start docker</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>设置开机自启</p>
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">sudo systemctl enable docker</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>Docker pull images</p>
|
||
<blockquote>
|
||
<p>将image下载到阿里云镜像仓库中<br>解决docker pull 镜像问题:<a href="https://github.com/shenjianZ/docker_image_pusher">https://github.com/shenjianZ/docker_image_pusher</a></p>
|
||
</blockquote>
|
||
<ol>
|
||
<li><p><strong>登录阿里云镜像服务</strong> <a href="https://cr.console.aliyun.com/">https://cr.console.aliyun.com/</a>,</p>
|
||
<p>启用个人实例，创建一个命名空间（<code>ALIYUN_NAME_SPACE</code>）</p>
|
||
</li>
|
||
<li><p>在<strong>访问凭证</strong> 可以看到账号 用户名（<code>ALIYUN_REGISTRY_USER</code>)</p>
|
||
<p>密码（<code>ALIYUN_REGISTRY_PASSWORD</code>)</p>
|
||
<p>仓库地址（<code>ALIYUN_REGISTRY</code>）</p>
|
||
</li>
|
||
<li><p><strong>启动Action</strong><br>进入您自己的项目，点击<code>Action</code>，启用<code>Github Action</code>功能</p>
|
||
</li>
|
||
<li><p><strong>配置环境变量</strong><br>进入Settings->Secret and variables->Actions->New Repository secret<br> 将上一步的四个值<br> <code>ALIYUN_NAME_SPACE</code>,</p>
|
||
<p> <code> ALIYUN_REGISTRY_USER</code>，</p>
|
||
<p> <code>ALIYUN_REGISTRY_PASSWORD</code>，</p>
|
||
<p> <code>ALIYUN_REGISTRY</code><br> 配置成环境变量</p>
|
||
</li>
|
||
<li><p><strong>添加镜像</strong><br> 打开<code>images.txt</code>文件，添加你想要的镜像 可以加tag</p>
|
||
</li>
|
||
<li><p>使用镜像<br>回到阿里云，镜像仓库，点击任意镜像，可查看镜像状态。(可以改成公开，拉取镜像免登录)</p>
|
||
</li>
|
||
</ol>
|
||
</li>
|
||
</ol>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>Hadoop集群搭建基础环境</title>
|
||
<url>/posts/61253.html</url>
|
||
<content><![CDATA[<h3 id="防火墙关闭"><a href="#防火墙关闭" class="headerlink" title="防火墙关闭"></a>防火墙关闭</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 在 6 台主机执行</span></span><br><span class="line">systemctl stop firewalld</span><br><span class="line">systemctl <span class="built_in">disable</span> firewalld</span><br></pre></td></tr></table></figure>
|
||
<h3 id="配置yum源"><a href="#配置yum源" class="headerlink" title="配置yum源"></a>配置yum源</h3><ul>
|
||
<li>下载 repo 文件：<br><a href="http://mirrors.aliyun.com/repo/Centos-7.repo">Centos-7.repo</a><br>并上传到<code>/tmp</code>，进入到<code>/tmp</code></li>
|
||
<li>备份并且替换系统的repo文件<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"> <span class="built_in">cp</span> Centos-7.repo /etc/yum.repos.d/ </span><br><span class="line"><span class="built_in">cd</span> /etc/yum.repos.d/ </span><br><span class="line"><span class="built_in">mv</span> CentOS-Base.repo CentOS-Base.repo.bak </span><br><span class="line"><span class="built_in">mv</span> Centos-7.repo CentOS-Base.repo</span><br></pre></td></tr></table></figure></li>
|
||
<li>将<code>nn1</code>上的<code>CentOS-Base.repo</code>拷贝到其他主机<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@nn2:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@nn3:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@s1:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@s2:/etc/yum.repos.d</span><br><span class="line">scp /etc/yum.repos.d/CentOS-Base.repo root@s3:/etc/yum.repos.d</span><br></pre></td></tr></table></figure></li>
|
||
<li>执行yum源更新命令<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">yum clean all</span><br><span class="line">yum makecache </span><br><span class="line">yum update -y </span><br></pre></td></tr></table></figure></li>
|
||
<li>安装常用软件 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl wget openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd ntp unrar bzip2</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="JDK安装"><a href="#JDK安装" class="headerlink" title="JDK安装"></a>JDK安装</h3><blockquote>
|
||
<p>注意需要在六台机器依次执行</p>
|
||
</blockquote>
|
||
<ul>
|
||
<li>上传到<code>/tmp</code>目录下，安装 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cd</span> /tmp</span><br><span class="line">rpm -ivh jdk-8u144-linux-x64.rpm</span><br></pre></td></tr></table></figure></li>
|
||
<li>配置环境变量 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">ln</span> -s /usr/java/jdk1.8.0_144/ /usr/java/jdk1.8</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export JAVA_HOME=/usr/java/jdk1.8'</span> >> /etc/profile.d/myEnv.sh </span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$JAVA_HOME/bin'</span> >> /etc/profile.d/myEnv.sh </span><br><span class="line"><span class="built_in">source</span> /etc/profile </span><br><span class="line">java -version</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="修改主机名和主机名映射"><a href="#修改主机名和主机名映射" class="headerlink" title="修改主机名和主机名映射"></a>修改主机名和主机名映射</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /etc/hostname</span><br></pre></td></tr></table></figure>
|
||
<p>6台机器分别为nn1、nn2、nn3、s1、s2、s3</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /etc/hosts</span><br></pre></td></tr></table></figure>
|
||
|
||
<p>修改为</p>
|
||
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">192.168.1.30 nn1</span><br><span class="line">192.168.1.31 nn2</span><br><span class="line">192.168.1.32 nn3</span><br><span class="line">192.168.1.33 s1</span><br><span class="line">192.168.1.34 s2</span><br><span class="line">192.168.1.35 s3</span><br></pre></td></tr></table></figure>
|
||
<h3 id="创建hadoop用户"><a href="#创建hadoop用户" class="headerlink" title="创建hadoop用户"></a>创建hadoop用户</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#创建hadoop用户 </span></span><br><span class="line">useradd hadoop </span><br><span class="line"><span class="comment">#给hadoop用户设置密码: 12345678 </span></span><br><span class="line">passwd hadoop</span><br></pre></td></tr></table></figure>
|
||
<h3 id="禁止非-wheel-组用户切换到root，配置免密切换root"><a href="#禁止非-wheel-组用户切换到root，配置免密切换root" class="headerlink" title="禁止非 wheel 组用户切换到root，配置免密切换root"></a>禁止非 wheel 组用户切换到root，配置免密切换root</h3><ul>
|
||
<li>修改/etc/pam.d/su配置 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">sed -i <span class="string">'s/#auth\t\trequired\tpam_wheel.so/auth\t\trequired\tpam_wheel.so/g'</span> <span class="string">'/etc/pam.d/su'</span> </span><br><span class="line">sed -i <span class="string">'s/#auth\t\tsufficient\tpam_wheel.so/auth\t\tsufficient\tpam_wheel.so/g'</span> <span class="string">'/etc/pam.d/su'</span></span><br></pre></td></tr></table></figure></li>
|
||
<li>修改/etc/login.defs文件 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">"SU_WHEEL_ONLY yes"</span> >> /etc/login.defs</span><br></pre></td></tr></table></figure></li>
|
||
<li>添加用户到管理员，禁止普通用户su 到 root <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#把hadoop用户加到wheel组里</span></span><br><span class="line">gpasswd -a hadoop wheel</span><br><span class="line"><span class="comment">#查看wheel组里是否有hadoop用户</span></span><br><span class="line"><span class="built_in">cat</span> /etc/group | grep wheel</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="给hadoop用户，配置SSH密钥"><a href="#给hadoop用户，配置SSH密钥" class="headerlink" title="给hadoop用户，配置SSH密钥"></a>给hadoop用户，配置SSH密钥</h3><h4 id="配置hadoop用户ssh免密码登录到hadoop"><a href="#配置hadoop用户ssh免密码登录到hadoop" class="headerlink" title="配置hadoop用户ssh免密码登录到hadoop"></a>配置hadoop用户ssh免密码登录到hadoop</h4><ul>
|
||
<li>仅在<code>nn1</code>执行这段脚本命令即可<br>但是 <code>su - hadoop </code> ,<code> mkdir ~/.ssh</code> 需要在其他主机执行一下 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#切换到hadoop用户 </span></span><br><span class="line">su - hadoop</span><br><span class="line"><span class="comment">#生成ssh公私钥 </span></span><br><span class="line">ssh-keygen -t rsa -f ~/.ssh/id_rsa -P <span class="string">''</span></span><br><span class="line">ssh-copy-id nn1</span><br><span class="line">ssh-copy-id nn2</span><br><span class="line">ssh-copy-id nn3</span><br><span class="line">ssh-copy-id s1</span><br><span class="line">ssh-copy-id s2</span><br><span class="line">ssh-copy-id s3</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@nn2:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@nn3:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@s1:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@s2:/home/hadoop/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa hadoop@s3:/home/hadoop/.ssh</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h4 id="配置hadoop用户ssh免密码登录到root"><a href="#配置hadoop用户ssh免密码登录到root" class="headerlink" title="配置hadoop用户ssh免密码登录到root"></a>配置hadoop用户ssh免密码登录到root</h4><ul>
|
||
<li>同上 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh-copy-id root@nn1</span><br><span class="line">ssh-copy-id root@ nn2</span><br><span class="line">ssh-copy-id root@nn3</span><br><span class="line">ssh-copy-id root@s1</span><br><span class="line">ssh-copy-id root@s2</span><br><span class="line">ssh-copy-id root@s3</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@nn2:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@nn3:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@s1:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@s2:/root/.ssh</span><br><span class="line">scp /home/hadoop/.ssh/id_rsa root@s3:/root/.ssh</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="脚本配置"><a href="#脚本配置" class="headerlink" title="脚本配置"></a>脚本配置</h3><ul>
|
||
<li><p><strong>ips</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /home/hadoop/bin/ips</span><br></pre></td></tr></table></figure>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">nn1 </span><br><span class="line">nn2</span><br><span class="line">nn3</span><br><span class="line">s1 </span><br><span class="line">s2 </span><br><span class="line">s3</span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>ssh_all.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /home/hadoop/bin/ssh_all.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># 进入到当前脚本所在目录</span></span><br><span class="line"><span class="built_in">cd</span> `<span class="built_in">dirname</span> <span class="variable">$0</span>`</span><br><span class="line"><span class="comment"># 获取当前脚本所在目录</span></span><br><span class="line">dir_path=`<span class="built_in">pwd</span>`</span><br><span class="line"><span class="comment">#echo $dir_path</span></span><br><span class="line"><span class="comment"># 读ips文件得到数组(里面是一堆主机名)</span></span><br><span class="line">ip_arr=(`<span class="built_in">cat</span> <span class="variable">$dir_path</span>/ips`)</span><br><span class="line"><span class="comment"># 遍历数组里的主机名</span></span><br><span class="line"><span class="keyword">for</span> ip <span class="keyword">in</span> <span class="variable">${ip_arr[*]}</span></span><br><span class="line"><span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 拼接ssh命令: ssh hadoop@nn1.hadoop ls</span></span><br><span class="line"> cmd_=<span class="string">"ssh hadoop@<span class="variable">${ip}</span> \"$*\" "</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="variable">$cmd_</span></span><br><span class="line"> <span class="comment"># 通过eval命令 执行 拼接的ssh 命令</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">eval</span> <span class="variable">${cmd_}</span> ; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"OK"</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"FAIL"</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"><span class="keyword">done</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>ssh_root.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># 进入到当前脚本所在目录</span></span><br><span class="line"><span class="built_in">cd</span> `<span class="built_in">dirname</span> <span class="variable">$0</span>`</span><br><span class="line"><span class="comment"># 获取当前脚本所在目录</span></span><br><span class="line">dir_path=`<span class="built_in">pwd</span>`</span><br><span class="line"><span class="comment">#echo $dir_path</span></span><br><span class="line"><span class="comment"># 读ips文件得到数组(里面是一堆主机名)</span></span><br><span class="line">ip_arr=(`<span class="built_in">cat</span> <span class="variable">$dir_path</span>/ips`)</span><br><span class="line"><span class="comment"># 遍历数组里的主机名</span></span><br><span class="line"><span class="keyword">for</span> ip <span class="keyword">in</span> <span class="variable">${ip_arr[*]}</span></span><br><span class="line"><span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 拼接ssh命令: ssh hadoop@nn1.hadoop ls</span></span><br><span class="line"> cmd_=<span class="string">"ssh hadoop@<span class="variable">${ip}</span> ~/bin/exe.sh \"$*\""</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="variable">$cmd_</span></span><br><span class="line"> <span class="comment"># 通过eval命令 执行 拼接的ssh 命令</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">eval</span> <span class="variable">${cmd_}</span> ; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"OK"</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"FAIL"</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"><span class="keyword">done</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>scp_all.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="meta">#! /bin/bash</span></span><br><span class="line"><span class="comment"># 进入到当前脚本所在目录</span></span><br><span class="line"><span class="built_in">cd</span> `<span class="built_in">dirname</span> <span class="variable">$0</span>`</span><br><span class="line"><span class="comment"># 获取当前脚本所在目录</span></span><br><span class="line">dir_path=`<span class="built_in">pwd</span>`</span><br><span class="line"><span class="comment">#echo $dir_path</span></span><br><span class="line"><span class="comment"># 读ips文件得到数组(里面是一堆主机名)</span></span><br><span class="line">ip_arr=(`<span class="built_in">cat</span> <span class="variable">$dir_path</span>/ips`)</span><br><span class="line"><span class="comment"># 源</span></span><br><span class="line">source_=<span class="variable">$1</span></span><br><span class="line"><span class="comment"># 目标</span></span><br><span class="line">target=<span class="variable">$2</span></span><br><span class="line"><span class="comment"># 遍历数组里的主机名</span></span><br><span class="line"><span class="keyword">for</span> ip <span class="keyword">in</span> <span class="variable">${ip_arr[*]}</span></span><br><span class="line"><span class="keyword">do</span></span><br><span class="line"> <span class="comment"># 拼接scp命令: scp 源 hadoop@nn1.hadoop:目标</span></span><br><span class="line"> cmd_=<span class="string">"scp -r <span class="variable">${source_}</span> hadoop@<span class="variable">${ip}</span>:<span class="variable">${target}</span>"</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="variable">$cmd_</span></span><br><span class="line"> <span class="comment"># 通过eval命令 执行 拼接的scp 命令</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">eval</span> <span class="variable">${cmd_}</span> ; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"OK"</span></span><br><span class="line"> <span class="keyword">else</span></span><br><span class="line"> <span class="built_in">echo</span> <span class="string">"FAIL"</span></span><br><span class="line"> <span class="keyword">fi</span></span><br><span class="line"><span class="keyword">done</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p><strong>exe.sh</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#切换到root用户执行cmd命令</span></span><br><span class="line">cmd=$*</span><br><span class="line">su - << <span class="string">EOF</span></span><br><span class="line"><span class="string">$cmd</span></span><br><span class="line"><span class="string">EOF</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p>赋予执行权限</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">chmod</span> +x ssh_all.sh </span><br><span class="line"><span class="built_in">chmod</span> +x scp_all.sh</span><br><span class="line"><span class="built_in">chmod</span> +x ssh_root.sh</span><br><span class="line"><span class="built_in">chmod</span> +x exe.sh</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>分发到其他主机</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">./ssh_all.sh <span class="built_in">mkdir</span> /home/hadoop/bin</span><br><span class="line">./scp_all.sh /home/hadoop/bin/ips /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/exe.sh /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/ssh_all.sh /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/scp_all.sh /home/hadoop/bin/</span><br><span class="line">./scp_all.sh /home/hadoop/bin/ssh_root.sh /home/hadoop/bin/</span><br></pre></td></tr></table></figure>
|
||
</li>
|
||
<li><p>将 <code>/home/hadoop/bin</code>添加到hadoop用户的环境变量，需要切换到<code>hadoop</code>用户</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:/home/hadoop/bin'</span> >> ~/.bashrc && <span class="built_in">source</span> ~/.bashrc</span><br><span class="line">scp_all.sh /home/hadoop/.bashrc /home/hadoop/</span><br><span class="line">ssh_all.sh <span class="built_in">source</span> ~/.bashrc </span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>Hadoop集群HDFS配置</title>
|
||
<url>/posts/61252.html</url>
|
||
<content><![CDATA[<h3 id="上传hadoop-3-1-4-tar-gz到-tmp，解压"><a href="#上传hadoop-3-1-4-tar-gz到-tmp，解压" class="headerlink" title="上传hadoop-3.1.4.tar.gz到/tmp，解压"></a>上传<code>hadoop-3.1.4.tar.gz</code>到<code>/tmp</code>，解压</h3><blockquote>
|
||
<p>注意在六台机器均上传到<code>/tmp</code></p>
|
||
</blockquote>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 在6台机器执行</span></span><br><span class="line"><span class="built_in">sudo</span> tar -zxvf /tmp/hadoop-3.1.4.tar.gz -C /usr/local/</span><br><span class="line"><span class="comment"># 分发到其他主机</span></span><br><span class="line">ssh_root.sh <span class="built_in">chown</span> -R hadoop:hadoop /usr/local/hadoop-3.1.4</span><br><span class="line">ssh_root.sh <span class="built_in">ln</span> -s /usr/local/hadoop-3.1.4/ /usr/local/hadoop</span><br></pre></td></tr></table></figure>
|
||
<h3 id="配置环境变量"><a href="#配置环境变量" class="headerlink" title="配置环境变量"></a>配置环境变量</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">'export HADOOP_HOME=/usr/local/hadoop'</span> >> /etc/profile.d/myEnv.sh</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$HADOOP_HOME/bin'</span> >> /etc/profile.d/myEnv.sh</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$HADOOP_HOME/sbin'</span> >> /etc/profile.d/myEnv.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 分发到nn2、nn3、s1、s2、s3</span></span><br><span class="line">scp_all.sh /etc/profile.d/myEnv.sh /etc/profile.d/</span><br><span class="line"><span class="comment"># source 环境变量</span></span><br><span class="line">ssh_root.sh <span class="built_in">source</span> /etc/profile</span><br></pre></td></tr></table></figure>
|
||
<blockquote>
|
||
<p>还需要创建 <code>/data</code>这个目录，由于nn1、nn2、nn3已经创建<code>/data</code>，其他三台需要创建一下</p>
|
||
</blockquote>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">### 在s1、s2、s3执行</span></span><br><span class="line"><span class="built_in">sudo</span> <span class="built_in">mkdir</span> /data</span><br><span class="line"><span class="built_in">sudo</span> <span class="built_in">chown</span> -R hadoop:hadoop /data</span><br></pre></td></tr></table></figure>
|
||
|
||
<h3 id="修改core-site-xml"><a href="#修改core-site-xml" class="headerlink" title="修改core-site.xml"></a>修改core-site.xml</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/core-site.xml </span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>fs.defaultFS<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hdfs://ns1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>默认文件服务的协议和NS逻辑名称，和hdfs-site.xml里的对应此配置替代了1.0里的fs.default.name<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.tmp.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/tmp<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>数据存储目录<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.proxyuser.root.groups<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hadoop<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span></span><br><span class="line"> hdfs dfsadmin –refreshSuperUserGroupsConfiguration,</span><br><span class="line"> yarn rmadmin –refreshSuperUserGroupsConfiguration</span><br><span class="line"> 使用这两个命令不用重启就能刷新</span><br><span class="line"> <span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.proxyuser.root.hosts<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>localhost<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>本地代理<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"><!-- zkfc的配置 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>ha.zookeeper.quorum<span class="tag"></<span class="name">name</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1:2181,nn2:2181,nn3:2181<span class="tag"></<span class="name">value</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>HA使用的zookeeper地址<span class="tag"></<span class="name">description</span>></span> </span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure>
|
||
<h3 id="修改hdfs-site-xml"><a href="#修改hdfs-site-xml" class="headerlink" title="修改hdfs-site.xml"></a>修改<code>hdfs-site.xml</code></h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml </span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.name.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/namenode<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>namenode本地文件存放地址<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.nameservices<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>ns1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>提供服务的NS逻辑名称，与core-site.xml里的对应<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="comment"><!-- namenode的配置 --></span></span><br><span class="line"> <span class="comment"><!-- 主要的 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.namenodes.ns1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1,nn2,nn3<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>列出该逻辑名称下的NameNode逻辑名称<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.rpc-address.ns1.nn1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1:9000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的RPC位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.http-address.ns1.nn1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn1:50070<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的Web Server位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.rpc-address.ns1.nn2<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn2:9000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的RPC位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.http-address.ns1.nn2<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn2:50070<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的Web Server位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.rpc-address.ns1.nn3<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn3:9000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的RPC位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.http-address.ns1.nn3<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>nn3:50070<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定NameNode的Web Server位置<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.handler.count<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>77<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>namenode的工作线程数<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- journaldata配置,使得其他两个namenode同步第一个namenode数据 --></span> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.shared.edits.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>qjournal://nn1:8485;nn2:8485;nn3:8485/ns1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定用于HA存放edits的共享存储，通常是namenode的所在机器<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.journalnode.edits.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/journaldata/<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>journaldata服务存放文件的地址<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>ipc.client.connect.max.retries<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>10<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>namenode和journalnode的链接重试次数10次<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>ipc.client.connect.retry.interval<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>10000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>重试的间隔时间10s<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- zkfc的配置 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.fencing.methods<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>sshfence<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定HA做隔离的方法，缺省是ssh，可设为shell，稍后详述<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.fencing.ssh.private-key-files<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/home/hadoop/.ssh/id_rsa<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>杀死命令脚本的免密配置秘钥<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.client.failover.proxy.provider.ns1<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>指定客户端用于HA切换的代理类，不同的NS可以用不同的代理类以上示例为Hadoop 2.0自带的缺省代理类<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.client.failover.proxy.provider.auto-ha<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span> </span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.ha.automatic-failover.enabled<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>true<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="comment"><!-- datanode配置 --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.datanode.data.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/data/datanode<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>datanode本地文件存放地址<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.replication<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>3<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>文件复本数<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.datanode.registration.ip-hostname-check<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>false<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.client.use.datanode.hostname<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>true<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.datanode.use.datanode.hostname<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>true<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure>
|
||
<h3 id="修改hadoop-env-sh"><a href="#修改hadoop-env-sh" class="headerlink" title="修改hadoop-env.sh"></a>修改<code>hadoop-env.sh</code></h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 添加这两行</span></span><br><span class="line"><span class="built_in">source</span> /etc/profile </span><br><span class="line"><span class="built_in">export</span> HADOOP_HEAPSIZE_MAX=512</span><br></pre></td></tr></table></figure>
|
||
<h3 id="分发这些配置文件"><a href="#分发这些配置文件" class="headerlink" title="分发这些配置文件"></a>分发这些配置文件</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/core-site.xml /usr/local/hadoop/etc/hadoop/</span><br><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/hdfs-site.xml /usr/local/hadoop/etc/hadoop/</span><br><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/hadoop-env.sh /usr/local/hadoop/etc/hadoop/</span><br></pre></td></tr></table></figure>
|
||
<h3 id="集群初始化"><a href="#集群初始化" class="headerlink" title="集群初始化"></a>集群初始化</h3><ul>
|
||
<li><p>需要先启动zookeeper集群</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh start</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 第一次启动先启动journalnode，便于3个namenode的元数据同步</span></span><br><span class="line">ssh_all_zookeeper.sh hadoop-daemon.sh start journalnode</span><br></pre></td></tr></table></figure></li>
|
||
<li><p><code>zkfc</code>搭建</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#在nn1节点执行 </span></span><br><span class="line">hdfs zkfc -formatZK</span><br><span class="line"><span class="comment">#nn1 nn2 nn3启动zkfc </span></span><br><span class="line">hadoop-daemon.sh start zkfc</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>初始化nn1的namenode，在nn1执行</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">hdfs namenode -format </span><br><span class="line">hadoop-daemon.sh start namenode</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>格式化第二台和第三台namenode，并且启动namenode,在nn2、nn3执行</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">hdfs namenode -bootstrapStandby </span><br><span class="line">hadoop-daemon.sh start namenode</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>修改<strong>workers</strong></p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">vim /usr/local/hadoop/etc/hadoop/workers</span><br></pre></td></tr></table></figure>
|
||
<p> 修改为</p>
|
||
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">s1</span><br><span class="line">s2</span><br><span class="line">s3</span><br></pre></td></tr></table></figure>
|
||
<p> 分发给其他机器</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp_all.sh /usr/local/hadoop/etc/hadoop/workers /usr/local/hadoop/etc/hadoop</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>启动datanode节点，在s1、s2、s3执行</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#启动各个节点的datanode</span></span><br><span class="line">hadoop-daemons.sh start datanode</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="集群启动"><a href="#集群启动" class="headerlink" title="集群启动"></a>集群启动</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">start-dfs.sh</span><br></pre></td></tr></table></figure>]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>Hadoop集群Zookeeper配置</title>
|
||
<url>/posts/61251.html</url>
|
||
<content><![CDATA[<h3 id="Zookeeper脚本配置"><a href="#Zookeeper脚本配置" class="headerlink" title="Zookeeper脚本配置"></a>Zookeeper脚本配置</h3><ul>
|
||
<li>拷贝ips<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cp</span> ips ips_zookeeper</span><br></pre></td></tr></table></figure>
|
||
修改为<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">nn1</span><br><span class="line">nn2</span><br><span class="line">nn3</span><br></pre></td></tr></table></figure></li>
|
||
<li>拷贝三个脚本<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cp</span> scp_all.sh scp_all_zookeeper.sh </span><br><span class="line"><span class="built_in">cp</span> ssh_all.sh ssh_all_zookeeper.sh </span><br><span class="line"><span class="built_in">cp</span> ssh_root.sh ssh_root_zookeeper.sh</span><br></pre></td></tr></table></figure></li>
|
||
<li>修改脚本<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line">vim scp_all_zookeeper.sh </span><br><span class="line">vim ssh_all_zookeeper.sh </span><br><span class="line">vim ssh_root_zookeeper.sh </span><br></pre></td></tr></table></figure>
|
||
|
||
将三个脚本中的ips改为ips_zookeeper</li>
|
||
</ul>
|
||
<h3 id="Zookeeper安装"><a href="#Zookeeper安装" class="headerlink" title="Zookeeper安装"></a>Zookeeper安装</h3><ul>
|
||
<li>上传到<code>/tmp</code>目录下，解压 <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">sudo</span> tar -zxvf /tmp/zookeeper-3.4.8.tar.gz -C /usr/local/</span><br><span class="line">scp -r /usr/local/zookeeper-3.4.8/ root@nn2:/usr/local/</span><br><span class="line">scp -r /usr/local/zookeeper-3.4.8/ root@nn3:/usr/local/</span><br><span class="line">ssh_root_zookeeper.sh <span class="built_in">chown</span> -R hadoop:hadoop /usr/local/zookeeper-3.4.8</span><br><span class="line">ssh_root_zookeeper.s <span class="built_in">ln</span> -s /usr/local/zookeeper-3.4.8/ /usr/local/zookeeper</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="Zookeeper配置"><a href="#Zookeeper配置" class="headerlink" title="Zookeeper配置"></a>Zookeeper配置</h3><ul>
|
||
<li><p>zoo.cfg配置</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cd</span> /usr/local/zookeeper/conf/</span><br><span class="line"><span class="built_in">cp</span> zoo_sample.cfg zoo.cfg</span><br></pre></td></tr></table></figure>
|
||
<p>然后<code>vim zoo.cfg</code>,修改如下：</p>
|
||
<figure class="highlight properties"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 修改dataDir</span></span><br><span class="line"><span class="attr">dataDir</span>=<span class="string">/data/zookeeper</span></span><br><span class="line"><span class="comment"># 添加一下内容</span></span><br><span class="line"><span class="attr">server.1</span>=<span class="string">nn1:2888:3888 </span></span><br><span class="line"><span class="attr">server.2</span>=<span class="string">nn2:2888:3888 </span></span><br><span class="line"><span class="attr">server.3</span>=<span class="string">nn3:2888:3888</span></span><br></pre></td></tr></table></figure>
|
||
<p>分发给nn2、nn3<br> <code>scp_all_zookeeper.sh /usr/local/zookeeper/conf/zoo.cfg /usr/local/zookeeper/conf/</code></p>
|
||
</li>
|
||
<li><p><code>zkEnv.sh</code>配置<br><code>vim /usr/local/zookeeper/bin/zkEnv.sh</code><br><img src="/../../../img/000001.png" alt="000001.png"><br>分发到nn2、nn3</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">scp_all_zookeeper.sh /usr/local/zookeeper/bin/zkEnv.sh /usr/local/zookeeper/bin/</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>创建zookeeper数据目录</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh_root_zookeeper.sh <span class="built_in">mkdir</span> -p /data/zookeeper</span><br><span class="line">ssh_root_zookeeper.sh <span class="built_in">chown</span> -R hadoop:hadoop /data</span><br></pre></td></tr></table></figure></li>
|
||
<li><p>创建myid文件</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">ssh nn1 <span class="string">'echo "1" > /data/zookeeper/myid'</span></span><br><span class="line">ssh nn2 <span class="string">'echo "2" > /data/zookeeper/myid'</span></span><br><span class="line">ssh nn3 <span class="string">'echo "3" > /data/zookeeper/myid'</span></span><br></pre></td></tr></table></figure></li>
|
||
<li><p>配置Zookeeper环境变量</p>
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 在其他所有主机也执行</span></span><br><span class="line"><span class="built_in">sudo</span> <span class="built_in">chown</span> -R hadoop:hadoop /etc/profile.d/myEnv.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">'export ZOOKEEPER_HOME=/usr/local/zookeeper'</span> >> /etc/profile.d/myEnv.sh</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'export PATH=$PATH:$ZOOKEEPER_HOME/bin'</span> >> /etc/profile.d/myEnv.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 分发到nn2、nn3</span></span><br><span class="line">scp_all_zookeeper.sh /etc/profile.d/myEnv.sh /etc/profile.d/</span><br><span class="line"><span class="comment"># source 环境变量</span></span><br><span class="line">ssh_all_zookeeper.sh <span class="built_in">source</span> /etc/profile</span><br></pre></td></tr></table></figure></li>
|
||
</ul>
|
||
<h3 id="Zookeeper的命令"><a href="#Zookeeper的命令" class="headerlink" title="Zookeeper的命令"></a>Zookeeper的命令</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment">#启动zk服务</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh start</span><br><span class="line"><span class="comment">#查看每个机器ZK运行的状态</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh status</span><br><span class="line"><span class="comment">#整体停止服务</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh stop </span><br><span class="line"><span class="comment">#重启zk服务</span></span><br><span class="line">ssh_all_zookeeper.sh /usr/local/zookeeper/bin/zkServer.sh restart</span><br></pre></td></tr></table></figure>
|
||
|
||
<figure class="highlight shell"><table><tr><td class="code"><pre><span class="line"><span class="meta prompt_">#</span><span class="language-bash">启动zookeeper客户端，并连接zookeeper集群</span></span><br><span class="line">/usr/local/zookeeper/bin/zkCli.sh -server nn1:2181</span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">可以简化为:</span></span><br><span class="line">zkCli.sh</span><br></pre></td></tr></table></figure>
|
||
|
||
]]></content>
|
||
</entry>
|
||
<entry>
|
||
<title>无法访问外网？需要订阅代理服务？</title>
|
||
<url>/posts/14011.html</url>
|
||
<content><![CDATA[<div class="note info flat"><p><strong>由于中国大陆的GFW（防火墙）限制，无法访问外网网络，因此需要访问像GitHub、YouTube这样的<br>的网站将被屏蔽拦截，接下来我将给出一种使用<code>VPN</code>服务的可行的方案来保证服务的可靠性。</strong></p>
|
||
</div>
|
||
|
||
<h3 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h3><blockquote>
|
||
<p>根据测试，许多提供服务的提供商所在的网站需要使用<code>外部网络</code>才能打开，仅有少部分的网站(<strong>比较贵</strong>)可以直接使用<br>国内网络环境打开直接购买订阅服务。</p>
|
||
</blockquote>
|
||
<p>那么你现在可以有两个选择：</p>
|
||
<ol>
|
||
<li><strong>方案一</strong>：使用无需<code>外部网络</code>便能开通订阅服务的VPN，费用高，如果你选择此方案，那么你可自行搜索解决，此处仅仅讨论方案二。</li>
|
||
<li><strong>方案二</strong>：如果使用此方案，详见下方。</li>
|
||
</ol>
|
||
<h3 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h3><blockquote>
|
||
<p>采用<strong>方案二</strong>方式</p>
|
||
<p>这是一些订阅服务推广的链接： <a href="https://9.234456.xyz/abc.html?t=638586217737356738">https://9.234456.xyz/abc.html?t=638586217737356738</a> (此链接打开无需使用VPN，但进入对应的机场页面却仍无法打开)</p>
|
||
<p>此教程中我们使用的机场是 </p>
|
||
<ol>
|
||
<li><code>一元机场</code>: <a href="https://一元机场.com/">https://xn--4gq62f52gdss.com/</a></li>
|
||
<li><code>赔钱机场</code>：<a href="https://赔钱机场.com/">https://xn--mes358aby2apfg.com/</a></li>
|
||
</ol>
|
||
</blockquote>
|
||
<h3 id="机场选择的建议："><a href="#机场选择的建议：" class="headerlink" title="机场选择的建议："></a>机场选择的建议：</h3><ul>
|
||
<li><p><code>一元机场</code><br><img src="/img/yiyuan.png"><br>可以看到</p>
|
||
<ul>
|
||
<li><code>12元/年</code>,每月50GB的可用额度，下个月重置流量额度</li>
|
||
<li><code>15元/季</code>，即为<code>60元/年</code>，每月有4000GB的可用额度，下个月重置流量额度</li>
|
||
<li><code>7元/月</code>，即为<code>84元/年</code>，每个月8000GB的可用额度，下个月重置流量额度<br>根据我个人的使用情况，大多数情况下我每月的流量使用未超过50GB，如果没有频繁的流量使用，<br>建议选择<code>12元/年</code>，否则可以选择<code>15元/季</code>，这意味着每月将有4000GB的可用额度</li>
|
||
</ul>
|
||
</li>
|
||
<li><p><code>赔钱机场</code></p>
|
||
<p><img src="/img/peiqian.png"><br><code>赔钱机场</code>的订阅共有9种方案，这里我仅显示自己正在使用的，个人认为十分优惠：</p>
|
||
<ul>
|
||
<li><code>34.99元/年</code>,每月500GB的可用额度，根据我观察和使用，这个订阅方案比<code>一元机场</code>的性价比更高，且流量使用额度也不用担心</li>
|
||
</ul>
|
||
</li>
|
||
</ul>
|
||
<h3 id="如何订阅？"><a href="#如何订阅？" class="headerlink" title="如何订阅？"></a>如何订阅？</h3><div class="note success flat"><p>由于需要外部网络才能完成订阅服务的购买，你可以向我的邮箱<code>15202078626@163.com</code>发送你的订阅计划方案，<br>扫描付款二维码，我将为你开通订阅（您只需要付款对应的订阅金额即可）</p>
|
||
</div>
|
||
<img src="/img/dingyue.png" width='150px'>
|
||
|
||
<h3 id="完成订阅后如何使用？"><a href="#完成订阅后如何使用？" class="headerlink" title="完成订阅后如何使用？"></a>完成订阅后如何使用？</h3><blockquote>
|
||
<p>你可以在 <code>Windows</code>、<code>Mac</code>、<code>Android</code>等平台使用此服务<br>使用订阅的对应链接： <a href="https://flowus.cn/shenjian/22f76d4f-e7b3-4b8a-8a89-561566f6eb60">https://flowus.cn/shenjian/22f76d4f-e7b3-4b8a-8a89-561566f6eb60</a></p>
|
||
</blockquote>
|
||
]]></content>
|
||
<tags>
|
||
<tag>网络代理</tag>
|
||
</tags>
|
||
</entry>
|
||
</search>
|