-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
1752 lines (1542 loc) · 286 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>I am cmp-cc</title>
<subtitle>一直在打杂，从未被超越。</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://cmp-cc.github.io/"/>
<updated>2016-10-12T02:52:52.260Z</updated>
<id>http://cmp-cc.github.io/</id>
<author>
<name>cmp-cc</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>win10 下php 环境搭建异常处理</title>
<link href="http://cmp-cc.github.io/2016/09/21/2016/win%2010%20%E4%B8%8Bphp%20%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA%E5%BC%82%E5%B8%B8/"/>
<id>http://cmp-cc.github.io/2016/09/21/2016/win 10 下php 环境搭建异常/</id>
<published>2016-09-21T07:30:00.000Z</published>
<updated>2016-10-12T02:52:52.260Z</updated>
<content type="html"><![CDATA[<p>最近要看php的一个CMS系统，简单学习了一下php，零接触Php，遇到一些问题，简单总结一下。</p>
<ul>
<li><p>在win10 上安装php，提示缺少 <code>msvcr110.dll</code>文件</p>
<ul>
<li>解决方案：<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">https://download.microsoft.com/download/9/C/D/9CD480DC-0301-41B0-AAAB-FE9AC1F60237/VSU4/vcredist_x64.exe</div></pre></td></tr></table></figure>
</li>
</ul>
</li>
<li><p>运行apache httpd 出现缺少 vcruntime140.dll</p>
<ul>
<li>解决方案：<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">https://download.microsoft.com/download/9/3/F/93FCF1E7-E6A4-478B-96E7-D4B285925B00/vc_redist.x64.exe</div></pre></td></tr></table></figure>
</li>
</ul>
</li>
<li><p>运行apache ,Cannot load modules/mod_access_compat.so into server:</p>
<ul>
<li><p>异常详情</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">httpd: Syntax error on line 72 of D:/Program-Software/Server-Tools/apache/httpd-2.4.23-win64-VC14/Apache24/conf/httpd.conf: Cannot load modules/mod_access_compat.so into server: \xd5\xd2\xb2\xbb\xb5\xbd\xd6\xb8\xb6\xa8\xb5\xc4\xc4\xa3\xbf\xe9\xa1\xa3</div></pre></td></tr></table></figure>
</li>
<li><p>解决方案</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">由于没有安装到默认目录，httpd.conf 中 ServerRoot默认指定地址为：C\Apache24 路径。</div><div class="line">修改此路径为真实安装目录，打开httpd.conf 修改即可（control+h 修改所有地址）</div></pre></td></tr></table></figure>
</li>
</ul>
</li>
<li><p>httpd: Could not reliably determine the server’s fully qualified domain name</p>
<ul>
<li><p>异常详情</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"> D:\Program-Software\Server-Tools\apache\httpd-2.4.23-win64-VC14\Apache24\bin>httpd</div><div class="line">AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using fe80::a174:5e03:3852:dadd. Set the 'ServerName' directive globally to suppress this message</div></pre></td></tr></table></figure>
</li>
<li><p>解决方案：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">编辑 httpd.conf 加入一句 ServerName localhost:80</div></pre></td></tr></table></figure>
</li>
</ul>
</li>
<li><p>apache 无法解析php文件</p>
<blockquote>
<p>需要让 apache 识别 php 模块并且用 php模块去处理 *.php 文件 ,linux 下，安装完就支持了，win 下需要自己折腾 - -</p>
<ul>
<li>解决方案<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">// 切记下载完整版php (php 下载一般有两个版本)，不然没有php5apache2_4.dll文件。 (httpd.conf 配置如下即可)</div><div class="line">LoadFile "D:\Program-Software\Programming-Language\Php\php\php-5.5.38-Win32-VC11-x64/libeay32.dll"</div><div class="line">LoadFile "D:\Program-Software\Programming-Language\Php\php\php-5.5.38-Win32-VC11-x64/ssleay32.dll"</div><div class="line">LoadFile "D:\Program-Software\Programming-Language\Php\php\php-5.5.38-Win32-VC11-x64/libpq.dll"</div><div class="line">LoadModule php5_module "D:\Program-Software\Programming-Language\Php\php\php-5.5.38-Win32-VC11-x64\php5apache2_4.dll"</div><div class="line">PHPIniDir "D:\Program-Software\Programming-Language\Php\php\php-5.5.38-Win32-VC11-x64"</div><div class="line">AddType application/x-httpd-php .php</div></pre></td></tr></table></figure>
</li>
</ul>
</blockquote>
</li>
<li><p>mysql 链接异常</p>
<ul>
<li>解决方案<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">在php目录下 打开php.ini</div><div class="line">打开：window on extension_dir</div><div class="line">打开: extension=php_mysqli.dll</div></pre></td></tr></table></figure>
</li>
</ul>
</li>
<li><p>时间格式化异常</p>
<ul>
<li><p>异常详情</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">Warning: date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone</div></pre></td></tr></table></figure>
</li>
<li><p>解决方案</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">php.ini 增加 `date.timezone = Asia/Shanghai`</div></pre></td></tr></table></figure></li>
</ul>
</li>
</ul>
]]></content>
<summary type="html">
<p>最近要看php的一个CMS系统，简单学习了一下php，零接触Php，遇到一些问题，简单总结一下。</p>
<ul>
<li><p>在win10 上安装php，提示缺少 <code>msvcr110.dll</code>文件</p>
<ul>
<li>解决方案：<figure
</summary>
<category term="Programming Language" scheme="http://cmp-cc.github.io/categories/Programming-Language/"/>
<category term="Php" scheme="http://cmp-cc.github.io/categories/Programming-Language/Php/"/>
<category term="Php" scheme="http://cmp-cc.github.io/tags/Php/"/>
</entry>
<entry>
<title>Titan 数据库使用</title>
<link href="http://cmp-cc.github.io/2016/08/25/Titan%20%E5%9B%BE%E6%95%B0%E6%8D%AE%E5%BA%93%E4%BD%BF%E7%94%A8/"/>
<id>http://cmp-cc.github.io/2016/08/25/Titan 图数据库使用/</id>
<published>2016-08-25T11:57:00.000Z</published>
<updated>2016-09-05T10:50:21.305Z</updated>
<content type="html"><![CDATA[<blockquote>
<p>之前有小规模的使用Neo4j，Neo4j 需要商业授权，且对于大规模的分布式支持并不友好。 案例也不多。<br>Titan 作为分布式图数据，构建于Elasticsearch、Hbase、Hadoop 之上， 分布式支持较好，且有部分商业案例. 主要应用:<code>知识图谱</code>、<code>情报分析</code>等。</p>
</blockquote>
<h1 id="Titan-基本介绍"><a href="#Titan-基本介绍" class="headerlink" title="Titan 基本介绍"></a>Titan 基本介绍</h1><p>Titan是一个分布式的图数据库，支持横向扩展，可容纳数千亿个顶点和边。 Titan支持事务，并且可以支撑上千并发用户和 计算复杂图形遍历。</p>
<p>Titan包含下面这些特性:</p>
<ul>
<li>弹性与线性扩展</li>
<li>分布式架构，可容错</li>
<li>支持多数据中心的高可用和热备</li>
<li>支持 ACID and 和最终一致性.</li>
<li>支持多种存储后端:<ul>
<li>Apache Cassandra</li>
<li>Apache HBase</li>
<li>Oracle BerkeleyDB</li>
</ul>
</li>
<li>支持大数据平台集成，完成图数据 分析、报表、ETL 。<ul>
<li>Apache Spark</li>
<li>Apache Giraph</li>
<li>Apache Hadoop</li>
</ul>
</li>
<li>支持位置、数字范围和全文检索:<ul>
<li>ElasticSearch</li>
<li>Solr</li>
<li>Apache Lucene</li>
</ul>
</li>
<li>原生支持 TinkerPop 软件栈:<ul>
<li>Gremlin graph query language</li>
<li>Frames object-to-graph mapper</li>
<li>Rexster graph server</li>
<li>Blueprints standard graph API</li>
</ul>
</li>
<li>开源协议Apache 2 license</li>
</ul>
<h1 id="Titan-开发环境搭建"><a href="#Titan-开发环境搭建" class="headerlink" title="Titan 开发环境搭建:"></a>Titan 开发环境搭建:</h1><ul>
<li>数据存储层为：Hbase</li>
<li>数据索引层为：Elasticsearch</li>
</ul>
<p>Titan + Hbase + Elasticsearch + Hadoop = Titan 图数据库</p>
<p>这里使用 Titan + CDH5.7 + Elastcisearch集群 = Titan 图数据库</p>
<blockquote>
<p>这篇文章将详细的演示CDH + ES + Titan 构建分布式图数据库</p>
</blockquote>
<p><strong>前置条件</strong></p>
<ul>
<li>已经安装Java8 且配置完成环境变量</li>
<li>已经安装Elasticsearch1.5.z 或 Elasticsearch1.5.z 集群</li>
<li>已经安装Hadoop、Hbase 或 CDH 平台</li>
</ul>
<blockquote>
<p>当前ES最高版本为2.3， 但是只能使用ES1.5 版本。 详情参看： <a href="http://s3.thinkaurelius.com/docs/titan/1.0.0/version-compat.html" target="_blank" rel="external">Titan1.0 兼容表</a></p>
</blockquote>
<h2 id="下载Titan-hadoop1"><a href="#下载Titan-hadoop1" class="headerlink" title="下载Titan-hadoop1"></a>下载Titan-hadoop1</h2><p>现在地址：<a href="https://github.com/thinkaurelius/titan/wiki/Downloads，并查看titan-hadoop1" target="_blank" rel="external">https://github.com/thinkaurelius/titan/wiki/Downloads，并查看titan-hadoop1</a> 和titan-hadoop2 日志，现在本质并没有太大区别。 </p>
<ul>
<li>服务器（这里使用为Contos）中下载Titan-hadoop1</li>
</ul>
<img src="/2016/08/25/Titan%20图数据库使用/d6b915a1-8d65-4714-9eee-addb84c19d5b.png" alt="titan安装文件" title="titan安装文件">
<ul>
<li><p>解压<code>Titan</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">unzip titan-1.0.0-hadoop1.zip</span><br></pre></td></tr></table></figure>
</li>
<li><p>运行<code>Gremlin 控制台</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">./bin/gremlin.sh</span><br></pre></td></tr></table></figure>
</li>
</ul>
<hr>
<blockquote>
<p>提示如果使用<code>titan-hadoop2</code> 需要引入<code>titan-hadoop.jar</code>,如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">cd titan-1.0.0-hadoop2</span><br><span class="line">cd lib/</span><br><span class="line">wget http://search.maven.org/remotecontent?filepath=com/thinkaurelius/titan/titan-hadoop/1.0.0/titan-hadoop-1.0.0.jar</span><br></pre></td></tr></table></figure></p>
</blockquote>
<p>如果没有引入<code>titan-hadoop.jar</code>,通过<code>./bin/gremlin.sh</code>启动<code>Gremlin 控制台</code>将会跑出如下异常:<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Invalid import definition: 'com.thinkaurelius.titan.hadoop.MapReduceIndexManagement'; reason: startup failed:</span><br><span class="line">script14719348643282042298222.groovy: 1: unable to resolve class com.thinkaurelius.titan.hadoop.MapReduceIndexManagement</span><br><span class="line"> @ line 1, column 1.</span><br><span class="line"> import com.thinkaurelius.titan.hadoop.MapReduceIndexManagement</span><br><span class="line"> ^</span><br><span class="line"></span><br><span class="line">1 error</span><br></pre></td></tr></table></figure></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager</span><br></pre></td></tr></table></figure>
<h2 id="Hbase-链接测试"><a href="#Hbase-链接测试" class="headerlink" title="Hbase 链接测试"></a>Hbase 链接测试</h2><p><strong>使用Hbase 作为数据存储层</strong></p>
<ul>
<li>进入<code>Gremlin 控制台</code>./bin/gremlin.sh`<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">hBaseConf = new BaseConfiguration();</span><br><span class="line">hBaseConf.setProperty("storage.backend","hbase");</span><br><span class="line">hBaseConf.setProperty("storage.hostname","worker01.hadoop,worker02.hadoop,worker03.hadoop");</span><br><span class="line">hBaseConf.setProperty("storage.hbase.ext.hbase.zookeeper.property.clientPort","2181");</span><br><span class="line">hBaseConf.setProperty("storage.hbase.table","titan");</span><br><span class="line">titanGraph = TitanFactory.open(hBaseConf);</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<ul>
<li>我们需要配置<code>Hbase连接地址</code>、<code>客户端端口</code>、<code>存储表</code></li>
<li>如何确定<code>hbase hostname地址</code>？ hbase 使用<code>zoopeeker</code>作为分布式协调服务。找到<code>hbase-site.xml</code>中<code>hbase.zookeeper.quorum</code>值<br>进入<code>CDH Manage</code></li>
</ul>
</blockquote>
<img src="/2016/08/25/Titan%20图数据库使用/7b7f63d9-a299-4d93-8b00-46671256a153.png" alt="CDH Manager 页面" title="CDH Manager 页面">
<img src="/2016/08/25/Titan%20图数据库使用/32641fe4-8946-4176-b9a7-3bc968ace9a1.png" alt="Hbase 配置URL" title="Hbase 配置URL">
<img src="/2016/08/25/Titan%20图数据库使用/380a7f69-ef58-4d3d-94ca-eee1fe0621e1.png" alt="zk quorum" title="zk quorum">
<ul>
<li>链接成功</li>
</ul>
<p>查看Hbase所有表，你将看到<code>titan</code>表,表示<code>titan</code>能够与hbase正常通信。</p>
<img src="/2016/08/25/Titan%20图数据库使用/86b1776f-c2e3-43e8-b7de-61e91c63118a.png" alt="hbase 整合成功" title="hbase 整合成功">
<h2 id="配置titan-hbase-es-properties"><a href="#配置titan-hbase-es-properties" class="headerlink" title="配置titan-hbase-es.properties"></a>配置<code>titan-hbase-es.properties</code></h2><p>我们只需要加载配置即可完成链接：</p>
<ul>
<li>配置Hbase，作用于存储层</li>
<li>配置Elastichsearch 应用于索引层<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">storage.backend=hbase</span><br><span class="line">storage.hostname=worker01.hadoop,worker02.hadoop,worker03.hadoop</span><br><span class="line"></span><br><span class="line">index.search.backend=elasticsearch</span><br><span class="line">index.search.hostname=192.168.8.14</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h2 id="Elasticsearch，测试安装"><a href="#Elasticsearch，测试安装" class="headerlink" title="Elasticsearch，测试安装"></a>Elasticsearch，测试安装</h2><p><strong>由于限定版本，只好重新安装一个新的ES服务器，在本地服务，并安装<code>head插件</code></strong><br><strong>下载 <code>ES1.5.1</code>，并安装</strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.1.tar.gz</span><br><span class="line">tar zxvf elasticsearch-1.5.1.tar.gz</span><br><span class="line">cd elasticsearch-1.5.1</span><br><span class="line">// 安装head 插件</span><br><span class="line">./bin/plugin -i mobz/elasticsearch-head</span><br><span class="line"></span><br><span class="line">// 后台启动</span><br><span class="line">./bin/elasticsearch -d</span><br></pre></td></tr></table></figure>
<ul>
<li>如下是链接Elasticsearch 集群</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">conf = new BaseConfiguration()</span><br><span class="line"></span><br><span class="line">conf .setProperty("storage.backend","hbase");</span><br><span class="line">conf .setProperty("storage.hostname","worker01.hadoop,worker02.hadoop,worker03.hadoop");</span><br><span class="line">conf .setProperty("storage.hbase.ext.hbase.zookeeper.property.clientPort","2181");</span><br><span class="line">conf .setProperty("storage.hbase.table","titan");</span><br><span class="line"></span><br><span class="line">conf.setProperty("index.search.backend","elasticsearch")</span><br><span class="line">conf.setProperty("index.search.hostname", "127.0.0.1")</span><br><span class="line">conf.setProperty("index.search.client-only",true)</span><br><span class="line">titanGraph = TitanFactory.open(conf );</span><br></pre></td></tr></table></figure>
<ul>
<li>我们访问新安装的elasticsearch: </li>
</ul>
<p><strong><a href="http://192.168.xx.xx:9200/_plugin/head" target="_blank" rel="external">http://192.168.xx.xx:9200/_plugin/head</a> ,将会发现titan 索引 </strong></p>
<img src="/2016/08/25/Titan%20图数据库使用/73418101-b3f8-44c8-8216-467dc3b82cb2.png" alt="es链接测试成功" title="es链接测试成功">
<h2 id="将测试成功配置参数，写在配置文件中"><a href="#将测试成功配置参数，写在配置文件中" class="headerlink" title="将测试成功配置参数，写在配置文件中"></a>将测试成功配置参数，写在配置文件中</h2><p><strong>编辑 <code>vi conf/titan-hbase-es.properties</code></strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">========== Titatn Config =======</span><br><span class="line"></span><br><span class="line">storage.backend=hbase</span><br><span class="line">storage.hostname=worker01.hadoop,worker02.hadoop,worker03.hadoop</span><br><span class="line"># Must use for MapR Hadoop Distributions ( Tells Titan to contact Zookeeper on port 5181 instead of 2181 )</span><br><span class="line">storage.hbase.ext.hbase.zookeeper.property.clientPort=2181</span><br><span class="line">storage.hbase.table = titan</span><br><span class="line"></span><br><span class="line">cache.db-cache = true</span><br><span class="line">cache.db-cache-clean-wait = 20</span><br><span class="line">cache.db-cache-time = 180000</span><br><span class="line">cache.db-cache-size = 0.5</span><br><span class="line"></span><br><span class="line">index.search.backend=elasticsearch</span><br><span class="line">index.search.hostname=127.0.0.1</span><br><span class="line">index.search.elasticsearch.client-only=true</span><br></pre></td></tr></table></figure>
<p><strong>控制台 中可以如下加载配置文件</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">g = TitanFactory.open('conf/titan-hbase-es.properties');</span><br></pre></td></tr></table></figure></p>
<hr>
<p><strong>这里基本的环境已经搭建完毕，你可以创建点、边、属性 等等</strong></p>
]]></content>
<summary type="html">
<blockquote>
<p>之前有小规模的使用Neo4j，Neo4j 需要商业授权，且对于大规模的分布式支持并不友好。 案例也不多。<br>Titan 作为分布式图数据，构建于Elasticsearch、Hbase、Hadoop 之上， 分布式支持较好，且有部分商业案例. 主
</summary>
<category term="Distributed Development" scheme="http://cmp-cc.github.io/categories/Distributed-Development/"/>
<category term="Titan" scheme="http://cmp-cc.github.io/categories/Distributed-Development/Titan/"/>
<category term="Titan" scheme="http://cmp-cc.github.io/tags/Titan/"/>
</entry>
<entry>
<title>Azkaban 短信预警插件开发</title>
<link href="http://cmp-cc.github.io/2016/06/09/2016/Azkaban%20%E7%9F%AD%E4%BF%A1%E9%A2%84%E8%AD%A6%E6%8F%92%E4%BB%B6%E5%BC%80%E5%8F%91/"/>
<id>http://cmp-cc.github.io/2016/06/09/2016/Azkaban 短信预警插件开发/</id>
<published>2016-06-09T14:53:00.000Z</published>
<updated>2016-06-24T09:31:50.121Z</updated>
<content type="html"><![CDATA[<p>Azkaban本身具有邮件预警功能，我们可以通过设置SLA，进行邮件失败警告或成功提示。但是邮件预警一般不能及时响应处理。（我们并不会及时查看邮件）。 为增加强应急响应，开发Azkaban 短信预警插件，及时向管理员、运维人员发出警告提示。</p>
<h2 id="短信-API"><a href="#短信-API" class="headerlink" title="短信 API"></a>短信 API</h2><p><strong>这里采用<a href="http://www.yunpian.com/" target="_blank" rel="external">云片</a> 手机短信服务，在此基础上封装一层API，使其发短信更加容易。</strong></p>
<p>服务器端首先生成<code>apikey</code>，用于身份认证，无消亡时间。<br>API 接口：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl -k -X POST --data "apikey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&mobile=18271932456&text=Azkaban 短信预警API服务" http://sms.avlyun.org/send.json</span><br></pre></td></tr></table></figure></p>
<h2 id="Azkaban-插件开发"><a href="#Azkaban-插件开发" class="headerlink" title="Azkaban 插件开发"></a>Azkaban 插件开发</h2><p>先吐槽：Azkaban 的文档写的稀巴烂(太多残缺，很多错误)，项目维护不积极，很多人已经投身<code>Airflow</code>。 还有LinkedIn 这家公司确定要卖掉啦~。</p>
<p><strong>Azkaban 源码提供了约定俗成预警插件写法：</strong></p>
<ul>
<li>继承 <code>azkaban.executor.ExecutableFlow.Alerter</code> 接口，实现相关预警方法。</li>
<li><code>azkaban.properties</code>通过<code>alerter.plugin.dir</code>指定预警插件存放目录，或者默认为：<code>plugins/alerter</code></li>
<li>插件包含<code>conf/plugin.properties</code>、<code>extlib</code>、<code>lib</code>、插件代码。</li>
<li><code>plugin.properties</code>必须包含<code>alerter.name</code>(名称)、<code>alerter.external.classpaths</code>(额外jar,就是extlib)、<code>alerter.class</code>(插件主类)</li>
</ul>
<h3 id="SMS-插件编码"><a href="#SMS-插件编码" class="headerlink" title="SMS 插件编码"></a>SMS 插件编码</h3><p><strong>原Azkaban Plugin 使用Ant构建，这里就使用Mavne 进行构建。 （效果是一模一样的）</strong></p>
<ul>
<li>创建Maven项目，其目录结构如下</li>
</ul>
<img src="/2016/06/09/2016/Azkaban%20短信预警插件开发/232f9c5a-ec8a-43c9-8907-88a9c4d8cc80.png" alt="项目目录结构" title="项目目录结构">
<ul>
<li><p>pom.xml 依赖及其项目构建插件</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br></pre></td><td class="code"><pre><span class="line"><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"</span><br><span class="line"> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"></span><br><span class="line"> <modelVersion>4.0.0</modelVersion></span><br><span class="line"> <groupId>org.freecode</groupId></span><br><span class="line"> <artifactId>azkaban-sms</artifactId></span><br><span class="line"> <version>0.0.1-SNAPSHOT</version></span><br><span class="line"> <packaging>jar</packaging></span><br><span class="line"> <name>azkaban-sms</name></span><br><span class="line"> <url>http://maven.apache.org</url></span><br><span class="line"> <properties></span><br><span class="line"> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding></span><br><span class="line"> </properties></span><br><span class="line"> <dependencies></span><br><span class="line"> <dependency></span><br><span class="line"> <groupId>junit</groupId></span><br><span class="line"> <artifactId>junit</artifactId></span><br><span class="line"> <version>4.9</version></span><br><span class="line"> <scope>test</scope></span><br><span class="line"> </dependency></span><br><span class="line"> <dependency></span><br><span class="line"> <groupId>com.linkedin.azkaban</groupId></span><br><span class="line"> <artifactId>azkaban</artifactId></span><br><span class="line"> <version>2.5.0</version></span><br><span class="line"> <scope>provided</scope></span><br><span class="line"> </dependency></span><br><span class="line"> <dependency></span><br><span class="line"> <groupId>log4j</groupId></span><br><span class="line"> <artifactId>log4j</artifactId></span><br><span class="line"> <version>1.2.16</version></span><br><span class="line"> <scope>provided</scope></span><br><span class="line"> </dependency></span><br><span class="line"> <dependency></span><br><span class="line"> <groupId>org.apache.httpcomponents</groupId></span><br><span class="line"> <artifactId>httpclient</artifactId></span><br><span class="line"> <version>4.5.1</version></span><br><span class="line"> </dependency></span><br><span class="line"> </dependencies></span><br><span class="line"> <build></span><br><span class="line"> <plugins></span><br><span class="line"> <plugin></span><br><span class="line"> <groupId>org.apache.maven.plugins</groupId></span><br><span class="line"> <artifactId>maven-compiler-plugin</artifactId></span><br><span class="line"> <version>3.5.1</version></span><br><span class="line"> <configuration></span><br><span class="line"> <source>1.8</source></span><br><span class="line"> <target>1.8</target></span><br><span class="line"> </configuration></span><br><span class="line"> </plugin></span><br><span class="line"> <plugin></span><br><span class="line"> <artifactId>maven-assembly-plugin</artifactId></span><br><span class="line"> <version>2.6</version></span><br><span class="line"> <executions></span><br><span class="line"> <execution></span><br><span class="line"> <id>Azkaban-sms</id></span><br><span class="line"> <phase>package</phase></span><br><span class="line"> <goals></span><br><span class="line"> <goal>single</goal></span><br><span class="line"> </goals></span><br><span class="line"> <configuration></span><br><span class="line"> <descriptors> </span><br><span class="line"> <descriptor>src/assembly/build.xml</descriptor></span><br><span class="line"> </descriptors></span><br><span class="line"> </configuration></span><br><span class="line"> </execution></span><br><span class="line"> </executions></span><br><span class="line"> </plugin></span><br><span class="line"> </plugins></span><br><span class="line"> </build></span><br><span class="line"></project></span><br></pre></td></tr></table></figure>
</li>
<li><p>配置<code>src/assembly/build.xml</code> 文件</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><assembly</span><br><span class="line"> xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"</span><br><span class="line"> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"</span><br><span class="line"> xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd"></span><br><span class="line"> <id>release</id></span><br><span class="line"> <formats></span><br><span class="line"> <format>zip</format></span><br><span class="line"> </formats></span><br><span class="line"> <baseDirectory>azkaban-sms</baseDirectory></span><br><span class="line"> <fileSets></span><br><span class="line"> <fileSet></span><br><span class="line"> <directory>src/main/resources</directory></span><br><span class="line"> <outputDirectory>conf</outputDirectory></span><br><span class="line"> <includes></span><br><span class="line"> <include>plugin.properties</include></span><br><span class="line"> </includes></span><br><span class="line"> </fileSet></span><br><span class="line"> <fileSet></span><br><span class="line"> <directory>src/assembly</directory></span><br><span class="line"> <outputDirectory>extlib</outputDirectory></span><br><span class="line"> <excludes></span><br><span class="line"> <exclude>*</exclude></span><br><span class="line"> </excludes></span><br><span class="line"> </fileSet></span><br><span class="line"> </fileSets></span><br><span class="line"> <dependencySets></span><br><span class="line"> <dependencySet></span><br><span class="line"> <useProjectArtifact>true</useProjectArtifact></span><br><span class="line"> <outputDirectory>lib</outputDirectory></span><br><span class="line"> <scope>runtime</scope></span><br><span class="line"> </dependencySet></span><br><span class="line"> </dependencySets></span><br><span class="line"></assembly></span><br></pre></td></tr></table></figure>
</li>
<li><p>SMSAlerter 代码编写</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br></pre></td><td class="code"><pre><span class="line">package org.freecode.azkaban;</span><br><span class="line">import java.io.IOException;</span><br><span class="line">import java.text.SimpleDateFormat;</span><br><span class="line">import java.util.ArrayList;</span><br><span class="line">import java.util.List;</span><br><span class="line">import java.util.regex.Pattern;</span><br><span class="line">import org.apache.http.Consts;</span><br><span class="line">import org.apache.http.NameValuePair;</span><br><span class="line">import org.apache.http.client.ClientProtocolException;</span><br><span class="line">import org.apache.http.client.entity.UrlEncodedFormEntity;</span><br><span class="line">import org.apache.http.client.methods.CloseableHttpResponse;</span><br><span class="line">import org.apache.http.client.methods.HttpPost;</span><br><span class="line">import org.apache.http.impl.client.CloseableHttpClient;</span><br><span class="line">import org.apache.http.impl.client.HttpClients;</span><br><span class="line">import org.apache.http.message.BasicNameValuePair;</span><br><span class="line">import org.apache.log4j.Logger;</span><br><span class="line">import azkaban.alert.Alerter;</span><br><span class="line">import azkaban.executor.ExecutableFlow;</span><br><span class="line">import azkaban.sla.SlaOption;</span><br><span class="line">import azkaban.utils.Props;</span><br><span class="line">import azkaban.utils.Utils;</span><br><span class="line">public class SMSAlerter implements Alerter {</span><br><span class="line"> private static final Logger logger = Logger.getLogger(SMSAlerter.class);</span><br><span class="line"> private SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss");</span><br><span class="line"> private CloseableHttpClient httpclient = HttpClients.createDefault();</span><br><span class="line"> </span><br><span class="line"> // 手机号验证</span><br><span class="line"> private static final Pattern phoneValidPattern = Pattern</span><br><span class="line"> .compile("1[0-9]{10}");</span><br><span class="line"> // 短信API地址</span><br><span class="line"> private String smsHost;</span><br><span class="line"> // ApiKey</span><br><span class="line"> private String smsTokenKey;</span><br><span class="line"> // ApiKey</span><br><span class="line"> private String smsToken;</span><br><span class="line"> // 短信手机号参数名称</span><br><span class="line"> private String smsPhoneKey;</span><br><span class="line"> // 预警手机号</span><br><span class="line"> private List<String> smsPhone;</span><br><span class="line"> // 短信内容参数名称</span><br><span class="line"> private String smsTextKey;</span><br><span class="line"> // 测试模式 : 当为true时,只打印日志，不发送短信。</span><br><span class="line"> // 当为false时,打印日志，且发送短信。</span><br><span class="line"> private boolean testMode;</span><br><span class="line"> public SMSAlerter(Props props) {</span><br><span class="line"> /*</span><br><span class="line"> * props.getString 第一个参数获取plugin.propertiy 文件获取对应值，如果没有找到使用第二默认参数</span><br><span class="line"> * </span><br><span class="line"> * 这里没有设置默认值，你可以通过修改源码设置它</span><br><span class="line"> */</span><br><span class="line"> this.smsHost = props.getString("sms.host");</span><br><span class="line"> this.smsTokenKey = props.getString("sms.token.key");</span><br><span class="line"> this.smsToken = props.getString("sms.token");</span><br><span class="line"> this.smsPhoneKey = props.getString("sms.phone.key", "mobile");</span><br><span class="line"> // 多个手机号使用空格、,、:、#、%、& 进行分割</span><br><span class="line"> this.smsPhone = props.getStringList("sms.phone", "[ ,:#%&]");</span><br><span class="line"> this.smsTextKey = props.getString("sms.text.key");</span><br><span class="line"> this.testMode = props.getBoolean("sms.test.model", true);</span><br><span class="line"> </span><br><span class="line"> /**</span><br><span class="line"> * 可配置一个配置项，通知一个手机号。 提示Azkaban SMS Plugin 加载成功。 当然也可以通过日志 </span><br><span class="line"> */</span><br><span class="line"> //sendSMS(Arrays.asList("18271932456"), "Azkaban SMS Plugin 加载成功!");</span><br><span class="line"> logger.info("Azkaban SMS Plugin 加载成功!");</span><br><span class="line"> }</span><br><span class="line"> // 向配置文件中的手机号发送短信</span><br><span class="line"> private void sendSMS(String msg) {</span><br><span class="line"> sendSMS(this.smsPhone, msg);</span><br><span class="line"> }</span><br><span class="line"> private void sendSMS(List<String> phoneList, String msg) {</span><br><span class="line"> if (smsPhone != null) {</span><br><span class="line"> logger.info(String.format("欲发送%s位联系人,分别为:%s", smsPhone.size(),</span><br><span class="line"> smsPhone.toString()));</span><br><span class="line"> }</span><br><span class="line"> for (String phone : phoneList) {</span><br><span class="line"> if (!phoneValidPattern.matcher(phone).matches()) {</span><br><span class="line"> logger.error("输入手机号无效,请确保手机号长度为11位");</span><br><span class="line"> continue;</span><br><span class="line"> }</span><br><span class="line"> List<NameValuePair> formparams = new ArrayList<NameValuePair>();</span><br><span class="line"> formparams.add(new BasicNameValuePair(this.smsTokenKey,</span><br><span class="line"> this.smsToken));</span><br><span class="line"> formparams.add(new BasicNameValuePair(this.smsTextKey, msg));</span><br><span class="line"> formparams.add(new BasicNameValuePair(this.smsPhoneKey, phone));</span><br><span class="line"> UrlEncodedFormEntity entity = new UrlEncodedFormEntity(formparams,</span><br><span class="line"> Consts.UTF_8);</span><br><span class="line"> HttpPost httpPost = new HttpPost(this.smsHost);</span><br><span class="line"> httpPost.setEntity(entity);</span><br><span class="line"> CloseableHttpResponse response = null;</span><br><span class="line"> try {</span><br><span class="line"> if (!testMode) {</span><br><span class="line"> response = httpclient.execute(httpPost);</span><br><span class="line"> logger.info(String.format("当前发送至：%s,发送状态为: %s", phone,</span><br><span class="line"> response.getStatusLine()));</span><br><span class="line"> } else {</span><br><span class="line"> logger.info(String.format("当前为测试模式,欲发送至：%s", phone));</span><br><span class="line"> }</span><br><span class="line"> Thread.sleep(1000); // 延迟一秒</span><br><span class="line"> } catch (ClientProtocolException e) {</span><br><span class="line"> logger.error("failed to send message " + msg + " to " + phone,</span><br><span class="line"> e);</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> logger.error("failed to send message " + msg + " to " + phone,</span><br><span class="line"> e);</span><br><span class="line"> } catch (InterruptedException e) {</span><br><span class="line"> logger.warn("send message interrupted ", e);</span><br><span class="line"> Thread.interrupted();</span><br><span class="line"> } finally {</span><br><span class="line"> if (response != null) {</span><br><span class="line"> try {</span><br><span class="line"> response.close();</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> // TODO Auto-generated catch block</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> /**</span><br><span class="line"> * </span><br><span class="line"> * @param flow</span><br><span class="line"> * @param status 工作流执行状态，应当为一个枚举值： SUCCESS 和 FAILD、FIRST FAILD</span><br><span class="line"> */</span><br><span class="line"> private void sendEmail(ExecutableFlow flow,List<String> sendList,String status) {</span><br><span class="line"> List<String> phoneList = new ArrayList<>();</span><br><span class="line"> for (String phone : sendList) {</span><br><span class="line"> if (phoneValidPattern.matcher(phone).matches()) {</span><br><span class="line"> phoneList.add(phone);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> logger.info(String.format("捕获Email列表为：%s,过滤手机号列表为:%s",sendList,phoneList ));</span><br><span class="line"> </span><br><span class="line"> StringBuffer textMsg = new StringBuffer();</span><br><span class="line"> textMsg.append(String.format("Antiy Azkaban 3.0 工作流执行状态: %s \n", status));</span><br><span class="line"> textMsg.append(String.format(" 工作流项目名称为：%s,目前已经执行%s!\n",</span><br><span class="line"> flow.getFlowId(),status));</span><br><span class="line"> textMsg.append(String.format("Execution ID:%s\n", flow.getExecutionId()));</span><br><span class="line"> textMsg.append(String.format("运行开始时间:%s\n", dateFormat.format(flow.getStartTime())));</span><br><span class="line"> textMsg.append(String.format("运行结束时间:%s\n", dateFormat.format(flow.getStartTime())));</span><br><span class="line"> textMsg.append(String.format("共耗时：%s\n",</span><br><span class="line"> Utils.formatDuration(flow.getStartTime(), flow.getEndTime())));</span><br><span class="line"> // 发送至全局手机配置</span><br><span class="line"> sendSMS(textMsg.toString());</span><br><span class="line"> // 发送至局部手机配置</span><br><span class="line"> sendSMS(phoneList, textMsg.toString());</span><br><span class="line"> }</span><br><span class="line"> @Override</span><br><span class="line"> public void alertOnSuccess(ExecutableFlow exflow) throws Exception {</span><br><span class="line"> sendEmail(exflow,exflow.getExecutionOptions().getSuccessEmails(),"成功");</span><br><span class="line"> }</span><br><span class="line"> @Override</span><br><span class="line"> public void alertOnError(ExecutableFlow exflow, String... extraReasons)</span><br><span class="line"> throws Exception {</span><br><span class="line"> sendEmail(exflow,exflow.getExecutionOptions().getFailureEmails(),"失败");</span><br><span class="line"> }</span><br><span class="line"> @Override</span><br><span class="line"> public void alertOnFirstError(ExecutableFlow exflow) throws Exception {</span><br><span class="line"> sendEmail(exflow,exflow.getExecutionOptions().getFailureEmails(),"First 失败");</span><br><span class="line"> }</span><br><span class="line"> @Override</span><br><span class="line"> public void alertOnSla(SlaOption slaOption, String slaMessage)</span><br><span class="line"> throws Exception {</span><br><span class="line"> @SuppressWarnings("unchecked")</span><br><span class="line"> List<String> emailList = (List<String>) slaOption.getInfo().get(</span><br><span class="line"> SlaOption.INFO_EMAIL_LIST);</span><br><span class="line"> logger.info("SAL 信息：" + emailList.toString());</span><br><span class="line"> logger.info("SAL info :" + slaOption.getInfo().toString());</span><br><span class="line"> logger.info("SAL type :" + slaOption.getType().toString());</span><br><span class="line"> logger.info("SAL actions :" + slaOption.getActions().toString());</span><br><span class="line"> List<String> phoneList = new ArrayList<>();</span><br><span class="line"> for (String email : emailList) {</span><br><span class="line"> if (!phoneValidPattern.matcher(email).matches()) {</span><br><span class="line"> phoneList.add(email);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> sendSMS(phoneList, slaMessage);</span><br><span class="line"> sendSMS(slaMessage);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
</li>
<li><p>配置<code>plugin.properties</code> 文件</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">alerter.name=sms</span><br><span class="line">alerter.external.classpaths=extlib/</span><br><span class="line">alerter.class=org.freecode.azkaban.SMSAlerter</span><br><span class="line"></span><br><span class="line">#SMSAlerter conf</span><br><span class="line"></span><br><span class="line"># sms api host</span><br><span class="line">sms.host=http://sms.avlyun.org/send.json</span><br><span class="line"># 认证参数名称</span><br><span class="line">sms.token.key=apikey</span><br><span class="line"># 认证token</span><br><span class="line">sms.token=xxxxxxxxxxxxxxxx</span><br><span class="line"># 手机参数名称</span><br><span class="line">sms.phone.key=mobile</span><br><span class="line"># 配置全局手机，可配置多个</span><br><span class="line">sms.phone=18271932456</span><br><span class="line"># 短信内容参数名称</span><br><span class="line">sms.text.key=text</span><br><span class="line"># 是否开启测试模式（false 表示发送短信）</span><br><span class="line">sms.test.model=false</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<p>你可以需要修改部分代码，来完成与短信接口的映射。</p>
</blockquote>
<ul>
<li><p>构建项目</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mvn package</span><br></pre></td></tr></table></figure>
</li>
<li><p>查看项目构建</p>
</li>
</ul>
<img src="/2016/06/09/2016/Azkaban%20短信预警插件开发/21023a17-a161-44e4-bed3-5255ddb801f3.png" alt="Azkaban SMS构建成功" title="Azkaban SMS构建成功">
<blockquote>
<p><code>azkaban-sms-0.0.1-SNAPSHOT-release.zip</code> 为Azkaban SMS插件</p>
</blockquote>
<h2 id="部署Azkaban-SMS-Plugin"><a href="#部署Azkaban-SMS-Plugin" class="headerlink" title="部署Azkaban SMS Plugin"></a>部署Azkaban SMS Plugin</h2><ul>
<li><p><code>plugins</code>目录 创建<code>alerter</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">cd /usr/local/azkaban3.0/azkaban-web-server-3.0.0/plugins</span><br><span class="line">mkdir alerter</span><br><span class="line">cd alerter</span><br></pre></td></tr></table></figure>
</li>
<li><p>拷贝至<code>alerter</code>目录，解压、配置</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">unzip azkaban-sms-0.0.1-SNAPSHOT-release.zip</span><br><span class="line">rm -f azkaban-sms-0.0.1-SNAPSHOT-release.zip</span><br><span class="line">cd azkaban-sms/</span><br><span class="line">vi conf/plugin.properties </span><br><span class="line"></span><br><span class="line">#配置短信API接口</span><br><span class="line">#配置全局手机接收人（可以为空）</span><br><span class="line">#配置是否为测试模式</span><br></pre></td></tr></table></figure>
</li>
</ul>
<p><strong>重新启动<code>Azkaban-web</code>，你可以通过日志<code>logs/azkaban-web.log</code> 参数执行状态</strong></p>
<h2 id="配置job作业"><a href="#配置job作业" class="headerlink" title="配置job作业"></a>配置job作业</h2><p>想要启动短信预警，必须配置<code>alert.type</code>属性为<code>sms</code>，只有配置<code>alert.type=sms</code>，局部短信和全局短信才能发送，有如下两种方式配置。</p>
<ul>
<li>在<code>job</code>文件中添加<code>alert.type=sms</code></li>
<li>在图形界面执行时配置，如下：</li>
</ul>
<img src="/2016/06/09/2016/Azkaban%20短信预警插件开发/ab788c20-ba61-4016-9360-2821f5a110e9.png" alt="图形界面配置alert.type参数" title="图形界面配置alert.type参数">
<h3 id="配置全局短信发送"><a href="#配置全局短信发送" class="headerlink" title="配置全局短信发送"></a>配置全局短信发送</h3><p>在<code>/plugin.properties</code>中通过sms.phone配置全局接收人。（每一个设置了<code>alerttype=sms</code> 的工作流作业，全局接收人都会接受到作业状态，无论成功或失败）<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">sms.phone=18271932456#15290043676#xxxxx</span><br><span class="line"></span><br><span class="line">或者不进行设置</span><br><span class="line"></span><br><span class="line">sms.phone=</span><br></pre></td></tr></table></figure></p>
<h3 id="配置局部短信发送"><a href="#配置局部短信发送" class="headerlink" title="配置局部短信发送"></a>配置局部短信发送</h3><p>在执行任务时，进行配置，配置如下： （包含手机联系人、Email联系人）<br><img src="/2016/06/09/2016/Azkaban%20短信预警插件开发/49c6172e-6fd4-42c5-9dae-2c5825e41855.png" alt="任务短信配置" title="任务短信配置"></p>
<h2 id="日志信息"><a href="#日志信息" class="headerlink" title="日志信息"></a>日志信息</h2><ul>
<li>你可以通过日志信息查看发送状态，如下在测试模式下进行短信发送。</li>
</ul>
<img src="/2016/06/09/2016/Azkaban%20短信预警插件开发/35af5fe8-f88b-4aed-8957-46d627cdfa90.png" alt="测试模式日志信息" title="测试模式日志信息">
<ul>
<li>当然你也会看到<code>javax.mail.SendFailedException: Invalid Addresses;</code>异常信息<br><strong>这里并没有开发一个新的页面，而是在局部邮件发送，填写了联系人号码（请忽略这个异常）</strong><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">javax.mail.SendFailedException: Invalid Addresses;</span><br><span class="line"> nested exception is:</span><br><span class="line"> com.sun.mail.smtp.SMTPAddressFailedException: 500 Error: bad syntax</span><br><span class="line">;</span><br><span class="line"> nested exception is:</span><br><span class="line"> com.sun.mail.smtp.SMTPAddressFailedException: 500 Error: bad syntax</span><br></pre></td></tr></table></figure></li>
</ul>
]]></content>
<summary type="html">
<p>Azkaban本身具有邮件预警功能，我们可以通过设置SLA，进行邮件失败警告或成功提示。但是邮件预警一般不能及时响应处理。（我们并不会及时查看邮件）。 为增加强应急响应，开发Azkaban 短信预警插件，及时向管理员、运维人员发出警告提示。</p>
<h2 id="短信-A
</summary>
<category term="Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/"/>
<category term="Bigdata Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/Bigdata-Workflow/"/>
<category term="Workflow" scheme="http://cmp-cc.github.io/tags/Workflow/"/>
<category term="Azkaban" scheme="http://cmp-cc.github.io/tags/Azkaban/"/>
</entry>
<entry>
<title>Gradle 使用异常处理</title>
<link href="http://cmp-cc.github.io/2016/05/30/2016/Gradle%20%E4%BD%BF%E7%94%A8%E5%BC%82%E5%B8%B8%E5%A4%84%E7%90%86/"/>
<id>http://cmp-cc.github.io/2016/05/30/2016/Gradle 使用异常处理/</id>
<published>2016-05-30T06:43:00.000Z</published>
<updated>2016-05-30T06:44:14.693Z</updated>
<content type="html"><![CDATA[<p><strong>Gradle 使用遇到的异常及其解决方案，持续更新</strong></p>
<h2 id="Xlint-unchecked"><a href="#Xlint-unchecked" class="headerlink" title="-Xlint:unchecked"></a>-Xlint:unchecked</h2><h3 id="异常描述"><a href="#异常描述" class="headerlink" title="异常描述"></a>异常描述</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">:azkaban-common:compileJava </span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()': 找不到javax.annotation.concurrent.GuardedBy的类文件</span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()'</span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()'</span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()'</span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()'</span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()'</span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()'</span><br><span class="line">C:\Users\root\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\13.0.1\d6f22b1e60a2f1ef99e22c9f5fde270b2088365\guava-13.0.1.jar(com/google/common/util/concurrent/Monitor.class): 警告: 无法找到类型 'GuardedBy' 的注释方法 'value()'</span><br><span class="line">C:\Users\root\Desktop\笔记\Azkaban\Resources\azkaban-new\azkaban-master\azkaban-common\src\main\java\azkaban\executor\ExecutorInfo.java:109: 警告: [EqualsHashCode] Classes that override equals should also override hashCode.</span><br><span class="line"> public boolean equals(Object obj)</span><br><span class="line"> ^</span><br><span class="line"> (see http://errorprone.info/bugpattern/EqualsHashCode)</span><br><span class="line">注: 某些输入文件使用了未经检查或不安全的操作。</span><br><span class="line">注: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。</span><br><span class="line">9 个警告</span><br></pre></td></tr></table></figure>
<h3 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h3><p>参考：<a href="http://stackoverflow.com/questions/18689365/how-to-add-xlintunchecked-to-my-android-gradle-based-project" target="_blank" rel="external">http://stackoverflow.com/questions/18689365/how-to-add-xlintunchecked-to-my-android-gradle-based-project</a></p>
<p>Gradle 添加如下配置，可消除Java编译警告<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">/**</span><br><span class="line"> * Gradle wrapper task.</span><br><span class="line"> * 当前的Gradle 版本</span><br><span class="line"> */</span><br><span class="line">task wrapper(type: Wrapper) {</span><br><span class="line"> gradleVersion = '2.13'</span><br><span class="line">} </span><br><span class="line"> </span><br><span class="line">allprojects {</span><br><span class="line"> gradle.projectsEvaluated {</span><br><span class="line"> tasks.withType(JavaCompile) {</span><br><span class="line"> options.compilerArgs << "-Xlint:unchecked" << "-Xlint:deprecation"</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
]]></content>
<summary type="html">
<p><strong>Gradle 使用遇到的异常及其解决方案，持续更新</strong></p>
<h2 id="Xlint-unchecked"><a href="#Xlint-unchecked" class="headerlink" title="-Xlint:unche
</summary>
<category term="Project Build" scheme="http://cmp-cc.github.io/categories/Project-Build/"/>
<category term="Gradle" scheme="http://cmp-cc.github.io/categories/Project-Build/Gradle/"/>
<category term="Gradle" scheme="http://cmp-cc.github.io/tags/Gradle/"/>
</entry>
<entry>
<title>Azkaban 插件</title>
<link href="http://cmp-cc.github.io/2016/05/29/2016/Azkaban%20%E6%8F%92%E4%BB%B6/"/>
<id>http://cmp-cc.github.io/2016/05/29/2016/Azkaban 插件/</id>
<published>2016-05-29T11:53:00.000Z</published>
<updated>2016-06-22T04:05:06.551Z</updated>
<content type="html"><![CDATA[<p><strong>Azkaban 官网只提供了2.5版本的zip，3.0 Plugin 需要手工构建，这篇文章介绍如下：</strong></p>
<ul>
<li>Azkaban 3.0 Plugin 手工构建过程</li>
<li><code>HDFS Browser Plugin</code>的安装过程</li>
</ul>
<h2 id="编译Azkaban-Plugins"><a href="#编译Azkaban-Plugins" class="headerlink" title="编译Azkaban Plugins"></a>编译<code>Azkaban Plugins</code></h2><p><strong>推荐在Linux环境下进行构建</strong></p>
<ul>
<li><p>安装Ant<br>Azkaban Plugins 使用Ant 构建，确保有Ant环境</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install ant-optional ant-junit.x86_64</span><br></pre></td></tr></table></figure>
</li>
<li><p>Clone Azkaban Plugins Resources</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">git clone https://github.com/azkaban/azkaban-plugins.git</span><br><span class="line">cd azkaban-plugins-master</span><br><span class="line">ant package</span><br></pre></td></tr></table></figure>
</li>
</ul>
<p><strong>构建输出</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line">[root@salt azkaban-plugins-master]# ant package</span><br><span class="line">Buildfile: build.xml</span><br><span class="line"></span><br><span class="line">createDist:</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist</span><br><span class="line"></span><br><span class="line">clean:</span><br><span class="line"> [echo] Deleting generated files in dist</span><br><span class="line"></span><br><span class="line">git.info:</span><br><span class="line"> [exec] Result: 128</span><br><span class="line"> [exec] Result: 128</span><br><span class="line"> [echo] Git tag found to be HEAD with tag from repo https://github.com/azkaban/azkaban-plugins.git</span><br><span class="line"></span><br><span class="line">repo.file:</span><br><span class="line"></span><br><span class="line">package-dep:</span><br><span class="line"></span><br><span class="line">clean:</span><br><span class="line"> [echo] Deleting generated files in dist</span><br><span class="line"></span><br><span class="line">build:</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/hadoopsecuritymanager/classes</span><br><span class="line"> [javac] Compiling 7 source files to /root/Azkaban_Plugins/azkaban-plugins-master/dist/hadoopsecuritymanager/classes</span><br><span class="line"></span><br><span class="line">jars:</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/hadoopsecuritymanager/jars</span><br><span class="line"> [jar] Building jar: /root/Azkaban_Plugins/azkaban-plugins-master/dist/hadoopsecuritymanager/jars/azkaban-hadoopsecuritymanager-.jar</span><br><span class="line">.....</span><br><span class="line">.....</span><br><span class="line"></span><br><span class="line">package-jobsummary:</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/conf</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/lib</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/extlib</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/web</span><br><span class="line"> [copy] Copying 1 file to /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/conf</span><br><span class="line"> [copy] Copying 1 file to /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/lib</span><br><span class="line"> [copy] Copying 3 files to /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/lib</span><br><span class="line"> [copy] Copying 2 files to /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/web</span><br><span class="line"> [copy] Copying 1 file to /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages</span><br><span class="line"> [tar] Building tar: /root/Azkaban_Plugins/azkaban-plugins-master/dist/jobsummary/packages/azkaban-jobsummary-.tar.gz</span><br><span class="line"></span><br><span class="line">package:</span><br><span class="line"></span><br><span class="line">package:</span><br><span class="line"> [mkdir] Created dir: /root/Azkaban_Plugins/azkaban-plugins-master/dist/packages</span><br><span class="line"> [copy] Copying 12 files to /root/Azkaban_Plugins/azkaban-plugins-master/dist/packages/viewer/hdfs</span><br><span class="line"> [copy] Copying 309 files to /root/Azkaban_Plugins/azkaban-plugins-master/dist/packages/jobtypes</span><br><span class="line"> [copy] Copying 310 files to /root/Azkaban_Plugins/azkaban-plugins-master/dist/packages/viewer/javaviewer</span><br><span class="line"> [tar] Building tar: /root/Azkaban_Plugins/azkaban-plugins-master/dist/packages/azkaban-plugins-.tar.gz</span><br><span class="line"></span><br><span class="line">BUILD SUCCESSFUL</span><br><span class="line">Total time: 1 minute 32 seconds</span><br></pre></td></tr></table></figure></p>
<ul>
<li>构建结果<br><strong>打开dist 目录查看构建结果</strong><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">cd dist</span><br><span class="line"></span><br><span class="line">drwxr-xr-x 5 root root 4096 Jun 02 16:20 hadoopsecuritymanager</span><br><span class="line">drwxr-xr-x 5 root root 4096 Jun 02 16:20 hadoopsecuritymanager-yarn</span><br><span class="line">drwxr-xr-x 7 root root 4096 Jun 02 16:20 hdfsviewer</span><br><span class="line">drwxr-xr-x 6 root root 4096 Jun 02 16:21 javaviewer</span><br><span class="line">drwxr-xr-x 5 root root 4096 Jun 02 16:21 jobsummary</span><br><span class="line">drwxr-xr-x 6 root root 4096 Jun 02 16:20 jobtype</span><br><span class="line">drwxr-xr-x 4 root root 4096 Jun 02 16:21 packages</span><br><span class="line">-rw-r--r-- 1 root root 75 Jun 02 16:20 package.version</span><br><span class="line">drwxr-xr-x 7 root root 4096 Jun 02 16:21 pigvisualizer</span><br><span class="line">drwxr-xr-x 5 root root 4096 Jun 02 16:21 reportal</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<ul>
<li>每一个文件包含一个<code>.tar.gz</code>文件，如<code>azkaban-plugins/dist/hdfsviewer/packages/azkaban-hdfs-viewer-3.0.0.tar.gz</code></li>
</ul>
</blockquote>
<ul>
<li>同样你可以只构建一个<code>Plugin</code>，如下<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd plugins/jobtype/</span><br><span class="line">ant package</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h2 id="HDFS-Browser-3-0-安装"><a href="#HDFS-Browser-3-0-安装" class="headerlink" title="HDFS Browser 3.0 安装"></a>HDFS Browser 3.0 安装</h2><p>Azkaban HDFS Browser插件允许你查看HDFS 文件系统和文件类型。<br>默认情况下 Azkaban HDFS Browser 使用启动Linux 用户冒充用户登陆 （Linux 用户 == Hadoop HDFS 用户）。</p>
<ul>
<li><p>Azkaban <code>azkaban-web-server-3.0.0/conf/azkaban.properties</code> 配置文件 配置加载<code>hdfs viewer plugin</code>:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">cd /usr/local/azkaban3.0/azkaban-web-server-3.0.0/conf/</span><br><span class="line">vi azkaban.properties</span><br><span class="line"></span><br><span class="line">// 增加如下</span><br><span class="line">viewer.plugins=hdfs</span><br></pre></td></tr></table></figure>
</li>
<li><p>将<code>viewer plugin tar</code> 文件移至<code>plugins</code>目录，并解压它，重命名为<code>hdfs</code>,移至<code>plugin/viewer</code>目录<br>plugins/viewer/hdfs/conf plugins/viewer/hdfs/lib </p>
</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">cd /usr/local/azkaban3.0/azkaban-web-server-3.0.0/plugins/</span><br><span class="line">mkdir viewer && cd viewer</span><br><span class="line">cp /root/Azkaban_Plugins/azkaban-plugins-master/dist/hdfsviewer/packages/azkaban-hdfs-viewer-3.0.0.tar.gz .</span><br><span class="line">tar zxvf azkaban-hdfs-viewer-3.0.0.tar.gz</span><br><span class="line">mv azkaban-hdfs-viewer-3.0.0 hdfs</span><br><span class="line">rm -f azkaban-hdfs-viewer-3.0.0.tar.gz</span><br></pre></td></tr></table></figure>
<ul>
<li><p>配置<code>conf/plugin.properites</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">viewer.name=HDFS</span><br><span class="line">viewer.path=hdfs</span><br><span class="line">viewer.order=1</span><br><span class="line">viewer.hidden=false</span><br><span class="line">viewer.external.classpaths=extlib/</span><br><span class="line">viewer.servlet.class=azkaban.viewer.hdfs.HdfsBrowserServlet</span><br><span class="line"></span><br><span class="line"># 这里使用hadoop2.x </span><br><span class="line"></span><br><span class="line">hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0</span><br><span class="line">azkaban.should.proxy=false</span><br><span class="line">proxy.user=azkaban</span><br><span class="line">proxy.keytab.location=</span><br><span class="line">allow.group.proxy=false</span><br><span class="line">file.max.lines=1000</span><br></pre></td></tr></table></figure>
</li>
<li><p>将Hadoop 2.x 相关Jar 放到<code>extlib</code>目录中</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">// 当前我使用的 CDH 版本</span><br><span class="line">[root@salt conf]# hadoop version</span><br><span class="line">Hadoop 2.6.0-cdh5.5.2</span><br></pre></td></tr></table></figure>
</li>
</ul>
<img src="/2016/05/29/2016/Azkaban%20插件/68b459b3-dc98-49da-a80a-848f0f8ccc66.png" alt="HDFS3.0 所需Jar" title="HDFS3.0 所需Jar">
<blockquote>
<p>原<code>extlib</code>目录<code>hadoop-core-1.0.4.jar</code> 删除或者重命名<code>mv hadoop-core-1.0.4.jar hadoop-core-1.0.4.jar.bak</code></p>
</blockquote>
<ul>
<li><code>Hadoop 2.x Securiy</code> 依赖 (这个是官方项目构建的Bug)<br><strong>将构建文件中<code>azkaban-plugins-master/dist/hadoopsecuritymanager-yarn/jars/azkaban-hadoopsecuritymanageryarn-3.0.0.jar</code> 加入到<code>lib</code>目录下</strong><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cp /root/Azkaban_Plugins/azkaban-plugins-master/dist/hadoopsecuritymanager-yarn/jars/azkaban-hadoopsecuritymanageryarn-3.0.0.jar hdfs/lib</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<p>推荐将<code>plugin extlib</code>中的jar移至<code>azkaban-web/extlib</code>目录中,Hadoop 环境检查成功可能会找到不到<code>plugins extlib</code>目录</p>
</blockquote>
<ul>
<li>配置 Hadoop<br>确保当前环境存在Hadoop环境变量<code>HADOOP_HOME</code>和<code>HADOOP_CONF_DIR</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/profile</span><br><span class="line"></span><br><span class="line">// 添加如下</span><br><span class="line">export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4</span><br><span class="line">export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/conf</span><br></pre></td></tr></table></figure>
</li>
</ul>
<p>或者<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">vi hdfs/conf/plugin.properties</span><br><span class="line"></span><br><span class="line">// 添加类似如下</span><br><span class="line">hadoop.home=/etc/hadoop</span><br><span class="line">hadoop.conf.dir=/etc/hadoop/conf</span><br></pre></td></tr></table></figure></p>
<blockquote>
<p>如果当前环境已经配置，无需在配置</p>
</blockquote>
<h2 id="HDFS-Browser-2-5-安装"><a href="#HDFS-Browser-2-5-安装" class="headerlink" title="HDFS Browser 2.5 安装"></a>HDFS Browser 2.5 安装</h2><p>如果你安装3.0失败，可以安装2.5版本。 显示效果一模一样</p>
<p> 将Hadoop 2.x 相关Jar 放到<code>extlib</code>目录中。 （不像3.0，如下即可）</p>
<img src="/2016/05/29/2016/Azkaban%20插件/11f40929-8ecb-4f67-9e0c-6795f3f32c4d.png" alt="HDFS2.5 所需Jar" title="HDFS2.5 所需Jar">
<p><strong>配置成功</strong></p>
<img src="/2016/05/29/2016/Azkaban%20插件/97579ced-061a-4780-9536-223bd9a08c2a.png" alt="配置成功" title="配置成功">
<h2 id="异常处理"><a href="#异常处理" class="headerlink" title="异常处理"></a>异常处理</h2><h3 id="Cannot-get-FileSystem"><a href="#Cannot-get-FileSystem" class="headerlink" title="Cannot get FileSystem"></a>Cannot get FileSystem</h3><h4 id="异常信息"><a href="#异常信息" class="headerlink" title="异常信息"></a>异常信息</h4><img src="/2016/05/29/2016/Azkaban%20插件/8b9a38f6-5004-4240-a015-2d55749c3d9a.png" alt="错误信息" title="错误信息">
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">ity.HadoopSecurityManager_H_2_0java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not found</span><br><span class="line">java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not found</span><br><span class="line"> at azkaban.viewer.hdfs.HdfsBrowserServlet.loadHadoopSecurityManager(HdfsBrowserServlet.java:<span class="number">138</span>)</span><br><span class="line"> at azkaban.viewer.hdfs.HdfsBrowserServlet.init(HdfsBrowserServlet.java:<span class="number">99</span>)</span><br><span class="line"> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:<span class="number">440</span>)</span><br><span class="line"> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:<span class="number">263</span>)</span><br><span class="line"> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:<span class="number">50</span>)</span><br><span class="line"> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:<span class="number">736</span>)</span><br><span class="line"> at org.mortbay.jetty.servlet.Context.startContext(Context.java:<span class="number">140</span>)</span><br><span class="line"> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:<span class="number">518</span>)</span><br><span class="line"> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:<span class="number">50</span>)</span><br></pre></td></tr></table></figure>
<h4 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h4><ul>
<li>没有找到 <code>plugin extlib jar</code> ，你可以将<code>extlib</code> 文件放置到<code>lib</code>目录或<code>azkaban/extlib</code>目录中 </li>
</ul>
<h3 id="HTTP-ERROR-500"><a href="#HTTP-ERROR-500" class="headerlink" title="HTTP ERROR 500"></a>HTTP ERROR 500</h3><p>点击Browser HDFS，返回服务器内部错误</p>
<h4 id="异常信息-1"><a href="#异常信息-1" class="headerlink" title="异常信息"></a>异常信息</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">HTTP ERROR 500</span><br><span class="line">Problem accessing /hdfs. Reason:</span><br><span class="line"> INTERNAL_SERVER_ERROR</span><br><span class="line">Caused by:</span><br><span class="line"></span><br><span class="line">java.lang.AbstractMethodError</span><br><span class="line"> at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:121)</span><br><span class="line"> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)</span><br><span class="line"> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)</span><br><span class="line"> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)</span><br><span class="line"> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)</span><br><span class="line"> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)</span><br><span class="line"> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)</span><br><span class="line"> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)</span><br><span class="line"> at org.mortbay.jetty.Server.handle(Server.java:326)</span><br><span class="line"> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)</span><br><span class="line"> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)</span><br><span class="line"> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)</span><br><span class="line"> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)</span><br><span class="line"> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)</span><br><span class="line"> at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)</span><br><span class="line"> at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)</span><br><span class="line"> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)</span><br></pre></td></tr></table></figure>
<h4 id="解决方案-1"><a href="#解决方案-1" class="headerlink" title="解决方案"></a>解决方案</h4><p>参考日志你会发现： <code>[HadoopSecurityManager] [Azkaban] DFS name file:///</code> <strong><code>HADOOP_HOME</code>和<code>HADOOP_CONF_DIR</code>配置失败，请检查</strong></p>
]]></content>
<summary type="html">
<p><strong>Azkaban 官网只提供了2.5版本的zip，3.0 Plugin 需要手工构建，这篇文章介绍如下：</strong></p>
<ul>
<li>Azkaban 3.0 Plugin 手工构建过程</li>
<li><code>HDFS Browser P
</summary>
<category term="Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/"/>
<category term="Bigdata Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/Bigdata-Workflow/"/>
<category term="Workflow" scheme="http://cmp-cc.github.io/tags/Workflow/"/>
<category term="Azkaban" scheme="http://cmp-cc.github.io/tags/Azkaban/"/>
</entry>
<entry>
<title>Azkaban 工作流升级3.0</title>
<link href="http://cmp-cc.github.io/2016/05/27/2016/Azkaban%20%E5%B7%A5%E4%BD%9C%E6%B5%81%E5%8D%87%E7%BA%A73.0/"/>
<id>http://cmp-cc.github.io/2016/05/27/2016/Azkaban 工作流升级3.0/</id>
<published>2016-05-27T06:06:00.000Z</published>
<updated>2016-06-03T03:49:06.577Z</updated>
<content type="html"><![CDATA[<p>前一段时间安装部署Azkaban 2.5 版本，数据运维的童鞋在使用中，确实解决了脚本工作流（依赖）、调度、监控等问题。</p>
<ul>
<li>但是Azkaban还是存在很多问题和不足<ul>
<li>不支持远程脚本调度（必须在固定的机器、没有远程登陆）</li>
<li>创建项目不支持中文 （这个Bug一直没有修复）</li>
<li>日志使用高级查询，通过Filter过滤时间，会出现时间格式转换异常。</li>
<li>等等</li>
</ul>
</li>
</ul>
<p><strong>Azkaban 3.0 虽然没有正式推出，但可以投入使用，也希望解决一下简单问题</strong></p>
<p>Azkaban 3.0 目前只能源码安装。</p>
<ul>
<li>升级Azkaban 3.0 分为如下三个步骤<ul>
<li>构建Azkaban源码，生成Zip或者Gz包</li>
<li>导入Azkaban3.0 数据库</li>
<li>配置启动<code>azkaban-web-server</code> 和 <code>azkaban-exec-server</code></li>
</ul>
</li>
</ul>
<h2 id="构建Azkaban-源码"><a href="#构建Azkaban-源码" class="headerlink" title="构建Azkaban 源码"></a>构建Azkaban 源码</h2><p>你可以选择如下方式的一种</p>
<h3 id="一个新环境下构建"><a href="#一个新环境下构建" class="headerlink" title="一个新环境下构建"></a>一个新环境下构建</h3><p><strong>安装Linux 基础工具、Java环境、Git</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo yum groupinstall -y "Development Tools"</span><br><span class="line">sudo yum install -y java-1.7.0-openjdk.x86_64 java-1.7.0-openjdk-devel.x86_64 git mysql-connector-java</span><br></pre></td></tr></table></figure></p>
<blockquote>
<p>如果有<code>Java环境 和Git</code>忽略如上</p>
</blockquote>
<p><strong>编译azkaban</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">git clone https://github.com/azkaban/azkaban.git</span><br><span class="line">cd azkaban</span><br><span class="line">./gradlew distZip</span><br></pre></td></tr></table></figure></p>
<blockquote>
<p><code>distZip</code> 构建zip文件。如果使用<code>distTar</code>表示生成tar文件。 </p>
</blockquote>
<hr>
<h3 id="可能遇到的异常"><a href="#可能遇到的异常" class="headerlink" title="可能遇到的异常"></a>可能遇到的异常</h3><p><strong>你需要多少执行，甚至需要手工编译</strong></p>
<ul>
<li>无法更新依赖<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">FAILURE: Build failed with an exception.</span><br><span class="line"></span><br><span class="line">* What went wrong:</span><br><span class="line">A problem occurred configuring root project 'azkaban'.</span><br><span class="line">> Could not resolve all dependencies for configuration ':classpath'.</span><br><span class="line"> > Could not resolve net.ltgt.gradle:gradle-errorprone-plugin:0.0.8.</span><br><span class="line"> Required by:</span><br><span class="line"> com.linkedin:azkaban:3.0.0</span><br><span class="line"> > Could not GET 'https://plugins.gradle.org/m2/net/ltgt/gradle/gradle-errorprone-plugin/0.0.8/gradle-errorprone-plugin-0.0.8.pom'.</span><br><span class="line"> > peer not authenticated</span><br><span class="line"></span><br><span class="line">* Try:</span><br><span class="line">Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.</span><br><span class="line"></span><br><span class="line">BUILD FAILED</span><br><span class="line"></span><br><span class="line">Total time: 45.497 secs</span><br><span class="line">[root@test-liupengwei azkaban]# vi build.gradle</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<p>如上是网络的原因，无法到<code>https://plugins.gradle.org/m2/</code>下载Gradle 插件</p>
</blockquote>
<ul>
<li>构建失败<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"> (see http://errorprone.info/bugpattern/WaitNotInLoop)</span><br><span class="line">/root/build2/azkaban-3.0.0/azkaban-execserver/src/test/java/azkaban/execapp/event/BlockingStatusTest.java:86: warning: [WaitNotInLoop] Because of spurious wakeups, wait(long) must always be called in a loop</span><br><span class="line"> wait(3000);</span><br><span class="line"> ^</span><br><span class="line"> (see http://errorprone.info/bugpattern/WaitNotInLoop)</span><br><span class="line">/root/build2/azkaban-3.0.0/azkaban-execserver/src/test/java/azkaban/execapp/event/BlockingStatusTest.java:92: warning: [WaitNotInLoop] Because of spurious wakeups, wait(long) must always be called in a loop</span><br><span class="line"> wait(1000);</span><br><span class="line"> ^</span><br><span class="line"> (see http://errorprone.info/bugpattern/WaitNotInLoop)</span><br><span class="line">/root/build2/azkaban-3.0.0/azkaban-execserver/src/test/java/azkaban/execapp/event/BlockingStatusTest.java:111: warning: [WaitNotInLoop] Because of spurious wakeups, wait(long) must always be called in a loop</span><br><span class="line"> wait(2000);</span><br><span class="line"> ^</span><br><span class="line"> (see http://errorprone.info/bugpattern/WaitNotInLoop)</span><br><span class="line">/root/build2/azkaban-3.0.0/azkaban-execserver/src/test/java/azkaban/execapp/event/BlockingStatusTest.java:118: warning: [WaitNotInLoop] Because of spurious wakeups, wait(long) must always be called in a loop</span><br><span class="line"> wait(2000);</span><br><span class="line"> ^</span><br><span class="line"> (see http://errorprone.info/bugpattern/WaitNotInLoop)</span><br><span class="line">16 warnings</span><br><span class="line">:azkaban-execserver:processTestResources UP-TO-DATE</span><br><span class="line">:azkaban-execserver:testClasses</span><br><span class="line">:azkaban-execserver:test</span><br><span class="line"></span><br><span class="line">azkaban.execapp.event.BlockingStatusTest > testUnfinishedBlockSeveralChanges FAILED</span><br><span class="line"> java.lang.AssertionError at BlockingStatusTest.java:100</span><br><span class="line"></span><br><span class="line">71 tests completed, 1 failed, 40 skipped</span><br><span class="line">:azkaban-execserver:test FAILED</span><br><span class="line"></span><br><span class="line">FAILURE: Build failed with an exception.</span><br><span class="line"></span><br><span class="line">* What went wrong:</span><br><span class="line">Execution failed for task ':azkaban-execserver:test'.</span><br><span class="line">> There were failing tests. See the report at: file:///root/build2/azkaban-3.0.0/azkaban-execserver/build/reports/tests/index.html</span><br><span class="line"></span><br><span class="line">* Try:</span><br><span class="line">Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.</span><br><span class="line"></span><br><span class="line">BUILD FAILED</span><br><span class="line"></span><br><span class="line">Total time: 15 mins 44.721 secs</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<p>如上异常是因为构建过程中部分单元测试无法通过。</p>
</blockquote>
<h3 id="手工-构建（Gradle）Azkaban"><a href="#手工-构建（Gradle）Azkaban" class="headerlink" title="手工 构建（Gradle）Azkaban"></a>手工 构建（Gradle）Azkaban</h3><ul>
<li><p>下载Gradle，并配置环境变量</p>
<ul>
<li><a href="https://services.gradle.org/distributions/gradle-2.13-all.zip" target="_blank" rel="external">https://services.gradle.org/distributions/gradle-2.13-all.zip</a></li>
<li>环境path 下添加bin路径</li>
</ul>
</li>
<li><p>修改build.gradle Maven数据源 （可选）<br><strong>如果下载依赖缓慢可配置</strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">buildscript {</span><br><span class="line"> repositories {</span><br><span class="line"> /**</span><br><span class="line"> * 如下增加 jcenter() 源 或 oschain源</span><br><span class="line"> */</span><br><span class="line"> jcenter() </span><br><span class="line"> mavenCentral()</span><br><span class="line"> maven {</span><br><span class="line"> url 'https://plugins.gradle.org/m2/'</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> dependencies {</span><br><span class="line"> classpath 'com.linkedin:gradle-dustjs-plugin:1.0.0'</span><br><span class="line"> classpath 'de.obqo.gradle:gradle-lesscss-plugin:1.0-1.3.3'</span><br><span class="line"> classpath 'net.ltgt.gradle:gradle-errorprone-plugin:0.0.8'</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">apply plugin: 'distribution'</span><br><span class="line"></span><br><span class="line">allprojects {</span><br><span class="line"> repositories {</span><br><span class="line"> /**</span><br><span class="line"> * 如下增加 jcenter() 源 或 oschain源</span><br><span class="line"> */</span><br><span class="line"> jcenter()</span><br><span class="line"> mavenCentral()</span><br><span class="line"> mavenLocal()</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
</li>
<li><p>忽略部分单元测试<br><strong>有些Unit命令无法执行,并不推荐在Window中进行Azkaban源码构建</strong></p>
<ul>
<li><p>修改<code>azkaban\azkaban-common\src\test\java\azkaban\jobExecutor\ProcessJobTest.java</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">/*</span><br><span class="line"> @Test</span><br><span class="line"> public void testFailedUnixCommand() throws Exception {</span><br><span class="line"> // Initialize the Props</span><br><span class="line"> props.put(ProcessJob.COMMAND, "xls -al");</span><br><span class="line"></span><br><span class="line"> try {</span><br><span class="line"> job.run();</span><br><span class="line"> } catch (RuntimeException e) {</span><br><span class="line"> Assert.assertTrue(true);</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">*/</span><br><span class="line">/* @Test</span><br><span class="line"> public void testMultipleUnixCommands() throws Exception {</span><br><span class="line"> // Initialize the Props</span><br><span class="line"> props.put(ProcessJob.COMMAND, "pwd"); //window 没有pwd命令</span><br><span class="line"> props.put("command.1", "date");</span><br><span class="line"> props.put("command.2", "whoami");</span><br><span class="line"></span><br><span class="line"> job.run();</span><br><span class="line"> }</span><br><span class="line">*/</span><br></pre></td></tr></table></figure>
</li>
<li><p>修改<code>azkaban\azkaban-common\src\test\java\azkaban\jobExecutor\PythonJobTest.java</code><br><strong>Window 构建修改，Linux构建不需要修改</strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"> @BeforeClass</span><br><span class="line">public static void init() {</span><br><span class="line"></span><br><span class="line"> long time = (new Date()).getTime();</span><br><span class="line"> scriptFile = "d:/tmp/azkaban_python" + time + ".py"; //选择一个window盘符</span><br><span class="line"> // dump script file</span><br><span class="line"> try {</span><br><span class="line"> Utils.dumpFile(scriptFile, scriptContent);</span><br><span class="line"> } catch (IOException e) {</span><br><span class="line"> e.printStackTrace(System.err);</span><br><span class="line"> Assert.fail("error in creating script file:" + e.getLocalizedMessage());</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
</li>
</ul>
</li>
<li><p>执行如下命令</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">gradle distZip --debug</span><br></pre></td></tr></table></figure>
</li>
<li><p>构建成功<br><strong>查看：azkaban\build\distributions目录</strong></p>
</li>
</ul>
<img src="/2016/05/27/2016/Azkaban%20工作流升级3.0/a3c3e5d8-87b1-46fc-bd60-36892b1861e1.png" alt="安装成功文件" title="安装成功文件">
<h2 id="安装Azkaban"><a href="#安装Azkaban" class="headerlink" title="安装Azkaban"></a>安装Azkaban</h2><p>关于Azkaban的安装，请参考<a href="http://cmp-cc.github.io/2016/04/27/2016/%E5%A4%A7%E6%95%B0%E6%8D%AE%E5%B7%A5%E4%BD%9C%E6%B5%81%20Azkaban%20%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA/">这篇文件：Azkaban2.5安装指南</a></p>
<ul>
<li>不同之处，你需要使用自己编译的Azkaban3.0进行配置。 数据库导入<code>azkaban.sql-3.0.0.zip</code>中<code>create-all-sql-3.0.0.sql</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">cd azkaban/build/distributions/ </span><br><span class="line">zunip azkaban-sql-3.0.0.zip</span><br><span class="line">cd azkaban-sql-3.0.0</span><br><span class="line">mysql -uroot -p -D azkaban2 < create-all-sql-3.0.0.sql</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<p>注意：azkaban2 数据库必须存在</p>
</blockquote>
<ul>
<li>同时你需要要执行<code>azkaban.sql-3.0.0.zip</code>中<code>update.active_executing_flows.3.0.sql</code>和<code>update.execution_flows.3.0.sql</code>文件,如果不执行，可能会出现如下异常<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">Exception in thread "main" azkaban.executor.ExecutorManagerException: Error fetching active flows</span><br><span class="line"> at azkaban.executor.JdbcExecutorLoader.fetchActiveFlows(JdbcExecutorLoader.java:216)</span><br><span class="line"> at azkaban.executor.ExecutorManager.loadRunningFlows(ExecutorManager.java:431)</span><br><span class="line"> at azkaban.executor.ExecutorManager.<init>(ExecutorManager.java:126)</span><br><span class="line"> at azkaban.webapp.AzkabanWebServer.loadExecutorManager(AzkabanWebServer.java:261)</span><br><span class="line"> at azkaban.webapp.AzkabanWebServer.<init>(AzkabanWebServer.java:197)</span><br><span class="line"> at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:739)</span><br><span class="line">Caused by: java.sql.SQLException: Unknown column 'ex.executor_id' in 'on clause' Query: SELECT ex.exec_id exec_id, ex.enc_type enc_type, ex.flow_data flow_data, et.host host, et.port port, ax.update_time axUpdateTime, et.id executorId, et.active executorStatus FROM execution_flows ex INNER JOIN active_executing_flows ax ON ex.exec_id = ax.exec_id INNER JOIN executors et ON ex.executor_id = et.id Parameters: []</span><br><span class="line"> at org.apache.commons.dbutils.AbstractQueryRunner.rethrow(AbstractQueryRunner.java:363)</span><br><span class="line"> at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:350)</span><br><span class="line"> at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:306)</span><br><span class="line"> at azkaban.executor.JdbcExecutorLoader.fetchActiveFlows(JdbcExecutorLoader.java:212)</span><br><span class="line"> ... 5 more</span><br></pre></td></tr></table></figure>
</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">mysql -uroot -p -D azkaban2 < update.active_executing_flows.3.0.sql</span><br><span class="line">mysql -uroot -p -D azkaban2 < update.execution_flows.3.0.sql</span><br></pre></td></tr></table></figure>
<p>其他相关步骤查看如上文章，安装过程过程大同小异。</p>
<h2 id="可能会遇到的异常"><a href="#可能会遇到的异常" class="headerlink" title="可能会遇到的异常"></a>可能会遇到的异常</h2><ul>
<li><p><strong>解压后的<code>azkaban-web-server-3.0.0</code>和<code>azkaban-exec-server-3.0.0</code>的bin目录下的<code>.sh</code> 均没有可执行权限</strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">cd azkaban-web-server-3.0.0</span><br><span class="line">chmod -R 777 bin/*.sh</span><br><span class="line"></span><br><span class="line">// exec 同理</span><br></pre></td></tr></table></figure>
</li>
<li><p><strong>执行sh出现/bin/bash^M: bad interpreter: No such file or directory</strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">// 如上原因是因为使用了window构建，linux不识别此文件</span><br><span class="line"></span><br><span class="line">vi azkaban-web-start.sh</span><br><span class="line"></span><br><span class="line">:set ff=nuix</span><br><span class="line">:wq</span><br></pre></td></tr></table></figure></li>
</ul>
]]></content>
<summary type="html">
<p>前一段时间安装部署Azkaban 2.5 版本，数据运维的童鞋在使用中，确实解决了脚本工作流（依赖）、调度、监控等问题。</p>
<ul>
<li>但是Azkaban还是存在很多问题和不足<ul>
<li>不支持远程脚本调度（必须在固定的机器、没有远程登陆）</li>
<l
</summary>
<category term="Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/"/>
<category term="Bigdata Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/Bigdata-Workflow/"/>
<category term="Workflow" scheme="http://cmp-cc.github.io/tags/Workflow/"/>
<category term="Azkaban" scheme="http://cmp-cc.github.io/tags/Azkaban/"/>
</entry>
<entry>
<title>Jython 异常及其解决方案</title>
<link href="http://cmp-cc.github.io/2016/05/18/2016/Jython%20%E5%BC%82%E5%B8%B8%E5%8F%8A%E5%85%B6%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88/"/>
<id>http://cmp-cc.github.io/2016/05/18/2016/Jython 异常及其解决方案/</id>
<published>2016-05-18T09:26:00.000Z</published>
<updated>2016-08-22T04:37:31.452Z</updated>
<content type="html"><![CDATA[<p>Jython 确实很少玩，也记录一下异常信息和解决方案把,如果以后有涉及，持续更新。</p>
<h2 id="cp0"><a href="#cp0" class="headerlink" title="cp0"></a>cp0</h2><h3 id="异常信息"><a href="#异常信息" class="headerlink" title="异常信息"></a>异常信息</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">console: Failed to install '': java.nio.charset.UnsupportedCharsetException: cp0.</span><br></pre></td></tr></table></figure>
<h3 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">eclipse.ini 设置 -Dpython.console.encoding=UTF-8</span><br><span class="line"></span><br><span class="line">或者</span><br><span class="line"></span><br><span class="line">Window -> Preferences -> Java -> installed JRES -> Edit 设置如下：</span><br><span class="line">Default VM arguments: -Dpython.console.encoding=UTF-8</span><br></pre></td></tr></table></figure>
<img src="/2016/05/18/2016/Jython%20异常及其解决方案/6c1429b8-0e58-4c24-b2c4-58e63a4b771e.png" alt="JVM 设置" title="JVM 设置">
<h2 id="No-module-named-site"><a href="#No-module-named-site" class="headerlink" title="No module named site"></a>No module named site</h2><h3 id="异常信息-1"><a href="#异常信息-1" class="headerlink" title="异常信息"></a>异常信息</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">ImportError: Cannot import site module and its dependencies: No module named site</span><br><span class="line">Determine if the following attributes are correct:</span><br><span class="line"> * sys.path: ['C:\\Users\\root\\.m2\\repository\\org\\python\\jython\\2.7.0\\Lib', '__classpath__', '__pyclasspath__/']</span><br><span class="line"> This attribute might be including the wrong directories, such as from CPython</span><br><span class="line"> * sys.prefix: C:\Users\root\.m2\repository\org\python\jython\2.7.0</span><br><span class="line"> This attribute is set by the system property python.home, although it can</span><br><span class="line"> be often automatically determined by the location of the Jython jar file</span><br><span class="line"></span><br><span class="line">You can use the -S option or python.import.site=false to not import the site module</span><br></pre></td></tr></table></figure>
<h3 id="解决方案-1"><a href="#解决方案-1" class="headerlink" title="解决方案"></a>解决方案</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">static {</span><br><span class="line"> System.setProperty("python.import.site", "false");</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<p>Jython 确实很少玩，也记录一下异常信息和解决方案把,如果以后有涉及，持续更新。</p>
<h2 id="cp0"><a href="#cp0" class="headerlink" title="cp0"></a>cp0</h2><h3 id="异常信息"><a hr
</summary>
<category term="Programming Language" scheme="http://cmp-cc.github.io/categories/Programming-Language/"/>
<category term="Jython" scheme="http://cmp-cc.github.io/categories/Programming-Language/Jython/"/>
<category term="Jython" scheme="http://cmp-cc.github.io/tags/Jython/"/>
</entry>
<entry>
<title>图片相似度搜索计算</title>
<link href="http://cmp-cc.github.io/2016/05/16/2016/%E5%9B%BE%E7%89%87%E7%9B%B8%E4%BC%BC%E5%BA%A6%E6%90%9C%E7%B4%A2/"/>
<id>http://cmp-cc.github.io/2016/05/16/2016/图片相似度搜索/</id>
<published>2016-05-15T16:00:00.000Z</published>
<updated>2016-08-18T11:30:00.177Z</updated>
<content type="html"><![CDATA[<ul>
<li>首先介绍Image相似度计算方法（PHash、AHash、DHash 等）。</li>
<li>Image相似度算法比较</li>
<li>案例：构建一个3千万图片（1T+）相似度搜索系统。（这里主要运用Elasticsearch 分布式计算分析能力）</li>
</ul>
<h2 id="PHash"><a href="#PHash" class="headerlink" title="PHash"></a>PHash</h2><h3 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h3><h3 id="算法实现"><a href="#算法实现" class="headerlink" title="算法实现"></a>算法实现</h3><p> 更多的phash应用，使用<code>http://www.phash.org/</code> 提供的Phash算法。</p>
<p>如下Phash算法为Java实现。 说明:<a href="http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html" target="_blank" rel="external">http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html</a><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br></pre></td><td class="code"><pre><span class="line">package org.freecode.image.phash;</span><br><span class="line">import java.awt.Graphics2D;</span><br><span class="line">import java.awt.color.ColorSpace;</span><br><span class="line">import java.awt.image.BufferedImage;</span><br><span class="line">import java.awt.image.ColorConvertOp;</span><br><span class="line">import java.io.File;</span><br><span class="line">import java.io.FileInputStream;</span><br><span class="line">import java.io.FileNotFoundException;</span><br><span class="line">import java.io.InputStream;</span><br><span class="line"> </span><br><span class="line">import java.math.BigInteger;</span><br><span class="line">import javax.imageio.ImageIO;</span><br><span class="line">/*</span><br><span class="line"> * pHash-like image hash.</span><br><span class="line"> * Author: Elliot Shepherd (elliot@jarofworms.com</span><br><span class="line"> * Based On: http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html</span><br><span class="line"> */</span><br><span class="line">public class ImagePHash {</span><br><span class="line"> </span><br><span class="line"> private int size = 32;</span><br><span class="line"> private int smallerSize = 8;</span><br><span class="line"> </span><br><span class="line"> public ImagePHash() {</span><br><span class="line"> initCoefficients();</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> public ImagePHash(int size, int smallerSize) {</span><br><span class="line"> this.size = size;</span><br><span class="line"> this.smallerSize = smallerSize;</span><br><span class="line"> </span><br><span class="line"> initCoefficients();</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> public static int hammingDistance(String lhs, String rhs){ </span><br><span class="line"> return new BigInteger(lhs, 16).xor(new BigInteger(rhs, 16)).bitCount();</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> public int distance(String s1, String s2) {</span><br><span class="line"> int counter = 0;</span><br><span class="line"> for (int k = 0; k < s1.length();k++) {</span><br><span class="line"> if(s1.charAt(k) != s2.charAt(k)) {</span><br><span class="line"> counter++;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> return counter;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> // Returns a 'binary string' (like. 001010111011100010) which is easy to do a hamming distance on.</span><br><span class="line"> public String getHash(InputStream is) throws Exception {</span><br><span class="line"> BufferedImage img = ImageIO.read(is);</span><br><span class="line"> </span><br><span class="line"> /* 1. Reduce size.</span><br><span class="line"> * Like Average Hash, pHash starts with a small image.</span><br><span class="line"> * However, the image is larger than 8x8; 32x32 is a good size.</span><br><span class="line"> * This is really done to simplify the DCT computation and not</span><br><span class="line"> * because it is needed to reduce the high frequencies.</span><br><span class="line"> */</span><br><span class="line"> img = resize(img, size, size);</span><br><span class="line"> </span><br><span class="line"> /* 2. Reduce color.</span><br><span class="line"> * The image is reduced to a grayscale just to further simplify</span><br><span class="line"> * the number of computations.</span><br><span class="line"> */</span><br><span class="line"> img = grayscale(img);</span><br><span class="line"> </span><br><span class="line"> double[][] vals = new double[size][size];</span><br><span class="line"> </span><br><span class="line"> for (int x = 0; x < img.getWidth(); x++) {</span><br><span class="line"> for (int y = 0; y < img.getHeight(); y++) {</span><br><span class="line"> vals[x][y] = getBlue(img, x, y);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> /* 3. Compute the DCT.</span><br><span class="line"> * The DCT separates the image into a collection of frequencies</span><br><span class="line"> * and scalars. While JPEG uses an 8x8 DCT, this algorithm uses</span><br><span class="line"> * a 32x32 DCT.</span><br><span class="line"> */</span><br><span class="line"> long start = System.currentTimeMillis();</span><br><span class="line"> double[][] dctVals = applyDCT(vals);</span><br><span class="line"> System.out.println("DCT: " + (System.currentTimeMillis() - start));</span><br><span class="line"> </span><br><span class="line"> /* 4. Reduce the DCT.</span><br><span class="line"> * This is the magic step. While the DCT is 32x32, just keep the</span><br><span class="line"> * top-left 8x8. Those represent the lowest frequencies in the</span><br><span class="line"> * picture.</span><br><span class="line"> */</span><br><span class="line"> /* 5. Compute the average value.</span><br><span class="line"> * Like the Average Hash, compute the mean DCT value (using only</span><br><span class="line"> * the 8x8 DCT low-frequency values and excluding the first term</span><br><span class="line"> * since the DC coefficient can be significantly different from</span><br><span class="line"> * the other values and will throw off the average).</span><br><span class="line"> */</span><br><span class="line"> double total = 0;</span><br><span class="line"> </span><br><span class="line"> for (int x = 0; x < smallerSize; x++) {</span><br><span class="line"> for (int y = 0; y < smallerSize; y++) {</span><br><span class="line"> total += dctVals[x][y];</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> total -= dctVals[0][0];</span><br><span class="line"> </span><br><span class="line"> double avg = total / (double) ((smallerSize * smallerSize) - 1);</span><br><span class="line"> </span><br><span class="line"> /* 6. Further reduce the DCT.</span><br><span class="line"> * This is the magic step. Set the 64 hash bits to 0 or 1</span><br><span class="line"> * depending on whether each of the 64 DCT values is above or</span><br><span class="line"> * below the average value. The result doesn't tell us the</span><br><span class="line"> * actual low frequencies; it just tells us the very-rough</span><br><span class="line"> * relative scale of the frequencies to the mean. The result</span><br><span class="line"> * will not vary as long as the overall structure of the image</span><br><span class="line"> * remains the same; this can survive gamma and color histogram</span><br><span class="line"> * adjustments without a problem.</span><br><span class="line"> */</span><br><span class="line"> String hash = "";</span><br><span class="line"> </span><br><span class="line"> for (int x = 0; x < smallerSize; x++) {</span><br><span class="line"> for (int y = 0; y < smallerSize; y++) {</span><br><span class="line"> if (x != 0 && y != 0) {</span><br><span class="line"> hash += (dctVals[x][y] > avg?"1":"0");</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> return hash;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> private BufferedImage resize(BufferedImage image, int width, int height) {</span><br><span class="line"> BufferedImage resizedImage = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);</span><br><span class="line"> Graphics2D g = resizedImage.createGraphics();</span><br><span class="line"> g.drawImage(image, 0, 0, width, height, null);</span><br><span class="line"> g.dispose();</span><br><span class="line"> return resizedImage;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> private ColorConvertOp colorConvert = new ColorConvertOp(ColorSpace.getInstance(ColorSpace.CS_GRAY), null);</span><br><span class="line"> </span><br><span class="line"> private BufferedImage grayscale(BufferedImage img) {</span><br><span class="line"> colorConvert.filter(img, img);</span><br><span class="line"> return img;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> private static int getBlue(BufferedImage img, int x, int y) {</span><br><span class="line"> return (img.getRGB(x, y)) & 0xff;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> // DCT function stolen from http://stackoverflow.com/questions/4240490/problems-with-dct-and-idct-algorithm-in-java</span><br><span class="line"> </span><br><span class="line"> private double[] c;</span><br><span class="line"> private void initCoefficients() {</span><br><span class="line"> c = new double[size];</span><br><span class="line"> </span><br><span class="line"> for (int i=1;i<size;i++) {</span><br><span class="line"> c[i]=1;</span><br><span class="line"> }</span><br><span class="line"> c[0]=1/Math.sqrt(2.0);</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> private double[][] applyDCT(double[][] f) {</span><br><span class="line"> int N = size;</span><br><span class="line"> </span><br><span class="line"> double[][] F = new double[N][N];</span><br><span class="line"> for (int u=0;u<N;u++) {</span><br><span class="line"> for (int v=0;v<N;v++) {</span><br><span class="line"> double sum = 0.0;</span><br><span class="line"> for (int i=0;i<N;i++) {</span><br><span class="line"> for (int j=0;j<N;j++) {</span><br><span class="line"> sum+=Math.cos(((2*i+1)/(2.0*N))*u*Math.PI)*Math.cos(((2*j+1)/(2.0*N))*v*Math.PI)*(f[i][j]);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> sum*=((c[u]*c[v])/4.0);</span><br><span class="line"> F[u][v] = sum;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> return F;</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> </span><br><span class="line"> public static void main(String[] args) throws FileNotFoundException, Exception {</span><br><span class="line"> ImagePHash p = new ImagePHash();</span><br><span class="line"> </span><br><span class="line"> String image1 = p.getHash(new FileInputStream(new File("F:/cc/DEF3A6A96A4012ED042AB88DC4DA24B4.png")));</span><br><span class="line"> String image2 = p.getHash(new FileInputStream(new File("F:/cc/F16FE1485F11302B5F196B04F7D9BA5B.png")));</span><br><span class="line"> System.out.println(image1+"\t"+image2);</span><br><span class="line"> </span><br><span class="line"> System.out.println("Score is " + p.distance(image1, image2));</span><br><span class="line"> System.out.println("Score is " + ImagePHash.hammingDistance(image1, image2));</span><br><span class="line"> </span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<h2 id="暂略。"><a href="#暂略。" class="headerlink" title="暂略。"></a>暂略。</h2><p>最近没时间。</p>
<p>构建Image相似度搜索系统步骤如下：</p>
<ul>
<li>通过图像相似度算法，离线将图像（MD5、相似度Hash值、等信息）存储到Elasticsearch中。线下保证增量数据增加。</li>
<li>开启Elasticsearch <code>script.disable_dynamic: false</code> 。 编写Elasticsearch 嵌入式脚本，用于计算<code>海明距离</code>。</li>
<li>将<code>海明距离</code>结果交给<code>score</code>值。</li>
<li>写相关图像相似度搜索APP。 （通过上诉步骤后，可以指定搜索阈值，并输入一张图片，计算当前图片相似度Hash值到Elasticsearch进行搜索，吧啦吧啦）</li>
</ul>
]]></content>
<summary type="html">
<ul>
<li>首先介绍Image相似度计算方法（PHash、AHash、DHash 等）。</li>
<li>Image相似度算法比较</li>
<li>案例：构建一个3千万图片（1T+）相似度搜索系统。（这里主要运用Elasticsearch 分布式计算分析能力）</li>
</summary>
<category term="ImageProcessing" scheme="http://cmp-cc.github.io/categories/ImageProcessing/"/>
<category term="图像处理" scheme="http://cmp-cc.github.io/tags/%E5%9B%BE%E5%83%8F%E5%A4%84%E7%90%86/"/>
</entry>
<entry>
<title>使用国内yum源</title>
<link href="http://cmp-cc.github.io/2016/05/04/2016/%E4%BD%BF%E7%94%A8%E5%9B%BD%E5%86%85yum%E6%BA%90/"/>
<id>http://cmp-cc.github.io/2016/05/04/2016/使用国内yum源/</id>
<published>2016-05-04T08:07:00.000Z</published>
<updated>2016-05-04T08:16:21.740Z</updated>
<content type="html"><![CDATA[<p><strong>国内yum源列表 Centos6</strong></p>
<table>
<thead>
<tr>
<th>来源</th>
<th>地址</th>
</tr>
</thead>
<tbody>
<tr>
<td>中科大</td>
<td><a href="http://centos.ustc.edu.cn/CentOS-Base.repo" target="_blank" rel="external">http://centos.ustc.edu.cn/CentOS-Base.repo</a></td>
</tr>
<tr>
<td>搜狐</td>
<td><a href="http://mirrors.sohu.com/help/CentOS-Base-sohu.repo" target="_blank" rel="external">http://mirrors.sohu.com/help/CentOS-Base-sohu.repo</a></td>
</tr>
<tr>
<td>网易</td>
<td><a href="http://mirrors.163.com/.help/CentOS6-Base-163.repo" target="_blank" rel="external">http://mirrors.163.com/.help/CentOS6-Base-163.repo</a></td>
</tr>
<tr>
<td>阿里云</td>
<td><a href="http://mirrors.aliyun.com/repo/Centos-6.repo" target="_blank" rel="external">http://mirrors.aliyun.com/repo/Centos-6.repo</a></td>
</tr>
</tbody>
</table>
<h2 id="使用国内yum"><a href="#使用国内yum" class="headerlink" title="使用国内yum"></a>使用国内yum</h2><p>系统配置yum的目录为<code>/etc/yum.repos.d</code></p>
<ul>
<li><p>备份系统yum配置文件</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd /etc/yum.repos.d</span><br><span class="line">mv CentOS-Base.repo CentOS-Base.repo.bk</span><br></pre></td></tr></table></figure>
</li>
<li><p>下载相关yum源文件（这里使用阿里云）</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo</span><br></pre></td></tr></table></figure>
</li>
<li><p>更新yum配置</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum clean all</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h2 id="使用yum-fastestmirror"><a href="#使用yum-fastestmirror" class="headerlink" title="使用yum-fastestmirror"></a>使用yum-fastestmirror</h2><p>每次用yum安装，yum-fastestmirror就会自动检查速度最快的镜像。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">yum install yum-fastestmirror</span><br><span class="line">yum clean all</span><br></pre></td></tr></table></figure>
<blockquote>
<p>通过<code>yum --disableplugin=fastestmirror update</code> 禁用此插件</p>
</blockquote>
<h2 id="被数落了"><a href="#被数落了" class="headerlink" title="被数落了~"></a>被数落了~</h2><p><strong>公司原来有开放的yum源，我还傻乎乎的使用别人家的。</strong><br><strong>Antiy: <a href="http://mirrors.avlyun.org/" target="_blank" rel="external">http://mirrors.avlyun.org/</a> 提供各种系统的yum镜像，欢迎使用。</strong></p>
<ul>
<li>修改为自家的东西<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">curl http://mirrors.avlyun.org/repo/centos6.repo > /etc/yum.repos.d/CentOS-Base.repo</span><br><span class="line">yum makecache</span><br></pre></td></tr></table></figure>
</li>
</ul>
<blockquote>
<p>我并没有使用yum-fastestmirror，自家的东东快的多。</p>
</blockquote>
]]></content>
<summary type="html">
<p><strong>国内yum源列表 Centos6</strong></p>
<table>
<thead>
<tr>
<th>来源</th>
<th>地址</th>
</tr>
</thead>
<tbody>
<tr>
<td>中科大</td>
<td><a href="
</summary>
<category term="Operating System" scheme="http://cmp-cc.github.io/categories/Operating-System/"/>
<category term="Linux" scheme="http://cmp-cc.github.io/categories/Operating-System/Linux/"/>
<category term="Linux" scheme="http://cmp-cc.github.io/tags/Linux/"/>
</entry>
<entry>
<title>大数据工作流 Azkaban 环境搭建</title>
<link href="http://cmp-cc.github.io/2016/04/27/2016/%E5%A4%A7%E6%95%B0%E6%8D%AE%E5%B7%A5%E4%BD%9C%E6%B5%81%20Azkaban%20%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA/"/>
<id>http://cmp-cc.github.io/2016/04/27/2016/大数据工作流 Azkaban 环境搭建/</id>
<published>2016-04-27T12:55:00.000Z</published>
<updated>2016-07-29T01:03:26.237Z</updated>
<content type="html"><![CDATA[<p>大数据运维的同学Python、Shell脚本泛滥，且依赖混乱，不易管理。 需要符合自身要求的任务调度系统，来减轻痛苦~~。</p>
<p>之前调研了Oozie、AzkaBan、Airflow、Luigi 评估了一下诉求，选择AzkaBan。</p>
<p>这篇文章将演示如何在一个全新的Linux(CentOS)环境安装Azkaban 。</p>
<h2 id="Azkaban-介绍"><a href="#Azkaban-介绍" class="headerlink" title="Azkaban 介绍"></a>Azkaban 介绍</h2><p>LinkedIn/Azkaban 为解决Hadoop的任务依赖问题。</p>
<ul>
<li>它有三个重要组件：<ul>
<li>关系数据库（目前仅支持mysql）</li>
<li>web管理服务器－AzkabanWebServer</li>
<li>执行服务器－AzkabanExecutorServer</li>
</ul>
</li>
</ul>
<img src="/2016/04/27/2016/大数据工作流%20Azkaban%20环境搭建/d82665b7-7a34-4272-ac43-6beaf6546f68.png" alt="azkaban组件" title="azkaban组件">
<h3 id="Mysql"><a href="#Mysql" class="headerlink" title="Mysql"></a>Mysql</h3><p>Azkaban使用MySQL来存储它的状态信息，Azkaban Executor Server和Azkaban Web Server均使用到了MySQL数据库。</p>
<ul>
<li><p>AzkabanExecutorServer在如下几个方面使用到了数据库：<br>1、获取project的信息<br>2、执行工作流<br>3、存储工作流运行日志</p>
</li>
<li><p>AzkabanWebServer在如下几个方面使用到了数据库：<br>1、Project管理<br>2、跟踪工作流执行进度<br>3、访问历史工作流的运行信息<br>4、定时执行工作流任务</p>
</li>
</ul>
<h3 id="AzkabanWebServer"><a href="#AzkabanWebServer" class="headerlink" title="AzkabanWebServer"></a>AzkabanWebServer</h3><p>AzkabanWebserver是整个Azkaban工作流系统的主要管理者，它负责project管理、用户登录认证、定时执行工作流、跟踪工作流执 行进度等一系列任务。同时，它还提供Web服务操作的接口，利用该接口，用户可以使用curl或其他ajax的方式，来执行azkaban的相关操作。操 作包括：用户登录、创建project、上传workflow、执行workflow、查询workflow的执行进度、杀掉workflow等一系列操 作，且这些操作的返回结果均是json的格式。</p>
<h3 id="AzkabanExecutorServer"><a href="#AzkabanExecutorServer" class="headerlink" title="AzkabanExecutorServer"></a>AzkabanExecutorServer</h3><p>之所以将AzkabanWebServer和AzkabanExecutorServer分开，主要是因为在某个任务流失败后，可以更方便的将重新执行。而且也更有利于Azkaban系统的升级。</p>
<h2 id="Azkaban-开始"><a href="#Azkaban-开始" class="headerlink" title="Azkaban 开始"></a>Azkaban 开始</h2><ul>
<li>solo server mode：<br><strong>最简单的模式，数据库内置的H2数据库，且webServer和executorServer运行在同一个进程中，没有单独分开。该模式适用于小规模的使用。</strong></li>
<li>two server mode：<br><strong>该模式使用MySQL数据库，webServer和executorServer运行在不同进程中，webServer和executorServer互不影响，该模式适用于大规模应用。</strong></li>
<li>multiple executor mode：<br><strong>该模式下，执行服务器和管理服务器在不同主机上，且执行服务器可以有多</strong></li>
</ul>
<h2 id="Azkaban-安装"><a href="#Azkaban-安装" class="headerlink" title="Azkaban 安装"></a>Azkaban 安装</h2><p>Azkaban2 仅仅只用Mysql 作为数据存储</p>
<h3 id="安装JDK1-8"><a href="#安装JDK1-8" class="headerlink" title="安装JDK1.8"></a>安装JDK1.8</h3><p><a href="http://cmp-cc.github.io/2014/11/07/2014/Linux%20%E5%BC%80%E5%8F%91%E7%8E%AF%E5%A2%83%E5%AE%89%E8%A3%85%E6%8C%87%E5%8D%97/">参考安装JDK1.8 安装</a></p>
<h3 id="安装Mysql"><a href="#安装Mysql" class="headerlink" title="安装Mysql"></a>安装Mysql</h3><p><a href="http://cmp-cc.github.io/2014/11/07/2014/Linux%20%E5%BC%80%E5%8F%91%E7%8E%AF%E5%A2%83%E5%AE%89%E8%A3%85%E6%8C%87%E5%8D%97/">参考安装Mysql5.7.11 </a></p>
<h4 id="创建数据库"><a href="#创建数据库" class="headerlink" title="创建数据库"></a>创建数据库</h4><h5 id="创建Azkaban数据库"><a href="#创建Azkaban数据库" class="headerlink" title="创建Azkaban数据库"></a><strong>创建Azkaban数据库</strong></h5><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">CREATE DATABASE azkaban;</span><br></pre></td></tr></table></figure>
<h5 id="创建数据库用户于Azkaban"><a href="#创建数据库用户于Azkaban" class="headerlink" title="创建数据库用户于Azkaban"></a><strong>创建数据库用户于Azkaban</strong></h5><p>如下：</p>
<ul>
<li>用户名:azkaban</li>
<li>地址：localhost</li>
<li>密码：azkaban<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">CREATE USER 'azkaban'@'localhost' IDENTIFIED BY 'azkaban' PASSWORD EXPIRE;</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h5 id="设置权限于Azkaban-用户"><a href="#设置权限于Azkaban-用户" class="headerlink" title="设置权限于Azkaban 用户"></a><strong>设置权限于Azkaban 用户</strong></h5><p><strong>这里执行如下，给予所有权限</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">grant all privileges on azkaban.* to 'azkaban'@'localhost' identified by 'azkaban';</span><br></pre></td></tr></table></figure></p>
<p><strong>也可以如下限定访问权限</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* TO 'azkaban'@'localhost' identified by 'azkaban';</span><br></pre></td></tr></table></figure></p>
<p><strong>配置数据包大小，修改属性<code>max_allowed_packet</code>值为<code>1024M</code>或者更高</strong></p>
<ul>
<li><p>编辑<code>/etc/my.cnf</code>文件，增加如下</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">vi /etc/my.cnf</span><br><span class="line"></span><br><span class="line">[mysqld]</span><br><span class="line">...</span><br><span class="line">max_allowed_packet=2048M</span><br></pre></td></tr></table></figure>
</li>
<li><p>重启 mysql</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">service mysql restart</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h5 id="创建Azkaban-表"><a href="#创建Azkaban-表" class="headerlink" title="创建Azkaban 表"></a>创建Azkaban 表</h5><ul>
<li>首先下载<a href="http://azkaban.github.io/downloads.html" target="_blank" rel="external"><code>azkaban-sql-script</code></a>压缩文件，这个档案包含在表创建脚本。</li>
<li>从下载页面选择合适版本进行下载。(这里选择最新版azkaban-sql-script-2.5.0.tar.gz)<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget –no-check-certificate https://s3.amazonaws.com/azkaban2/azkaban2/2.5.0/azkaban-sql-script-2.5.0.tar.gz</span><br></pre></td></tr></table></figure>
</li>
</ul>
<p><strong>create-all-sql 脚本包含所有单条语句,忽略所有<code>update</code>前缀的脚本</strong></p>
<ul>
<li><p>解压并导入azkaban DB</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">tar zxvf azkaban-sql-script-2.5.0.tar.gz</span><br><span class="line"></span><br><span class="line">mysql -u root -p azkaban < /user/cmp-cc/azkaban-2.5.0/create-all-sql-2.5.0.sql </span><br><span class="line"></span><br><span class="line">// 提示你输入密码即可</span><br></pre></td></tr></table></figure>
</li>
<li><p>检查是否创建成功</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">mysql> show tables;</span><br><span class="line">+------------------------+</span><br><span class="line">| Tables_in_azkaban |</span><br><span class="line">+------------------------+</span><br><span class="line">| active_executing_flows |</span><br><span class="line">| active_sla |</span><br><span class="line">| execution_flows |</span><br><span class="line">| execution_jobs |</span><br><span class="line">| execution_logs |</span><br><span class="line">| project_events |</span><br><span class="line">| project_files |</span><br><span class="line">| project_flows |</span><br><span class="line">| project_permissions |</span><br><span class="line">| project_properties |</span><br><span class="line">| project_versions |</span><br><span class="line">| projects |</span><br><span class="line">| properties |</span><br><span class="line">| schedules |</span><br><span class="line">| triggers |</span><br><span class="line">+------------------------+</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h3 id="安装-Azkaban-Web-Server"><a href="#安装-Azkaban-Web-Server" class="headerlink" title="安装 Azkaban Web Server"></a>安装 Azkaban Web Server</h3><p>Azkaban Web Server 处理项目管理、身份验证、调度和触发器的执行。</p>
<h4 id="安装-Web-Server"><a href="#安装-Web-Server" class="headerlink" title="安装 Web Server"></a>安装 Web Server</h4><ul>
<li>选择合适的版本，<a href="http://azkaban.github.io/downloads.html" target="_blank" rel="external">下载页面</a>。 </li>
<li>你也以可以<a href="https://github.com/azkaban/azkaban2" target="_blank" rel="external"><code>git clone 最新版</code></a>,查看文档，<a href="http://azkaban.github.io/azkaban/docs/latest/#building-from-source" target="_blank" rel="external">如何进行源码安装</a></li>
<li><p>这里为<code>azkaban-web-server-2.5.0.tar.gz</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget –no-check-certificate https://s3.amazonaws.com/azkaban2/azkaban2/2.5.0/azkaban-web-server-2.5.0.tar.gz</span><br></pre></td></tr></table></figure>
</li>
<li><p>解压到指定目录</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p /usr/local/azkaban/</span><br><span class="line"></span><br><span class="line">tar zxvf azkaban-web-server-2.5.0.tar.gz -C /usr/local/azkaban/</span><br></pre></td></tr></table></figure>
</li>
<li><p>相关目录描述</p>
</li>
</ul>
<table>
<thead>
<tr>
<th>Folder</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>bin</td>
<td>The scripts to start Azkaban jetty server</td>
</tr>
<tr>
<td>conf</td>
<td>The configurations for Azkaban solo server</td>
</tr>
<tr>
<td>lib</td>
<td>The jar dependencies for Azkaban</td>
</tr>
<tr>
<td>extlib</td>
<td>Additional jars that are added to extlib will be added to Azkaban’s classpath</td>
</tr>
<tr>
<td>plugins</td>
<td>the directory where plugins can be installed</td>
</tr>
<tr>
<td>web</td>
<td>The web (css, javascript, image) files for Azkaban web server.</td>
</tr>
</tbody>
</table>
<ul>
<li><code>conf</code>目录，有如下三个文件<ul>
<li>azkaban.properties - Used by Azkaban for runtime parameters</li>
<li>global.properties - Global static properties that are passed as shared properties to every workflow and job.</li>
<li>azkaban-users.xml - Used to add users and roles for authentication. This file is not used if the XmLUserManager is not set up to use it.</li>
</ul>
</li>
</ul>
<p>The <code>azkaban.properties</code> file will be the main configuration file that is necessary to setup Azkaban.</p>
<h4 id="获取SSL-KeyStore（可选）"><a href="#获取SSL-KeyStore（可选）" class="headerlink" title="获取SSL KeyStore（可选）"></a>获取SSL KeyStore（可选）</h4><p>Azkaban 使用<code>SSL socket</code> 链接器,设置SSL可以通过使用<code>https://</code> 访问页面，使其更加安全。<br><strong>SSL 并是必须的，我们可以通过<code>azkaban.properties</code>配置取消，请参考SSL配置</strong></p>
<h5 id="创建SSL-配置"><a href="#创建SSL-配置" class="headerlink" title="创建SSL 配置"></a>创建SSL 配置</h5><p><strong>这里keytool 是JDK提供，并将其密钥生成或复制于<code>/azkaban/web</code>目录</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">cd /usr/local/azkaban/azkaban-web-2.5.0/web/</span><br><span class="line"></span><br><span class="line">keytool -genkey -keystore keystore -alias keystore -keyalg RSA</span><br></pre></td></tr></table></figure></p>
<p>将有如下提示，密码长度 >= 6 (这里为:password)<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">Enter keystore password: </span><br><span class="line">Re-enter new password: </span><br><span class="line">What is your first and last name?</span><br><span class="line"> [Unknown]: azkaban</span><br><span class="line">What is the name of your organizational unit?</span><br><span class="line"> [Unknown]: Jetty</span><br><span class="line">What is the name of your organization?</span><br><span class="line"> [Unknown]: cmp-cc</span><br><span class="line">What is the name of your City or Locality?</span><br><span class="line"> [Unknown]: WuHan</span><br><span class="line">What is the name of your State or Province?</span><br><span class="line"> [Unknown]: WuHan</span><br><span class="line">What is the two-letter country code for this unit?</span><br><span class="line"> [Unknown]: 86</span><br><span class="line">Is CN=azkaban, OU=Jetty, O=cmp-cc, L=WuHan, ST=WuHan, C=86 correct?</span><br><span class="line"> [no]: yes</span><br><span class="line"></span><br><span class="line">Enter key password for <jetty-azkaban></span><br><span class="line"> (RETURN if same as keystore password): // 这里回车即可</span><br></pre></td></tr></table></figure></p>
<h4 id="配置azkaban-properties文件"><a href="#配置azkaban-properties文件" class="headerlink" title="配置azkaban.properties文件"></a>配置<code>azkaban.properties</code>文件</h4><p> 修改<code>./conf/azkaban.properties</code>文件,进行如下修改<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">vi /usr/local/azkaban/azkaban-web-2.5.0/conf/azkaban.properties</span><br></pre></td></tr></table></figure></p>
<p><strong>我的完整配置信息，邮件通知自己配置即可</strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line">#Azkaban Personalization Settings</span><br><span class="line">azkaban.name=Test</span><br><span class="line">azkaban.label=My Local Azkaban</span><br><span class="line">azkaban.color=#FF3601</span><br><span class="line">azkaban.default.servlet.path=/index</span><br><span class="line">web.resource.dir=web/</span><br><span class="line">default.timezone.id=Asia/Shanghai</span><br><span class="line"></span><br><span class="line">#Azkaban UserManager class</span><br><span class="line">user.manager.class=azkaban.user.XmlUserManager</span><br><span class="line">user.manager.xml.file=conf/azkaban-users.xml</span><br><span class="line"></span><br><span class="line">#Loader for projects</span><br><span class="line">executor.global.properties=conf/global.properties</span><br><span class="line">azkaban.project.dir=projects</span><br><span class="line"></span><br><span class="line">database.type=mysql</span><br><span class="line">mysql.port=3306</span><br><span class="line">mysql.host=localhost</span><br><span class="line">mysql.database=azkaban</span><br><span class="line">mysql.user=azkaban</span><br><span class="line">mysql.password=azkaban</span><br><span class="line">mysql.numconnections=100</span><br><span class="line"></span><br><span class="line"># Velocity dev mode</span><br><span class="line">velocity.dev.mode=false</span><br><span class="line"></span><br><span class="line"># Azkaban Jetty server properties.</span><br><span class="line">jetty.maxThreads=25</span><br><span class="line">jetty.ssl.port=8443</span><br><span class="line">jetty.port=8081</span><br><span class="line">jetty.keystore=web/keystore</span><br><span class="line">jetty.password=password</span><br><span class="line">jetty.keypassword=password</span><br><span class="line">jetty.truststore=web/keystore</span><br><span class="line">jetty.trustpassword=password</span><br><span class="line"></span><br><span class="line"># Azkaban Executor settings</span><br><span class="line">executor.port=12321</span><br><span class="line"></span><br><span class="line"># mail settings</span><br><span class="line">mail.sender=</span><br><span class="line">mail.host=</span><br><span class="line">job.failure.email=</span><br><span class="line">job.success.email=</span><br><span class="line"></span><br><span class="line">lockdown.create.projects=false</span><br><span class="line"></span><br><span class="line">cache.directory=cache</span><br></pre></td></tr></table></figure>
<h5 id="SSL-密钥配置"><a href="#SSL-密钥配置" class="headerlink" title="SSL 密钥配置"></a>SSL 密钥配置</h5><p>修改<code>jetty.xxxx</code>信息为keystore 设置，类似如下：</p>
<ul>
<li><p>如果已经生成keystore SSL文件，如下配置</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"># Azkaban Jetty server properties.</span><br><span class="line">jetty.maxThreads=25</span><br><span class="line">jetty.ssl.port=8443</span><br><span class="line">jetty.port=8081</span><br><span class="line">jetty.keystore=web/keystore</span><br><span class="line">jetty.password=password</span><br><span class="line">jetty.keypassword=password</span><br><span class="line">jetty.truststore=web/keystore</span><br><span class="line">jetty.trustpassword=password</span><br></pre></td></tr></table></figure>
</li>
<li><p>取消SSL 设置</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">jetty.maxThreads=25</span><br><span class="line">jetty.port=8081</span><br><span class="line">jetty.use.ssl=false</span><br><span class="line"></span><br><span class="line">#jetty.ssl.port=8443</span><br><span class="line">#jetty.keystore=web/keystore</span><br><span class="line">#jetty.password=keystore</span><br><span class="line">#jetty.keypassword=keystore</span><br><span class="line">#jetty.truststore=web/keystore</span><br><span class="line">#jetty.trustpassword=keystore</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h5 id="DB-设置"><a href="#DB-设置" class="headerlink" title="DB 设置"></a>DB 设置</h5><ul>
<li>将MySQL数据库链接添加至<code>extlib</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd /usr/local/azkaban/azkaban-web-2.5.0/extlib</span><br><span class="line">wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar</span><br></pre></td></tr></table></figure>
</li>
</ul>
<p>确保<code>mysql jdbc连接器</code>已经添加到<code>extlib</code>目录,配置大致如下<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">database.type=mysql</span><br><span class="line">mysql.port=3306</span><br><span class="line">mysql.host=localhost</span><br><span class="line">mysql.database=azkaban</span><br><span class="line">mysql.user=azkaban</span><br><span class="line">mysql.password=azkaban</span><br><span class="line">mysql.numconnections=100</span><br></pre></td></tr></table></figure></p>
<h5 id="UserManager-设置"><a href="#UserManager-设置" class="headerlink" title="UserManager 设置"></a>UserManager 设置</h5><p>Azkaban 使用<code>UserManager</code>支持用户认证和用户角色<br>默认：Azkaban使用<code>XmlUserManager</code>从<code>azkaban-users.xml</code>获取用户名、密码、角色信息。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">user.manager.class=azkaban.user.XmlUserManager</span><br><span class="line">user.manager.xml.file=conf/azkaban-users.xml</span><br></pre></td></tr></table></figure></p>
<h5 id="jetty运行设置"><a href="#jetty运行设置" class="headerlink" title="jetty运行设置"></a>jetty运行设置</h5><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">jetty.maxThreads=25</span><br><span class="line">jetty.ssl.port=8443</span><br></pre></td></tr></table></figure>
<h4 id="Azkaban-web-运行和关闭"><a href="#Azkaban-web-运行和关闭" class="headerlink" title="Azkaban-web 运行和关闭"></a>Azkaban-web 运行和关闭</h4><ul>
<li><p>start AzkabanWebServer.</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/azkaban-web-start.sh</span><br></pre></td></tr></table></figure>
</li>
<li><p>shutdown AzkabanWebServer</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/azkaban-web-shutdown.sh</span><br></pre></td></tr></table></figure>
</li>
<li><p>浏览器端口</p>
<ul>
<li>设置SSL=true 默认访问： <a href="https://192.168.xx.xxxx:8443" target="_blank" rel="external">https://192.168.xx.xxxx:8443</a></li>
<li>设置SSL=false 默认访问： <a href="http://192.168.xx.xxxx:8081" target="_blank" rel="external">http://192.168.xx.xxxx:8081</a></li>
</ul>
</li>
</ul>
<blockquote>
<p>SSL 是可以关闭的，azkaban.properties 文件中添加<code>jetty.use.ssl=false</code> </p>
</blockquote>
<p><strong>运行Azkaban-web 并访问浏览器验证是否成功</strong></p>
<blockquote>
<p>记得关闭防火墙，或开放8443 和 8010 端口</p>
</blockquote>
<h3 id="安装-Azkaban-Executor-Server"><a href="#安装-Azkaban-Executor-Server" class="headerlink" title="安装 Azkaban Executor Server"></a>安装 Azkaban Executor Server</h3><p><strong>Azkaban Executor Server</strong> 处理工作流和作业的实际执行。</p>
<h4 id="安装Executor-Server"><a href="#安装Executor-Server" class="headerlink" title="安装Executor Server"></a>安装Executor Server</h4><ul>
<li><a href="http://azkaban.github.io/downloads.html" target="_blank" rel="external">下载页面选择版本</a> 与 Azkaban-Web 保持一致。</li>
<li>你也以可以<a href="https://github.com/azkaban/azkaban2" target="_blank" rel="external"><code>git clone 最新版</code></a>,查看文档，<a href="http://azkaban.github.io/azkaban/docs/latest/#building-from-source" target="_blank" rel="external">如何进行源码安装</a></li>
<li><p>这里是下载<code>azkaban-executor-server-2.5.0.tar.gz</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget –no-check-certificate https://s3.amazonaws.com/azkaban2/azkaban2/2.5.0/azkaban-executor-server-2.5.0.tar.gz</span><br></pre></td></tr></table></figure>
</li>
<li><p>解压到指定目录</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tar zxvf azkaban-executor-server-2.5.0.tar.gz -C /usr/local/azkaban/</span><br></pre></td></tr></table></figure>
</li>
<li><p>相关目录描述<br>Folder | Description<br>— | —<br>bin | The scripts to start Azkaban jetty server<br>conf | The configurations for Azkaban solo server<br>lib | The jar dependencies for Azkaban<br>extlib | Additional jars that are added to extlib will be added to Azkaban’s classpath<br>plugins | the directory where plugins can be installed</p>
</li>
</ul>
<h4 id="配置azkaban-properties文件-1"><a href="#配置azkaban-properties文件-1" class="headerlink" title="配置azkaban.properties文件"></a>配置<code>azkaban.properties</code>文件</h4><p><strong>修改<code>./conf/azkaban.properties</code>文件,进行如下修改</strong></p>
<p><strong>我的配置文件</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">default.timezone.id=Asia/Shanghai</span><br><span class="line"></span><br><span class="line"># Azkaban JobTypes Plugins</span><br><span class="line">azkaban.jobtype.plugin.dir=plugins/jobtypes</span><br><span class="line"></span><br><span class="line">#Loader for projects</span><br><span class="line">executor.global.properties=conf/global.properties</span><br><span class="line">azkaban.project.dir=projects</span><br><span class="line"></span><br><span class="line">database.type=mysql</span><br><span class="line">mysql.port=3306</span><br><span class="line">mysql.host=localhost</span><br><span class="line">mysql.database=azkaban</span><br><span class="line">mysql.user=azkaban</span><br><span class="line">mysql.password=azkaban</span><br><span class="line">mysql.numconnections=100</span><br><span class="line"></span><br><span class="line"># Azkaban Executor settings</span><br><span class="line">executor.maxThreads=50</span><br><span class="line">executor.port=12321</span><br><span class="line">executor.flow.threads=30</span><br></pre></td></tr></table></figure></p>
<h5 id="设置DB"><a href="#设置DB" class="headerlink" title="设置DB"></a>设置DB</h5><ul>
<li><p>将MySQL数据库链接添加至<code>extlib</code> 可重新下载，如下</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd /usr/local/azkaban/azkaban-executor-2.5.0/extlib/</span><br><span class="line">wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar</span><br></pre></td></tr></table></figure>
</li>
<li><p>DB设置(覆盖azkaban.properties)<br><strong>如下：</strong></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">database.type=mysql</span><br><span class="line">mysql.port=3306</span><br><span class="line">mysql.host=localhost</span><br><span class="line">mysql.database=azkaban</span><br><span class="line">mysql.user=azkaban</span><br><span class="line">mysql.password=azkaban</span><br><span class="line">mysql.numconnections=100</span><br></pre></td></tr></table></figure>
</li>
</ul>
<h5 id="配置AzabanWebServer-和-AzkabanExecutorServer"><a href="#配置AzabanWebServer-和-AzkabanExecutorServer" class="headerlink" title="配置AzabanWebServer 和 AzkabanExecutorServer"></a>配置AzabanWebServer 和 AzkabanExecutorServer</h5><p><strong>Executor server 需要一个端口，Web server 需要知道这个端口，使用如下默认即可</strong><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"># Azkaban Executor settings</span><br><span class="line">executor.maxThreads=50</span><br><span class="line">executor.port=12321</span><br><span class="line">executor.flow.threads=30</span><br></pre></td></tr></table></figure></p>
<h4 id="运行和关闭"><a href="#运行和关闭" class="headerlink" title="运行和关闭"></a>运行和关闭</h4><ul>
<li><p>运行AzkabanExecutorServer</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/azkaban-exec-start.sh</span><br></pre></td></tr></table></figure>
</li>
<li><p>关闭 AzkabanExecutorServer</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/azkaban-exec-shutdown.sh</span><br></pre></td></tr></table></figure>
</li>
</ul>
<p><strong>这里所有Azkaban 环境已经安装完毕，你需要启动azkaban-web 和 azkaban-executor 进行工作流管理（作业调度）</strong></p>
<hr>
]]></content>
<summary type="html">
<p>大数据运维的同学Python、Shell脚本泛滥，且依赖混乱，不易管理。 需要符合自身要求的任务调度系统，来减轻痛苦~~。</p>
<p>之前调研了Oozie、AzkaBan、Airflow、Luigi 评估了一下诉求，选择AzkaBan。</p>
<p>这篇文章将演示如何
</summary>
<category term="Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/"/>
<category term="Bigdata Workflow" scheme="http://cmp-cc.github.io/categories/Workflow/Bigdata-Workflow/"/>
<category term="Workflow" scheme="http://cmp-cc.github.io/tags/Workflow/"/>
<category term="Azkaban" scheme="http://cmp-cc.github.io/tags/Azkaban/"/>
</entry>
<entry>
<title>Python + OpenCV 环境配置</title>
<link href="http://cmp-cc.github.io/2016/04/26/2016/Python%20+%20OpenCV%20%E7%8E%AF%E5%A2%83%E9%85%8D%E7%BD%AE/"/>
<id>http://cmp-cc.github.io/2016/04/26/2016/Python + OpenCV 环境配置/</id>
<published>2016-04-26T11:12:00.000Z</published>
<updated>2016-04-28T07:55:30.213Z</updated>
<content type="html"><![CDATA[<p>OpenCV的全称是：Open Source Computer Vision Library。OpenCV是一个基于（开源）发行的跨平台计算机视觉库，可以运行在Linux、Windows和Mac OS操作系统上。它轻量级而且高效——由一系列 C 函数和少量 C++ 类构成，同时提供了Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。</p>
<p>CV2是OpenCV官方的一个扩展库，属于一个Python模块，处理图像函数主要还是和CV2打交道。</p>
<p>安装详情参考:<br><a href="http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_setup/py_setup_in_windows/py_setup_in_windows.html" target="_blank" rel="external">http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_setup/py_setup_in_windows/py_setup_in_windows.html</a></p>
<h2 id="安装步骤"><a href="#安装步骤" class="headerlink" title="安装步骤"></a>安装步骤</h2><h3 id="安装Python-2-7"><a href="#安装Python-2-7" class="headerlink" title="安装Python 2.7"></a>安装Python 2.7</h3><p>当前最新版OpenCV 支持Python2.7 （64bit or 32 bit）<br><strong>这里确定你已经安装了</strong></p>
<h3 id="安装OpenCV"><a href="#安装OpenCV" class="headerlink" title="安装OpenCV"></a>安装OpenCV</h3><p>下载地址：<a href="https://sourceforge.net/projects/opencvlibrary/files/" target="_blank" rel="external">https://sourceforge.net/projects/opencvlibrary/files/</a></p>
<ul>
<li>选择合适的系统版本，这里选择Window，版本为opencv-3.1.0.exe。</li>
<li>安装opencv-3.1.0.exe ，选择解压目录<code>D:\Program-Software\Programming-Language\Python\Image\opencv</code></li>
<li><p>配置环境变量，Path添加类似如下</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">D:\Program-Software\Programming-Language\Python\Image\opencv\build\x64\vc14\bin</span><br></pre></td></tr></table></figure>
</li>
<li><p>将opencv\build\python\2.7\x64添加到Python27\Lib\site-packages目录（选择不同位数的版本）</p>
</li>
</ul>
<h3 id="安装Numpy"><a href="#安装Numpy" class="headerlink" title="安装Numpy"></a>安装Numpy</h3><p><strong>opencv 依赖</strong></p>