-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathDeep Clustering Algorithms.html
50 lines (50 loc) · 14.8 KB
/
Deep Clustering Algorithms.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
<h1 style="text-align: center;"><span style="font-family: 'comic sans ms', sans-serif;">Deep Clustering Algorithms</span></h1>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;">作者:凯鲁嘎吉 - 博客园 <a href="http://www.cnblogs.com/kailugaji/" target="_blank">http://www.cnblogs.com/kailugaji/</a></span></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 本文研究路线:深度自编码器(Deep Autoencoder)->Deep Embedded Clustering(DEC)->Improved Deep Embedded clustering(IDEC)->Deep Convolutional Embedded Clustering(DCEC)->Deep Fuzzy K-means(DFKM),其中Deep Autoencoder已经在<a id="cb_post_title_url" class="postTitle2" href="https://www.cnblogs.com/kailugaji/p/11599870.html">深度自编码器(Deep Autoencoder)MATLAB解读</a>中提到,也有很多深度自编码器的改进方法,不详细讲解,重点谈深度聚类算法。如有不对之处,望指正。</span></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 深度聚类算法的网络架构图</span></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227140054377-126489743.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227145354853-1317725683.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 深度聚类算法的损失函数</span></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227145447861-41732470.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></span></p>
<h2><span style="font-family: 'comic sans ms', sans-serif;">1. Deep Embedded Clustering</span></h2>
<h3><span style="font-family: 'comic sans ms', sans-serif;">1.1 Stochastic Neighbor Embedding (SNE)</span></h3>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> SNE是一种非线性降维策略,两个特征之间存在非线性相关性,主要用于数据可视化,PCA(主成成分分析)是一种线性降维策略,两个特征之间存在线性相关性。SNE在原始空间(高维空间)中利用Gauss分布将数据点之间的距离度量转化为条件概率,在映射空间(低维空间)中利用Gauss分布将映射点之间的距离度量转化为条件概率,并利用KL散度来最小化高维空间与低维空间的条件概率。</span></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227104924927-1822871307.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> SNE面临的问题有两个:(1)KL散度是一种非对称度量,(2)拥挤问题。对于非对称问题,定义pij,将非对称度量转化为对称度量。但对称度量仍然面临拥挤问题,映射到低维空间中,映射点之间不能根据数据本身的特性很好地分开。</span></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227104936236-1544771479.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 对于拥挤问题(The Crowding Problem)的解决,提出t-SNE,一种非线性降维策略,主要用于可视化数据。引入厚尾部的学生t分布,将低维空间映射点之间的距离度量转化为概率分布t分布qij,使得不同簇之间的点能很好地分开。</span></p>
<h3><span style="font-family: 'comic sans ms', sans-serif;">1.2 t-SNE</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227104947073-301337495.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<h3><span style="font-family: 'comic sans ms', sans-serif;">1.3 Deep Embedded Clustering(DEC)</span></h3>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 受t-SNE的启发,提出DEC算法,重新定义pij,它是根据qij得到的,相当于对qij增加权重,使得数据更尖锐化,隐层软分配凸的更凸。微调阶段,舍弃掉编码器层,最小化KL散度作为损失函数,迭代更新参数。DEC通过降噪自编码,逐层贪婪训练后组合成栈式自编码,然后撤去解码层,仅使用编码层,对提取出来的特征使用相对熵作为损失函数对网络进行微调,该结构可以同时对数据进行特征学习和聚类。但是DEC算法没有考虑微调会扭曲嵌入式空间,削弱嵌入式特征的代表性,从而影响聚类效果。</span></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> <span class="src" data-group="0-0">DEC算法<span class="src" data-group="0-1">先使用整个网络进行预训练,得到原始数据经过非线性映射到潜在特征空间的数据表示,即特征。<span class="src" data-group="0-2">然后对得到的特征用K-means算法进行网络初始化,得到初始聚类中心。<span class="src" data-group="0-3">再使用相对熵迭代,微调网络,直至满足收敛性判定准则停止。</span></span></span></span></span></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"><span class="src" data-group="0-0"><span class="src" data-group="0-1"><span class="src" data-group="0-2"><span class="src" data-group="0-3"> 补充一点,在得到隐层特征z之后,外加一层聚类层,聚类中心μ就是z与聚类层的连接权重。通过聚类层,得到KL散度损失函数。</span></span></span></span></span></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227104958225-1060460026.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<h2><span style="font-family: 'comic sans ms', sans-serif;">2. Improved Deep Embedded Clustering(IDEC)</span></h2>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> DEC丢弃解码器层,并使用聚类损失Lc对编码器进行微调。作者认为这种微调会扭曲嵌入空间,削弱嵌入特征的代表性,从而影响聚类性能。因此,提出保持解码器层不变,直接将聚类损失附加到嵌入空间。IDEC算法是对DEC算法的改进,通过保存局部结构防止微调对嵌入式空间的扭曲,即在预训练时,使用欠完备自编码,微调时的损失函数采用相对熵和重建损失之和,以此来保障嵌入式空间特征的代表性。</span></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 基于局部结构保留的深度嵌入聚类IDEC是对DEC算法的改进,通过保存局部结构方式避免微调时对嵌入空间的扭曲。IDEC算法在预训练结束后,对重建损失和聚类损失的加权和进行微调,在最大限度保证不扭曲嵌入空间的前提下,得到最优聚类结果。</span></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227150138030-1871860051.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><span style="font-size: 16px; font-family: 'comic sans ms', sans-serif;">代码中q计算了两遍,进batch之前计算了所有的q,并根据q得到总体的p。在每个batch里面,又更新了各自的q,但是p仍然调用之前的p(从之前的p中找到该batch相应序号的p),这是由于batch里面无法获取全部的更新的q的信息,也就无法计算p。在每个batch中计算每个batch的目标函数,p没有根据q更新而更新,不过下一个epoch会更新p。</span></p>
<h2><span style="font-family: 'comic sans ms', sans-serif;">3. Deep Convolutional Embedded Clustering(DCEC)</span></h2>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 深度卷积嵌入聚类算法(deep convolutional embedded clustering, DCEC),是在DEC原有网络基础上,加入了卷积自编码操作,并在特征空间保留数据局部结构,从而取得了更好聚类效果。</span></p>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> 深度卷积嵌入聚类算法DCEC是在IDEC算法基础上进行的改进,将编码层和解码层中的全连接换成卷积操作,这样可以更好地提取层级特征。图中编码层和解码层各有3层卷积,卷积层后加了一个flatten操作拉平特征向量,以获得10维特征。DCEC只是将IDEC的所有全连接操作换成卷积操作,其损失函数依旧是重建损失和聚类损失之和。</span></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227151436867-1455103743.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<h2><span style="font-family: 'comic sans ms', sans-serif;">4. Deep Fuzzy K-means(DFKM)</span></h2>
<p><span style="font-family: 'comic sans ms', sans-serif; font-size: 16px;"> Deep Fuzzy K-means同样在低维映射空间中加入聚类过程,将特征提取与聚类同时进行,引入熵加权的模糊K-means,不采用原来的欧氏距离,而是自己重新定义度量准则,权值偏置的正则化项防止过拟合,提高泛化能力。</span></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227105022088-2127887599.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227105032805-995175228.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227105043394-886581489.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<p><img src="https://img2018.cnblogs.com/blog/1027447/201912/1027447-20191227105054045-2029819812.png" alt="" style="display: block; margin-left: auto; margin-right: auto;" /></p>
<h2><span style="font-family: 'comic sans ms', sans-serif;">5. 参考文献</span></h2>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[1] Maaten L, Hinton G. <a href="http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf" target="_blank">Visualizing data using t-SNE</a>[J]. Journal of machine learning research, 2008, 9(Nov): 2579-2605.</span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[2] Vincent P, Larochelle H, Lajoie I, et al. <a href="http://www.jmlr.org/papers/volume11/vincent10a/vincent10a.pdf" target="_blank">Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion</a>[J]. Journal of machine learning research, 2010, 11(Dec): 3371-3408.</span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[3] Xie J, Girshick R, Farhadi A. <a href="http://proceedings.mlr.press/v48/xieb16.pdf" target="_blank">Unsupervised deep embedding for clustering analysis</a>[C]//International conference on machine learning. 2016: 478-487.</span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[4] Guo X, Gao L, Liu X, et al. <a href="https://www.ijcai.org/proceedings/2017/0243.pdf" target="_blank">Improved deep embedded clustering with local structure preservation</a>[C]//IJCAI. 2017: 1753-1759.</span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[5] Guo X, Liu X, Zhu E, et al. <a href="https://www.researchgate.net/profile/Xifeng_Guo/publication/320658590_Deep_Clustering_with_Convolutional_Autoencoders/links/5a2ba172aca2728e05dea395/Deep-Clustering-with-Convolutional-Autoencoders.pdf" target="_blank">Deep clustering with convolutional autoencoders</a>[C]//International Conference on Neural Information Processing. Springer, Cham, 2017: 373-382.</span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[6] Zhang R, Li X, Zhang H, et al. <a href="https://sci-hub.shop/10.1109/TFUZZ.2019.2945232" target="_blank">Deep Fuzzy K-Means with Adaptive Loss and Entropy Regularization</a>[J]. IEEE Transactions on Fuzzy Systems, 2019.</span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[7] t-SNE相关资料:<a href="http://www.datakit.cn/blog/2017/02/05/t_sne_full.html" target="_blank">t-SNE完整笔记</a>、<a href="https://www.oreilly.com/learning/an-illustrated-introduction-to-the-t-sne-algorithm" target="_blank">An illustrated introduction to the t-SNE algorithm</a>、<a href="http://bindog.github.io/blog/2016/06/04/from-sne-to-tsne-to-largevis/" target="_blank">从SNE到t-SNE再到LargeVis</a>、<a href="https://blog.csdn.net/sinat_20177327/article/details/80298645" target="_blank">t-SNE算法-CSDN</a></span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[8] </span><span style="font-size: 16px;"><span style="font-family: 'times new roman', times;"><span style="font-family: 'times new roman', times;"><a href="https://github.com/XifengGuo/IDEC" target="_blank">DEC与IDEC的Python代码-Github</a>、</span></span><span style="font-family: 'times new roman', times;"><a href="https://github.com/XifengGuo/DEC-keras" target="_blank">DEC-Keras-Github</a>、<span class="author"><a href="https://github.com/piiswrong/dec" target="_blank">piiswrong<span class="path-divider">/dec-Github</span></a>、<a href="https://github.com/XifengGuo/DCEC" target="_blank">DCEC-Github</a></span></span></span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[9] <a href="https://github.com/hyzhang98/DFKM" target="_blank">DFKM的Python代码-Github</a></span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[10] 谢娟英,侯琦,曹嘉文. <a href="http://fcst.ceaj.org/CN/article/downloadArticleFile.do?attachType=PDF&id=1763" target="_blank">深度卷积自编码图像聚类算法</a>[J]. 计算机科学与探索, 2019, 13(4): 586-595.DOI:10.3778/j.issn.1673-9418.1806029. </span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[11] </span><span style="font-family: 'times new roman', times; font-size: 16px;"><a href="https://github.com/zhoushengisnoob/DeepClustering" target="_blank">Deep Clustering: methods and implements-Github </a>深度聚类会议论文汇总</span></p>
<p><span style="font-family: 'times new roman', times; font-size: 16px;">[12] </span><span style="font-family: 'times new roman', times; font-size: 16px;"><a href="https://deepnotes.io/deep-clustering" target="_blank">Deep Clustering | Deep Learning Notes</a></span></p>