dc.contributor.author |
Pretorius, Arnu
|
|
dc.contributor.author |
Kroon, Steve
|
|
dc.contributor.author |
Kamper, Herman
|
|
dc.date.accessioned |
2019-03-07T09:46:57Z |
|
dc.date.available |
2019-03-07T09:46:57Z |
|
dc.date.issued |
2018-07 |
|
dc.identifier.citation |
Pretorius, A., Kroon, S. and Kamper, H. 2018. Learning dynamics of linear denoising autoencoders. ICML 2018: The 35th International Conference on Machine Learning, Stockholm, Sweden, 10-15 Jul 2018, pp. 4141-4150 |
en_US |
dc.identifier.uri |
https://arxiv.org/abs/1806.05413
|
|
dc.identifier.uri |
http://hdl.handle.net/10204/10746
|
|
dc.description |
Copyright 2018 The author(s). |
en_US |
dc.description.abstract |
Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic expressions that exactly describe their learning dynamics. We verify our theoretical predictions with simulations as well as experiments on MNIST and CIFAR-10. The theory illustrates how, when tuned correctly, noise allows DAEs to ignore low variance directions in the inputs while learning to reconstruct them. Furthermore, in a comparison of the learning dynamics of DAEs to standard regularised autoencoders, we show that noise has a similar regularisation effect to weight decay, but with faster training dynamics. We also show that our theoretical predictions approximate learning dynamics on real-world data and qualitatively match observed dynamics in nonlinear DAEs. |
en_US |
dc.language.iso |
en |
en_US |
dc.relation.ispartofseries |
Worklist;22132 |
|
dc.subject |
Denoising autoencoders |
en_US |
dc.subject |
DAEs |
en_US |
dc.subject |
Learning dynamics |
en_US |
dc.subject |
Linear DAEs |
en_US |
dc.title |
Learning dynamics of linear denoising autoencoders |
en_US |
dc.type |
Conference Presentation |
en_US |
dc.identifier.apacitation |
Pretorius, A., Kroon, S., & Kamper, H. (2018). Learning dynamics of linear denoising autoencoders. http://hdl.handle.net/10204/10746 |
en_ZA |
dc.identifier.chicagocitation |
Pretorius, Arnu, Steve Kroon, and Herman Kamper. "Learning dynamics of linear denoising autoencoders." (2018): http://hdl.handle.net/10204/10746 |
en_ZA |
dc.identifier.vancouvercitation |
Pretorius A, Kroon S, Kamper H, Learning dynamics of linear denoising autoencoders; 2018. http://hdl.handle.net/10204/10746 . |
en_ZA |
dc.identifier.ris |
TY - Conference Presentation
AU - Pretorius, Arnu
AU - Kroon, Steve
AU - Kamper, Herman
AB - Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic expressions that exactly describe their learning dynamics. We verify our theoretical predictions with simulations as well as experiments on MNIST and CIFAR-10. The theory illustrates how, when tuned correctly, noise allows DAEs to ignore low variance directions in the inputs while learning to reconstruct them. Furthermore, in a comparison of the learning dynamics of DAEs to standard regularised autoencoders, we show that noise has a similar regularisation effect to weight decay, but with faster training dynamics. We also show that our theoretical predictions approximate learning dynamics on real-world data and qualitatively match observed dynamics in nonlinear DAEs.
DA - 2018-07
DB - ResearchSpace
DP - CSIR
KW - Denoising autoencoders
KW - DAEs
KW - Learning dynamics
KW - Linear DAEs
LK - https://researchspace.csir.co.za
PY - 2018
T1 - Learning dynamics of linear denoising autoencoders
TI - Learning dynamics of linear denoising autoencoders
UR - http://hdl.handle.net/10204/10746
ER -
|
en_ZA |