A simple demo to verify the symmetry of Feature and Weights in DNNs [1]. The weights of Y-X network (right) should coincide with the feature extracted by X-Y network (left):
A detailed analyisis can be found in [2] (with demos in this repo).
[1] Xu, Xiangxiang, Shao-Lun Huang, Lizhong Zheng, and Lin Zhang. "The geometric structure of generalized softmax learning." In 2018 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2018.
[2] Xu, Xiangxiang, Shao-Lun Huang, Lizhong Zheng, and Gregory W. Wornell. 2022. "An Information Theoretic Interpretation to Deep Neural Networks" Entropy 24, no. 1: 135. https://doi.org/10.3390/e24010135