SFFTT: A Shared-Parameter and Fast Fourier Transform Lite Transformer
學年 113
學期 2
發表日期 2025-07-15
作品名稱 SFFTT: A Shared-Parameter and Fast Fourier Transform Lite Transformer
作品名稱（其他語言） SFFTT: 結合參數共享與快速傅立葉轉換的輕量化Transformer
著者 G. Y. Chen；M. L. Wu
作品所屬單位
出版者
會議名稱 智慧運算論壇 (SCF 2025)
會議地點 臺北市，台灣
摘要 In recent years, Transformer models have achieved remarkable success in natural language processing, yet their enormous number of parameters and high computational complexity restrict their application in resource-constrained environments. This thesis proposes a lightweight Transformer variant, termed SFFTT, which replaces the traditional self-attention mechanism with Fast Fourier Transform (FFT) in the first half of the encoder and employs parameter sharing along with attention threshold filtering in the latter encoder layers and the decoder. Additionally, we introduce SFFTTwithDyT by substituting all Layer Normalization layers with Dynamic Tanh normalization to enhance training stability and model expressiveness. Experimental results demonstrate that the SFFTT series models maintain competitive performance while significantly reducing parameter count and computational cost, offering an effective solution for lightweight Transformer applications.
關鍵字 Lightweight Transformer; Parameter Sharing; Fast Fourier Transform; Dynamic Tanh Normalization; Natural Language Processing
語言 en_US
收錄於
會議性質 國內
校內研討會地點 無
研討會時間 20250715~20250717
通訊作者
國別 TWN
公開徵稿
出版型式
出處
相關連結

機構典藏連結 ( http://tkuir.lib.tku.edu.tw:8080/dspace/handle/987654321/127934 )