On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding

View a PDF of the paper titled On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding, by Kevin Xu and Issei Sato

View PDF
HTML (experimental)

Abstract:Looped Transformers offer advantages in parameter efficiency and Turing completeness. However, their expressive power for function approximation and approximation rate remains underexplored. In this paper, we establish approximation rates of Looped Transformers by defining the concept of the modulus of continuity for sequence-to-sequence functions. This reveals a limitation specific to the looped architecture. That is, the analysis prompts us to incorporate scaling parameters for each loop, conditioned on timestep encoding. Experimental results demonstrate that increasing the number of loops enhances performance, with further gains achieved through the timestep encoding architecture.