The Implications of Linguistic Illegibility for LLM Security

LLMs are trained to generate natural language.

LLMs are trained to generate natural language. However, various strands of evidence indicate that an LLM's externalized linguistic outputs and mechanistically-extracted linguistic features can be an unreliable lens for understanding internal model computation. We introduce the…