There is an ongoing debate in the NLP community whether modern language models contain linguistic knowledge, recovered through so-called probes. This work examines whether linguistic knowledge is a necessary condition ...