Turkish Political Compass Test of LLMs

The goal of this project is to evaluate the political compass of large language models (LLMs) within the Turkish political landscape. While much of the existing research in this area focuses on American politics, particularly the Democrat-Republican divide, or is conducted in English, this project aims to address the gap by exploring political understanding in Turkish. The objective is to identify a relevant political survey or test in Turkish, either multiple-choice or open-ended, and use it to assess the political alignment and bias of various instruction-tuned LLMs. By leveraging such a test, the project will provide insights into how well these models represent and respond to diverse Turkish political perspectives, ranging from conservative to progressive ideologies.

Relevant links:

• [2402.16786] Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models

• [2306.16388] Towards Measuring the Representation of Subjective Global Opinions in Language Models

• https://www.pewresearch.org/short-reads/2024/06/03/ratings-of-the-eu-remain-broadly-positive-in-member-countries-but-have-gone-down-slightly-since-2022/

• https://www.pewresearch.org/international-surveys 

Suitable for Cmpe492