OpenAI introduces IndQA benchmark to evaluate AI systems on Indian culture, languages

OpenAI announced the launch of a new benchmark, IndQA, for evaluating AI systems that focus on Indian languages, communication patterns in the country, and popular disciplines there.

The ChatGPT-maker noted that it worked with over 250 Indian experts across languages and domains in order to develop its new IndQA benchmark, spanning 2,278 questions across 12 languages and 10 cultural domains.

“While our aim is to create similar benchmarks for other languages and regions, India is an obvious starting point. India has about a billion people who don’t use English as their primary language, 22 official languages (including at least seven with over 50 million speakers), and is ChatGPT’s second largest market,” said OpenAI in a blog post.

The topics that IndQA focuses on are Architecture & Design, Arts & Culture, Everyday Life, Food & Cuisine, History, Law & Ethics, Literature & Linguistics, Media & Entertainment, Religion & Spirituality, and Sports & Recreation.

The languages covered include Bengali, English, Hindi, Kannada, Marathi, Odia, Telugu, Gujarati, Malayalam, Punjabi, Tamil, and even Hinglish.

“We use IndQA to evaluate how recent frontier models perform and chart progress over the last couple years. With IndQA we can see that OpenAI’s models have improved significantly over time on Indian languages (with caveats⁠), but still have substantial room for improvement. We look forward to improving performance and sharing results for future models,” noted OpenAI in its blog post.

The company is seeking to expand its presence in India, as it sets up its first India office in Delhi and looks to hire talent in the country, as well as advertises its products to Indian customers.

Last month, OpenAI announced that users in India could get its low-cost offering, ChatGPT Go, available for free for a year.