As the saying goes, there is always someone stronger, and behind every capable person is someone else pushing the limits.
Ever since OpenAI threw the first brick, this field has been buried under an avalanche of new bricks. By now, there are enough of them to build several houses.
Every time a large model is released, the surrounding articles are full of praise: it tops a benchmark here, leads among models with the same parameter count there, or reaches more than 90% of GPT-3.5 capability on certain questions. But before it can enjoy the spotlight for long, a newer model often rolls right over it and turns it into a defeated baseline in the next paper.
You cannot judge the spring scenery with your eyes closed; the mud shows only after you step into the water
From a user's perspective, choosing the right model is not easy. Whenever a new model appears, sponsored articles praise it to the skies, while real users who have tried it give sharply mixed feedback.
The main reason is that very few people have tested the model's capabilities fairly, objectively, and in detail. Most people enter a handful of cases and look at the results. When the output is good, they sincerely say "awesome"; when it falls short, they close the laptop, take a deep breath, and quietly say "trash."
But whether the results from these small samples are good or bad, they cannot fully represent a large model's capability. Only systematic testing can show which model is strong, where it is strong, which model is weak, and exactly where it falls short. Only then can we find the model that fits the subtasks in our own domain.
If it is a mule or a horse, take it out for a run
Of course, many LLMs are released with benchmark results on datasets such as MMLU, C-Eval, and GSM8K. Some focus on general knowledge, some on reasoning, and others on mathematical ability.
These tests are important, but using leaderboards as the main evaluation mechanism for large models may not be the best answer. Leaderboards have already been over-optimized and are easy to fit, whether through pretraining, SFT with benchmark-style questions, or specially designed prompts. The methods vary widely, but the result is the same: they do not fully reflect a model's comprehensive capability.
As model users, we care more about whether a model performs well in the domain we care about and whether it improves the subtasks we actually need. That is why we designed an evaluation system combining multiple data dimensions and multiple subtasks, with the goal of finding models that better fit our own work.
Chinese Cognitive Capability
As we all know, the language distribution in many models' training corpora is highly uneven. English usually dominates, while Chinese accounts for a very small share. Domestic open-source models tend to do better on this point, with higher Chinese corpus proportions and, in many cases, specific optimization for Chinese.
So how good are these models at Chinese? Because our own data is also mainly in Chinese, the answer is especially important.
On GitHub, one researcher has a project dedicated to studying this exact question. The reference link is at the end of this article.
He used three common character sets: the Table of General Standard Chinese Characters, the Standard Form of National Characters, and Unicode CJK Unified Ideographs, totaling 21,267 Chinese characters. He first tested character recognition with WordPiece and BBPE, then used token distributions to plot the model's input embedding distribution and output embedding distribution. The former is the initial semantic representation obtained after a word or token enters the embedding layer; the latter is the semantic representation the model finally understands after multiple layers of computation.
What can we learn from this work?
- For character recognition, if a model encodes a Chinese character as unk_token_id in WordPiece, the model is considered unable to recognize that character. The more a character is split by BBPE, the weaker the model's understanding of that character.
- Through vocabulary vector distributions, we can see whether a language model has achieved multilingual alignment. If words with the same meaning in different languages cluster together, the model has performed multilingual alignment, and we can directly observe its level of support for different languages.
- Vector distributions can also give us a rough view of semantic understanding. Well-trained characters and words show visible connectedness and a certain topological relationship after plotting; undertrained characters and words show a noticeably uniform distribution with no meaningful distance relationships, suggesting that they retain the features of the original random distribution.
Datasets
Dataset selection is extremely important because different data sources have different distributions. Even with the same LLM and the same subtask, results can vary across data sources. Chinese and English data from the same type of source may also produce different test results.
Therefore, when building our test datasets, we combined data from Chinese and English medical literature, Chinese and English patient education content, medical consultations, clinical trials, and other dimensions to ensure richness and randomness.
Subtasks
We designed seven dimensional subtasks, which we call the Seven True Tasks.
Some of these subtasks are common in NLP, but they are closely tied to our business. They help us solve practical problems and strengthen our capabilities. For example:
- Relation extraction, entity-level sentiment analysis, and drug perception extraction help us enrich our medical knowledge graph faster, structure medical texts more effectively, understand physicians' treatment perceptions more precisely, and build richer HCP360 profiles.
- Entity linking strengthens our entity recognition tasks.
- Summary extraction, tag extraction, and determining whether an article focuses on a specific drug are strong business requirements.
Through these seven subtasks, we hope to identify the large models best suited to our business.
Because different subtasks have different characteristics, we use different evaluation metrics accordingly so that results can be calculated more appropriately:
- For subtasks with clearly bounded outputs, we use micro precision, recall, and F1 scores.
- For subtasks with less clearly bounded outputs, we use ROUGE Score and BLEU Score.
We also designed a general-purpose prompt for all subtask research. When evaluating each model's capability, we use the same prompt to eliminate the impact of prompt differences. Before scoring subtask results, we first test whether each model can understand the general prompt and output in the required format. That, too, reflects model capability. Outputs that meet the requirements are then evaluated with the corresponding calculation method.
You say you are fair, I say I am fair; only testing tells us whether it is truly fair
Judging from the performance of major models across different subtasks, GPT-4 stands at the top and holds an overwhelming advantage in most tasks. But it is not omnipotent. In more complex textual contexts, its relation extraction capability still falls short of our expectations, and in some subtasks, several of our industry-specific models can actually compete with GPT-4 on fairly even terms.
By comparison, domestic large models each have their strengths and weaknesses, but overall there is still a noticeable gap with OpenAI.
Today, new models emerge one after another. Only with a unified large-model evaluation system in hand can we choose the model that fits our current business instead of simply following the crowd.
Reference: Analysis of Chinese Cognitive Capability in Language Models https://github.com/twang2218/vocab-coverage
