Enhance Kanji Recognition in Japanese Language Detection

[michaelbennieUFL]

Related Issues

• Add Kanji support #152
• Can distinguish between Simplified Chinese and Japanese Kanji? greyblake/whatlang-rs#122

Solution

This pull request improves the language detection between Japanese and Chinese, specifically when dealing with Kanji characters that are common in Japanese texts. By introducing a new script called Japanese_Han (abbreviated as JHAN), which includes Kanji characters commonly used in Japan, the detection logic is adjusted to better distinguish Japanese from Chinese when only Kanji characters are present. While there is a slight decrease in accuracy for single and dual-character inputs in the high-accuracy model, the overall sentence-level accuracy remains at 100%. This enhancement allows for accurate detection of Japanese sentences that use only Kanji.

Context

In previous versions, the language detector struggled to differentiate between Japanese and Chinese texts that contained only Kanji characters. This issue arose because both languages share many Han characters (Kanji in Japanese), leading to incorrect classification of Japanese Kanji-only texts as Chinese with high confidence. For example, words like "経済" (economy), "労働" (labor), and "勉強中" (studying) are uniquely Japanese but were being detected as Chinese.

Technical Approach

1. Introducing Japanese_Han Script

• Created a new script Japanese_Han (JHAN) that contains Kanji characters commonly used in Japanese.
• The JHAN script is added to the src/script.rs file with ranges of Kanji characters specific to Japanese usage.
• This script excludes Han characters that are unique to Chinese, focusing on those prevalent in Japanese texts.

2. Adjusting Detection Logic

• Updated the JAPANESE_CHARACTER_SET in src/constant.rs to use the new Japanese_Han script:

  pub(crate) static JAPANESE_CHARACTER_SET: Lazy<CharSet> =
      Lazy::new(|| CharSet::from_char_classes(&["Hiragana", "Katakana", "Japanese_Han"]));

• Modified the detect_language_with_rules method in src/detector.rs:

  • When a character is not matched by any of the one-language alphabets, it checks against JAPANESE_CHARACTER_SET and Alphabet::Han.
  • If the character matches JAPANESE_CHARACTER_SET, it increments the count for Japanese.
  • If the character matches Alphabet::Han, it increments the count for Chinese.
  • Adjusted the logic to handle uncertainty when both Chinese and Japanese counts are present, deciding based on counts and enabling a low-accuracy mode if necessary.

3. Handling Uncertainty

• Introduced a variable cjk_lang_uncertainty to track cases where it's challenging to distinguish between Chinese and Japanese.
• If both languages are detected with high uncertainty, the detector compares the counts and returns the language with the higher count when is_low_accuracy_mode_enabled is true.

Limits

• Single Character Accuracy: The accuracy for single-character inputs has decreased slightly due to overlapping Kanji characters between Japanese and Chinese.
• Dual Character Accuracy: Similar decrease observed in dual-character inputs for the same reasons.
• Ambiguity with Shared Kanji: Some Kanji characters are used in both languages, making it inherently challenging to distinguish based solely on character analysis.
• Complexity of Kanji Usage: The Japanese language occasionally uses Kanji characters that are uncommon or used differently in Chinese, and vice versa.

4. Test Cases and Accuracy Reports

• Updated the accuracy reports to reflect the changes:
  • Slight decrease in accuracy for single-character and dual-character inputs in the high-accuracy model.
  • Sentence-level accuracy remains at 100%.
  • Is now able to detect Kanji only Japanese sentences as Japanese
• Tested the updated detector with various Kanji-only Japanese texts:

Test Case 1: "勉強中" 
--------------------
  Chinese: 0.50
  Japanese: 0.50

Test Case 2: "沒"  (This is meant to be Chinese)
--------------------
  Chinese: 1.00

Test Case 3: "我是你的"  (This is meant to be Chinese)
--------------------
  Chinese: 1.00

Test Case 4: "労働"
--------------------
  Japanese: 1.00

Test Case 5: "御免"
--------------------
  Japanese: 0.55
  Chinese: 0.45

Test Case 6: "漢字"  (This can be both)
--------------------
  Chinese: 0.79
  Japanese: 0.21

Test Case 7: "桜"
--------------------
  Japanese: 1.00

Test Case 8: "峠" (It doesn't catch this one)
--------------------
  Chinese: 1.00

Test Case 9: "畑"
--------------------
  Japanese: 1.00
  Chinese: 0.00

Test Case 10: "塀"
--------------------
  Japanese: 1.00
  Chinese: 0.00

Test Case 11: "経済"
--------------------
  Japanese: 1.00
  Chinese: 0.00

Test Case 12: "和製漢字"  (This can be both)
--------------------
  Chinese: 0.78
  Japanese: 0.22

Test Case 13: "雫"
--------------------
  Japanese: 0.95
  Chinese: 0.05

Test Case 14: "労働"
--------------------
  Japanese: 1.00

Test Case 15: "豆腐"  (This can be both)
--------------------
  Chinese: 0.73
  Japanese: 0.27

Test Case 16: "自動販売機"
--------------------
  Japanese: 0.88
  Chinese: 0.12

Test Case 17: "関西国際空港"
--------------------
  Chinese: 0.59
  Japanese: 0.41

Test Case 18: "関西国际空港" (This is meant to be Chinese)
--------------------
  Chinese: 1.00

Test Case 19: "大阪" (This can be both)
--------------------
  Japanese: 0.80
  Chinese: 0.20

Test Case 20: "東京"  (This can be both)
--------------------
  Japanese: 0.53
  Chinese: 0.47

Test Case 21: "今日は"
--------------------
  Japanese: 1.00

[michaelbennieUFL]

It also fixes these issues in the python release

pemistahl/lingua-py#231

pemistahl/lingua-py#202

[pemistahl]

Hi Michael, thank you very much for this PR. Finally there is someone who knows Chinese and Japanese well enough to help me distinguish them better. Awesome. :) As I'm planning to make a new release of my library in October, I will evaluate your changes soon and most likely merge them, eventually. Great work!

[pemistahl]

Please add unit tests for the test cases you have listed in your PR.

[pemistahl]

Why is this rule applied in low accuracy mode only? The rule engine should operate independently of the selected accuracy mode.

[michaelbennieUFL]

It's due to to the fact that on low accuracy mode, the lingua-rs doesn't use the n-gram model after running detect_language_with_rules (idk if that's right). Regardless, by adding this case in, a lot more Chinese words get recognized as Chinese in low accuracy. Otherwise, they would be misidentified as unknown. If you want I can move this logic to compute_language_confidence_values_for_languages .

[michaelbennieUFL]

Where do you want me to add the unit tests?

[pemistahl]

Just add a new unit test method in file detector.rs. I think it's best to use a parameterized test method. Just take a look at the other test methods and do it analogously.

[pemistahl]

Please replace Language::Chinese with Language::from_str("Chinese") as it was before. The same goes for Japanese. Otherwise, the code won't compile if Chinese and / or Japanese are not among the selected language dependencies.