{"id":5754,"date":"2012-06-14T17:42:24","date_gmt":"2012-06-14T12:12:24","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=5754"},"modified":"2016-12-19T15:16:40","modified_gmt":"2016-12-19T09:46:40","slug":"normalizing-accented-words","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/normalizing-accented-words\/","title":{"rendered":"Normalizing Accented Words"},"content":{"rendered":"

We all often need to work on data aggregated together from different sources, and before we analyse it, we often need to normalize it to a certain standard, A normalization process typically includes removing special characters, converting all text to lower case , We can also have certain rules that words like “saint” will always be normalized to “st.” etc.<\/p>\n

An important part of such normalization is to account for ‘accented’ characters like \u00e9 or \u00e8 , and generally you would want them to normalize to normal\u00a0English\u00a0alphabet ‘e’, as that would help in sorting\/searching words containing these characters. For eg : you would want “Indianap\u00f2lis” to be normalized to “Indianapolis”.<\/p>\n

We can achieve this by using java.text.Normalizer<\/strong> class, all we need to do is<\/p>\n

[java]
\nNormalizer.normalize(&quot;Indianap\u00f2lis&quot;, Normalizer.Form.NFKD).replaceAll(&quot;\\\\p{InCombiningDiacriticalMarks}+&quot;, &quot;&quot;)
\n[\/java]<\/p>\n

Lets understand what is happening here, clearly we are calling a static function normalize in java.text.Normalizer<\/strong> class, the first parameter we passed is the string we want to normalize, the second parameter is the normalization form<\/a>. There are 4 normalization forms<\/p>\n

    \n
  1. NFC – Canonical Decomposition, followed by Canonical Composition.<\/li>\n
  2. NFD – Canonical Decomposition<\/li>\n
  3. NFKC – Compatibility Decomposition, followed by Canonical Composition<\/li>\n
  4. NFKD – Compatibility Decomposition<\/li>\n<\/ol>\n

    So in the second parameter we pass in the NFKD form, which is an enum of type\u00a0Form<\/strong>.<\/p>\n

    The normalizer function will still return us “Indianap\u00f2lis”. so, what happened there?
    \n
    \nLets understand, we can create an \u00f2 using two ways. It can either be a unicode character (U+00F2) or it can be a normal english ‘o’ with a grave accent added to it (o+ (U+0060)).
    \n
    What normalizer function did was it normalized both cases to english ‘o’ with an accent (o+ (U+0060))
    \n
    The next part is a replaceAll with a regex, the\u00a0{InCombiningDiacriticalMarks} is a Unicode block property, which matches the accented characters.
    \n
    The second part replaces all accented characters (grave accent (U+0060) in this case) with empty string. So what we finally get is “Indianapolis”, and we are done.
    \n
    \nHope it helped!<\/p>\n

    Sachin Anand<\/p>\n

    sachin[at]intelligrape[dot]com<\/p>\n

    @babasachinanand<\/p>\n","protected":false},"excerpt":{"rendered":"

    We all often need to work on data aggregated together from different sources, and before we analyse it, we often need to normalize it to a certain standard, A normalization process typically includes removing special characters, converting all text to lower case , We can also have certain rules that words like “saint” will always […]<\/p>\n","protected":false},"author":14,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":11,"footnotes":""},"categories":[1],"tags":[829,830,828,827,831],"class_list":["post-5754","post","type-post","status-publish","format-standard","hentry","category-technology","tag-nfc","tag-nfkc","tag-nfkd","tag-normalization","tag-text-normalization"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t