Log in to create and rate content, and to follow, bookmark, and share content with other members. Not a member? Join Now!

AnsweredAssumed Answered

Chinese documents

Question asked by chiho80 on Nov 7, 2005
Latest reply on Mar 22, 2006 by derek

Like • Show 0 Likes0
Comment • 13

I have added some chinese "word documents" and created a chinese HTML document. I cannot search with the documents when I use chinese terms(Using English term is ok).

I have looked into the index using Luke, the chinese words seem not indexed. I suspected that is the problem of the StandardAnalyzer, but I tried different analyzers (custom and two from lucene sandbox), recreted the index on each attempt, but still in vain. I have checked the analyzers with lucene and they worked fine (in my test script).

Can anyone tell me if there are different ways to check what happened? Is it the web layers problem?

Thanks in advance. :o

No one else has this question

Outcomes

13 Replies

andy Nov 7, 2005 3:23 PM
Mark Correct
Correct Answer
Hi

The tokeniser is specified at the type level and is localisable.
If there is no localisation then the values from dictionaryModel.xml are used.

In the default configuration there is a default localisation bundle that specifies the tokenisers that are used. This is in the file

dataTypeAnalyzers.properties

You can set these for a particular locale by adding something like

dataTypeAnalyzers_zh_CN.properties

in the language pack. I don't think any of the language bundles include this as yet. If no locale specific file is found it falls back to the default.

The next question is: "How is the locale found?"
The default locale is picked up from the server or set when interacting with the client. It is possible that interactively adding a document may produce different tokenisation from indexing as a result of a rule as one could set the locale via the client and the other via the default Java locale in the repository.

I hope this helps. Let me know how you get on.

Regards

Andy
Like • Show 0 Likes0
Actions
chiho80 Nov 9, 2005 5:40 PM
Mark Correct
Correct Answer
Hi Andy,

Thanks for your advices. Actually I tried to change dataTypeAnalyzers.properties to use the custom analyzer (which can recognize zh, zh_CN and en). But it has no effect at all.

This time, I have read through the index and seem like the index is not written properly. The locale used to write the index made the chinese characters not readable.

In Hong Kong, we use zh as the main chinese locale. zh_CN is used in China. When we try to write zh content using zh_CN locale, the characters gets strange. When I look into the index, the index looks like zh_CN encoded. I will try to set up a debugger to see where the problem is. :)

I am sorry that I cannot post the analyzer here for your testing coz the analyzer contains some copyrighted codes. I will try to seek for your advice during the course and post the result / changes in code here when the bug is fixed. :D
Like • Show 0 Likes0
Actions
chiho80 Nov 22, 2005 2:21 PM
Mark Correct
Correct Answer
The problem is solved. The problem is due to the reading of chinese file.
When the LuceneIndexerImpl tries to load the document from file system, it does not specify the charset. This would cause problem when reading CJK documents. Please find the code below I used to solve the problem.

[LuceneIndexerImpl ] At around line 1428, I changed the code to:
```
InputStreamReader isr = null;
InputStream ris = reader.getContentInputStream();
  try {
    isr = new InputStreamReader(ris,"UTF-8");
  } catch (UnsupportedEncodingException e) {
     isr = new InputStreamReader(ris);
  }
  doc.add(Field.Text("TEXT",isr));
}
```
Like • Show 0 Likes0
Actions
derek Nov 23, 2005 10:10 AM
Mark Correct
Correct Answer
Hi,

Thanks for this. You are right and we have fixed this.

During the indexing, we perform a transformation on any text that is not UTF-8 and force it to UTF-8.

Regards
Like • Show 0 Likes0
Actions
lhy719 Dec 22, 2005 3:17 AM
Mark Correct
Correct Answer
Hi Derek and Chiho,

I'm a Chinese in Taiwan. I've the Alfresoc 1.1 installed, but I still can't find Chinese file or folder names. Notheless it doesn't work with searching Chinese in any file content.

Does the fix method mentioned above has been apply in Alfresco 1.1? Or other things need to be done for Traditional Chinese?

Thanks!
Hammer Lee
Like • Show 0 Likes0
Actions
derek Jan 3, 2006 11:01 AM
Mark Correct
Correct Answer
Hi,

The default analyzer is for English. You will notice that chiho80 was writing a custom analyzer for Chinese. These are fairly standard components to write - you might search the Lucene forums for more information on where to get one.

Or you could beg chiho80 to convince his company to release the code.

Regards
Like • Show 0 Likes0
Actions
goodguy Jan 23, 2006 3:09 AM
Mark Correct
Correct Answer
derek

Is this going into Alfresco 1.2?
Like • Show 0 Likes0
Actions
derek Jan 23, 2006 9:13 AM
Mark Correct
Correct Answer
That depends on whether you can convince chiho80 to give it to us.
:wink:
Like • Show 0 Likes0
Actions
chiho80 Jan 25, 2006 6:25 AM
Mark Correct
Correct Answer
It's been a while since I last been here. I am sorry that it seems not possible to contribute the analyzer after a discussion with my boss, as this is one of the core technologies used in our system (non-alfresco).

If you want to find some open source alternatives, you may try CJKAnalyzer, this is not as accurate as dictionary based Analyzer though. You may contact me for help. :D
Like • Show 0 Likes0
Actions
akinori Feb 25, 2006 6:31 AM
Mark Correct
Correct Answer
Hi,

Have you succeeded to enable CJKAnalyzer in Alfresco?

I'm Japanese. There are two major options to analyze Japanese contents in Lucene. JapaneseAnalyzer and CJKAnalyzer. So far I failed both of them.
I tried JaneseAnalysis first. Because it looked better. JapaneseAnalyzer requires another software called Sen to make dictionaly in advance, and it may require System.property stuff which I'm not sure if I already set properly in Tomcat context or not. I saw lots of errors and warnings in the bootstrap message.

So now I'm trying to use CJKAnalyzer anyway. It seemed to work. No error message. But I cannot still find double byte charactors in index.

What I did…
1. Copy lucene-ja.jar which includes CJKAnalyzer to alfresco's lib directory.
2. Make dataTypeAnalyzers_ja_JP.properies pointing to CJKAnalyzer.
3. Restart Alfresco (1.2 RC2)
4. Import a word file including English and Japanese text.

After that,
1. the word file was found by a query with an English word.
2. the word file wasn't found by any query with a Japanese word (longer than 3 charactors I tried).
Like • Show 0 Likes0
Actions
akinori Mar 1, 2006 7:08 AM
Mark Correct
Correct Answer
Hi again,

Still I've never succeeded to search with Japanese Contents.
Anyway I found more facts.

1. Both of CJKAnalyzer and JapaneseAnalyzer ran without any error. (Only when I remove the jar file of the analyzer, Alfresco gave me exceptions messages. I believe this means my properties file points to the analyzer properly)
2. I can find Japanese file name and topic titile, but I cannot find any files whose contents include Japanese charactors with Japanese keywords.
3. English keywords still work. If the target file includes both of English words and Japanese words, I can find it with any of those English words.

Sorry, I know it's off-topic in strict sense. Title is Chinese and I'm writing about Japanese issue. But I supposed my situation looked similar to chiho80's case.
I was wondering if someone could instruct me the way to check how lucene works in Alfresco's context now.

Akinori
Like • Show 0 Likes0
Actions
andy Mar 22, 2006 9:53 AM
Mark Correct
Correct Answer
Hi

Apologies, this issue has mainly been discussed under French alaysis.

There was a bug tokenising the TEXT attribute in the index - it was always using the standard analyser and not picking up the one defined for text attributres. This is fixed in the future 1.2.1 build (tested with the french analyser). These changes will be merged into the HEAD code stream in due course. The code can be obtained from svn.

I will look into merge and 1.2.1 timings.

Regards

Andy
Like • Show 0 Likes0
Actions
derek Mar 22, 2006 4:23 PM
Mark Correct
Correct Answer
The HEAD now contains all the latest bug fixes from the V1.2.0 branch (future 1.2.1 release).
Like • Show 0 Likes0
Actions

Chinese documents

Outcomes

Related Content

Recommended Content