Language Fluency as a Factor in Machine Translation Comprehension

Machine translations have been used for nearly 50 years, and although accuracy steadily improves, results do not often attain the standard set for human interpretation. Some people can understand poor translations better than others, but little is known about why this is so. Here, we show that language fluency was not a major factor in the ability of a human reader to comprehend a passage of text translated by computer, but prior topic knowledge could be


INTRODUCTION
Every day, millions of text passages are translated from one language to another by machine due to the high speed and low cost compared to human interpretation [4]. The major disadvantage of this automated technique is its relatively poor accuracy. However, people reading the same translated passage with errors can have varying amounts of understanding depending upon their familiarity with the topic, their fluency in the target language, the amount of effort they are willing to spend trying to understand, or other factors [8]. Here, we focus on language fluency as a predictor of reading comprehension. Results of the study show that there was no significant, positive correlation between tests of grammar, vocabulary, and reading comprehension and translated text understanding. However, significant differences between the reading comprehension tests suggests prior knowledge of the topic could be the major factor.

BACKGROUND Machine Translation
Since the 1950s, researchers have been trying to develop software that enables computers to translate natural languages such as English, French, and German. While great improvments have been made, most maching translation (MT) systems still cannot match the accuracy of human interpreters [7]. However, for relatively unimportant matters where only the gist of a passage is necessary, the speed and low cost of MT ensures that such systems are used every day.
Several standalone and Web-based MT systems are now available, but perhaps Google Translate (http://translate.google.com/) is used most often. Although its accuracy varies considerably with the language pairs chosen [1,2], good results are obtained frequently, and it usually outperforms other MT systems. In addition, Google Translate supports far more languages (currently 71 in 4,970 language-pair combinations).

Translation Comprehension
Previous studies have shown that people reading the same passage can have considerably different opinions about how much was understood. For example, in one study [3], 66 students evaluated the same passage of text translated from Spanish, German, Chinese, Malay, Korean, or Hindi to English using Google Translate. An additional 9 students evaluated the passage in English without translation. Table 1 shows large variances in understanding for most translations (with the exceptions of Spanish to English and no translation). What factors cause variations in reading comprehension? Some people might be more fluent in their native language (in this case, English), more knowledgeable about the subject matter, or mentally agile, and can more likely guess what a translation really means if the wording is not exactly right [10]. Some people might have experience with a different language. For example, if a person knows some Spanish, even if the translation is in English, he or she might be able to understand better because of previous exposure to translation errors. Some people might have different emotional reactions to the grammatical inadequacies of translations, and some might try harder than others to understand the material [9]. Even the amount of a person's 'working memory' could affect comprehension [5].

Tests of Language Proficiency
In an attempt to determine what affects a person's reading comprehension, we focused on native language proficiency. Although several tests are available to measure this language fluency, we chose two: the Maze test and the Test of English as a Foreign Language (TOEFL). A u g 10, 2 0 1 3 The Maze test is a multiple-choice variation of the Cloze test [13] and it requires a person to have the ability to understand context and vocabulary in order to identify the correct words that belong in deleted passages of a text [11,12]. However, high scores do necessarily indicate that the reader actually comprehends the material. Rather, it concentrates on the reader's knowledge of vocabulary and grammar.
The TOEFL is used by many universities and colleges in the United States for admission of students from non-Englishspeaking countries, and the reading comprehension portion is one way to determine how much information a person can extract from a passage of text [6]. Interestingly, these comprehension scores might not correspond to a person's selfassessed rating of understanding. For example, in one study [3], 75 students got 73% of the TOEFL reading comprehension questions correct but reported being able to understand only about 63% of the text.

Method
A total of 80 undergraduate students (70% male, 21% non-native English speakers) from two universities in the northeastern region of the United States participated in a simple survey. First, each student answered questions in a Maze test to measure his or her English fluency. Two paragraphs (P1 and P2) of text were presented, each with five questions (correct answers are shown in bold): Can we see (1) ___ the earth is a globe? Yes, we can, when we watch a ship that sails out to sea. If we watch closely, we see that the ship begins (2) ___ . The bottom of the ship disappears first, and then the ship seems to sink lower and lower, (3) ___ we can see only the top of the ship, and then we see nothing at all. What is hiding the ship from us? It is the earth. Stick a pin most of the way into an orange, and (4) ___ turn the orange away from you. You will see the pin disappear, (5) ___ a ship does on the earth. A. yet B. therefore C. since D. whereas E. and Next, each student read a passage (P3) from the TOEFL in English (no translation) and answered seven questions (correct answers are shown in bold): Marie Curie was one of the most accomplished scientists in history. Together with her husband, Pierre, she discovered radium, an element widely used for treating cancer, and studied uranium and other radioactive substances. Pierre and Marie's amicable collaboration later helped to unlock the secrets of the atom. Marie was born in 1867 in Warsaw, Poland, where her father was a professor of physics. At an early age, she displayed a brilliant mind and a blithe personality. Her great exuberance for learning prompted her to continue with her studies after high school. She became disgruntled, however, when she learned that the university in Warsaw was closed to women. Determined to receive a higher education, she defiantly left Poland and in 1891 entered the Sorbonne, a French university, where she earned her master's degree and doctorate in physics.
Marie was fortunate to have studied at the Sorbonne with some of the greatest scientists of her day, one of whom was Pierre Curie. Marie and Pierre were married in 1895 and spent many productive years working together in the physics laboratory. A short time after they discovered radium, Pierre was killed by a horse-drawn wagon in 1906. Marie was stunned by this horrible misfortune and endured heartbreaking anguish. Despondently she recalled their close relationship and the joy that they had shared in scientific research. The fact that she had two young daughters to raise by herself greatly increased her distress.
Curie's feeling of desolation finally began to fade when she was asked to succeed her husband as a physics professor at the Sorbonne. She was the first woman to be given a professorship at the world-famous university.
In 1911 she received the Nobel Prize in chemistry for isolating radium. Although Marie Curie eventually suffered a fatal illness from her long exposure to radium, she never became disillusioned about her work. Regardless of the consequences, she had dedicated herself to science and to revealing the mysteries of the physical world.

The Curies' ____ collaboration helped to unlock the secrets of the atom.
A Finally, each student read another passage (P4) of text from the TOEFL that was translated by Google Translate from Malay to English. This language pair was chosen because of the many errors that resulted (a possible 'worst case'). The complete translated passage is shown below: "Frank Towers", whether you heard about it? He probably was a member of the dianggapi safe after sailing with three ships ditakdir unfortunate that in 1900-an. There also recognize that as a Berjiwa most fortunate. He is a member of the fire berlengkuran part of middle-aged working in the engine room.
There is also a normal life as a mala menganggapinya a hard worker but he has the ability to avoid fatal accidents associated with several ships dahshiat ever recorded. He also said the ship had become a member on the Titanic when the ship is menghempas the ice mountain. Two years later, he was working in the Empress of Ireland when the ship collided with Storstad. More than a thousand people were killed as a result of the disaster. Account after that, he found work in the Lusitania in May 1915 when hit by U-20 torpedoes. He continued to live with berrupanya not pierced directly. If you start kewujutan doubtful about this person, you may not be far from the truth. It did not register that there dicatakan a man named Frank Tower, who has worked in one of three vessels. Legendary Frank Towers seems one of the case which originated from urban folklore, and menimbul due keringinan human to experience victory over the disastrous situation. (Although) Fact or design, Frank Towers is one of the multicultural character arcs that help polish your history books.
In this translation, unknown words (e.g. "mala menganggapinya") were simply repeated. The correct translation to English (not given to the students) is shown here: Frank Tower, have you ever heard of him? He is the questionable figure who supposedly survived three doomed ships in the 1900's. Some consider him one of the luckiest men alive. He was touted to be a middle-aged fireman in the engine room. Some considered him an ordinary, hardworking person, but he had the ability to avoid dying in some of the most horrendous ocean liner accidents ever recorded.
He was said to have once been a crew member on the Titanic at the time that the ship hit the iceberg. Two years later, he was working on the Empress of Ireland when she collided with the Storstad. Over one thousand people died in that disaster. He was then employed in May of 1915 on the Lusitania when it was hit by a U-20 torpedo. He apparently lived through that without a scratch as well. If you are beginning to doubt this man's existence, you are probably not too far from the truth. No records have been found ever listing a man by Frank Towers working on any of the three ships.
The legend of Frank Towers seems to be another case of an urban folk tale, humanity's desire to see triumph over a tragic situation. Fact or fiction, Frank Towers is one of the multiple characters that help color the history books.
Students were asked to answer three questions about this translated text (correct answers are shown in bold):

This passage sheds doubt on
A. The fact that three ships sank.
B. The thought that America was at war with other countries.
C. Frank Towers causing the ships to sink.
D. Urban folk tales are not always based on real people.

According to this passage, which of the following phrases best captures the intent of the article.
A. A stitch in time saves nine.
B. Don't believe everything you read.
C. Seeing is believing.
D. There are many layers of an onion.

The article states that urban folk tales are created because
A. They are fun to hear.
B. Humanity likes to see someone triumph over tragedy C. They trick people into believing lies.
D. People do not remember details clearly.
In addition, they were asked what percentage of the text they thought they understood, and how much they should have been able to understand.

Results
Students reported they were able to understand on average only 55.91% (std dev 23.67%) of the translated text (P4), but should have been able to understand 77.10% (std dev 23.51%), a significant difference (T = -6.971, p < 0.001). On average, they got 56.67% of the reading comprehsion questions correct (std dev 28.76%), very close to their reported rate of understanding. In contrast, they got only 41.25 % of the non-translated passage (P3) questions correct (std dev 15.91%). The lower score for the non-translated text with no word-choice or grammatical errors shows that the subject matter could be a major factor in determining the ultimate comprehension.
Students did much better on the grammar-focused Maze tests, however, with mean correct answers of 79.75% for P1 and 92.50% for P2 (std dev 16.07% and 14.71%, respectively).
There was no significant difference between the 63 native-and 17 non-native-English speakers on any of the measures except for their performance on the second Maze test P2 in which the non-native speakers did better (Chi-square = 5.942, 0 =0.015). Also, surprisingly, the non-native-English speakers performed better comprehending the translated text, but not significantly.
As expected, significant correlations were found between the first and second Maze test paragraph average scores (P1 and P2) (R = 0.335, p = 0.002 ). That is, students who did well on one word choice and grammatical test did well on the other. There was also a significant correlation between the first Maze test and the translated text scores (P1 and P4) (R = -0.225 , p = 0.045). Thus, word choice and grammatical knowledge might be a predictor of translation comprehension, although there was no significant correlation between the second Maze test P2 and P4, between P1 and between the nontranslated text P3, or between P2 and P3. However, the correlation between P1 and P4 was negative. That is, those who did well on the first Maze test did worse on the translated comprehension test. Finally, the correlation between the nontranslated text P3 and translated text P4 accuracies was close to being significant (R = 0.202, p = 0.073 ), perhaps because of the change in subject matter. That is, those who comprehended the non-translated text tended to comprehend the translated text, but not significantly.

Discussion
The study suffers from several limitations. First, only a small amount of text was used in two Maze and two TOEFL reading comprehension tests. Other language fluency and comprehension tests might have given other results.
Second, samples of translated text with different degrees of errors might produce other outcomes. In particular, in this study, students understood the non-translated text with no errors less than the translated text with many errors, indicating prior knowledge of the subject matter could have the most influence on the comprehension of material.
Third, the study could benefit from subjects with larger variations in fluency. Here, even those who spoke English as a second language were very knowledgeable as there were few differences with native-English speakers. In fact, nonnative-English speakers did better on the second Maze test.

CONCLUSION
For the first time, this study has focused on the factors that influence a reader's comprehension of machine translated text.
The study concentrated on language fluency, however, and few significant correlations were found. In fact, the study indicated that vocabulary and grammatical knowledge might be negatively correlated with reading comprehension.
Perhaps the major finding, however, was that students comprehended a non-translated text passage with no errors less than they comprehended the transated text (on a different subject) with many errors. Thus, future studies should focus on the affect of readers' prior topic knowledge rather than language fluency as a possible predictor of reading comprehension.