en

A Probabilistic Answer Type Model

Christopher Pinchak Department of Computing Science University of Alberta Edmonton, Alberta, Canada pinchak@cs.ualberta.ca Dekang Lin Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA lindek@google.com

Abstract All questions are implicitly associated with an expected answer type. Unlike previous approaches that require a prede- fined set of question types, we present a method for dynamically constructing a probability-based answer type model for each different question. Our model evaluates the appropriateness of a poten- tial answer by the probability that it fits into the question contexts. Evaluation is performed against manual and semi- automatic methods using a fixed set of an- swer labels. Results show our approach to be superior for those questions classified as having a miscellaneous answer type. 1 Introduction Given a question, people are usually able to form an expectation about the type of the answer, even if they do not know the actual answer. An accu- rate expectation of the answer type makes it much easier to select the answer from a sentence that contains the query words. Consider the question “What is the capital of Norway?” We would ex- pect the answer to be a city and could filter out most of the words in the following sentence: The landed aristocracy was virtually crushed by Hakon V, who reigned from 1299 to 1319, and Oslo became the capital of Norway, re- placing Bergen as the principal city of the kingdom. The goal of answer typing is to determine whether a word’s semantic type is appropriate as an answer for a question. Many previous ap- proaches to answer typing, e.g., (Ittycheriah et al., 2001; Li and Roth, 2002; Krishnan et al., 2005), employ a predefined set of answer types and use supervised learning or manually constructed rules to classify a question according to expected an- swer type. A disadvantage of this approach is that there will always be questions whose answers do not belong to any of the predefined types. Consider the question: “What are tourist attrac- tions in Reims?” The answer may be many things: a church, a historic residence, a park, a famous intersection, a statue, etc. A common method to deal with this problem is to define a catch-all class. This class, however, tends not to be as effective as other answer types. Another disadvantage of predefined answer types is with regard to granularity. If the types are too specific, they are more difficult to tag. If they are too general, too many candidates may be identified as having the appropriate type. In contrast to previous approaches that use a su- pervised classifier to categorize questions into a predefined set of types, we propose an unsuper- vised method to dynamically construct a proba- bilistic answer type model for each question. Such a model can be used to evaluate whether or not a word fits into the question context. For exam- ple, given the question “What are tourist attrac- tions in Reims?”, we would expect the appropriate answers to fit into the context “X is a tourist attrac- tion.” From a corpus, we can find the words that appeared in this context, such as: A-Ama Temple, Aborigine, addition, Anak Krakatau, archipelago, area, baseball, Bletchley Park, brewery, cabaret, Cairo, Cape Town, capital, center, ... Using the frequency counts of these words in the context, we construct a probabilistic model to compute P ( in ( w, Γ) | w ) , the probability for a word w to occur in a set of contexts Γ , given an occurrence of w . The parameters in this model are obtained from a large, automatically parsed, un- labeled corpus. By asking whether a word would occur in a particular context extracted from a ques-

393

 en

tion, we avoid explicitly specifying a list of pos- sible answer types. This has the added benefit of being easily adapted to different domains and corpora in which a list of explicit possible answer types may be difficult to enumerate and/or identify within the text. The remainder of this paper is organized as fol- lows. Section 2 discusses the work related to an- swer typing. Section 3 discusses some of the key concepts employed by our probabilistic model, in- cluding word clusters and the contexts of a ques- tion and a word. Section 4 presents our probabilis- tic model for answer typing. Section 5 compares the performance of our model with that of an or- acle and a semi-automatic system performing the same task. Finally, the concluding remarks in are made in Section 6. 2 Related Work Light et al. (2001) performed an analysis of the effect of multiple answer type occurrences in a sentence. When multiple words of the same type appear in a sentence, answer typing with fixed types must assign each the same score. Light et al. found that even with perfect answer sentence identification, question typing, and semantic tag- ging, a system could only achieve 59% accuracy over the TREC-9 questions when using their set of 24 non-overlapping answer types. By computing the probability of an answer candidate occurring in the question contexts directly, we avoid having multiple candidates with the same level of appro- priateness as answers. There have been a variety of approaches to de- termine the answer types, which are also known as Qtargets (Echihabi et al., 2003). Most previous approaches classify the answer type of a question as one of a set of predefined types. Many systems construct the classification rules manually (Cui et al., 2004; Greenwood, 2004; Hermjakob, 2001). The rules are usually triggered by the presence of certain words in the question. For example, if a question contains “author” then the expected answer type is Person. The number of answer types as well as the num- ber of rules can vary a great deal. For example, (Hermjakob, 2001) used 276 rules for 122 answer types. Greenwood (2004), on the other hand, used 46 answer types with unspecified number of rules. The classification rules can also be acquired with supervised learning. Ittycheriah, et al. (2001) describe a maximum entropy based question clas- sification scheme to classify each question as hav- ing one of the MUC answer types. In a similar ex- periment, Li & Roth (2002) train a question clas- sifier based on a modified version of SNoW using a richer set of answer types than Ittycheriah et al. The LCC system (Harabagiu et al., 2003) com- bines fixed types with a novel loop-back strategy. In the event that a question cannot be classified as one of the fixed entity types or semantic concepts derived from WordNet (Fellbaum, 1998), the an- swer type model backs off to a logic prover that uses axioms derived form WordNet, along with logic rules, to justify phrases as answers. Thus, the LCC system is able to avoid the use of a miscel- laneous type that often exhibits poor performance. However, the logic prover must have sufficient ev- idence to link the question to the answer, and gen- eral knowledge must be encoded as axioms into the system. In contrast, our answer type model derives all of its information automatically from unannotated text. Answer types are often used as filters. It was noted in (Radev et al., 2002) that a wrong guess about the answer type reduces the chance for the system to answer the question correctly by as much as 17 times. The approach presented here is less brittle. Even if the correct candidate does not have the highest likelihood according to the model, it may still be selected when the answer extraction module takes into account other factors such as the proximity to the matched keywords. Furthermore, a probabilistic model makes it eas- ier to integrate the answer type scores with scores computed by other components in a question an- swering system in a principled fashion. 3 Resources Before introducing our model, we first describe the resources used in the model. 3.1 Word Clusters Natural language data is extremely sparse. Word clusters are a way of coping with data sparseness by abstracting a given word to a class of related words. Clusters, as used by our probabilistic an- swer typing system, play a role similar to that of named entity types. Many methods exist for clus- tering, e.g., (Brown et al., 1990; Cutting et al., 1992; Pereira et al., 1993; Karypis et al., 1999). We used the Clustering By Committee (CBC)

394

 en

Table 1: Words and their clusters 720 3048 302 840 1642 Word Clusters suite software, network, wireless,... rooms, bathrooms, restrooms, ... meeting room, conference room, ... ghost rabbit, squirrel, duck, elephant, frog, ... goblins, ghosts, vampires, ghouls, ... punk, reggae, folk, pop, hip-pop, ... huge, larger, vast, significant, ... coming-of-age, true-life, ... clouds, cloud, fog, haze, mist, ... algorithm (Pantel and Lin, 2002) on a 10 GB En- glish text corpus to obtain 3607 clusters. The fol- lowing is an example cluster generated by CBC: tension, anger, anxiety, tensions, frustration, resentment, uncertainty, confusion, conflict, discontent, insecurity, controversy, unease, bitterness, dispute, disagreement, nervous- ness, sadness, despair, animosity, hostility, outrage, discord, pessimism, anguish, ... In the clustering generated by CBC, a word may belong to multiple clusters. The clusters to which a word belongs often represent the senses of the word. Table 1 shows two example words and their clusters. 3.2 Contexts The context in which a word appears often im- poses constraints on the semantic type of the word. This basic idea has been exploited by many pro- posals for distributional similarity and clustering, e.g., (Church and Hanks, 1989; Lin, 1998; Pereira et al., 1993). Similar to Lin and Pantel (2001), we define the contexts of a word to be the undirected paths in dependency trees involving that word at either the beginning or the end. The following diagram shows an example dependency tree: obj Which city hosted the 1988 Winter Olympics? The links in the tree represent dependency rela- tionships. The direction of a link is from the head to the modifier in the relationship. Labels associ- ated with the links represent types of relations. In a context, the word itself is replaced with a variable X. We say a word is the filler of a context if it replaces X. For example, the contexts for the word “Olympics” in the above sentence include the following paths: 384 326 764 792 1406 302 423 398 403 428 Context of “Olympics” Explanation X NN Winter Winter X X NN 1988 1988 X X obj host host X X obj subj host city city hosted X In these paths, words are reduced to their root forms and proper names are reduced to their entity tags (we used MUC7 named entity tags). Paths allow us to balance the specificity of con- texts and the sparseness of data. Longer paths typ- ically impose stricter constraints on the slot fillers. However, they tend to have fewer occurrences, making them more prone to errors arising from data sparseness. We have restricted the path length to two (involving at most three words) and require the two ends of the path to be nouns. We parsed the AQUAINT corpus (3GB) with Minipar (Lin, 2001) and collected the frequency counts of words appearing in various contexts. Parsing and database construction is performed off-line as the database is identical for all ques- tions. We extracted 527,768 contexts that ap- peared at least 25 times in the corpus. An example context and its fillers are shown in Figure 1. subj obj X host Olympics Africa 2 grant 1 readiness 2 AP 1 he 2 Rio de Janeiro 1 Argentina 1 homeland 3 Rome 1 Athens 16 IOC 1 Salt Lake City 2 Atlanta 3 Iran 2 school 1 Bangkok 1 Jakarta 1 S. Africa 1 . . . . . . . . . decades 1 president 2 Zakopane 4 facility 1 Pusan 1 government 1 race 1 Figure 1: An example context and its fillers 3.2.1 Question Contexts To build a probabilistic model for answer typ- ing, we extract a set of contexts, called question contexts, from a question. An answer is expected to be a plausible filler of the question contexts. Question contexts are extracted from a question with two rules. First, if the wh-word in a ques- tion has a trace in the parse tree, the question con- texts are the contexts of the trace. For example, the

det subj

det

NN

NN

395

 en

question “What do most tourists visit in Reims?” is parsed as: 2486 1896 130 i in What i do most tourists visit e i in Reims? The symbol e i is the trace of what i . Minipar generates the trace to indicate that the word what is the object of visit in the deep structure of the sentence. The following question contexts are ex- tracted from the above question: 374 327 432 326 754 1665 298 427 408 Context Explanation X obj visit subj tourist tourist visits X X obj visit in Reims visit X in Reims The second rule deals with situations where the wh-word is a determiner, as in the question “Which city hosted the 1988 Winter Olympics?” (the parse tree for which is shown in section 3.2). In such cases, the question contexts consist of a single context involving the noun that is modified by the determiner. The context for the above sen- subj tence is X city , corresponding to the sentence “X is a city.” This context is used because the question explicitly states that the desired answer is a city. The context overrides the other contexts be- cause the question explicitly states the desired an- swer type. Experimental results have shown that using this context in conjunction with other con- texts extracted from the question produces lower performance than using this context alone. In the event that a context extracted from a ques- tion is not found in the database, we shorten the context in one of two ways. We start by replac- ing the word at the end of the path with a wildcard that matches any word. If this fails to yield en- tries in the context database, we shorten the con- text to length one and replace the end word with automatically determined similar words instead of a wildcard. 3.2.2 Candidate Contexts Candidate contexts are very similar in form to question contexts, save for one important differ- ence. Candidate contexts are extracted from the parse trees of the answer candidates rather than the question. In natural language, some words may be polysemous. For example, Washington may re- fer to a person, a city, or a state. The occurrences of Washington in “Washington’s descendants” and “suburban Washington” should not be given the same score when the question is seeking a loca- tion. Given that the sense of a word is largely de- termined by its local context (Choueka and Lusig- nan, 1985), candidate contexts allow the model to take into account the candidate answers’ senses implicitly. 4 Probabilistic Model The goal of an answer typing model is to evalu- ate the appropriateness of a candidate word as an answer to the question. If we assume that a set of answer candidates is provided to our model by some means (e.g., words comprising documents extracted by an information retrieval engine), we wish to compute the value P ( in ( w, F Q ) I w ) . That is, the appropriateness of a candidate answer w is proportional to the probability that it will occur in the question contexts F Q extracted from the ques- tion. To mitigate data sparseness, we can introduce a hidden variable C that represents the clusters to which the candidate answer may belong. As a can- didate may belong to multiple clusters, we obtain: P ( in ( w, P Q ) Iw ) = E P ( in ( w, P Q ) , CIw ) (1) C = E P ( CIw ) P ( in ( w, P Q ) IC, w ) (2) C Given that a word appears, we assume that it has the same probability to appear in a context as all other words in the same cluster. Therefore: P ( in ( w, P Q ) IC, w ) � P ( in ( C, P Q ) IC ) (3) We can now rewrite the equation in (2) as: P ( in ( w, P Q ) Iw ) Pz� E P ( CIw ) P ( in ( C, P Q ) IC ) (4) C This equation splits our model into two parts: one models which clusters a word belongs to and the other models how appropriate a cluster is to the question contexts. When F Q consists of multi- ple contexts, we make the naive Bayes assumption that each individual context -y Q E F Q is indepen- dent of all other contexts given the cluster C . P ( in ( w, P Q ) Iw ) Pz� E P ( CIw ) H P ( in ( C,γ Q ) IC ) (5) C γ Q ∈ Γ Q Equation (5) needs the parameters P ( C l w ) and P ( in ( C, -y Q ) IC ) , neither of which are directly available from the context-filler database. We will discuss the estimation of these parameters in Sec- tion 4.2.

det

det

subj

obj

396

 en

4.1 Using Candidate Contexts The previous model assigns the same likelihood to every instance of a given word. As we noted in section 3.2.2, a word may be polysemous. To take into account a word’s context, we can instead com- pute P ( in ( w, F Q ) Iw, in ( w, F w )) , where F w is the set of contexts for the candidate word w in a re- trieved passage. By introducing word clusters as intermediate variables as before and making a similar assump- tion as in equation (3), we obtain: P ( in ( w, r Q ) lw, in ( w, r w )) = E P ( in ( w, r Q ) , Cl w, in ( w, r w )) (6) C E � P ( Clw, in ( w, r w )) P ( in ( C, r Q ) lC ) (7) C Like equation (4), equation (7) partitions the model into two parts. Unlike P ( CIw ) in equation (4), the probability of the cluster is now based on the particular occurrence of the word in the candi- date contexts. It can be estimated by: P ( Clw,in ( w, r w )) = P ( in ( w, r w ) lw, C ) P ( w, C ) (8) P ( in ( w, r w ) lw ) P ( w ) H P ( in ( w, 7 w ) l w, C ) 7 w E r w H P ( in ( w,7 w ) lw ) 7 w E r w 4.2 Estimating Parameters Our probabilistic model requires the parameters P ( CIw ) , P ( CIw, in ( w, -y )) , and P ( in ( C,-y ) IC ) , where w is a word, C is a cluster that w belongs to, and -y is a question or candidate context. This sec- tion explains how these parameters are estimated without using labeled data. The context-filler database described in Sec- tion 3.2 provides the joint and marginal fre- quency counts of contexts and words ( Iin ( -y, w ) I , Iin ( *,-y ) I and Iin ( w,* ) I ). These counts al- low us to compute the probabilities P ( in ( w,-y )) , P ( in ( w, * )) , and P ( in ( *,-y )) . We can also com- pute P ( in ( w, -y ) Iw ) , which is smoothed with add- one smoothing (see equation (11) in Figure 2). The estimation of P ( CIw ) presents a challenge. We have no corpus from which we can directly measure P ( CIw ) because word instances are not labeled with their clusters. P ( in ( w, 7 ) lw ) = lin ( w, 7 ) l + P ( in ( *,7 )) (11) lin ( w, * ) l + 1 � 1 if w E C , P u ( Clw ) = IT C I 1 w E C I I I (12) / 0 otherwise l E sim ( w,w ' ) x P u ( Clw ' ) 1190 674 1883 635 173 341 P ( Clw ) = w I E S ( w ) (13) E sim ( w,w ' ) x P u ( C ' lw ' ) I C I I w E C I I , w I E S ( w ) P ( in ( C, 7 ) lC ) = w I E C E P ( Clw ' ) x lin ( w ' ,7 ) l + P ( in ( *,7 )) (14) E P ( Clw ' ) x lin ( w ' , * ) l + 1 w I E C Figure 2: Probability estimation We use the average weighted “guesses” of the top similar words of w to compute P ( CIw ) (see equation 13). The intuition is that if w ' and w are similar words, P ( CIw ' ) and P ( CIw ) tend to have similar values. Since we do not know P ( CIw ' ) either, we substitute it with uniform dis- tribution P ,, ( CIw ' ) as in equation (12) of Fig- ure 2. Although P ,, ( CIw ' ) is a very crude guess, the weighted average of a set of such guesses can often be quite accurate. The similarities between words are obtained as a byproduct of the CBC algorithm. For each word, we use S ( w ) to denote the top- n most similar words ( n =50 in our experiments) and sim ( w, w ' ) to denote the similarity between words w and w ' . The following is a sample similar word list for the word suit: S ( suit ) _ { lawsuit 0.49, suits 0.47, com- plaint 0.29, lawsuits 0.27, jacket 0.25, coun- tersuit 0.24, counterclaim 0.24, pants 0.24, trousers 0.22, shirt 0.21, slacks 0.21, case 0.21, pantsuit 0.21, shirts 0.20, sweater 0.20, coat 0.20, ... 1 The estimation for P ( CIw, in ( w, -y w )) is sim- ilar to that of P ( CIw ) except that instead of all w ' E S ( w ) , we instead use {w ' Iw ' E S ( w ) n in ( w ' , -y w ) � . By only looking at a particular con- text -y w , we may obtain a different distribution over C than P ( CIw ) specifies. In the event that the data are too sparse to estimate P ( CIw, in ( w, -y w )) , we fall back to using P ( CIw ) . P ( in ( C,-y ) IC ) is computed in (14) by assum- ing each instance of w contains a fractional in- stance of C and the fractional count is P ( CIw ) . Again, add-one smoothing is used.

�

x P ( Clw ) (9)

3086 1296 550 = H �lw ) P ( Clw, in ( w, 7 w )) 1 7 w E r w P ( C J x P ( Clw ) (10)

397

 en

1502 980 940 1032 1032 1032 230 226 206 207 206 217 System Median % Top 1% Top 5% Top 10% Top 50% Oracle 0.7% 89(57%) 123(79%) 131(85%) 154(99%) Frequency 7.7% 31(20%) 67(44%) 86(56%) 112(73%) Our model 1.2% 71(46%) 106(69%) 119(77%) 146(95%) no cand. contexts 2.2% 58(38%) 102(66%) 113(73%) 145(94%) ANNIE 4.0% 54(35%) 79(51%) 93(60%) 123(80%) Table 2: Summary of Results

5 Experimental Setup & Results We evaluate our answer typing system by using it to filter the contents of documents retrieved by the information retrieval portion of a question an- swering system. Each answer candidate in the set of documents is scored by the answer typing sys- tem and the list is sorted in descending order of score. We treat the system as a filter and observe the proportion of candidates that must be accepted by the filter so that at least one correct answer is accepted. A model that allows a low percentage of candidates to pass while still allowing at least one correct answer through is favorable to a model in which a high number of candidates must pass. This represents an intrinsic rather than extrinsic evaluation (Moll´a and Hutchinson, 2003) that we believe illustrates the usefulness of our model. The evaluation data consist of 154 questions from the TREC-2003 QA Track (Voorhees, 2003) satisfying the following criteria, along with the top 10 documents returned for each question as iden- tified by NIST using the PRISE 1 search engine. • the question begins with What, Which, or Who. We restricted the evaluation such ques- tions because our system is designed to deal with questions whose answer types are often semantically open-ended noun phrases. • There exists entry for the question in the an- swer patterns provided by Ken Litkowski 2 . • One of the top-10 documents returned by PRISE contains a correct answer. We compare the performance of our prob- abilistic model with that of two other sys- tems. Both comparison systems make use of a small, predefined set of manually-assigned MUC- 7 named-entity types (location, person, organiza- tion, cardinal, percent, date, time, duration, mea- sure, money) augmented with thing-name (proper 1 www.itl.nist.gov/iad/894.02/works/papers/zp2/zp2.html 2 trec.nist.gov/data/qa/2003 qadata/03QA.tasks/t12.pats.txt names of inanimate objects) and miscellaneous (a catch-all answer type of all other candidates). Some examples of thing-name are Guinness Book of World Records , Thriller , Mars Pathfinder , and Grey Cup . Examples of miscellaneous answers are copper , oil , red , and iris . The differences in the comparison systems is with respect to how entity types are assigned to the words in the candidate documents. We make use of the ANNIE (Maynard et al., 2002) named entity recognition system, along with a manual assigned “oracle” strategy, to assign types to candidate an- swers. In each case, the score for a candidate is either 1 if it is tagged as the same type as the ques- tion or 0 otherwise. With this scoring scheme pro- ducing a sorted list we can compute the probability of the first correct answer appearing at rank R = k as follows: k - 2 T � - 77 t — c—i c (15) 11 t — i t—k +1 where t is the number of unique candidate answers that are of the appropriate type and c is the number of unique candidate answers that are correct. Using the probabilities in equation (15), we compute the expected rank, E ( R ) , of the first cor- rect answer of a given question in the system as: kP ( R = k ) (16) Answer candidates are the set of ANNIE- identified tokens with stop words and punctuation removed. This yields between 900 and 8000 can- didates for each question, depending on the top 10 documents returned by PRISE. The oracle system represents an upper bound on using the predefined set of answer types. The ANNIE system repre- sents a more realistic expectation of performance. The median percentage of candidates that are accepted by a filter over the questions of our eval- uation data provides one measure of performance and is preferred to the average because of the ef- fect of large values on the average. In QA, a sys- tem accepting 60% of the candidates is not signif- icantly better or worse than one accepting 100%,

P ( R = k ) =

i =0

t - c +1 E k =1

E ( R ) =

398

 en

1013 888 629 873 730 1171 1152 706 705 211 418 230 197 202 196 221 226 201 197 202 216 230 197 197 201 198 System Measure Question Type All (154) Location Location (57) Person (17) Organization Organization (19) Thing-Name Thing-Name (17) Misc (37) Other (7) Our model Median 1.2% 0.8% 2.0% 1.3% 3.7% 3.5% 12.2% Top 1% 71 34 6 9 7 13 2 Top 5% 106 53 11 11 10 19 2 Top 10% 119 55 12 17 10 22 3 Top 50% 146 56 16 18 17 34 5 Oracle Median 0.7% 0.4% 1.0% 0.3% 0.4% 16.0% 0.3% Top 1% 89 44 8 16 14 1 6 Top 5% 123 57 17 19 17 6 7 Top 10% 131 57 17 19 17 14 7 Top 50% 154 57 17 19 17 37 7 ANNIE Median 4.0% 0.6% 1.4% 6.1% 100% 16.7% 50.0% Top 1% 54 39 5 7 0 0 3 Top 5% 79 53 12 9 0 2 3 Top 10% 93 54 13 11 0 12 3 Top 50% 123 56 16 15 5 28 3 Table 3: Detailed breakdown of performance

but the effect on average is quite high. Another measure is to observe the number of questions with at least one correct answer in the top N % for various values of N . By examining the number of correct answers found in the top N % we can better understand what an effective cutoff would be. The overall results of our comparison can be found in Table 2. We have added the results of a system that scores candidates based on their fre- quency within the document as a comparison with a simple, yet effective, strategy. The second col- umn is the median percentage of where the highest scored correct answer appears in the sorted candi- date list. Low percentage values mean the answer is usually found high in the sorted list. The re- maining columns list the number of questions that have a correct answer somewhere in the top N % of their sorted lists. This is meant to show the ef- fects of imposing a strict cutoff prior to running the answer type model. The oracle system performs best, as it bene- fits from both manual question classification and manual entity tagging. If entity assignment is performed by an automatic system (as it is for ANNIE), the performance drops noticeably. Our probabilistic model performs better than ANNIE and achieves approximately 2/3 of the perfor- mance of the oracle system. Table 2 also shows that the use of candidate contexts increases the performance of our answer type model. Table 3 shows the performance of the oracle system, our model, and the ANNIE system broken down by manually-assigned answer types. Due to insufficient numbers of questions, the cardinal, percent, time, duration, measure, and money types are combined into an “Other” category. When compared with the oracle system, our model per- forms worse overall for questions of all types ex- cept for those seeking miscellaneous answers. For miscellaneous questions, the oracle identifies all tokens that do not belong to one of the other known categories as possible answers. For all questions of non-miscellaneous type, only a small subset of the candidates are marked appropriate. In particular, our model performs worse than the oracle for questions seeking persons and thing- names. Person questions often seek rare person names, which occur in few contexts and are diffi- cult to reliably cluster. Thing-name questions are easy for a human to identify but difficult for au- tomatic system to identify. Thing-names are a di- verse category and are not strongly associated with any identifying contexts. Our model outperforms the ANNIE system in general, and for questions seeking organizations, thing-names, and miscellaneous targets in partic- ular. ANNIE may have low coverage on organi- zation names, resulting in reduced performance. Like the oracle, ANNIE treats all candidates not assigned one of the categories as appropriate for miscellaneous questions. Because ANNIE cannot identify thing-names, they are treated as miscella- neous. ANNIE shows low performance on thing- names because words incorrectly assigned types are sorted to the bottom of the list for miscella- neous and thing-name questions. If a correct an- swer is incorrectly assigned a type it will be sorted near the bottom, resulting in a poor score.

399

 en

6 Conclusions We have presented an unsupervised probabilistic answer type model. Our model uses contexts de- rived from the question and the candidate answer to calculate the appropriateness of a candidate an- swer. Statistics gathered from a large corpus of text are used in the calculation, and the model is constructed to exploit these statistics without be- ing overly specific or overly general. The method presented here avoids the use of an explicit list of answer types. Explicit answer types can exhibit poor performance, especially for those questions not fitting one of the types. They must also be redefined when either the domain or corpus substantially changes. By avoiding their use, our answer typing method may be easier to adapt to different corpora and question answering domains (such as bioinformatics). In addition to operating as a stand-alone answer typing component, our system can be combined with other existing answer typing strategies, es- pecially in situations in which a catch-all answer type is used. Our experimental results show that our probabilistic model outperforms the oracle and a system using automatic named entity recognition under such circumstances. The performance of our model is better than that of the semi-automatic system, which is a better indication of the expected performance of a comparable real-world answer typing system. Acknowledgments The authors would like to thank the anonymous re- viewers for their helpful comments on improving the paper. The first author is supported by the Nat- ural Sciences and Engineering Research Council of Canada, the Alberta Ingenuity Fund, and the Al- berta Informatics Circle of Research Excellence. References P.F. Brown, V.J. Della Pietra, P.V. deSouza, J.C. Lai, and R.L. Mercer. 1990. Class-based n-gram Models of Natural Language. Computational Linguistics , 16(2):79–85. Y. Choueka and S. Lusignan. 1985. Disambiguation by Short Contexts. Computer and the Humanities , 19:147–157. K. Church and P. Hanks. 1989. Word Association Norms, Mutual Information, and Lexicography. In Proceedings of ACL-89 , pages 76–83, Vancouver, British Columbia, Canada. H. Cui, K. Li, R. Sun, T-S. Chua, and M-K. Kan. 2004. Na- tional University of Singapore at the TREC-13 Question Answering Main Task. In Notebook of TREC 2004 , pages 34–42, Gaithersburg, Maryland. D.R. Cutting, D. Karger, J. Pedersen, and J.W. Tukey. 1992. Scatter/Gather: A Cluster-based Approach to Browsing Large Document Collections. In Proceedings of SIGIR- 92 , pages 318–329, Copenhagen, Denmark. A. Echihabi, U. Hermjakob, E. Hovy, D. Marcu, E. Melz, and D. Ravichandran. 2003. Multiple-Engine Question Answering in TextMap. In Proceedings of TREC 2003 , pages 772–781, Gaithersburg, Maryland. C. Fellbaum. 1998. WordNet. An Electronic Lexical Database . MIT Press, Cambridge, Massachusetts. M.A. Greenwood. 2004. AnswerFinder: Question Answer- ing from your Desktop. In Proceedings of the Seventh Annual Colloquium for the UK Special Interest Group for Computational Linguistics (CLUK ’04) , University of Birmingham, UK. S. Harabagiu, D. Moldovan, C. Clark, M. Bowden, J. Williams, and J. Bensley. 2003. Answer Mining by Combining Extraction Techniques with Abductive Rea- soning. In Proceedings of TREC 2003 , pages 375–382, Gaithersburg, Maryland. U. Hermjakob. 2001. Parsing and Question Classification for Question Answering. In Proceedings of the ACL Work- shop on Open-Domain Question Answering , Toulouse, France. A. Ittycheriah, M. Franz, W-J. Zhu, and A. Ratnaparkhi. 2001. Question Answering Using Maximum Entropy Components. In Proceedings of NAACL 2001 , Pittsburgh, Pennsylvania. G. Karypis, E.-H. Han, and V. Kumar. 1999. Chameleon: A Hierarchical Clustering Algorithm using Dynamic Model- ing. IEEE Computer. Special Issue on Data Analysis and Mining , 32(8):68–75. V. Krishnan, S. Das, and S. Chakrabarti. 2005. Enhanced Answer Type Inference from Questions using Sequential Models. In Proceedings of HLT/EMNLP 2005 , pages 315–322, Vancouver, British Columbia, Canada. X. Li and D. Roth. 2002. Learning Question Classifiers. In Proceedings of COLING 2002 , pages 556–562, Taipei, Taiwan. M. Light, G. Mann, E. Riloff, and E. Breck. 2001. Analyses for Elucidating Current Question Answering Technology. Natural Language Engineering , 7(4):325–342. D. Lin and P. Pantel. 2001. Discovery of Inference Rules for Question Answering. Natural Language Engineering , 7(4):343–360. D. Lin. 1998. Automatic Retrieval and Clustering of Similar Words. In Proceedings of COLING-ACL 1998 , Montreal, Qu´ebec, Canada. D. Lin. 2001. Language and Text Analysis Tools. In Pro- ceedings of HLT 2001 , pages 222–227, San Diego, Cali- fornia. D. Maynard, V. Tablan, H. Cunningham, C. Ursu, H. Sag- gion, K. Bontcheva, and Y. Wilks. 2002. Architectural Elements of Language Engineering Robustness. Natural Language Engineering , 8(2/3):257–274. D. Moll´a and B. Hutchinson. 2003. Intrinsic versus Extrinsic Evaluations of Parsing Systems. In Proceedings of EACL Workshop on Evaluation Initiatives in Natural Language Processing , pages 43–50, Budapest, Hungary. P. Pantel and D. Lin. 2002. Document Clustering with Com- mittees. In Proceedings of SIGIR 2002 , pages 199–206, Tampere, Finland. F. Pereira, N. Tishby, and L. Lee. 1993. Distributional Clus- tering of English Words. In Proceedings of ACL 1992 , pages 183–190. D. Radev, W. Fan, H. Qi, H. Wu, and A. Grewal. 2002. Prob- ablistic Question Answering on the Web. In Proceedings of the Eleventh International World Wide Web Conference . E.M. Voorhees. 2003. Overview of the TREC 2003 Ques- tion Answering Track. In Proceedings of TREC 2003 , Gaithersburg, Maryland.

400