Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Suppose i have Two Text Articles (article 'a' => x words, article 'b' => y words)

i find the total number of words in 'a' = x the total number of occurances of the word "the" in the article = x1

probability of the word 'the' in article 'a' => x1/x (i have stored this as decimal, so i dont have exact x1 and x at a later stage)

now i want to merge(Add ?) the probability of the same word 'the' in article 'b', which is y1/y

how should i do that, i suppose simply adding will be wrong.

as a later stage i will get more articles 'c', 'd'... and i want to keep the probability updates, how should i do this ?

Thank you.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

You need to keep track of the total size of the articles that you've processed to date. Suppose that $p$ is the probability that a randomly chosen word in the articles processed so far is 'the', and those articles contain altogether $n$ words. You now get a new article with $k$ instances of 'the' in $m$ words. You should then update $p$ to $\frac{pn+k}{n+m}$ and $n$ to $n+m$ in preparation for processing the next article.

share|improve this answer

If I understand what you are doing correctly, then you want as your final case (total number of occurrences of "the") / (total number of words). So in this case, you will take

$$\dfrac{x_1 + y_1}{x + y}$$

Now if you didn't record $x, y, x_1, y_1$ separately, then I guess you'll have to recount them. Or you at least need to know the proportion of words in one article to the number of words in the other article.

share|improve this answer
    
You don't need to record all of $x,y,x_1,y_1$. For example, if you only know $x' = x_1/x$ and $y' = y_1/y$ then you can calculate $(x'x + y'y)/(x+y)$. –  Yuval Filmus Jul 10 '11 at 21:16
    
Yuval is absolutely right. Knowing x and y, or the proportion of x to y would both suffice. –  mixedmath Jul 10 '11 at 21:18
    
@mixedmath at a later stage, i only have x1/x (in decimal) and y1 and y, ............... also you say other option is that i need to know the proportion of words in one article to the number of words in the other article...... how does this solve when i have a 3rd, 4th ... and more articles and have to update the probability ?? Thanks. –  Pheonix Jul 10 '11 at 21:19
    
@Phoenix: Being able to do it with 2 is all you need. Just treat the first 2 articles as a single article and then rinse, wash, and repeat. What does having a second question mark mean at the end of an interrogative?? Is it intoned differently? ;p –  mixedmath Jul 10 '11 at 21:20
    
@mixedmath: sorry i did not understood what you meant by you question, thanks for the answer though :) –  Pheonix Jul 10 '11 at 21:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.