Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I'm not very mathematically oriented (working on fixing that) so excuse me if this is basic :)

I need to calculate "fragmentation ratio" - the best name I could come up with the thing I'm trying to achieve.

I have 400 different lists where each list contains 20 named items (with some meta data). Same name item can exist in multiple lists. I calculated that in my dataset there's 326 unique items among those 8000 items (400*20).

So where I stumble is that how do I calculate a meaningful value for fragmentation of items between those lists?

--- edit ---

It looks like I was even more vague and hard to understand than I thought.

I researched this a bit more and I think better term for this would be similarities between lists and calculating that from the collection.

Here's a small example of the data structures:

List 1: ['this','is','a','list']
List 2: ['is','it']
List 3: ['yes,'it','is','list']

Now what I would like to quantify is how much similarities these arrays share.

Does this explain my problem any better? :)

share|improve this question
    
It's OK to ask a vague question and make specifying the question more precisely a part of the question, but you've taken that to an extreme -- you haven't made any attempt at all at describing what you're trying to quantify, beyond calling it "fragmentation". I for one don't can't form a clear idea of what you're after from just that one word. –  joriki Dec 21 '12 at 15:33
    
I added a bit more to hopefully make it more understandable. –  James D. Dec 21 '12 at 17:45

1 Answer 1

up vote 0 down vote accepted

One option would be to define similarity as the number of common items between two lists. Does that do what you want?

share|improve this answer
    
I thought of that but it seems highly inefficient when you have millions of lists. Hmm, however with map/reduce I could make it run in parallel which would then make it actually feasible solution. –  James D. Dec 22 '12 at 9:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.