Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

7
votes
2answers
101 views

How do I find the formula (or rules) that created a list of numbers with seemingly no pattern?

Newbie here, and I apologize if this is the wrong forum for this type of question... I have a group of 200 or so alphanumeric codes from an unknown source. Here's an example piece of the data set: <...
4
votes
1answer
5k views

Normalization of data in decision tree

After reading through a few references, I have come to know that for machine learning in general, it is necessary to normalize features so that no features are arbitrarily large ($centering$) and all ...
4
votes
1answer
5k views

Mutual Information for clustering

I'm working on a document clustering application and decided to use Normalized Mutual Information as one of the measures of effectivenes. But I don't really understand how to implement this in that ...
4
votes
1answer
65 views

How to find almost periodic lattices in a set of high-dimensional points?

sorry for lame question, but I just have no maturity in this direction. Let's say I have very large set ( millions ) of high-dimensional vectors ( typical dimensionality is 64). These vectors ...
3
votes
1answer
458 views

What is KL-Divergence? Why Do I need it? How do I use it?

I am currently studying KL Divergence. But It seems very confusing that I don't maybe understand why do I ever need it and what is that for? As I have been reading stuff about Mutual Information, it ...
3
votes
2answers
1k views

Distance between unequal-dimension vectors (and data)?

It is easy to find simple distance measures for equal-dimension vectors, such as Euclidean Distance or Correlation. What about unequal-dimension vectors, such as, for instance, $(a,b,c)$ and $(d,e)$? ...
3
votes
1answer
2k views

How can one prove that mahalanobis distance is a metric?

How can one prove that mahalanobis distance is a metric? How can one show that these four properties of a metric are valid for mahalanobis distance? 1) d(x, y) ≥ 0 (non-negativity, or separation ...
3
votes
0answers
45 views

Number of rules (Data Mining)

This is a paraphrasing of a statement from Data Mining by Witten et al., 4th edition, on section 1.6. Consider the set $S = \{1, \dots, 288\}$. A rule is defined as a one-element singleton set ...
3
votes
0answers
212 views

Does there exist closed-form solutions for calculations in the EM-algorithm for Gaussian Mixtures?

Our data can be represented as $X_{n\times d}$ matrix, where we have $n$ data points lying in $\mathbb{R}^d$. We assume that there are $k$ underlying Gaussian models, $\mathcal{N}_d(\mu_j, \Sigma_j)$, ...
3
votes
1answer
35 views

Clustering elements according to covariance matrix

I'm doing a little bit of topic modelling (which is not really my area) with twitter tweets. The situation is the following: I have a (sort of) covariance matrix where the entrie $C_{ij}$ corresponds ...
3
votes
0answers
355 views

Gini coefficient vs Gini impurity - decision trees

The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - ...
3
votes
2answers
173 views

Are A and B conditionally independent

Are A and B conditionally independent given the class label? I calculated that $$P(A=1) = \frac{1}{2}$$ $$P(B=1) = \frac{2}{5}$$ $$P(A=1,B=1)=\frac{1}{5}$$ My answer is yes. I do it by anding $(A\...
2
votes
1answer
2k views

why use a small learning rate in gradient descent

I am new to neural networks and recently found out about gradient descent. Something does not sit right with me. x←x−λ∇fk(x) Why does this formula work? Wouldn'...
2
votes
1answer
851 views

Naive Bayes Classification Example

Given the following data: $$\begin{array}{c|c|c|c|c|} \text{Instance} & \text{A} & \text{B} &\text{C} &\text{Class} \\ \hline \text{1} & 1 & 2 & 1 & 1 \\ \hline \text{...
2
votes
2answers
87 views

Are mathematicians discovering algorithms by mining them with computers? [closed]

Throughout history many famous equations have been intuitively derived by famous mathematicians - Einstein's discovery of mass–energy equivalence, Schrödinger's equation for the orbitals of electrons, ...
2
votes
2answers
96 views

Plain English interpretation needed for the sentence to understand EM-algorithm?

I am trying to read an EM-algorithm article on the web, however, as soon as I started I have face a sentence interpretation problem with this like "... in the presence of missing or hidden data" in ...
2
votes
3answers
135 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
2
votes
2answers
78 views

How could we know the Dirichlet allocation is describing the topic rather than something else?

Dirichlet distribution is widely used in document modelling and document clustering. I tried to understand its rational. I read from this article that: Different Dirichlet distributions can be ...
2
votes
1answer
278 views

Creating a lift chart for a classification tree

This is likely a simple question but I'm new to data mining techniques and am trying to compare two different predictive models. I've created a logistic regression and a classification tree and would ...
2
votes
0answers
426 views

Spectral relaxation of k-means clustering

I am working on a presentation on Spectral relaxation of k-means clustering (http://papers.nips.cc/paper/1992-spectral-relaxation-for-k-means-clustering.pdf) and I am a bit stuck. I understand ...
2
votes
0answers
278 views

Optimization of English Braille: Using the fewest dots

Background: The English Braille system is laid out in such a way so that the letters can be referenced by their position in the alphabet. Of the six dots available for each character, the top four ...
2
votes
0answers
56 views

How can I infer order from partially ordered discrete sequences?

A really interesting problem that I can't stop thinking about! Have run in to this a couple of times but yet to find a smart approach to either solve or frame this problem. This is my try at ...
2
votes
0answers
750 views

How do I measure the goodness of cosine similarity scores across different vector spaces?

I am a computer scientist working on a problem that requires some statistical measures, though (not being very well versed in statistics) I am not quite sure what statistics to use. Overview: I ...
1
vote
2answers
70 views

How can I interpret the image for prior probability?

The image included below is about Bayesian statistics. While looking at the lecture, the lecturer expressed the probability distribution of prior probability as a uniform distribution. Somehow, I feel ...
1
vote
1answer
47 views

Why/when is there $\frac{1}{2}$ in front of Least Squares Estimator?

So in PCA I encountered a formulation for LSE, which is: $$\frac{1}{2} \sum_{i=1}^N ||x_i - \tilde{x_i}||^2$$ Where $\tilde{x}$ is a "restriction" of $x$ such that only parts of the observations are ...
1
vote
2answers
109 views

F-measure of a binary classifier

Prove that the F-measure of any binary classifier is $\leq\dfrac{precision+recall}{2}$ Let $P=precision$ and $R=recall$ I have that the F measure $=\dfrac{2PR}{P+R}$ Note that $precision=\dfrac{tp}...
1
vote
2answers
63 views

Are there $a,b,c,d \in \mathbb N$ such that $\frac{a + b}{a + b + c + d} < \frac{a}{a + c} < \frac{b}{b + d}$?

Consider the following $2 \times 2$ contingency table: \begin{array}{c|cc|c} & C & \overline C & \Sigma \\ \hline V & 4000 & 3500 & 7500 \\ \overline V & 2000 & 500 &...
1
vote
1answer
35 views

Efficient methods to compute PCA for an incrementally growing matrix

I am wondering if there exists an efficient method to compute PCA. I am drafting the question into a presudo code: ...
1
vote
1answer
33 views

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n u^T_k x \space u_k$?

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n (u^T_k x) \space u_k$? Where $\{u_1,...,u_n\}$ is orthonormal basis and $||u||^2=u^T_i u_i=1 \forall i$. Is this some standard vector ...
1
vote
1answer
2k views

How Jaccard similarity can be approximated with minhash similarity?

In Page 81 of this book, Mining of Massive Data Sets. It says the following: Now, consider the probability that $h(S_1) = h(S_2)$. If we imagine the rows permuted randomly, and we proceed from the ...
1
vote
1answer
241 views

Smoothing Spline Example

I am learning the smoothing spline method. I saw that smoothing spline is a penalty term to reduce overfitting in linear regression. Given dataset {$(x_1,y_1),(x_2,y_2)..(x_n,y_n)$}So the formular ...
1
vote
1answer
115 views

Physical Meaning Behind Matrix Factorization

As we all know, Matrix Factorization is an effective method to do rating prediction jobs in recommender systems. Thanks to the work of Yahuda Koren. My question is why MF can do this job ? What's the ...
1
vote
1answer
64 views

Given two arrays, how can one determine the the intersection?

I have two datasets (A & B). They each have 1000 numbers. 99% of the time: A < x <= B However, 1% of the time B < x < A. How can I solve for x, ...
1
vote
1answer
18 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
1
vote
1answer
120 views

Intuitive difference between cosine similarity and bilinear similarity

Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows: $cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$ Similarly, bilinear similarity is defined as: ...
1
vote
1answer
155 views

Probability and Statistics Books for Distributions and Introduction to Data Mining/Machine Learning

In college, I took a probability class using Sheldon Ross' A First Course in Probability. It was not my best semester to say the least. However, I am returning back to probability and statistics as it ...
1
vote
1answer
18 views

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do?

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do? That is $$\frac{\sum_{i<j}(\hat{d_{ij}}-d_{ij})^2}{\sum_{i<j} d_{ij}^2}$$ Where the nominator is the raw stress. I ...
1
vote
1answer
41 views

Unification of data set using machine learning

I have just started learning data science so pardon me the statements that does not make any sense. Consider this situation - I have a data set which is made of examples containing personal info ...
1
vote
1answer
337 views

Hierarchical Clustering using Centroids

Perform a hierarchical clustering (with five clusters) of the one-dimensional set of points $2, 3, 5, 7, 11, 13, 17, 19, 23$ assuming clusters are represented by their centroid (average) and at each ...
1
vote
1answer
72 views

Is EM-algorithm only for missing data?

Currently studying EM algorithm and have been through a few articles, they all say it is for missing data. I believe there is some implication in the term "missing data". I wonder if EM is designed ...
1
vote
1answer
672 views

What is the relationship or difference between MLE and EM algorithm?

I am trying to study EM algorithm and Maximum Likelihood Estimation. Somehow, they both sound the same to me but can't really say the difference. Maybe I don't really understand any of them. I have ...
1
vote
1answer
91 views

How to find the minimum of $f(a_1, b_1, \ldots , a_m, b_m) = \sum_{j=1}^{n} (y_j - \sum_{k=1}^{m} a_kx_j^{b_k})^2$?

$$f(a_1, b_1, \ldots , a_m, b_m) = \sum_{j=1}^{n} (y_j - \sum_{k=1}^m a_kx_j^{b_k})^2$$ $$2m < n$$ $x$ and $y$ are constants, and $a$ and $b$ are variables to find. I took deviation out of it and ...
1
vote
1answer
165 views

Unsupervised learning algorithms to detect anomaly in waves.

I have a sample of graphs (more than 10000...). that look like in the image below: I am searching for an unsupervised learning algorithms that can help me to detect anomalous observations. Here what ...
1
vote
3answers
90 views

Separating populations and estimating line-fit parameters

Given a dataset containing two populations, each of which can be described by a linear relationship between two variables in each sample with high R$^2$, how does one separate the two populations (and ...
1
vote
2answers
54 views

Scaling data into $[-1,1]$

I have a data in the matrix for: \begin{bmatrix} 1 & 2 & 3 & 9 & 6\\ 8 & 2 & 7 & 4 & 6 \\ 1 & 2 & 8 & 7 & 4 \end{bmatrix} Each row corresponds ...
1
vote
1answer
1k views

Naive Bayes to Predict a class label

How do I use Naïve Bayes to predict a class label for a test sample $(A=1, B=1, C=1)$ I know Bayes Theorem is: $$P(C|A) = [P(A|C) P(C)]/P(A)$$ I have no idea how to do this, please help.
1
vote
1answer
25 views

Can deep learning be a good way to learn a “High-quality” simple functions for images? For example, identical transformation, rotation, translation. [closed]

Can deep learning be a good way to learn a "High-quality" simple functions for images? For example, identical transformation, rotation, translation, even a linear mapping.
1
vote
0answers
34 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
1
vote
0answers
12 views

In Fisher's LDA, what is $a_i =w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation?

In Fisher's LDA, what is $a_i = w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation? My notes write that: projection lengths of data into it (unit vector $w$) as $a_i = w^T x_i$ So is $...
1
vote
0answers
19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...