Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

-1
votes
0answers
27 views

Party Invitations Problem [on hold]

There is a party on some Sunday next year, it is open to all. a)Everyday at some point, a person comes and drops a slip with contact details in a dropbox. b)A person on a given day could comeback ...
0
votes
0answers
16 views

looking for a simple example in machine learning with step-by-step procedure

I am looking for a simple example in the area of machine learning as well as a step-by-step procedure. For example, if I have 3 measurement devices and each one reports 5 data hourly how can I ...
0
votes
1answer
18 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
0
votes
1answer
28 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
1
vote
1answer
25 views

Can deep learning be a good way to learn a “High-quality” simple functions for images? For example, identical transformation, rotation, translation. [closed]

Can deep learning be a good way to learn a "High-quality" simple functions for images? For example, identical transformation, rotation, translation, even a linear mapping.
0
votes
2answers
705 views

Factor Analysis in Text Mining task

I was faced with the task of determining the topics of big text massive. For example you have 1 million any text phrases or sentences. I want factorize the main topic from this massive. The ordinary ...
0
votes
1answer
72 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-...
0
votes
0answers
51 views

Predict a subset from sequence of sets

I want to solve this data mining problem: There is a sequence $S(t) = [A_1,...,A_{t-1}]$ . Each $A_i \subset U$ are set. Current time is $t \in \mathbb{N}$. $B$ is a subset of $A_t$. ($B \subset A_t$)...
0
votes
1answer
13 views

If there are any curve databases with structured data

I found these curve lists: http://www.lmfdb.org/EllipticCurve/ https://en.wikipedia.org/wiki/List_of_curves http://www-groups.dcs.st-and.ac.uk/~history/Curves/Curves.html http://old.nationalcurvebank....
1
vote
1answer
18 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
1
vote
0answers
34 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
1
vote
1answer
286 views

Does Gini index considers only a binary split for each attribute or can it have multi way spliting?

Does Gini-index based classification split values for any attribute always as a binary split or can it split into more than $2$ branches (multi-way split)? For more clarification, if a split on $A$ ...
0
votes
1answer
47 views

Finding the max number of pairs in a market basket problem

Is my reasoning here correct or is there a better way to solve this problem? Given a set of items $I$ and a set of baskets $B$, where each basket contains $k$ items, what is the maximum number of ...
1
vote
1answer
120 views

Intuitive difference between cosine similarity and bilinear similarity

Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows: $cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$ Similarly, bilinear similarity is defined as: ...
1
vote
1answer
34 views

Efficient methods to compute PCA for an incrementally growing matrix

I am wondering if there exists an efficient method to compute PCA. I am drafting the question into a presudo code: ...
1
vote
1answer
155 views

Probability and Statistics Books for Distributions and Introduction to Data Mining/Machine Learning

In college, I took a probability class using Sheldon Ross' A First Course in Probability. It was not my best semester to say the least. However, I am returning back to probability and statistics as it ...
0
votes
1answer
65 views

Is variance of mean equals mean of covariance?

I am trying to finish a problem, my method requires to prove variance of mean equals mean of covariance, but I have trouble proving it. Is it correct? Or more condition needed? Now I use $$Var(X)=E(X^...
0
votes
0answers
29 views

Extrapolation of measurement data

I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it ...
0
votes
1answer
26 views

What's matrix $W$ in nonlinear PCA?

Nonlinear PCA is based on minimizing wrt matrix $W$ the function: $$I = E \{ \|x-Wg(W^Tx)\|^2\}$$ where $g$ is an odd function. However, what is $W$?
0
votes
1answer
14 views

What are correlation coefficients used for in PCA?

What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for?
1
vote
1answer
47 views

Why/when is there $\frac{1}{2}$ in front of Least Squares Estimator?

So in PCA I encountered a formulation for LSE, which is: $$\frac{1}{2} \sum_{i=1}^N ||x_i - \tilde{x_i}||^2$$ Where $\tilde{x}$ is a "restriction" of $x$ such that only parts of the observations are ...
0
votes
1answer
27 views

Why approximate pairwise distances only over lower triangle of distance matrix?

In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix? $$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$...
0
votes
1answer
23 views

How to interpret maximizing “separability and reciprocal of scattering” in Fisher's LDA?

How to interpret maximizing "separability and reciprocal of scattering" in Fisher's LDA? That is, if $s_1$ is minimized scattering inside (projected) class 1 and $s_2$ is the same for class 2. Then ...
0
votes
0answers
19 views

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean?

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean? The projection seems to generally be defined as $\tilde{X} = XR$, but what does the tilde do there?
1
vote
1answer
18 views

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do?

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do? That is $$\frac{\sum_{i<j}(\hat{d_{ij}}-d_{ij})^2}{\sum_{i<j} d_{ij}^2}$$ Where the nominator is the raw stress. I ...
1
vote
1answer
33 views

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n u^T_k x \space u_k$?

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n (u^T_k x) \space u_k$? Where $\{u_1,...,u_n\}$ is orthonormal basis and $||u||^2=u^T_i u_i=1 \forall i$. Is this some standard vector ...
0
votes
0answers
21 views

Does “more points” necessarily imply “smoother boundary”?

So in the context of Naive Bayes I encountered a postulate that the more "nearest neighbours" one includes (i.e. the larger the $k$ in $k$-NN), the smoother the boundaries of classes become. Is this ...
1
vote
0answers
12 views

In Fisher's LDA, what is $a_i =w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation?

In Fisher's LDA, what is $a_i = w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation? My notes write that: projection lengths of data into it (unit vector $w$) as $a_i = w^T x_i$ So is $...
1
vote
0answers
19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...
3
votes
1answer
2k views

How can one prove that mahalanobis distance is a metric?

How can one prove that mahalanobis distance is a metric? How can one show that these four properties of a metric are valid for mahalanobis distance? 1) d(x, y) ≥ 0 (non-negativity, or separation ...
0
votes
1answer
248 views

K-Means Clustering

Perform a k-means clustering (with 3 clusters) of the one-dimensional set of points $1,1,2,3,4,7,8,8,12,14,16,21,22,24$ and with the inital point $=2$ To start I would select $3$ centroids that are ...
0
votes
0answers
32 views

Can I use independence for bayesian network?

I'm struggling trying to understand bayesian network. Having trouble finding P(F&G) I know independence and conditional independence are different things. Conditional independence is when A ...
1
vote
1answer
41 views

Unification of data set using machine learning

I have just started learning data science so pardon me the statements that does not make any sense. Consider this situation - I have a data set which is made of examples containing personal info ...
2
votes
2answers
78 views

How could we know the Dirichlet allocation is describing the topic rather than something else?

Dirichlet distribution is widely used in document modelling and document clustering. I tried to understand its rational. I read from this article that: Different Dirichlet distributions can be ...
0
votes
1answer
43 views

Elementary reference on Support Vector Machines

I am having difficulties in reading the book Support vector mathine by Ingo Steinwart with some notations and mathematical terms . Can you please recommend a prerequisite book or online course by ...
0
votes
1answer
315 views

Converting categorical data to binary variabels

when we have both categorical and numerical attributes in our data, it is said we can convert our categorical attributes to numerical by using some methods like binary variables. my question is should ...
3
votes
2answers
1k views

Distance between unequal-dimension vectors (and data)?

It is easy to find simple distance measures for equal-dimension vectors, such as Euclidean Distance or Correlation. What about unequal-dimension vectors, such as, for instance, $(a,b,c)$ and $(d,e)$? ...
0
votes
1answer
42 views

Can't understanding information entropy and data compression, for beginners

I watched a very well done video about information entropy, and I thought I got the concept in my mind but then I asked my self a question that denied every certainty I had. The video if you are ...
0
votes
1answer
500 views

Proving triangle inequality holds for Jaccard Distance

The following explanation is given in the Mining of Massive Datasets Textbook to prove that the triangle inequality holds for the Jaccard Distance (this sign $!=$ means does not equal): "For the ...
1
vote
1answer
72 views

Is EM-algorithm only for missing data?

Currently studying EM algorithm and have been through a few articles, they all say it is for missing data. I believe there is some implication in the term "missing data". I wonder if EM is designed ...
1
vote
0answers
27 views

data analysing graphs from excel

i have a question at excel. if you can help me?? i need to do data analysing graphs from excel, and they look more or the less the same like this one. and i want to separate them to steps ...
0
votes
1answer
28 views

Analyzing multiple time series describing the same feature

I'm currently facing a problem for which I am given several time series that all describe the feature. E.g. the height of several trees of the same kind was measured over a period of time each. ...
1
vote
0answers
26 views

Data set generation

For my research work, I need to generate data set by using some standard statistical distribution. I have seen Unknown distribution for data set generation. I do not have any idea about unknown ...
1
vote
1answer
668 views

What is the relationship or difference between MLE and EM algorithm?

I am trying to study EM algorithm and Maximum Likelihood Estimation. Somehow, they both sound the same to me but can't really say the difference. Maybe I don't really understand any of them. I have ...
1
vote
2answers
108 views

F-measure of a binary classifier

Prove that the F-measure of any binary classifier is $\leq\dfrac{precision+recall}{2}$ Let $P=precision$ and $R=recall$ I have that the F measure $=\dfrac{2PR}{P+R}$ Note that $precision=\dfrac{tp}...
1
vote
0answers
31 views

Calculate PageRank for small web

Calculate PageRank for: A links to B, B links to C and C links to B and C where the damping factor $\beta=0.8$ I have: $M=\begin{bmatrix} 0&0&\frac{1}{2} \\ 1&0&\frac{1}{2} \\ 0&...
0
votes
2answers
701 views

Prove that the Jaccard Similarity of bags is always less than or equal to 1/2

Prove that the Jaccard Similarity of bags is always less than or equal to 1/2 I know that the Jaccard Similarity of bags is $= \dfrac{|B\cap C|}{|B\cup C|}$ and if the bags are the same then the ...
1
vote
1answer
332 views

Hierarchical Clustering using Centroids

Perform a hierarchical clustering (with five clusters) of the one-dimensional set of points $2, 3, 5, 7, 11, 13, 17, 19, 23$ assuming clusters are represented by their centroid (average) and at each ...
2
votes
1answer
849 views

Naive Bayes Classification Example

Given the following data: $$\begin{array}{c|c|c|c|c|} \text{Instance} & \text{A} & \text{B} &\text{C} &\text{Class} \\ \hline \text{1} & 1 & 2 & 1 & 1 \\ \hline \text{...
1
vote
0answers
116 views

How do I calculate the upper bound probability using Hoeffding's Ineqaulity?

The image below show mathematical expressions for the meanings of training and testing. I can understand the meaning of testing but the meaning of training is not so clear because of $M$ there. I took ...