# Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

104 questions
27 views

### Party Invitations Problem [on hold]

There is a party on some Sunday next year, it is open to all. a)Everyday at some point, a person comes and drops a slip with contact details in a dropbox. b)A person on a given day could comeback ...
16 views

### looking for a simple example in machine learning with step-by-step procedure

I am looking for a simple example in the area of machine learning as well as a step-by-step procedure. For example, if I have 3 measurement devices and each one reports 5 data hourly how can I ...
18 views

### Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
28 views

### Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
25 views

### Can deep learning be a good way to learn a “High-quality” simple functions for images? For example, identical transformation, rotation, translation. [closed]

Can deep learning be a good way to learn a "High-quality" simple functions for images? For example, identical transformation, rotation, translation, even a linear mapping.
705 views

### Factor Analysis in Text Mining task

I was faced with the task of determining the topics of big text massive. For example you have 1 million any text phrases or sentences. I want factorize the main topic from this massive. The ordinary ...
72 views

51 views

### Predict a subset from sequence of sets

I want to solve this data mining problem: There is a sequence $S(t) = [A_1,...,A_{t-1}]$ . Each $A_i \subset U$ are set. Current time is $t \in \mathbb{N}$. $B$ is a subset of $A_t$. ($B \subset A_t$)...
13 views

### If there are any curve databases with structured data

I found these curve lists: http://www.lmfdb.org/EllipticCurve/ https://en.wikipedia.org/wiki/List_of_curves http://www-groups.dcs.st-and.ac.uk/~history/Curves/Curves.html http://old.nationalcurvebank....
18 views

### Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
34 views

### Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
286 views

### Does Gini index considers only a binary split for each attribute or can it have multi way spliting?

Does Gini-index based classification split values for any attribute always as a binary split or can it split into more than $2$ branches (multi-way split)? For more clarification, if a split on $A$ ...
47 views

### Finding the max number of pairs in a market basket problem

Is my reasoning here correct or is there a better way to solve this problem? Given a set of items $I$ and a set of baskets $B$, where each basket contains $k$ items, what is the maximum number of ...
120 views

### Intuitive difference between cosine similarity and bilinear similarity

Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows: $cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$ Similarly, bilinear similarity is defined as: ...
34 views

### Efficient methods to compute PCA for an incrementally growing matrix

I am wondering if there exists an efficient method to compute PCA. I am drafting the question into a presudo code: ...
155 views

### Probability and Statistics Books for Distributions and Introduction to Data Mining/Machine Learning

In college, I took a probability class using Sheldon Ross' A First Course in Probability. It was not my best semester to say the least. However, I am returning back to probability and statistics as it ...
I am trying to finish a problem, my method requires to prove variance of mean equals mean of covariance, but I have trouble proving it. Is it correct? Or more condition needed? Now I use $$Var(X)=E(X^... 0answers 29 views ### Extrapolation of measurement data I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it ... 1answer 26 views ### What's matrix W in nonlinear PCA? Nonlinear PCA is based on minimizing wrt matrix W the function:$$I = E \{ \|x-Wg(W^Tx)\|^2\}$$where g is an odd function. However, what is W? 1answer 14 views ### What are correlation coefficients used for in PCA? What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for? 1answer 47 views ### Why/when is there \frac{1}{2} in front of Least Squares Estimator? So in PCA I encountered a formulation for LSE, which is:$$\frac{1}{2} \sum_{i=1}^N ||x_i - \tilde{x_i}||^2$$Where \tilde{x} is a "restriction" of x such that only parts of the observations are ... 1answer 27 views ### Why approximate pairwise distances only over lower triangle of distance matrix? In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix?$$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$... 1answer 23 views ### How to interpret maximizing “separability and reciprocal of scattering” in Fisher's LDA? How to interpret maximizing "separability and reciprocal of scattering" in Fisher's LDA? That is, if s_1 is minimized scattering inside (projected) class 1 and s_2 is the same for class 2. Then ... 0answers 19 views ### In the context of PCA and random projection, what does \tilde{X} \sim XR mean? In the context of PCA and random projection, what does \tilde{X} \sim XR mean? The projection seems to generally be defined as \tilde{X} = XR, but what does the tilde do there? 1answer 18 views ### In raw stress, what does putting \sum_{i <j} d_{ij} in denominator do? In raw stress, what does putting \sum_{i <j} d_{ij} in denominator do? That is$$\frac{\sum_{i<j}(\hat{d_{ij}}-d_{ij})^2}{\sum_{i<j} d_{ij}^2}$$Where the nominator is the raw stress. I ... 1answer 33 views ### In PCA, why for every x \in \mathbb{R}^n, x=\sum_{k=1}^n u^T_k x \space u_k? In PCA, why for every x \in \mathbb{R}^n, x=\sum_{k=1}^n (u^T_k x) \space u_k? Where \{u_1,...,u_n\} is orthonormal basis and ||u||^2=u^T_i u_i=1 \forall i. Is this some standard vector ... 0answers 21 views ### Does “more points” necessarily imply “smoother boundary”? So in the context of Naive Bayes I encountered a postulate that the more "nearest neighbours" one includes (i.e. the larger the k in k-NN), the smoother the boundaries of classes become. Is this ... 0answers 12 views ### In Fisher's LDA, what is a_i =w^T x_i if w is an unit vector and x_i is an observation? In Fisher's LDA, what is a_i = w^T x_i if w is an unit vector and x_i is an observation? My notes write that: projection lengths of data into it (unit vector w) as a_i = w^T x_i So is ... 0answers 19 views ### Matching Metric - How to Normalize for different Amounts in Equation I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ... 1answer 2k views ### How can one prove that mahalanobis distance is a metric? How can one prove that mahalanobis distance is a metric? How can one show that these four properties of a metric are valid for mahalanobis distance? 1) d(x, y) ≥ 0 (non-negativity, or separation ... 1answer 248 views ### K-Means Clustering Perform a k-means clustering (with 3 clusters) of the one-dimensional set of points 1,1,2,3,4,7,8,8,12,14,16,21,22,24 and with the inital point =2 To start I would select 3 centroids that are ... 0answers 32 views ### Can I use independence for bayesian network? I'm struggling trying to understand bayesian network. Having trouble finding P(F&G) I know independence and conditional independence are different things. Conditional independence is when A ... 1answer 41 views ### Unification of data set using machine learning I have just started learning data science so pardon me the statements that does not make any sense. Consider this situation - I have a data set which is made of examples containing personal info ... 2answers 78 views ### How could we know the Dirichlet allocation is describing the topic rather than something else? Dirichlet distribution is widely used in document modelling and document clustering. I tried to understand its rational. I read from this article that: Different Dirichlet distributions can be ... 1answer 43 views ### Elementary reference on Support Vector Machines I am having difficulties in reading the book Support vector mathine by Ingo Steinwart with some notations and mathematical terms . Can you please recommend a prerequisite book or online course by ... 1answer 315 views ### Converting categorical data to binary variabels when we have both categorical and numerical attributes in our data, it is said we can convert our categorical attributes to numerical by using some methods like binary variables. my question is should ... 2answers 1k views ### Distance between unequal-dimension vectors (and data)? It is easy to find simple distance measures for equal-dimension vectors, such as Euclidean Distance or Correlation. What about unequal-dimension vectors, such as, for instance, (a,b,c) and (d,e)? ... 1answer 42 views ### Can't understanding information entropy and data compression, for beginners I watched a very well done video about information entropy, and I thought I got the concept in my mind but then I asked my self a question that denied every certainty I had. The video if you are ... 1answer 500 views ### Proving triangle inequality holds for Jaccard Distance The following explanation is given in the Mining of Massive Datasets Textbook to prove that the triangle inequality holds for the Jaccard Distance (this sign != means does not equal): "For the ... 1answer 72 views ### Is EM-algorithm only for missing data? Currently studying EM algorithm and have been through a few articles, they all say it is for missing data. I believe there is some implication in the term "missing data". I wonder if EM is designed ... 0answers 27 views ### data analysing graphs from excel i have a question at excel. if you can help me?? i need to do data analysing graphs from excel, and they look more or the less the same like this one. and i want to separate them to steps ... 1answer 28 views ### Analyzing multiple time series describing the same feature I'm currently facing a problem for which I am given several time series that all describe the feature. E.g. the height of several trees of the same kind was measured over a period of time each. ... 0answers 26 views ### Data set generation For my research work, I need to generate data set by using some standard statistical distribution. I have seen Unknown distribution for data set generation. I do not have any idea about unknown ... 1answer 668 views ### What is the relationship or difference between MLE and EM algorithm? I am trying to study EM algorithm and Maximum Likelihood Estimation. Somehow, they both sound the same to me but can't really say the difference. Maybe I don't really understand any of them. I have ... 2answers 108 views ### F-measure of a binary classifier Prove that the F-measure of any binary classifier is \leq\dfrac{precision+recall}{2} Let P=precision and R=recall I have that the F measure =\dfrac{2PR}{P+R} Note that precision=\dfrac{tp}... 0answers 31 views ### Calculate PageRank for small web Calculate PageRank for: A links to B, B links to C and C links to B and C where the damping factor \beta=0.8 I have: M=\begin{bmatrix} 0&0&\frac{1}{2} \\ 1&0&\frac{1}{2} \\ 0&... 2answers 701 views ### Prove that the Jaccard Similarity of bags is always less than or equal to 1/2 Prove that the Jaccard Similarity of bags is always less than or equal to 1/2 I know that the Jaccard Similarity of bags is = \dfrac{|B\cap C|}{|B\cup C|} and if the bags are the same then the ... 1answer 332 views ### Hierarchical Clustering using Centroids Perform a hierarchical clustering (with five clusters) of the one-dimensional set of points 2, 3, 5, 7, 11, 13, 17, 19, 23 assuming clusters are represented by their centroid (average) and at each ... 1answer 849 views ### Naive Bayes Classification Example Given the following data:$$\begin{array}{c|c|c|c|c|} \text{Instance} & \text{A} & \text{B} &\text{C} &\text{Class} \\ \hline \text{1} & 1 & 2 & 1 & 1 \\ \hline \text{...
The image below show mathematical expressions for the meanings of training and testing. I can understand the meaning of testing but the meaning of training is not so clear because of $M$ there. I took ...