Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

-1
votes
0answers
28 views

Party Invitations Problem [on hold]

There is a party on some Sunday next year, it is open to all. a)Everyday at some point, a person comes and drops a slip with contact details in a dropbox. b)A person on a given day could comeback ...
0
votes
0answers
16 views

looking for a simple example in machine learning with step-by-step procedure

I am looking for a simple example in the area of machine learning as well as a step-by-step procedure. For example, if I have 3 measurement devices and each one reports 5 data hourly how can I ...
0
votes
1answer
18 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
0
votes
1answer
28 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
1
vote
1answer
25 views

Can deep learning be a good way to learn a “High-quality” simple functions for images? For example, identical transformation, rotation, translation. [closed]

Can deep learning be a good way to learn a "High-quality" simple functions for images? For example, identical transformation, rotation, translation, even a linear mapping.
0
votes
1answer
13 views

If there are any curve databases with structured data

I found these curve lists: http://www.lmfdb.org/EllipticCurve/ https://en.wikipedia.org/wiki/List_of_curves http://www-groups.dcs.st-and.ac.uk/~history/Curves/Curves.html http://old.nationalcurvebank....
0
votes
0answers
51 views

Predict a subset from sequence of sets

I want to solve this data mining problem: There is a sequence $S(t) = [A_1,...,A_{t-1}]$ . Each $A_i \subset U$ are set. Current time is $t \in \mathbb{N}$. $B$ is a subset of $A_t$. ($B \subset A_t$)...
1
vote
1answer
18 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
1
vote
0answers
34 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
0
votes
1answer
72 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-...
0
votes
1answer
47 views

Finding the max number of pairs in a market basket problem

Is my reasoning here correct or is there a better way to solve this problem? Given a set of items $I$ and a set of baskets $B$, where each basket contains $k$ items, what is the maximum number of ...
1
vote
1answer
120 views

Intuitive difference between cosine similarity and bilinear similarity

Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows: $cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$ Similarly, bilinear similarity is defined as: ...
1
vote
1answer
34 views

Efficient methods to compute PCA for an incrementally growing matrix

I am wondering if there exists an efficient method to compute PCA. I am drafting the question into a presudo code: ...
1
vote
1answer
155 views

Probability and Statistics Books for Distributions and Introduction to Data Mining/Machine Learning

In college, I took a probability class using Sheldon Ross' A First Course in Probability. It was not my best semester to say the least. However, I am returning back to probability and statistics as it ...
0
votes
1answer
65 views

Is variance of mean equals mean of covariance?

I am trying to finish a problem, my method requires to prove variance of mean equals mean of covariance, but I have trouble proving it. Is it correct? Or more condition needed? Now I use $$Var(X)=E(X^...
0
votes
1answer
26 views

What's matrix $W$ in nonlinear PCA?

Nonlinear PCA is based on minimizing wrt matrix $W$ the function: $$I = E \{ \|x-Wg(W^Tx)\|^2\}$$ where $g$ is an odd function. However, what is $W$?
0
votes
1answer
14 views

What are correlation coefficients used for in PCA?

What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for?
0
votes
1answer
27 views

Why approximate pairwise distances only over lower triangle of distance matrix?

In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix? $$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$...
0
votes
0answers
19 views

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean?

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean? The projection seems to generally be defined as $\tilde{X} = XR$, but what does the tilde do there?
1
vote
1answer
47 views

Why/when is there $\frac{1}{2}$ in front of Least Squares Estimator?

So in PCA I encountered a formulation for LSE, which is: $$\frac{1}{2} \sum_{i=1}^N ||x_i - \tilde{x_i}||^2$$ Where $\tilde{x}$ is a "restriction" of $x$ such that only parts of the observations are ...
1
vote
1answer
33 views

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n u^T_k x \space u_k$?

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n (u^T_k x) \space u_k$? Where $\{u_1,...,u_n\}$ is orthonormal basis and $||u||^2=u^T_i u_i=1 \forall i$. Is this some standard vector ...
1
vote
1answer
18 views

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do?

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do? That is $$\frac{\sum_{i<j}(\hat{d_{ij}}-d_{ij})^2}{\sum_{i<j} d_{ij}^2}$$ Where the nominator is the raw stress. I ...
0
votes
0answers
21 views

Does “more points” necessarily imply “smoother boundary”?

So in the context of Naive Bayes I encountered a postulate that the more "nearest neighbours" one includes (i.e. the larger the $k$ in $k$-NN), the smoother the boundaries of classes become. Is this ...
0
votes
1answer
23 views

How to interpret maximizing “separability and reciprocal of scattering” in Fisher's LDA?

How to interpret maximizing "separability and reciprocal of scattering" in Fisher's LDA? That is, if $s_1$ is minimized scattering inside (projected) class 1 and $s_2$ is the same for class 2. Then ...
1
vote
0answers
12 views

In Fisher's LDA, what is $a_i =w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation?

In Fisher's LDA, what is $a_i = w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation? My notes write that: projection lengths of data into it (unit vector $w$) as $a_i = w^T x_i$ So is $...
1
vote
0answers
19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...
0
votes
0answers
29 views

Extrapolation of measurement data

I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it ...
0
votes
0answers
32 views

Can I use independence for bayesian network?

I'm struggling trying to understand bayesian network. Having trouble finding P(F&G) I know independence and conditional independence are different things. Conditional independence is when A ...
1
vote
1answer
41 views

Unification of data set using machine learning

I have just started learning data science so pardon me the statements that does not make any sense. Consider this situation - I have a data set which is made of examples containing personal info ...
0
votes
1answer
43 views

Elementary reference on Support Vector Machines

I am having difficulties in reading the book Support vector mathine by Ingo Steinwart with some notations and mathematical terms . Can you please recommend a prerequisite book or online course by ...
0
votes
1answer
315 views

Converting categorical data to binary variabels

when we have both categorical and numerical attributes in our data, it is said we can convert our categorical attributes to numerical by using some methods like binary variables. my question is should ...
0
votes
1answer
42 views

Can't understanding information entropy and data compression, for beginners

I watched a very well done video about information entropy, and I thought I got the concept in my mind but then I asked my self a question that denied every certainty I had. The video if you are ...
0
votes
1answer
501 views

Proving triangle inequality holds for Jaccard Distance

The following explanation is given in the Mining of Massive Datasets Textbook to prove that the triangle inequality holds for the Jaccard Distance (this sign $!=$ means does not equal): "For the ...
1
vote
0answers
27 views

data analysing graphs from excel

i have a question at excel. if you can help me?? i need to do data analysing graphs from excel, and they look more or the less the same like this one. and i want to separate them to steps ...
0
votes
1answer
28 views

Analyzing multiple time series describing the same feature

I'm currently facing a problem for which I am given several time series that all describe the feature. E.g. the height of several trees of the same kind was measured over a period of time each. ...
1
vote
0answers
26 views

Data set generation

For my research work, I need to generate data set by using some standard statistical distribution. I have seen Unknown distribution for data set generation. I do not have any idea about unknown ...
1
vote
0answers
31 views

Calculate PageRank for small web

Calculate PageRank for: A links to B, B links to C and C links to B and C where the damping factor $\beta=0.8$ I have: $M=\begin{bmatrix} 0&0&\frac{1}{2} \\ 1&0&\frac{1}{2} \\ 0&...
1
vote
2answers
109 views

F-measure of a binary classifier

Prove that the F-measure of any binary classifier is $\leq\dfrac{precision+recall}{2}$ Let $P=precision$ and $R=recall$ I have that the F measure $=\dfrac{2PR}{P+R}$ Note that $precision=\dfrac{tp}...
0
votes
2answers
706 views

Prove that the Jaccard Similarity of bags is always less than or equal to 1/2

Prove that the Jaccard Similarity of bags is always less than or equal to 1/2 I know that the Jaccard Similarity of bags is $= \dfrac{|B\cap C|}{|B\cup C|}$ and if the bags are the same then the ...
0
votes
1answer
248 views

K-Means Clustering

Perform a k-means clustering (with 3 clusters) of the one-dimensional set of points $1,1,2,3,4,7,8,8,12,14,16,21,22,24$ and with the inital point $=2$ To start I would select $3$ centroids that are ...
1
vote
1answer
334 views

Hierarchical Clustering using Centroids

Perform a hierarchical clustering (with five clusters) of the one-dimensional set of points $2, 3, 5, 7, 11, 13, 17, 19, 23$ assuming clusters are represented by their centroid (average) and at each ...
2
votes
1answer
850 views

Naive Bayes Classification Example

Given the following data: $$\begin{array}{c|c|c|c|c|} \text{Instance} & \text{A} & \text{B} &\text{C} &\text{Class} \\ \hline \text{1} & 1 & 2 & 1 & 1 \\ \hline \text{...
1
vote
0answers
116 views

How do I calculate the upper bound probability using Hoeffding's Ineqaulity?

The image below show mathematical expressions for the meanings of training and testing. I can understand the meaning of testing but the meaning of training is not so clear because of $M$ there. I took ...
0
votes
1answer
26 views

How to differentiate between two sets of data; one being a mixture of sinusoidal waves and other without any pattern

I have two sets of data; one is a mixture of different sinusoidal waves of different frequencies and amplitude while the other set does not have any regular pattern. I have the following questions: ...
7
votes
2answers
101 views

How do I find the formula (or rules) that created a list of numbers with seemingly no pattern?

Newbie here, and I apologize if this is the wrong forum for this type of question... I have a group of 200 or so alphanumeric codes from an unknown source. Here's an example piece of the data set: <...
1
vote
0answers
33 views

Can someone show me a simple example of finding simple squared error in k-means?

I'm working on some homework and one of our assignments asks us to find the final mean of each cluster and the SSE (sum of squared error). I haven't been able to find anything online and was wondering ...
0
votes
1answer
17 views

Analyze mixture of data attributes

When I have a list of persons (nodes) with the attributes: REVENUE | LINKS I can search for the people with most links and the people with most revenue. But when I want to know who has the ...
1
vote
0answers
37 views

Regression of 6 input parameters

I have a set of data that contains 6 inputs and 1 output (about 500 rows). Basically, I'm trying to figure out the complex relationship between the 6 inputs (say different component of concrete mix, i....
1
vote
2answers
70 views

How can I interpret the image for prior probability?

The image included below is about Bayesian statistics. While looking at the lecture, the lecturer expressed the probability distribution of prior probability as a uniform distribution. Somehow, I feel ...
0
votes
0answers
42 views

Difference between Data Mining and a model?

I am pretty much confused with the terms between pattern mining and a model in data mining. For example, I have a number of sequential patterns like below: No. 1: A->B->C->A = 0.5 No. 2: B->C->D = 0....