Questions tagged [data-mining]
This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.
137
questions
-1
votes
0
answers
22
views
Least Squares but with entries of Beta
I can only find Bj_hat but cannot find how Bj_~ is equals in the first problem.problem image. The notation I find for Bj_hat is sum from i=1 of ((y1 - B0 - sum l not j (xil*Bl)*xij)/ sum from i=1 to n ...
0
votes
0
answers
12
views
Proving the equation about the total number of possible rules, R, extracted from a data set that contains d items
Here is the equation about the total number of possible rules, R, extracted from a data set that contains d items:
$$ R = 3^d -2^{d+1}+1$$
Let's say we have 3 items in one itemset I:
$$I ={ \{milk, ...
1
vote
0
answers
103
views
Machine Learning over Finite Fields
My question is if machine learning over finite fields is a sensible idea, or if there is any literature regarding this. By that I mean, given some function with inputs and outputs on a finite field ...
- 57
0
votes
0
answers
13
views
Independence test for paired predictors
I know that the title of this question can be misleading but trust me, i didn't know what title to put. I'm doing a project in the field of data science and, before to use any machine learning ...
0
votes
0
answers
14
views
Proof of Nesterov Momentum change of variables
Please help me to know how the above phrase has been changed to a lower expression by changing a variable !?
show picture
1
vote
0
answers
110
views
Cosine similarity between a vector and a mean of vectors/a sum of vectors
I am trying to understand how cosine similarity works when applied to two vectors: considering vectors $u$ and $v$, their cosine similarity is computed as follows:
$cos(u,v)$ = $\frac{u\cdot v}{||u||*|...
- 428
0
votes
0
answers
20
views
How to group using percentiles?
I'm trying to understand this research below that I added:
The outcome variable was categorized using percentiles to Mild, Moderate, and Severe in the Symptom Severity Scale (SSS). The three ...
1
vote
0
answers
55
views
Probability that a randomly chosen centorid will be chosen from a cluster
k-means is a data mining algorithm useful for clustering purposes:
Given K equally sized clusters, the probability that a randomly chosen initial
centroid will come from any given cluster is 1/K, but ...
- 428
2
votes
0
answers
28
views
Maximum and minum value for the entropy summation formula
Suppose $t$ is a non empty set divided in $j$ different classes and consider $p_j$ the relative frequency of the elements of class $j$ with respect to all the elements of the set $t$. Consider the ...
- 428
0
votes
1
answer
145
views
Mathematics behind Gini Index for Decision Tree
In Data Science and ML, Gini index is a measure used to value how "homogeneous" a certain node of a decision tree is with respect to the distribution of categorical values in the node itself:...
- 428
1
vote
1
answer
39
views
eigenvctors using PCA
I have Y the observations, and I have the eigenvalue matrix V . and I want to calculate the eigenevctors and the principals compenents using PCA with R.
I'm trying $U=V * Y^T$ with V the eigenvectors ...
- 15
0
votes
1
answer
42
views
What does Quinlan mean by "the confidence limits for the binomial distribution"?
My classmates and I are trying to figure out what J. Ross Quinlan means on page 41 of C4.5: Programs for Machine Learning. He says:
The probability of error cannot be determined easily, but has ...
0
votes
0
answers
142
views
Prove that $\arccos(\cos(x, y))$ is a metric
I'm taking a data mining course in uni, and the professor asked us to prove that $\arccos(\cos(x, y))$ is a metric. I've never seen such a notation, having two numbers inside cosine function. Does ...
0
votes
0
answers
71
views
The Trace Ratio Problem
For two m × m positive semidefinite matrices $S_p$ and $S_l$, the trace ratio problem is defined as
finding a m × d (d < m) transform matrix W∗ that satisfies $$
W^{*}=\underset{W^{T} W=I}{\arg \...
0
votes
0
answers
25
views
given a symmetric matrix A, of dimension n*n and an arbitrary unitary matrix V of dimension n*d . Maximize the trace of $V^T A V$
$$
\left\{\begin{array}{l}
\max _{V \in \mathbb{R}^{n \times d}} \operatorname{Tr}\left[V^{T} A V\right]=\lambda_{1}+\cdots+\lambda_{d} \\
V^{T} V=I
\end{array}\right.
$$
-1
votes
1
answer
34
views
Hi , i want to prove this result: In order to maximize the variance, we can maximize the trace of the matrix. [closed]
$$
\arg \min _{Q \in \mathbf{R}^{d \times k}} \frac{1}{n} \sum_{i=1}^{n}\left\|\vec{x}_{i}-Q Q^{\top} \vec{x}_{i}\right\|^{2}=\arg \max _{Q \in \mathbf{R}^{d \times k}} \operatorname{tr}\left(Q^{\top}\...
1
vote
0
answers
163
views
maximized the trace of $V^T A V$
given a symmetric matrix A, of dimension $n \times n $ and an an arbitrary unitary matrix V of dimension $ n \times n$ the the trace of $V^T A V$ is an orthogonal basis of the eigenspace associated ...
0
votes
1
answer
105
views
Equally Distributed Data Set Measurement
I will be creating my own dataset with scores ranging from 50.00 to 100.00. How will I say that the dataset I chose is equally distributed and unbiased ? Is there a formula to know this?
- 223
0
votes
0
answers
29
views
Effectively extracting "real" data from a noisy dataset
Background & Motivation:
I have three lists of timestamps. They are not necessarily of the same size, however they are ordered. Each such list corresponds to a real-life instrument, of which there ...
- 1,281
0
votes
1
answer
38
views
What kind of properties can be defined over the fundamental matrix of absorbing random walks?
A connected d-regular graph $G = (V,E,d)$ is given, when each node has exactly d outlinks reaching the other nodes. Also, a function $l$ maps nodes with positive and negative labels, where $V_+$ are ...
- 11
0
votes
1
answer
28
views
Algorithm for Weighted Geospatial Clustering
Given a set of values (x, y, z) = (x-coord (long), y-coord (lat), revenue), is there an algorithm similar to k-means clustering that will help find the most revenue concentrated clusterings of a fixed ...
user848541
0
votes
0
answers
21
views
data prediction using statistics
Here is the data set for the engagement ratio (ER) of diabetes patients of 11 weeks. How can I anticipate the ER of the 12th week?
ER in every weeks
Weeks ER
1 0.17
2 ...
0
votes
1
answer
179
views
Calculate new ratings based on averages and counts?
For a while, I've been scraping a website for an item's "ratings" data.
It seems to me that if I know (for any given timeframe):
the start and end count of submitted ratings, and,
the start ...
- 1
0
votes
0
answers
440
views
Plot data points according to the pairwise distance matrix
Consider eight data points. The following matrix (i.e., a symmetric matrix with the lower triangle elements) shows the pairwise distances between any two points.
0
11 0
5 13 0
12 2 14 0
7 17 1 18 0
...
2
votes
0
answers
55
views
Hierarchical Clustering with Ward Distance
I know how hierarchical clustering (with a certain definition of inter-cluster distance) works. And I know that Ward's procedure is based on the goal of minimizing the sum of the squared errors ...
- 21
1
vote
0
answers
175
views
Derivation of the Chebyshev distance from the Minkowski distance
I know that the Minkowski distance is defined as follows:
$$d(x,y)=\lim_{r\to\infty}\left(\sum_{k=1}^n|x_k-y_k|^r\right)^{1/r}$$
and I also read that the Chebyshev distance could be considered as a ...
- 449
0
votes
0
answers
34
views
Approximate data of N elements as a sum of products of the target
Computer Engineer here. I need help with a personal project that is data science related. I am trying to approximate how to solve something like this. Each of these values are a PCA(principle ...
- 1
5
votes
2
answers
175
views
Courses to take during pure math masters to keep data science and applied work as a possibility
I was wondering what courses you can take in a pure math masters to preserve the opportunity to go into data science, economics, policy research or other applied work while preserving the opportunity ...
- 337
1
vote
0
answers
40
views
Classification of 32 bit integers into 2 classes with uneven probability
I've been given a set of integer data $x_t$ that are all 32-bit unsigned integers. These data have been previously divided into two classes using a function unknown to me as the following:
$$f(x)=\...
- 75
0
votes
0
answers
49
views
What metrics can be used to describe the clustering of a group of points?
Consider a set of points $\Omega \subset \mathbb{R}^n$ (to make it simple, consider $n=2$). So you basically have a bunch of points on a plane.
I need to find good metrics that can describe the ...
- 993
0
votes
1
answer
68
views
Multi Label Classification: Union of two Binary Sets
I have started Data Science on my own and was looking at evaluation metrics. I came across this measure of Accuracy with the equation $$ \frac{1}{p}\sum_{i=1}^{p}\frac{\vert Y_i \cap Z_i \vert}{\vert ...
- 11
1
vote
1
answer
93
views
Expected Optimism 0-1 Loss with 0-1 Response
Want to show that
$$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(\hat{Y}, Y_i)$$
For 0-1 loss function with 0-1 response.
Want I've done
$$op = l_{in} - l=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', \hat{Y})-\...
- 155
0
votes
1
answer
246
views
example 1.4 chapter 1 from mining of massive data sets book
I am reading the book mining of massive data sets. (http://mmds.org/)
In its chapter 1
http://infolab.stanford.edu/~ullman/mmds/ch1.pdf
following section is there on page 9.
Example 1.4: Suppose ...
2
votes
1
answer
931
views
exercise 1.3 from Mining of Massive Data Sets book
Hello there is a question given in Mining of Massive Data Sets book
http://infolab.stanford.edu/~ullman/mmds/ch1.pdf
it is on page 15 exercise 1.3.2
My solution is following:
as there are $10$ ...
0
votes
2
answers
1k
views
Bonferroni’s Principle discussed in Mining of Massive Data Sets book
I am reading a chapter of book Mining of Massive Data Sets
book is available here
http://www.mmds.org
Chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf
Now in Section 1.2.3 An example of ...
1
vote
0
answers
131
views
Definition of Entropy (Information Theory)
In Information Theory, entropy is defined as:
$$-\sum_{i}P_ilog(P_i)$$
where $-P_ilog(P_i)$ looks like this (using log base 2):
From just a generic English definition of entropy, meaning lack of ...
- 1,511
2
votes
1
answer
83
views
Can i use the coefficients of a trained model of Logistic Regression as a result itself without using the model on unseen data?
I'm trying to figure out if i can use a logistic regression as a predictive model, to estimate the probability of response of a user in CRM by having the predictors and i also have the class (...
1
vote
1
answer
264
views
How is Autoregressive Model used in longitudinal data analysis?
I know that in the analysis of time series, Auto-regressive Model such as AR(1) is frequently used. In the context of time series, there is no covariate (or the only covariate is time). In the context ...
- 43
0
votes
1
answer
216
views
why does singular values of M equal to square roots of eigenvalues $M^TM$
per wiki, there is a rule to compute singular values
The non-zero singular values of M (found on the diagonal entries of Σ)
are the square roots of the non-zero eigenvalues of both $M^*M$ and $MM^...
- 511
3
votes
1
answer
1k
views
Calculation of Intrinsic dimension of datasets
Currently I am following a Machine learning course and we are looking at the intrinsic dimension of datasets. The professor gave a few examples of the intrinsic dimension of some objects (ej. the ...
- 161
0
votes
1
answer
78
views
Dimensionality reduction for high dimensional curves?
I have a continuous curve in high dimensional space, and I'd like to visualize it in lower dimensional (2D or 3D) space to get an intuition of what it looks like. I'm familiar with PCA and t-SNE, but ...
- 61
0
votes
1
answer
47
views
Different order of insertion - different Bayesian network ? how to prove formally?
I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion.
If i ...
- 21
0
votes
1
answer
71
views
Which similarity formula should I use?
I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5
The author uses Cosine ...
0
votes
1
answer
32
views
If there are any curve databases with structured data
I found these curve lists:
http://www.lmfdb.org/EllipticCurve/
https://en.wikipedia.org/wiki/List_of_curves
http://www-groups.dcs.st-and.ac.uk/~history/Curves/Curves.html
http://old.nationalcurvebank....
- 3,472
1
vote
1
answer
30
views
Weighting function for a scatter plot of ratio and difference across several orders of magnitude
I am comparing straight line distances to the shortest discovered path.
I have millions of points with a pair (straight line distance,...
- 1,083
1
vote
0
answers
43
views
Finding statistically increasing and decreasing sub-sequences in a (noisy) vector
I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
- 41
1
vote
1
answer
155
views
Uniform Effect of K-means Clustering
In the following link is discussed the uniform Effect of K-means Clustering:
https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-p174318763
...
- 507
0
votes
1
answer
129
views
Finding the max number of pairs in a market basket problem
Is my reasoning here correct or is there a better way to solve this problem?
Given a set of items $I$ and a set of baskets $B$, where each basket contains $k$ items, what is the maximum number of ...
- 88
1
vote
1
answer
1k
views
Intuitive difference between cosine similarity and bilinear similarity
Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows:
$cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$
Similarly, bilinear similarity is defined as:
...
- 1,534
1
vote
1
answer
44
views
Efficient methods to compute PCA for an incrementally growing matrix
I am wondering if there exists an efficient method to compute PCA. I am drafting the question into a presudo code:
...
- 113