Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

Filter by
Sorted by
Tagged with
1 vote
1 answer
41 views

coefficients of the linear model $y=β_0+β_1x_1+β_2x_2$ that minimises the sum of squares error [closed]

Consider the following sample: $$(x_{11}, x_{12}, y_1) = (1, 3, 4), $$$$(x_{21}, x_{22}, y_2) = (2, 1, 5),$$$$ (x_{31}, x_{32}, y_3) = (3, 0, 7),$$$$ (x_{41}, x_{42}, y_4) = (4, −2, 6).$$ How can I ...
1Mathsss's user avatar
2 votes
1 answer
93 views

optimisation problem that minimises the sum of absolute errors of the simple linear model $y = a + bx$.

Consider the following sample: $(x_1, y_1) = (1, 1)$, $(x_2, y_2) = (2, 5)$, $(x_3, y_3) = (3, 8)$, $(x_4, y_4) = (4, 18)$. How can I write down an optimisation problem that minimises the sum of ...
1Mathsss's user avatar
0 votes
1 answer
57 views

Calculating the RMSE and $R^2$ with a linear model $y = a + bx$ so that the mean squared error of the model is minimised [closed]

Consider the following sample: $(x_1, y_1) = (1, 1)$, $(x_2, y_2) = (2, 5)$, $(x_3, y_3) = (3, 8)$, $(x_4, y_4) = (4, 18)$. Suppose you predict $y$ based on $x$ with a linear model $y = a + bx$ so ...
mathana's user avatar
  • 27
0 votes
0 answers
83 views

calculate all pairwise dot (inner) products between data points

Suppose we have $n$ data points in $\mathbb{R}^p$ and we would like to calculate all pairwise dot (inner) products between them. How can I write a single matrix multiplication that does this? (Some ...
user1052623's user avatar
6 votes
1 answer
145 views

What is the exact function or algorithm Windows 10 is using to calculate taskbar underline color from Registry values, based on data I have collected?

In earlier versions of Windows, it was easy for users to set the exact colors of interface elements. For example, in Windows 7, you could go to [Control Panel -> Personalization -> Window Color],...
WingedKnight's user avatar
0 votes
0 answers
23 views

FTT Features to use after time-domain is transformed to frequency-domain

Please forgive the question if it sounds trivial/naive, I am from computer science background, not electrical/computer engineering. I work with GPS trajectory dataset for classification. Data was ...
Amina Umar's user avatar
0 votes
1 answer
66 views

Looking far a formula giving the coordinates $(x,y)$ at a given instant of a point traversing $\mathbb{N}^2$ in a layered way

We are observing a point in a two-dimensional Cartesian coordinate system which, at the initial time, is at the origin (0,0) ; one second later, the point moves to coordinate (0,1), and another second ...
JJJohn's user avatar
  • 1,436
1 vote
0 answers
400 views

Machine Learning over Finite Fields

My question is if machine learning over finite fields is a sensible idea, or if there is any literature regarding this. By that I mean, given some function with inputs and outputs on a finite field ...
Kevin's user avatar
  • 57
0 votes
0 answers
19 views

Independence test for paired predictors

I know that the title of this question can be misleading but trust me, i didn't know what title to put. I'm doing a project in the field of data science and, before to use any machine learning ...
Francesco De Santis's user avatar
1 vote
0 answers
228 views

Cosine similarity between a vector and a mean of vectors/a sum of vectors

I am trying to understand how cosine similarity works when applied to two vectors: considering vectors $u$ and $v$, their cosine similarity is computed as follows: $cos(u,v)$ = $\frac{u\cdot v}{||u||*|...
PwNzDust's user avatar
  • 468
1 vote
0 answers
104 views

Probability that a randomly chosen centorid will be chosen from a cluster

k-means is a data mining algorithm useful for clustering purposes: Given K equally sized clusters, the probability that a randomly chosen initial centroid will come from any given cluster is 1/K, but ...
PwNzDust's user avatar
  • 468
2 votes
0 answers
30 views

Maximum and minum value for the entropy summation formula

Suppose $t$ is a non empty set divided in $j$ different classes and consider $p_j$ the relative frequency of the elements of class $j$ with respect to all the elements of the set $t$. Consider the ...
PwNzDust's user avatar
  • 468
0 votes
1 answer
198 views

Mathematics behind Gini Index for Decision Tree

In Data Science and ML, Gini index is a measure used to value how "homogeneous" a certain node of a decision tree is with respect to the distribution of categorical values in the node itself:...
PwNzDust's user avatar
  • 468
1 vote
1 answer
43 views

eigenvctors using PCA

I have Y the observations, and I have the eigenvalue matrix V . and I want to calculate the eigenevctors and the principals compenents using PCA with R. I'm trying $U=V * Y^T$ with V the eigenvectors ...
Augustin's user avatar
0 votes
1 answer
55 views

What does Quinlan mean by "the confidence limits for the binomial distribution"?

My classmates and I are trying to figure out what J. Ross Quinlan means on page 41 of C4.5: Programs for Machine Learning. He says: The probability of error cannot be determined easily, but has ...
William John Holden's user avatar
0 votes
0 answers
212 views

Prove that $\arccos(\cos(x, y))$ is a metric

I'm taking a data mining course in uni, and the professor asked us to prove that $\arccos(\cos(x, y))$ is a metric. I've never seen such a notation, having two numbers inside cosine function. Does ...
Тадеј Гојић's user avatar
0 votes
0 answers
85 views

The Trace Ratio Problem

For two m × m positive semidefinite matrices $S_p$ and $S_l$, the trace ratio problem is defined as finding a m × d (d < m) transform matrix W∗ that satisfies $$ W^{*}=\underset{W^{T} W=I}{\arg \...
mouad lmazini's user avatar
0 votes
0 answers
37 views

given a symmetric matrix A, of dimension n*n and an arbitrary unitary matrix V of dimension n*d . Maximize the trace of $V^T A V$

$$ \left\{\begin{array}{l} \max _{V \in \mathbb{R}^{n \times d}} \operatorname{Tr}\left[V^{T} A V\right]=\lambda_{1}+\cdots+\lambda_{d} \\ V^{T} V=I \end{array}\right. $$
mouad lmazini's user avatar
-1 votes
1 answer
34 views

Hi , i want to prove this result: In order to maximize the variance, we can maximize the trace of the matrix. [closed]

$$ \arg \min _{Q \in \mathbf{R}^{d \times k}} \frac{1}{n} \sum_{i=1}^{n}\left\|\vec{x}_{i}-Q Q^{\top} \vec{x}_{i}\right\|^{2}=\arg \max _{Q \in \mathbf{R}^{d \times k}} \operatorname{tr}\left(Q^{\top}\...
mouad lmazini's user avatar
1 vote
0 answers
295 views

maximized the trace of $V^T A V$

given a symmetric matrix A, of dimension $n \times n $ and an an arbitrary unitary matrix V of dimension $ n \times n$ the the trace of $V^T A V$ is an orthogonal basis of the eigenspace associated ...
mouad lmazini's user avatar
0 votes
1 answer
143 views

Equally Distributed Data Set Measurement

I will be creating my own dataset with scores ranging from 50.00 to 100.00. How will I say that the dataset I chose is equally distributed and unbiased ? Is there a formula to know this?
alyssaeliyah's user avatar
0 votes
0 answers
32 views

Effectively extracting "real" data from a noisy dataset

Background & Motivation: I have three lists of timestamps. They are not necessarily of the same size, however they are ordered. Each such list corresponds to a real-life instrument, of which there ...
10GeV's user avatar
  • 1,371
0 votes
1 answer
65 views

What kind of properties can be defined over the fundamental matrix of absorbing random walks?

A connected d-regular graph $G = (V,E,d)$ is given, when each node has exactly d outlinks reaching the other nodes. Also, a function $l$ maps nodes with positive and negative labels, where $V_+$ are ...
pizzibarbaro's user avatar
0 votes
1 answer
29 views

Algorithm for Weighted Geospatial Clustering

Given a set of values (x, y, z) = (x-coord (long), y-coord (lat), revenue), is there an algorithm similar to k-means clustering that will help find the most revenue concentrated clusterings of a fixed ...
user avatar
0 votes
0 answers
21 views

data prediction using statistics

Here is the data set for the engagement ratio (ER) of diabetes patients of 11 weeks. How can I anticipate the ER of the 12th week? ER in every weeks Weeks ER 1 0.17 2 ...
Aeron Akash's user avatar
0 votes
1 answer
218 views

Calculate new ratings based on averages and counts?

For a while, I've been scraping a website for an item's "ratings" data. It seems to me that if I know (for any given timeframe): the start and end count of submitted ratings, and, the start ...
ƬƦƖƝƛ's user avatar
0 votes
0 answers
572 views

Plot data points according to the pairwise distance matrix

Consider eight data points. The following matrix (i.e., a symmetric matrix with the lower triangle elements) shows the pairwise distances between any two points. 0 11 0 5 13 0 12 2 14 0 7 17 1 18 0 ...
Kurogin_Qin's user avatar
2 votes
0 answers
73 views

Hierarchical Clustering with Ward Distance

I know how hierarchical clustering (with a certain definition of inter-cluster distance) works. And I know that Ward's procedure is based on the goal of minimizing the sum of the squared errors ...
ozner's user avatar
  • 21
1 vote
0 answers
311 views

Derivation of the Chebyshev distance from the Minkowski distance

I know that the Minkowski distance is defined as follows: $$d(x,y)=\lim_{r\to\infty}\left(\sum_{k=1}^n|x_k-y_k|^r\right)^{1/r}$$ and I also read that the Chebyshev distance could be considered as a ...
Lila's user avatar
  • 479
0 votes
0 answers
37 views

Approximate data of N elements as a sum of products of the target

Computer Engineer here. I need help with a personal project that is data science related. I am trying to approximate how to solve something like this. Each of these values are a PCA(principle ...
apaul's user avatar
  • 1
5 votes
2 answers
233 views

Courses to take during pure math masters to keep data science and applied work as a possibility

I was wondering what courses you can take in a pure math masters to preserve the opportunity to go into data science, economics, policy research or other applied work while preserving the opportunity ...
MR1992's user avatar
  • 357
1 vote
0 answers
44 views

Classification of 32 bit integers into 2 classes with uneven probability

I've been given a set of integer data $x_t$ that are all 32-bit unsigned integers. These data have been previously divided into two classes using a function unknown to me as the following: $$f(x)=\...
PouJa's user avatar
  • 175
0 votes
0 answers
65 views

What metrics can be used to describe the clustering of a group of points?

Consider a set of points $\Omega \subset \mathbb{R}^n$ (to make it simple, consider $n=2$). So you basically have a bunch of points on a plane. I need to find good metrics that can describe the ...
Andry's user avatar
  • 1,033
0 votes
1 answer
101 views

Multi Label Classification: Union of two Binary Sets

I have started Data Science on my own and was looking at evaluation metrics. I came across this measure of Accuracy with the equation $$ \frac{1}{p}\sum_{i=1}^{p}\frac{\vert Y_i \cap Z_i \vert}{\vert ...
Aslan's user avatar
  • 11
1 vote
1 answer
103 views

Expected Optimism 0-1 Loss with 0-1 Response

Want to show that $$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(\hat{Y}, Y_i)$$ For 0-1 loss function with 0-1 response. Want I've done $$op = l_{in} - l=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', \hat{Y})-\...
Casper Lindberg's user avatar
0 votes
1 answer
302 views

example 1.4 chapter 1 from mining of massive data sets book

I am reading the book mining of massive data sets. (http://mmds.org/) In its chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf following section is there on page 9. Example 1.4: Suppose ...
political science's user avatar
2 votes
1 answer
1k views

exercise 1.3 from Mining of Massive Data Sets book

Hello there is a question given in Mining of Massive Data Sets book http://infolab.stanford.edu/~ullman/mmds/ch1.pdf it is on page 15 exercise 1.3.2 My solution is following: as there are $10$ ...
political science's user avatar
0 votes
2 answers
1k views

Bonferroni’s Principle discussed in Mining of Massive Data Sets book

I am reading a chapter of book Mining of Massive Data Sets book is available here http://www.mmds.org Chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf Now in Section 1.2.3 An example of ...
political science's user avatar
1 vote
0 answers
155 views

Definition of Entropy (Information Theory)

In Information Theory, entropy is defined as: $$-\sum_{i}P_ilog(P_i)$$ where $-P_ilog(P_i)$ looks like this (using log base 2): From just a generic English definition of entropy, meaning lack of ...
EllipticalInitial's user avatar
2 votes
1 answer
131 views

Can i use the coefficients of a trained model of Logistic Regression as a result itself without using the model on unseen data?

I'm trying to figure out if i can use a logistic regression as a predictive model, to estimate the probability of response of a user in CRM by having the predictors and i also have the class (...
Oscar Gutierrez's user avatar
1 vote
1 answer
307 views

How is Autoregressive Model used in longitudinal data analysis?

I know that in the analysis of time series, Auto-regressive Model such as AR(1) is frequently used. In the context of time series, there is no covariate (or the only covariate is time). In the context ...
YC_Xu's user avatar
  • 43
0 votes
1 answer
304 views

why does singular values of M equal to square roots of eigenvalues $M^TM$

per wiki, there is a rule to compute singular values The non-zero singular values of M (found on the diagonal entries of Σ) are the square roots of the non-zero eigenvalues of both $M^*M$ and $MM^...
Jay's user avatar
  • 511
3 votes
1 answer
1k views

Calculation of Intrinsic dimension of datasets

Currently I am following a Machine learning course and we are looking at the intrinsic dimension of datasets. The professor gave a few examples of the intrinsic dimension of some objects (ej. the ...
Oliver's user avatar
  • 171
0 votes
1 answer
84 views

Dimensionality reduction for high dimensional curves?

I have a continuous curve in high dimensional space, and I'd like to visualize it in lower dimensional (2D or 3D) space to get an intuition of what it looks like. I'm familiar with PCA and t-SNE, but ...
Austin Stone's user avatar
0 votes
1 answer
53 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
caffein's user avatar
  • 121
0 votes
1 answer
89 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
christouandr7's user avatar
0 votes
1 answer
38 views

If there are any curve databases with structured data

I found these curve lists: http://www.lmfdb.org/EllipticCurve/ https://en.wikipedia.org/wiki/List_of_curves http://www-groups.dcs.st-and.ac.uk/~history/Curves/Curves.html http://old.nationalcurvebank....
Lance's user avatar
  • 3,694
1 vote
1 answer
31 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
Stepan's user avatar
  • 1,093
1 vote
0 answers
43 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
Tomer's user avatar
  • 41
1 vote
1 answer
158 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-p174318763 ...
Patricio's user avatar
  • 567