Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Proof of Nesterov Momentum change of variables

Please help me to know how the above phrase has been changed to a lower expression by changing a variable !? show picture
user avatar
1 vote
0 answers
21 views

Cosine similarity between a vector and a mean of vectors/a sum of vectors

I am trying to understand how cosine similarity works when applied to two vectors: considering vectors $u$ and $v$, their cosine similarity is computed as follows: $cos(u,v)$ = $\frac{u\cdot v}{||u||*|...
user avatar
  • 422
0 votes
0 answers
16 views

How to group using percentiles?

I'm trying to understand this research below that I added: The outcome variable was categorized using percentiles to Mild, Moderate, and Severe in the Symptom Severity Scale (SSS). The three ...
user avatar
0 votes
0 answers
20 views

Time complexity of MICE

What is the time complexity of Multiple imputation by chained equation? Its hard to calculate Big O for MICE. Note: Using linear regression. I calculated myself and get the O(n^2logn).
user avatar
0 votes
0 answers
95 views

The probability classifier so obtained is identical to Centroid classifier

I'm studying data mining and I try to prove that the bayes classifier is statistically indentical to nearest centroid classifier. I try to assume for bayes classifier, $P(X=x|Y=c_{k})$ has normal ...
user avatar
0 votes
0 answers
7 views

I have been learning about covariance and came across this interesting question. How do you estimate U and D matrices just from this given plot?

Trying to figure out how to do this problem and I am stuck unable to solve it.
user avatar
  • 25
0 votes
0 answers
6 views

Posteriori of latent factor model

I'm reading this article https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3419391/. I want to compute the posterior distribution of the factor latent model. The generic form of a latent factor model is $$ ...
user avatar
  • 1
0 votes
0 answers
22 views

Sorting features based on performance number which is a linear combination of features

I have a dataset that looks like table of data with features as columns (on/off) and resulting performance number in performance column The way to read this data is as follows: When a combination of ...
user avatar
0 votes
0 answers
13 views

simulate the normal gamma inverse density using metropolis hastings

I want to simulate the normal gamma inverse density using Metropolis hasting. The density is defined as : $f\left(x, \sigma^{2} \mid \mu, \gamma, \alpha, \beta\right)=\frac{1}{\sigma \sqrt{2 \pi \...
user avatar
  • 1
1 vote
0 answers
28 views

Probability that a randomly chosen centorid will be chosen from a cluster

k-means is a data mining algorithm useful for clustering purposes: Given K equally sized clusters, the probability that a randomly chosen initial centroid will come from any given cluster is 1/K, but ...
user avatar
  • 422
2 votes
0 answers
20 views

Maximum and minum value for the entropy summation formula

Suppose $t$ is a non empty set divided in $j$ different classes and consider $p_j$ the relative frequency of the elements of class $j$ with respect to all the elements of the set $t$. Consider the ...
user avatar
  • 422
0 votes
1 answer
95 views

Mathematics behind Gini Index for Decision Tree

In Data Science and ML, Gini index is a measure used to value how "homogeneous" a certain node of a decision tree is with respect to the distribution of categorical values in the node itself:...
user avatar
  • 422
1 vote
1 answer
32 views

eigenvctors using PCA

I have Y the observations, and I have the eigenvalue matrix V . and I want to calculate the eigenevctors and the principals compenents using PCA with R. I'm trying $U=V * Y^T$ with V the eigenvectors ...
user avatar
0 votes
0 answers
5 views

Why does generating random samples of a specific proportion fit into the null hypothesis and sampling distribution?

A sampling distribution and null hypothesis problem goes as follows: We are given dataset 100,000 rows of website traffic user conversion data. So there is a column showing whether the user was ...
user avatar
0 votes
0 answers
183 views

Maximum f-score for a classifier when changing threshold.

everyone! I have the following problem: "The Antirobot Classifier must distinguish robots from humans. Ivan and Marya wrote classifiers for the stream, about which it is known that there are ...
user avatar
0 votes
0 answers
41 views

Probabilistic PCA

I just learned R and I need to write an R program that generates an array of N=20 rows of experimental units for P=10 species (in columns) following a centered normal distribution such that the rows ...
user avatar
0 votes
1 answer
35 views

What does Quinlan mean by "the confidence limits for the binomial distribution"?

My classmates and I are trying to figure out what J. Ross Quinlan means on page 41 of C4.5: Programs for Machine Learning. He says: The probability of error cannot be determined easily, but has ...
user avatar
0 votes
0 answers
36 views

calculate the eigenvalues of the trace ratio problem

Given a symmetric matrix A, of dimension n×n and an arbitrary unitary matrix V of dimension n×p, B a symmetric positive definite. How to prove that $f(\theta)=\max _{V^{T} V=/} \operatorname{Tr}\left[...
user avatar
0 votes
0 answers
97 views

Prove that $\arccos(\cos(x, y))$ is a metric

I'm taking a data mining course in uni, and the professor asked us to prove that $\arccos(\cos(x, y))$ is a metric. I've never seen such a notation, having two numbers inside cosine function. Does ...
user avatar
0 votes
0 answers
30 views

What does autocorrelation means for the following data?

Let's suppose we have a time series is a=[1,1,1,3,3,3,1,1,1,3,3,3] then the autocorrelation figure for this time series is enter image description here The lag here ...
user avatar
  • 1
0 votes
0 answers
20 views

Mathematics of Text Mining for high school students

I would like to organize a series of seminars for senior high school students. Since my seminars are in mathematics, and must be linked to seminars held by other teachers on "text mining", I ...
user avatar
  • 7,280
0 votes
0 answers
20 views

Determining the mathematical algorithm for “Products Frequently Bought Together” using previous sales data?

I am working on a project of making a billing & taxation software and I am stuck on one step. I have all the previous sales & purchase data for a shop. Now I want to determine, based on ...
user avatar
0 votes
0 answers
52 views

The Trace Ratio Problem

For two m × m positive semidefinite matrices $S_p$ and $S_l$, the trace ratio problem is defined as finding a m × d (d < m) transform matrix W∗ that satisfies $$ W^{*}=\underset{W^{T} W=I}{\arg \...
user avatar
0 votes
0 answers
23 views

given a symmetric matrix A, of dimension n*n and an arbitrary unitary matrix V of dimension n*d . Maximize the trace of $V^T A V$

$$ \left\{\begin{array}{l} \max _{V \in \mathbb{R}^{n \times d}} \operatorname{Tr}\left[V^{T} A V\right]=\lambda_{1}+\cdots+\lambda_{d} \\ V^{T} V=I \end{array}\right. $$
user avatar
-1 votes
1 answer
33 views

Hi , i want to prove this result: In order to maximize the variance, we can maximize the trace of the matrix. [closed]

$$ \arg \min _{Q \in \mathbf{R}^{d \times k}} \frac{1}{n} \sum_{i=1}^{n}\left\|\vec{x}_{i}-Q Q^{\top} \vec{x}_{i}\right\|^{2}=\arg \max _{Q \in \mathbf{R}^{d \times k}} \operatorname{tr}\left(Q^{\top}\...
user avatar
1 vote
0 answers
95 views

maximized the trace of $V^T A V$

given a symmetric matrix A, of dimension $n \times n $ and an an arbitrary unitary matrix V of dimension $ n \times n$ the the trace of $V^T A V$ is an orthogonal basis of the eigenspace associated ...
user avatar
0 votes
1 answer
46 views

Equally Distributed Data Set Measurement

I will be creating my own dataset with scores ranging from 50.00 to 100.00. How will I say that the dataset I chose is equally distributed and unbiased ? Is there a formula to know this?
user avatar
0 votes
0 answers
28 views

Effectively extracting "real" data from a noisy dataset

Background & Motivation: I have three lists of timestamps. They are not necessarily of the same size, however they are ordered. Each such list corresponds to a real-life instrument, of which there ...
user avatar
  • 1,309
0 votes
1 answer
34 views

What kind of properties can be defined over the fundamental matrix of absorbing random walks?

A connected d-regular graph $G = (V,E,d)$ is given, when each node has exactly d outlinks reaching the other nodes. Also, a function $l$ maps nodes with positive and negative labels, where $V_+$ are ...
user avatar
0 votes
1 answer
25 views

Algorithm for Weighted Geospatial Clustering

Given a set of values (x, y, z) = (x-coord (long), y-coord (lat), revenue), is there an algorithm similar to k-means clustering that will help find the most revenue concentrated clusterings of a fixed ...
user avatar
0 votes
0 answers
19 views

data prediction using statistics

Here is the data set for the engagement ratio (ER) of diabetes patients of 11 weeks. How can I anticipate the ER of the 12th week? ER in every weeks Weeks ER 1 0.17 2 ...
user avatar
0 votes
1 answer
134 views

Calculate new ratings based on averages and counts?

For a while, I've been scraping a website for an item's "ratings" data. It seems to me that if I know (for any given timeframe): the start and end count of submitted ratings, and, the start ...
user avatar
0 votes
0 answers
247 views

Plot data points according to the pairwise distance matrix

Consider eight data points. The following matrix (i.e., a symmetric matrix with the lower triangle elements) shows the pairwise distances between any two points. 0 11 0 5 13 0 12 2 14 0 7 17 1 18 0 ...
user avatar
1 vote
0 answers
44 views

Hierarchical Clustering with Ward Distance

I know how hierarchical clustering (with a certain definition of inter-cluster distance) works. And I know that Ward's procedure is based on the goal of minimizing the sum of the squared errors ...
user avatar
  • 11
0 votes
0 answers
101 views

Derivation of the Chebyshev distance from the Minkowski distance

I know that the Minkowski distance is defined as follows: $$d(x,y)=\lim_{r\to\infty}\left(\sum_{k=1}^n|x_k-y_k|^r\right)^{1/r}$$ and I also read that the Chebyshev distance could be considered as a ...
user avatar
  • 417
0 votes
0 answers
34 views

Approximate data of N elements as a sum of products of the target

Computer Engineer here. I need help with a personal project that is data science related. I am trying to approximate how to solve something like this. Each of these values are a PCA(principle ...
user avatar
  • 1
5 votes
2 answers
142 views

Courses to take during pure math masters to keep data science and applied work as a possibility

I was wondering what courses you can take in a pure math masters to preserve the opportunity to go into data science, economics, policy research or other applied work while preserving the opportunity ...
user avatar
  • 337
1 vote
0 answers
38 views

Classification of 32 bit integers into 2 classes with uneven probability

I've been given a set of integer data $x_t$ that are all 32-bit unsigned integers. These data have been previously divided into two classes using a function unknown to me as the following: $$f(x)=\...
user avatar
  • 65
0 votes
0 answers
35 views

What metrics can be used to describe the clustering of a group of points?

Consider a set of points $\Omega \subset \mathbb{R}^n$ (to make it simple, consider $n=2$). So you basically have a bunch of points on a plane. I need to find good metrics that can describe the ...
user avatar
  • 933
0 votes
1 answer
54 views

Multi Label Classification: Union of two Binary Sets

I have started Data Science on my own and was looking at evaluation metrics. I came across this measure of Accuracy with the equation $$ \frac{1}{p}\sum_{i=1}^{p}\frac{\vert Y_i \cap Z_i \vert}{\vert ...
user avatar
  • 11
1 vote
1 answer
91 views

Expected Optimism 0-1 Loss with 0-1 Response

Want to show that $$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(\hat{Y}, Y_i)$$ For 0-1 loss function with 0-1 response. Want I've done $$op = l_{in} - l=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', \hat{Y})-\...
user avatar
0 votes
1 answer
167 views

example 1.4 chapter 1 from mining of massive data sets book

I am reading the book mining of massive data sets. (http://mmds.org/) In its chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf following section is there on page 9. Example 1.4: Suppose ...
user avatar
2 votes
1 answer
795 views

exercise 1.3 from Mining of Massive Data Sets book

Hello there is a question given in Mining of Massive Data Sets book http://infolab.stanford.edu/~ullman/mmds/ch1.pdf it is on page 15 exercise 1.3.2 My solution is following: as there are $10$ ...
user avatar
0 votes
2 answers
861 views

Bonferroni’s Principle discussed in Mining of Massive Data Sets book

I am reading a chapter of book Mining of Massive Data Sets book is available here http://www.mmds.org Chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf Now in Section 1.2.3 An example of ...
user avatar
1 vote
0 answers
116 views

Definition of Entropy (Information Theory)

In Information Theory, entropy is defined as: $$-\sum_{i}P_ilog(P_i)$$ where $-P_ilog(P_i)$ looks like this (using log base 2): From just a generic English definition of entropy, meaning lack of ...
user avatar
  • 1,155
2 votes
1 answer
66 views

Can i use the coefficients of a trained model of Logistic Regression as a result itself without using the model on unseen data?

I'm trying to figure out if i can use a logistic regression as a predictive model, to estimate the probability of response of a user in CRM by having the predictors and i also have the class (...
user avatar
1 vote
1 answer
203 views

How is Autoregressive Model used in longitudinal data analysis?

I know that in the analysis of time series, Auto-regressive Model such as AR(1) is frequently used. In the context of time series, there is no covariate (or the only covariate is time). In the context ...
user avatar
  • 43
0 votes
1 answer
159 views

why does singular values of M equal to square roots of eigenvalues $M^TM$

per wiki, there is a rule to compute singular values The non-zero singular values of M (found on the diagonal entries of Σ) are the square roots of the non-zero eigenvalues of both $M^*M$ and $MM^...
user avatar
  • 491
2 votes
0 answers
1k views

Calculation of Intrinsic dimension of datasets

Currently I am following a Machine learning course and we are looking at the intrinsic dimension of datasets. The professor gave a few examples of the intrinsic dimension of some objects (ej. the ...
user avatar
  • 151
0 votes
1 answer
57 views

Dimensionality reduction for high dimensional curves?

I have a continuous curve in high dimensional space, and I'd like to visualize it in lower dimensional (2D or 3D) space to get an intuition of what it looks like. I'm familiar with PCA and t-SNE, but ...
user avatar