Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Asymptotically equivalent with the Bayesian Information Criterion (BIC)

Question Show that $$ln[\pi(n + p +2)\bar {\sigma}^2 ] - 2ln \Gamma (n/2) - ln|\mathbf{\Sigma}| $$ is asymptotically equivalent to the BIC approximation as $n \rightarrow \infty$ $$n\:[ln(2\pi\bar {\...
0
votes
0answers
41 views

Show Unbiased Learner

I want to show that $g_{\tau}(\mathbf{x}) = \mathbf{x}^T\hat{\beta}$ where $\hat{\beta} = \mathbf{X}^+\mathbf{y}$ and $\tau$ denotes a fixed training set is an unbiased learner, in the sense that: $$\...
0
votes
0answers
59 views

Expected Optimism 0-1 Loss with 0-1 Response

Want to show that $$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(\hat{Y}, Y_i)$$ For 0-1 loss function with 0-1 response. Want I've done $$op = l_{in} - l=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', \hat{Y})-\...
0
votes
0answers
32 views

Data Mining - Minimizer of the risk

To show Given: $$Loss(y, \hat{y}) = \alpha (\hat{y}-y)_+ + \beta (y-\hat{y})_+$$ Note: $c_+$ is equal to c if c > 0 and zero otherwise! Show that the minimizer of the risk $l(g) = E Loss(Y, g(X))$ ...
0
votes
1answer
15 views

example 1.4 chapter 1 from mining of massive data sets book

I am reading the book mining of massive data sets. (http://mmds.org/) In its chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf following section is there on page 9. Example 1.4: Suppose ...
2
votes
1answer
38 views

exercise 1.3 from Mining of Massive Data Sets book

Hello there is a question given in Mining of Massive Data Sets book http://infolab.stanford.edu/~ullman/mmds/ch1.pdf it is on page 15 exercise 1.3.2 My solution is following: as there are $10$ ...
0
votes
1answer
48 views

Bonferroni’s Principle discussed in Mining of Massive Data Sets book

I am reading a chapter of book Mining of Massive Data Sets book is available here http://www.mmds.org Chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf Now in Section 1.2.3 An example of ...
1
vote
0answers
62 views

Definition of Entropy (Information Theory)

In Information Theory, entropy is defined as: $$-\sum_{i}P_ilog(P_i)$$ where $-P_ilog(P_i)$ looks like this (using log base 2): From just a generic English definition of entropy, meaning lack of ...
2
votes
1answer
21 views

Can i use the coefficients of a trained model of Logistic Regression as a result itself without using the model on unseen data?

I'm trying to figure out if i can use a logistic regression as a predictive model, to estimate the probability of response of a user in CRM by having the predictors and i also have the class (...
0
votes
0answers
18 views

Given distances from known distributions, how to get the unknown distribution with which the distances are calucated with

Hi the question is actually pretty common in data mining contest: whenever you submit a result you will get a score (the distance). After a lot of trials you will have a lot of pairs of distribution ...
1
vote
1answer
19 views

How is Autoregressive Model used in longitudinal data analysis?

I know that in the analysis of time series, Auto-regressive Model such as AR(1) is frequently used. In the context of time series, there is no covariate (or the only covariate is time). In the context ...
0
votes
1answer
44 views

why does singular values of M equal to square roots of eigenvalues $M^TM$

per wiki, there is a rule to compute singular values The non-zero singular values of M (found on the diagonal entries of Σ) are the square roots of the non-zero eigenvalues of both $M^*M$ and $MM^...
1
vote
0answers
118 views

Calculation of Intrinsic dimension of datasets

Currently I am following a Machine learning course and we are looking at the intrinsic dimension of datasets. The professor gave a few examples of the intrinsic dimension of some objects (ej. the ...
0
votes
1answer
20 views

Dimensionality reduction for high dimensional curves?

I have a continuous curve in high dimensional space, and I'd like to visualize it in lower dimensional (2D or 3D) space to get an intuition of what it looks like. I'm familiar with PCA and t-SNE, but ...
0
votes
0answers
18 views

looking for a simple example in machine learning with step-by-step procedure

I am looking for a simple example in the area of machine learning as well as a step-by-step procedure. For example, if I have 3 measurement devices and each one reports 5 data hourly how can I ...
0
votes
1answer
23 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
0
votes
1answer
38 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
1
vote
1answer
29 views

Can deep learning be a good way to learn a “High-quality” simple functions for images? For example, identical transformation, rotation, translation. [closed]

Can deep learning be a good way to learn a "High-quality" simple functions for images? For example, identical transformation, rotation, translation, even a linear mapping.
0
votes
1answer
16 views

If there are any curve databases with structured data

I found these curve lists: http://www.lmfdb.org/EllipticCurve/ https://en.wikipedia.org/wiki/List_of_curves http://www-groups.dcs.st-and.ac.uk/~history/Curves/Curves.html http://old.nationalcurvebank....
0
votes
0answers
55 views

Predict a subset from sequence of sets

I want to solve this data mining problem: There is a sequence $S(t) = [A_1,...,A_{t-1}]$ . Each $A_i \subset U$ are set. Current time is $t \in \mathbb{N}$. $B$ is a subset of $A_t$. ($B \subset A_t$)...
1
vote
1answer
20 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
1
vote
0answers
36 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
0
votes
1answer
87 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-...
0
votes
1answer
57 views

Finding the max number of pairs in a market basket problem

Is my reasoning here correct or is there a better way to solve this problem? Given a set of items $I$ and a set of baskets $B$, where each basket contains $k$ items, what is the maximum number of ...
1
vote
1answer
254 views

Intuitive difference between cosine similarity and bilinear similarity

Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows: $cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$ Similarly, bilinear similarity is defined as: ...
1
vote
1answer
38 views

Efficient methods to compute PCA for an incrementally growing matrix

I am wondering if there exists an efficient method to compute PCA. I am drafting the question into a presudo code: ...
2
votes
1answer
194 views

Probability and Statistics Books for Distributions and Introduction to Data Mining/Machine Learning

In college, I took a probability class using Sheldon Ross' A First Course in Probability. It was not my best semester to say the least. However, I am returning back to probability and statistics as it ...
0
votes
1answer
67 views

Is variance of mean equals mean of covariance?

I am trying to finish a problem, my method requires to prove variance of mean equals mean of covariance, but I have trouble proving it. Is it correct? Or more condition needed? Now I use $$Var(X)=E(X^...
0
votes
1answer
26 views

What's matrix $W$ in nonlinear PCA?

Nonlinear PCA is based on minimizing wrt matrix $W$ the function: $$I = E \{ \|x-Wg(W^Tx)\|^2\}$$ where $g$ is an odd function. However, what is $W$?
0
votes
1answer
14 views

What are correlation coefficients used for in PCA?

What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for?
0
votes
1answer
32 views

Why approximate pairwise distances only over lower triangle of distance matrix?

In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix? $$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$...
0
votes
0answers
21 views

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean?

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean? The projection seems to generally be defined as $\tilde{X} = XR$, but what does the tilde do there?
1
vote
1answer
48 views

Why/when is there $\frac{1}{2}$ in front of Least Squares Estimator?

So in PCA I encountered a formulation for LSE, which is: $$\frac{1}{2} \sum_{i=1}^N ||x_i - \tilde{x_i}||^2$$ Where $\tilde{x}$ is a "restriction" of $x$ such that only parts of the observations are ...
1
vote
1answer
34 views

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n u^T_k x \space u_k$?

In PCA, why for every $x \in \mathbb{R}^n$, $x=\sum_{k=1}^n (u^T_k x) \space u_k$? Where $\{u_1,...,u_n\}$ is orthonormal basis and $||u||^2=u^T_i u_i=1 \forall i$. Is this some standard vector ...
1
vote
1answer
19 views

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do?

In raw stress, what does putting $\sum_{i <j} d_{ij}$ in denominator do? That is $$\frac{\sum_{i<j}(\hat{d_{ij}}-d_{ij})^2}{\sum_{i<j} d_{ij}^2}$$ Where the nominator is the raw stress. I ...
0
votes
1answer
28 views

How to interpret maximizing “separability and reciprocal of scattering” in Fisher's LDA?

How to interpret maximizing "separability and reciprocal of scattering" in Fisher's LDA? That is, if $s_1$ is minimized scattering inside (projected) class 1 and $s_2$ is the same for class 2. Then ...
1
vote
0answers
12 views

In Fisher's LDA, what is $a_i =w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation?

In Fisher's LDA, what is $a_i = w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation? My notes write that: projection lengths of data into it (unit vector $w$) as $a_i = w^T x_i$ So is $...
1
vote
0answers
19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...
0
votes
0answers
38 views

Extrapolation of measurement data

I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it ...
0
votes
0answers
35 views

Can I use independence for bayesian network?

I'm struggling trying to understand bayesian network. Having trouble finding P(F&G) I know independence and conditional independence are different things. Conditional independence is when A ...
1
vote
1answer
54 views

Unification of data set using machine learning

I have just started learning data science so pardon me the statements that does not make any sense. Consider this situation - I have a data set which is made of examples containing personal info ...
0
votes
1answer
44 views

Elementary reference on Support Vector Machines

I am having difficulties in reading the book Support vector mathine by Ingo Steinwart with some notations and mathematical terms . Can you please recommend a prerequisite book or online course by ...
0
votes
1answer
467 views

Converting categorical data to binary variabels

when we have both categorical and numerical attributes in our data, it is said we can convert our categorical attributes to numerical by using some methods like binary variables. my question is should ...
0
votes
1answer
47 views

Can't understanding information entropy and data compression, for beginners

I watched a very well done video about information entropy, and I thought I got the concept in my mind but then I asked my self a question that denied every certainty I had. The video if you are ...
0
votes
1answer
701 views

Proving triangle inequality holds for Jaccard Distance

The following explanation is given in the Mining of Massive Datasets Textbook to prove that the triangle inequality holds for the Jaccard Distance (this sign $!=$ means does not equal): "For the ...
1
vote
0answers
29 views

data analysing graphs from excel

i have a question at excel. if you can help me?? i need to do data analysing graphs from excel, and they look more or the less the same like this one. and i want to separate them to steps ...
0
votes
1answer
28 views

Analyzing multiple time series describing the same feature

I'm currently facing a problem for which I am given several time series that all describe the feature. E.g. the height of several trees of the same kind was measured over a period of time each. ...
1
vote
0answers
26 views

Data set generation

For my research work, I need to generate data set by using some standard statistical distribution. I have seen Unknown distribution for data set generation. I do not have any idea about unknown ...
2
votes
0answers
38 views

Calculate PageRank for small web

Calculate PageRank for: A links to B, B links to C and C links to B and C where the damping factor $\beta=0.8$ I have: $M=\begin{bmatrix} 0&0&\frac{1}{2} \\ 1&0&\frac{1}{2} \\ 0&...
1
vote
2answers
164 views

F-measure of a binary classifier

Prove that the F-measure of any binary classifier is $\leq\dfrac{precision+recall}{2}$ Let $P=precision$ and $R=recall$ I have that the F measure $=\dfrac{2PR}{P+R}$ Note that $precision=\dfrac{tp}...