Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

Filter by
Sorted by
Tagged with
-1
votes
0answers
13 views

Prove that lift(X->Y) < α ⇔ lift(X->¬Y)> α

I am learning about pattern and rule assesment of mined frequent pattern. One of such rules is lift, which is defined as the ratio of the observed joint probability of X and Y to the expected joint ...
0
votes
0answers
24 views

Find the center of a circle with fixed radius containing the maximal number of points in an 1D real dataset

Let us consider a dataset of real numbers, $x_1 \le x_2 \le... \le x_n$, where $n$ is a large number (~100 000) and a given $r > 0$ radius. I would like to determine an $\bar{x} \in [x_1,x_n]$ (not ...
0
votes
1answer
14 views

Calculate new ratings based on averages and counts?

For a while, I've been scraping a website for an item's "ratings" data. It seems to me that if I know (for any given timeframe): the start and end count of submitted ratings, and, the start ...
0
votes
0answers
37 views

Plot data points according to the pairwise distance matrix

Consider eight data points. The following matrix (i.e., a symmetric matrix with the lower triangle elements) shows the pairwise distances between any two points. 0 11 0 5 13 0 12 2 14 0 7 17 1 18 0 ...
1
vote
0answers
15 views

Hierarchical Clustering with Ward Distance

I know how hierarchical clustering (with a certain definition of inter-cluster distance) works. And I know that Ward's procedure is based on the goal of minimizing the sum of the squared errors ...
0
votes
0answers
10 views

How can I calculate the confidence of W -> Z\W

I am learning about Itemset mining and association rules. Studying, I came upon this definition To clarify, "minconf" its a variable designated by the user to determine the minimum ...
0
votes
0answers
20 views

MAR (Missing At Random) and its probability of missingness

I am aware that MAR (Missing at Random) means the propensity for a data point to be missing is not related to the missing data, but it is related to some of the observed data. However, what I am ...
0
votes
0answers
14 views

LOF - local reachability density - Why is it considered an average?

In the context of Local Outlier Factor (described in the following paper https://dl.acm.org/doi/10.1145/335191.335388 ), the local reachability density of a point p is defined as follow : $$ lrd_K (p) ...
0
votes
0answers
16 views

Most Important Variables Linear Regression

When determining the most important variables in linear regression, I understand that first we need to decrease multicollinearity, if present, by using some sort of variable reduction method such as ...
0
votes
0answers
19 views

Derivation of the Chebyshev distance from the Minkowski distance

I know that the Minkowski distance is defined as follows: $$d(x,y)=\lim_{r\to\infty}\left(\sum_{k=1}^n|x_k-y_k|^r\right)^{1/r}$$ and I also read that the Chebyshev distance could be considered as a ...
0
votes
0answers
32 views

Approximate data of N elements as a sum of products of the target

Computer Engineer here. I need help with a personal project that is data science related. I am trying to approximate how to solve something like this. Each of these values are a PCA(principle ...
4
votes
1answer
54 views

Courses to take during pure math masters to keep data science and applied work as a possibility

I was wondering what courses you can take in a pure math masters to preserve the opportunity to go into data science, economics, policy research or other applied work while preserving the opportunity ...
0
votes
0answers
9 views

Assigning Probabilities to Alarming Events for predicting System Outages

I have a problem. I have two sets of data, let's assume they are clean and well-organised. Data-set 1: Alarm Events. Data-set 2: System Outages. The two data-sets have only one feature in common - the ...
0
votes
0answers
22 views

For a set $\mathcal{S}\subset\mathbb{R}^n$, find a function $g(.)$, so that $s=g(x_1,…,x_r)$, $x_i\in\mathbb{R},r\leq n$, for each $s\in\mathcal{S}$

Thank you in advance for your help. I am trying to fight with a technical issue that I do not really know how to prove. I have a set of $n$ sensors that provide me with $n$ measurements –not ...
0
votes
0answers
19 views

Select correct sampling rate for a process, data science

For example, lets say I collected a days worth of a several variables every 0.1 seconds, and I want to collect them every day from now on. If I were to keep the 0.1 second sampling rate my database ...
0
votes
0answers
14 views

What is the difference between biclustering and clustering?

After reading the wiki page for biclustering (https://en.m.wikipedia.org/wiki/Biclustering), I am really confused on what is the difference between biclustering and clustering? Any explanation/...
0
votes
0answers
24 views

Is the Hamming distance sensitive to sequence of bits?

Two binary strings ($110$) and ($100$) have a hamming distance of $1$, similarly ($110$) and ($111$) also have hamming distance $1$. Hamming distance is not sensitive to the sequence of bits. Is there ...
1
vote
0answers
34 views

Classification of 32 bit integers into 2 classes with uneven probability

I've been given a set of integer data $x_t$ that are all 32-bit unsigned integers. These data have been previously divided into two classes using a function unknown to me as the following: $$f(x)=\...
0
votes
0answers
17 views

Show that the solution of optimization problem for K-means is not unique

Show that the solution of optimization problem for K-means is not unique: $$\text{arg min} \sum_{l=1}^k \sum_{i \in S_l} \vert \vert x_i - \mu_l \vert \vert ^2$$ s.t. $\mu_1,...,\mu_k \in R^p$ and $\...
0
votes
0answers
19 views

What metrics can be used to describe the clustering of a group of points?

Consider a set of points $\Omega \subset \mathbb{R}^n$ (to make it simple, consider $n=2$). So you basically have a bunch of points on a plane. I need to find good metrics that can describe the ...
0
votes
1answer
19 views

Multi Label Classification: Union of two Binary Sets

I have started Data Science on my own and was looking at evaluation metrics. I came across this measure of Accuracy with the equation $$ \frac{1}{p}\sum_{i=1}^{p}\frac{\vert Y_i \cap Z_i \vert}{\vert ...
0
votes
0answers
24 views

How to convert collection of data into equation

I want to convert some random numbers like $1, 5, 8, 10, 1, 5$ to $F(x)$ $F(1)=1, F(2)=5, F(3)=8, F(4)=10, F(5)=1$ and $ F(6)=5$ Like that I need some generalisation to convert any data, for ...
0
votes
0answers
10 views

Asymptotically equivalent with the Bayesian Information Criterion (BIC)

Question Show that $$ln[\pi(n + p +2)\bar {\sigma}^2 ] - 2ln \Gamma (n/2) - ln|\mathbf{\Sigma}| $$ is asymptotically equivalent to the BIC approximation as $n \rightarrow \infty$ $$n\:[ln(2\pi\bar {\...
0
votes
1answer
70 views

Expected Optimism 0-1 Loss with 0-1 Response

Want to show that $$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(\hat{Y}, Y_i)$$ For 0-1 loss function with 0-1 response. Want I've done $$op = l_{in} - l=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', \hat{Y})-\...
0
votes
1answer
21 views

example 1.4 chapter 1 from mining of massive data sets book

I am reading the book mining of massive data sets. (http://mmds.org/) In its chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf following section is there on page 9. Example 1.4: Suppose ...
2
votes
1answer
354 views

exercise 1.3 from Mining of Massive Data Sets book

Hello there is a question given in Mining of Massive Data Sets book http://infolab.stanford.edu/~ullman/mmds/ch1.pdf it is on page 15 exercise 1.3.2 My solution is following: as there are $10$ ...
0
votes
2answers
217 views

Bonferroni’s Principle discussed in Mining of Massive Data Sets book

I am reading a chapter of book Mining of Massive Data Sets book is available here http://www.mmds.org Chapter 1 http://infolab.stanford.edu/~ullman/mmds/ch1.pdf Now in Section 1.2.3 An example of ...
1
vote
0answers
92 views

Definition of Entropy (Information Theory)

In Information Theory, entropy is defined as: $$-\sum_{i}P_ilog(P_i)$$ where $-P_ilog(P_i)$ looks like this (using log base 2): From just a generic English definition of entropy, meaning lack of ...
2
votes
1answer
25 views

Can i use the coefficients of a trained model of Logistic Regression as a result itself without using the model on unseen data?

I'm trying to figure out if i can use a logistic regression as a predictive model, to estimate the probability of response of a user in CRM by having the predictors and i also have the class (...
1
vote
1answer
87 views

How is Autoregressive Model used in longitudinal data analysis?

I know that in the analysis of time series, Auto-regressive Model such as AR(1) is frequently used. In the context of time series, there is no covariate (or the only covariate is time). In the context ...
0
votes
1answer
88 views

why does singular values of M equal to square roots of eigenvalues $M^TM$

per wiki, there is a rule to compute singular values The non-zero singular values of M (found on the diagonal entries of Σ) are the square roots of the non-zero eigenvalues of both $M^*M$ and $MM^...
2
votes
0answers
548 views

Calculation of Intrinsic dimension of datasets

Currently I am following a Machine learning course and we are looking at the intrinsic dimension of datasets. The professor gave a few examples of the intrinsic dimension of some objects (ej. the ...
0
votes
1answer
33 views

Dimensionality reduction for high dimensional curves?

I have a continuous curve in high dimensional space, and I'd like to visualize it in lower dimensional (2D or 3D) space to get an intuition of what it looks like. I'm familiar with PCA and t-SNE, but ...
0
votes
1answer
29 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
0
votes
1answer
46 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
1
vote
1answer
29 views

Can deep learning be a good way to learn a “High-quality” simple functions for images? For example, identical transformation, rotation, translation. [closed]

Can deep learning be a good way to learn a "High-quality" simple functions for images? For example, identical transformation, rotation, translation, even a linear mapping.
0
votes
1answer
23 views

If there are any curve databases with structured data

I found these curve lists: http://www.lmfdb.org/EllipticCurve/ https://en.wikipedia.org/wiki/List_of_curves http://www-groups.dcs.st-and.ac.uk/~history/Curves/Curves.html http://old.nationalcurvebank....
1
vote
1answer
21 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
1
vote
0answers
38 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
0
votes
1answer
100 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-p174318763 ...
0
votes
1answer
64 views

Finding the max number of pairs in a market basket problem

Is my reasoning here correct or is there a better way to solve this problem? Given a set of items $I$ and a set of baskets $B$, where each basket contains $k$ items, what is the maximum number of ...
1
vote
1answer
418 views

Intuitive difference between cosine similarity and bilinear similarity

Given a pair of strings in vector form $(s_i,s_j)$, I can find cosine similarity of pairs as follows: $cosine(s_i,s_j)=s_i.*s_j / (\|s_i\|\|s_j\|)$ Similarly, bilinear similarity is defined as: ...
1
vote
1answer
40 views

Efficient methods to compute PCA for an incrementally growing matrix

I am wondering if there exists an efficient method to compute PCA. I am drafting the question into a presudo code: ...
2
votes
1answer
260 views

Probability and Statistics Books for Distributions and Introduction to Data Mining/Machine Learning

In college, I took a probability class using Sheldon Ross' A First Course in Probability. It was not my best semester to say the least. However, I am returning back to probability and statistics as it ...
0
votes
1answer
71 views

Is variance of mean equals mean of covariance?

I am trying to finish a problem, my method requires to prove variance of mean equals mean of covariance, but I have trouble proving it. Is it correct? Or more condition needed? Now I use $$Var(X)=E(X^...
0
votes
1answer
27 views

What's matrix $W$ in nonlinear PCA?

Nonlinear PCA is based on minimizing wrt matrix $W$ the function: $$I = E \{ \|x-Wg(W^Tx)\|^2\}$$ where $g$ is an odd function. However, what is $W$?
0
votes
1answer
16 views

What are correlation coefficients used for in PCA?

What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for?
0
votes
1answer
50 views

Why approximate pairwise distances only over lower triangle of distance matrix?

In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix? $$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$...
0
votes
0answers
22 views

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean?

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean? The projection seems to generally be defined as $\tilde{X} = XR$, but what does the tilde do there?
1
vote
1answer
54 views

Why/when is there $\frac{1}{2}$ in front of Least Squares Estimator?

So in PCA I encountered a formulation for LSE, which is: $$\frac{1}{2} \sum_{i=1}^N ||x_i - \tilde{x_i}||^2$$ Where $\tilde{x}$ is a "restriction" of $x$ such that only parts of the observations are ...