Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

54 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
66 views

How to find almost periodic lattices in a set of high-dimensional points?

sorry for lame question, but I just have no maturity in this direction. Let's say I have very large set ( millions ) of high-dimensional vectors ( typical dimensionality is 64). These vectors ...
3
votes
0answers
46 views

Number of rules (Data Mining)

This is a paraphrasing of a statement from Data Mining by Witten et al., 4th edition, on section 1.6. Consider the set $S = \{1, \dots, 288\}$. A rule is defined as a one-element singleton set ...
3
votes
0answers
237 views

Does there exist closed-form solutions for calculations in the EM-algorithm for Gaussian Mixtures?

Our data can be represented as $X_{n\times d}$ matrix, where we have $n$ data points lying in $\mathbb{R}^d$. We assume that there are $k$ underlying Gaussian models, $\mathcal{N}_d(\mu_j, \Sigma_j)$, ...
3
votes
1answer
38 views

Clustering elements according to covariance matrix

I'm doing a little bit of topic modelling (which is not really my area) with twitter tweets. The situation is the following: I have a (sort of) covariance matrix where the entrie $C_{ij}$ corresponds ...
3
votes
0answers
377 views

Gini coefficient vs Gini impurity - decision trees

The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - ...
2
votes
0answers
36 views

Calculate PageRank for small web

Calculate PageRank for: A links to B, B links to C and C links to B and C where the damping factor $\beta=0.8$ I have: $M=\begin{bmatrix} 0&0&\frac{1}{2} \\ 1&0&\frac{1}{2} \\ 0&...
2
votes
0answers
465 views

Spectral relaxation of k-means clustering

I am working on a presentation on Spectral relaxation of k-means clustering (http://papers.nips.cc/paper/1992-spectral-relaxation-for-k-means-clustering.pdf) and I am a bit stuck. I understand ...
2
votes
0answers
309 views

Optimization of English Braille: Using the fewest dots

Background: The English Braille system is laid out in such a way so that the letters can be referenced by their position in the alphabet. Of the six dots available for each character, the top four ...
2
votes
0answers
58 views

How can I infer order from partially ordered discrete sequences?

A really interesting problem that I can't stop thinking about! Have run in to this a couple of times but yet to find a smart approach to either solve or frame this problem. This is my try at ...
2
votes
0answers
776 views

How do I measure the goodness of cosine similarity scores across different vector spaces?

I am a computer scientist working on a problem that requires some statistical measures, though (not being very well versed in statistics) I am not quite sure what statistics to use. Overview: I ...
1
vote
0answers
61 views

Definition of Entropy (Information Theory)

In Information Theory, entropy is defined as: $$-\sum_{i}P_ilog(P_i)$$ where $-P_ilog(P_i)$ looks like this (using log base 2): From just a generic English definition of entropy, meaning lack of ...
1
vote
0answers
36 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
1
vote
0answers
12 views

In Fisher's LDA, what is $a_i =w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation?

In Fisher's LDA, what is $a_i = w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation? My notes write that: projection lengths of data into it (unit vector $w$) as $a_i = w^T x_i$ So is $...
1
vote
0answers
19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...
1
vote
0answers
29 views

data analysing graphs from excel

i have a question at excel. if you can help me?? i need to do data analysing graphs from excel, and they look more or the less the same like this one. and i want to separate them to steps ...
1
vote
0answers
26 views

Data set generation

For my research work, I need to generate data set by using some standard statistical distribution. I have seen Unknown distribution for data set generation. I do not have any idea about unknown ...
1
vote
0answers
129 views

How do I calculate the upper bound probability using Hoeffding's Ineqaulity?

The image below show mathematical expressions for the meanings of training and testing. I can understand the meaning of testing but the meaning of training is not so clear because of $M$ there. I took ...
1
vote
0answers
35 views

Can someone show me a simple example of finding simple squared error in k-means?

I'm working on some homework and one of our assignments asks us to find the final mean of each cluster and the SSE (sum of squared error). I haven't been able to find anything online and was wondering ...
1
vote
0answers
37 views

Regression of 6 input parameters

I have a set of data that contains 6 inputs and 1 output (about 500 rows). Basically, I'm trying to figure out the complex relationship between the 6 inputs (say different component of concrete mix, i....
1
vote
1answer
368 views

Does Gini index considers only a binary split for each attribute or can it have multi way spliting?

Does Gini-index based classification split values for any attribute always as a binary split or can it split into more than $2$ branches (multi-way split)? For more clarification, if a split on $A$ ...
1
vote
0answers
62 views

Odd Ratio and Logistic Regression

Windy and Play Tennis = 9 Windy and Not Playing Tennis = 8 Not Windy and Play Tennis = 14 Not Windy and Not Playing Tennis = 8 I performed logistic regression in Weka and got odd ratio as 0.3448 for ...
1
vote
0answers
10k views

How can I get the covariance just given the variance?

This is a fairly straight forward problem but I don't know how to calculate the covariance given just the variance of X and Y. Suppose X and Y are independent random variables such that Var(X)=1 and ...
1
vote
1answer
50 views

What is the meant by node role in Graph Theory?

Suppose that I have a graph with some numbers of nodes, and each node connected to other node with a relation, My question what is the node roles here ? Is the role = the relation type, weighted ...
1
vote
0answers
29 views

mutual information for power low distributed data?

I have a dataset with power-low distribution, I would like to measure some kind of correlation/mutual information between the data to its class,(for feature selection task), Can I use mutual-...
0
votes
0answers
34 views

Show Unbiased Learner

I want to show that $g_{\tau}(\mathbf{x}) = \mathbf{x}^T\hat{\beta}$ where $\hat{\beta} = \mathbf{X}^+\mathbf{y}$ and $\tau$ denotes a fixed training set is an unbiased learner, in the sense that: $$\...
0
votes
0answers
36 views

Expected Optimism 0-1 Loss with 0-1 Response

Want to show that $$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(\hat{Y}, Y_i)$$ For 0-1 loss function with 0-1 response. Want I've done $$op = l_{in} - l=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', \hat{Y})-\...
0
votes
0answers
29 views

Data Mining - Minimizer of the risk

To show Given: $$Loss(y, \hat{y}) = \alpha (\hat{y}-y)_+ + \beta (y-\hat{y})_+$$ Note: $c_+$ is equal to c if c > 0 and zero otherwise! Show that the minimizer of the risk $l(g) = E Loss(Y, g(X))$ ...
0
votes
0answers
18 views

Given distances from known distributions, how to get the unknown distribution with which the distances are calucated with

Hi the question is actually pretty common in data mining contest: whenever you submit a result you will get a score (the distance). After a lot of trials you will have a lot of pairs of distribution ...
0
votes
0answers
68 views

Calculation of Intrinsic dimension of datasets

Currently I am following a Machine learning course and we are looking at the intrinsic dimension of datasets. The professor gave a few examples of the intrinsic dimension of some objects (ej. the ...
0
votes
1answer
19 views

Dimensionality reduction for high dimensional curves?

I have a continuous curve in high dimensional space, and I'd like to visualize it in lower dimensional (2D or 3D) space to get an intuition of what it looks like. I'm familiar with PCA and t-SNE, but ...
0
votes
0answers
17 views

looking for a simple example in machine learning with step-by-step procedure

I am looking for a simple example in the area of machine learning as well as a step-by-step procedure. For example, if I have 3 measurement devices and each one reports 5 data hourly how can I ...
0
votes
1answer
23 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
0
votes
1answer
37 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
0
votes
0answers
55 views

Predict a subset from sequence of sets

I want to solve this data mining problem: There is a sequence $S(t) = [A_1,...,A_{t-1}]$ . Each $A_i \subset U$ are set. Current time is $t \in \mathbb{N}$. $B$ is a subset of $A_t$. ($B \subset A_t$)...
0
votes
1answer
85 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-...
0
votes
1answer
14 views

What are correlation coefficients used for in PCA?

What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for?
0
votes
1answer
32 views

Why approximate pairwise distances only over lower triangle of distance matrix?

In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix? $$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$...
0
votes
0answers
21 views

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean?

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean? The projection seems to generally be defined as $\tilde{X} = XR$, but what does the tilde do there?
0
votes
1answer
27 views

How to interpret maximizing “separability and reciprocal of scattering” in Fisher's LDA?

How to interpret maximizing "separability and reciprocal of scattering" in Fisher's LDA? That is, if $s_1$ is minimized scattering inside (projected) class 1 and $s_2$ is the same for class 2. Then ...
0
votes
0answers
33 views

Extrapolation of measurement data

I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it ...
0
votes
0answers
35 views

Can I use independence for bayesian network?

I'm struggling trying to understand bayesian network. Having trouble finding P(F&G) I know independence and conditional independence are different things. Conditional independence is when A ...
0
votes
1answer
44 views

Elementary reference on Support Vector Machines

I am having difficulties in reading the book Support vector mathine by Ingo Steinwart with some notations and mathematical terms . Can you please recommend a prerequisite book or online course by ...
0
votes
1answer
47 views

Can't understanding information entropy and data compression, for beginners

I watched a very well done video about information entropy, and I thought I got the concept in my mind but then I asked my self a question that denied every certainty I had. The video if you are ...
0
votes
1answer
17 views

Analyze mixture of data attributes

When I have a list of persons (nodes) with the attributes: REVENUE | LINKS I can search for the people with most links and the people with most revenue. But when I want to know who has the ...
0
votes
0answers
47 views

Difference between Data Mining and a model?

I am pretty much confused with the terms between pattern mining and a model in data mining. For example, I have a number of sequential patterns like below: No. 1: A->B->C->A = 0.5 No. 2: B->C->D = 0....
0
votes
0answers
41 views

Things between Maximum Likelihood Estimation and Ordinary Least Squares?

What are the similarities and differences between Ordinary Least Squares and Maximum Likelihood Estimation? Are they both parametric methods? If we have Ordinary Least Squares, why do we need ...
0
votes
0answers
25 views

Build the conclusion with confidence

(if this question doesn't fit here, please point me to right direction. Thanks.) We need to check a sales data S’s accuracy to determine if it can be used in downstream calculations. Currently, it's ...
0
votes
0answers
294 views

Markov chain modelling?

I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use ...
0
votes
0answers
131 views

Number of clusters in $k$-means clustering for higher-dimensional data.

I've read the Wikipedia article and a lot of posts on stackexchange (like this really thorough one) on determining the number of clusters in a data set. Based on that, I am currently using the ...
0
votes
0answers
383 views

How to calculate conditional probability for real data

I have my data in the form as: ...