Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

54 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
66 views

How to find almost periodic lattices in a set of high-dimensional points?

sorry for lame question, but I just have no maturity in this direction. Let's say I have very large set ( millions ) of high-dimensional vectors ( typical dimensionality is 64). These vectors ...
46 views

Number of rules (Data Mining)

This is a paraphrasing of a statement from Data Mining by Witten et al., 4th edition, on section 1.6. Consider the set $S = \{1, \dots, 288\}$. A rule is defined as a one-element singleton set ...
237 views

Does there exist closed-form solutions for calculations in the EM-algorithm for Gaussian Mixtures?

Our data can be represented as $X_{n\times d}$ matrix, where we have $n$ data points lying in $\mathbb{R}^d$. We assume that there are $k$ underlying Gaussian models, $\mathcal{N}_d(\mu_j, \Sigma_j)$, ...
38 views

Clustering elements according to covariance matrix

I'm doing a little bit of topic modelling (which is not really my area) with twitter tweets. The situation is the following: I have a (sort of) covariance matrix where the entrie $C_{ij}$ corresponds ...
377 views

Gini coefficient vs Gini impurity - decision trees

The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - ...
36 views

19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...
29 views

data analysing graphs from excel

i have a question at excel. if you can help me?? i need to do data analysing graphs from excel, and they look more or the less the same like this one. and i want to separate them to steps ...
26 views

Data set generation

For my research work, I need to generate data set by using some standard statistical distribution. I have seen Unknown distribution for data set generation. I do not have any idea about unknown ...
129 views

How do I calculate the upper bound probability using Hoeffding's Ineqaulity?

The image below show mathematical expressions for the meanings of training and testing. I can understand the meaning of testing but the meaning of training is not so clear because of $M$ there. I took ...
35 views

Can someone show me a simple example of finding simple squared error in k-means?

I'm working on some homework and one of our assignments asks us to find the final mean of each cluster and the SSE (sum of squared error). I haven't been able to find anything online and was wondering ...
37 views

Regression of 6 input parameters

I have a set of data that contains 6 inputs and 1 output (about 500 rows). Basically, I'm trying to figure out the complex relationship between the 6 inputs (say different component of concrete mix, i....
368 views

Does Gini index considers only a binary split for each attribute or can it have multi way spliting?

Does Gini-index based classification split values for any attribute always as a binary split or can it split into more than $2$ branches (multi-way split)? For more clarification, if a split on $A$ ...
62 views

Odd Ratio and Logistic Regression

Windy and Play Tennis = 9 Windy and Not Playing Tennis = 8 Not Windy and Play Tennis = 14 Not Windy and Not Playing Tennis = 8 I performed logistic regression in Weka and got odd ratio as 0.3448 for ...
10k views

How can I get the covariance just given the variance?

This is a fairly straight forward problem but I don't know how to calculate the covariance given just the variance of X and Y. Suppose X and Y are independent random variables such that Var(X)=1 and ...
50 views

What is the meant by node role in Graph Theory?

Suppose that I have a graph with some numbers of nodes, and each node connected to other node with a relation, My question what is the node roles here ? Is the role = the relation type, weighted ...
29 views

mutual information for power low distributed data?

I have a dataset with power-low distribution, I would like to measure some kind of correlation/mutual information between the data to its class,(for feature selection task), Can I use mutual-...
34 views

29 views

Data Mining - Minimizer of the risk

To show Given: $$Loss(y, \hat{y}) = \alpha (\hat{y}-y)_+ + \beta (y-\hat{y})_+$$ Note: $c_+$ is equal to c if c > 0 and zero otherwise! Show that the minimizer of the risk $l(g) = E Loss(Y, g(X))$ ...
18 views

Given distances from known distributions, how to get the unknown distribution with which the distances are calucated with

Hi the question is actually pretty common in data mining contest: whenever you submit a result you will get a score (the distance). After a lot of trials you will have a lot of pairs of distribution ...
68 views

Calculation of Intrinsic dimension of datasets

Currently I am following a Machine learning course and we are looking at the intrinsic dimension of datasets. The professor gave a few examples of the intrinsic dimension of some objects (ej. the ...
19 views

Dimensionality reduction for high dimensional curves?

I have a continuous curve in high dimensional space, and I'd like to visualize it in lower dimensional (2D or 3D) space to get an intuition of what it looks like. I'm familiar with PCA and t-SNE, but ...
17 views

looking for a simple example in machine learning with step-by-step procedure

I am looking for a simple example in the area of machine learning as well as a step-by-step procedure. For example, if I have 3 measurement devices and each one reports 5 data hourly how can I ...
23 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
37 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
55 views

Predict a subset from sequence of sets

I want to solve this data mining problem: There is a sequence $S(t) = [A_1,...,A_{t-1}]$ . Each $A_i \subset U$ are set. Current time is $t \in \mathbb{N}$. $B$ is a subset of $A_t$. ($B \subset A_t$)...
85 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-...
14 views

What are correlation coefficients used for in PCA?

What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for?
32 views

Why approximate pairwise distances only over lower triangle of distance matrix?

In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix? $$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$...
21 views

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean?

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean? The projection seems to generally be defined as $\tilde{X} = XR$, but what does the tilde do there?
27 views

How to interpret maximizing “separability and reciprocal of scattering” in Fisher's LDA?

How to interpret maximizing "separability and reciprocal of scattering" in Fisher's LDA? That is, if $s_1$ is minimized scattering inside (projected) class 1 and $s_2$ is the same for class 2. Then ...
33 views

Extrapolation of measurement data

I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it ...
35 views

Can I use independence for bayesian network?

I'm struggling trying to understand bayesian network. Having trouble finding P(F&G) I know independence and conditional independence are different things. Conditional independence is when A ...
44 views

Elementary reference on Support Vector Machines

I am having difficulties in reading the book Support vector mathine by Ingo Steinwart with some notations and mathematical terms . Can you please recommend a prerequisite book or online course by ...
47 views

Can't understanding information entropy and data compression, for beginners

I watched a very well done video about information entropy, and I thought I got the concept in my mind but then I asked my self a question that denied every certainty I had. The video if you are ...
17 views

Analyze mixture of data attributes

When I have a list of persons (nodes) with the attributes: REVENUE | LINKS I can search for the people with most links and the people with most revenue. But when I want to know who has the ...
47 views

Difference between Data Mining and a model?

I am pretty much confused with the terms between pattern mining and a model in data mining. For example, I have a number of sequential patterns like below: No. 1: A->B->C->A = 0.5 No. 2: B->C->D = 0....
41 views

Things between Maximum Likelihood Estimation and Ordinary Least Squares?

What are the similarities and differences between Ordinary Least Squares and Maximum Likelihood Estimation? Are they both parametric methods? If we have Ordinary Least Squares, why do we need ...
25 views

Build the conclusion with confidence

(if this question doesn't fit here, please point me to right direction. Thanks.) We need to check a sales data S’s accuracy to determine if it can be used in downstream calculations. Currently, it's ...
294 views

Markov chain modelling?

I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use ...