Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

This tag is for questions about data mining, which is an interdisciplinary subfield of computer science. It is the computational process of discovering patterns in large data sets.

0
votes
2answers
711 views

Factor Analysis in Text Mining task

I was faced with the task of determining the topics of big text massive. For example you have 1 million any text phrases or sentences. I want factorize the main topic from this massive. The ordinary ...
4
votes
1answer
66 views

How to find almost periodic lattices in a set of high-dimensional points?

sorry for lame question, but I just have no maturity in this direction. Let's say I have very large set ( millions ) of high-dimensional vectors ( typical dimensionality is 64). These vectors ...
3
votes
1answer
36 views

Clustering elements according to covariance matrix

I'm doing a little bit of topic modelling (which is not really my area) with twitter tweets. The situation is the following: I have a (sort of) covariance matrix where the entrie $C_{ij}$ corresponds ...
1
vote
1answer
291 views

Does Gini index considers only a binary split for each attribute or can it have multi way spliting?

Does Gini-index based classification split values for any attribute always as a binary split or can it split into more than $2$ branches (multi-way split)? For more clarification, if a split on $A$ ...
1
vote
1answer
45 views

What is the meant by node role in Graph Theory?

Suppose that I have a graph with some numbers of nodes, and each node connected to other node with a relation, My question what is the node roles here ? Is the role = the relation type, weighted ...
0
votes
1answer
18 views

Different order of insertion - different Bayesian network ? how to prove formally?

I have some Bayesian network which i constructed from some data, say it consists of nodes A, B, C and D and that was the initial order of insertion. If i ...
0
votes
1answer
29 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
0
votes
1answer
72 views

Uniform Effect of K-means Clustering

In the following link is discussed the uniform Effect of K-means Clustering: https://www.springer.com/cda/content/document/cda_downloaddocument/9783642298066-c2.pdf?SGWID=0-0-45-1338325-...
0
votes
1answer
14 views

What are correlation coefficients used for in PCA?

What are correlation coefficients used for in PCA? One can discover them through the PCA formulation, but what are they useful for?
0
votes
1answer
27 views

Why approximate pairwise distances only over lower triangle of distance matrix?

In the context of dimensionality reduction. Why approximate pairwise distances only over lower triangle of distance matrix? $$\min_{\{\hat{x_i}\}} I = \sum_{i <j} ({ \hat{ d_{ij} } - d_{ij} })^2$$...
0
votes
1answer
23 views

How to interpret maximizing “separability and reciprocal of scattering” in Fisher's LDA?

How to interpret maximizing "separability and reciprocal of scattering" in Fisher's LDA? That is, if $s_1$ is minimized scattering inside (projected) class 1 and $s_2$ is the same for class 2. Then ...
0
votes
1answer
43 views

Elementary reference on Support Vector Machines

I am having difficulties in reading the book Support vector mathine by Ingo Steinwart with some notations and mathematical terms . Can you please recommend a prerequisite book or online course by ...
0
votes
1answer
42 views

Can't understanding information entropy and data compression, for beginners

I watched a very well done video about information entropy, and I thought I got the concept in my mind but then I asked my self a question that denied every certainty I had. The video if you are ...
0
votes
1answer
17 views

Analyze mixture of data attributes

When I have a list of persons (nodes) with the attributes: REVENUE | LINKS I can search for the people with most links and the people with most revenue. But when I want to know who has the ...
0
votes
1answer
26 views

Looking for a ranking method to produce a list of products from best to worst given some features

I am pretty new to data mining and I am trying to help a friend of mine who is working on his graduate thesis in economics. The main purpose of his work is to rank a number of items (smart home ...
3
votes
0answers
45 views

Number of rules (Data Mining)

This is a paraphrasing of a statement from Data Mining by Witten et al., 4th edition, on section 1.6. Consider the set $S = \{1, \dots, 288\}$. A rule is defined as a one-element singleton set ...
3
votes
0answers
212 views

Does there exist closed-form solutions for calculations in the EM-algorithm for Gaussian Mixtures?

Our data can be represented as $X_{n\times d}$ matrix, where we have $n$ data points lying in $\mathbb{R}^d$. We assume that there are $k$ underlying Gaussian models, $\mathcal{N}_d(\mu_j, \Sigma_j)$, ...
3
votes
0answers
355 views

Gini coefficient vs Gini impurity - decision trees

The problem refers to decision trees building. According to Wikipedia 'Gini coefficient' should not be confused with 'Gini impurity'. However both measures can be used when building a decision tree - ...
2
votes
0answers
426 views

Spectral relaxation of k-means clustering

I am working on a presentation on Spectral relaxation of k-means clustering (http://papers.nips.cc/paper/1992-spectral-relaxation-for-k-means-clustering.pdf) and I am a bit stuck. I understand ...
2
votes
0answers
278 views

Optimization of English Braille: Using the fewest dots

Background: The English Braille system is laid out in such a way so that the letters can be referenced by their position in the alphabet. Of the six dots available for each character, the top four ...
2
votes
0answers
56 views

How can I infer order from partially ordered discrete sequences?

A really interesting problem that I can't stop thinking about! Have run in to this a couple of times but yet to find a smart approach to either solve or frame this problem. This is my try at ...
2
votes
0answers
750 views

How do I measure the goodness of cosine similarity scores across different vector spaces?

I am a computer scientist working on a problem that requires some statistical measures, though (not being very well versed in statistics) I am not quite sure what statistics to use. Overview: I ...
1
vote
0answers
34 views

Finding statistically increasing and decreasing sub-sequences in a (noisy) vector

I have a vector of real non-negative values of length ~60. The values represent a geometric property (can be area, circumference, etc.) of an object extracted from a movie of a biological sample, and ...
1
vote
0answers
12 views

In Fisher's LDA, what is $a_i =w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation?

In Fisher's LDA, what is $a_i = w^T x_i$ if $w$ is an unit vector and $x_i$ is an observation? My notes write that: projection lengths of data into it (unit vector $w$) as $a_i = w^T x_i$ So is $...
1
vote
0answers
19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...
1
vote
0answers
27 views

data analysing graphs from excel

i have a question at excel. if you can help me?? i need to do data analysing graphs from excel, and they look more or the less the same like this one. and i want to separate them to steps ...
1
vote
0answers
26 views

Data set generation

For my research work, I need to generate data set by using some standard statistical distribution. I have seen Unknown distribution for data set generation. I do not have any idea about unknown ...
1
vote
0answers
31 views

Calculate PageRank for small web

Calculate PageRank for: A links to B, B links to C and C links to B and C where the damping factor $\beta=0.8$ I have: $M=\begin{bmatrix} 0&0&\frac{1}{2} \\ 1&0&\frac{1}{2} \\ 0&...
1
vote
0answers
116 views

How do I calculate the upper bound probability using Hoeffding's Ineqaulity?

The image below show mathematical expressions for the meanings of training and testing. I can understand the meaning of testing but the meaning of training is not so clear because of $M$ there. I took ...
1
vote
0answers
34 views

Can someone show me a simple example of finding simple squared error in k-means?

I'm working on some homework and one of our assignments asks us to find the final mean of each cluster and the SSE (sum of squared error). I haven't been able to find anything online and was wondering ...
1
vote
0answers
37 views

Regression of 6 input parameters

I have a set of data that contains 6 inputs and 1 output (about 500 rows). Basically, I'm trying to figure out the complex relationship between the 6 inputs (say different component of concrete mix, i....
1
vote
0answers
61 views

Odd Ratio and Logistic Regression

Windy and Play Tennis = 9 Windy and Not Playing Tennis = 8 Not Windy and Play Tennis = 14 Not Windy and Not Playing Tennis = 8 I performed logistic regression in Weka and got odd ratio as 0.3448 for ...
1
vote
0answers
10k views

How can I get the covariance just given the variance?

This is a fairly straight forward problem but I don't know how to calculate the covariance given just the variance of X and Y. Suppose X and Y are independent random variables such that Var(X)=1 and ...
1
vote
0answers
29 views

mutual information for power low distributed data?

I have a dataset with power-low distribution, I would like to measure some kind of correlation/mutual information between the data to its class,(for feature selection task), Can I use mutual-...
0
votes
0answers
16 views

looking for a simple example in machine learning with step-by-step procedure

I am looking for a simple example in the area of machine learning as well as a step-by-step procedure. For example, if I have 3 measurement devices and each one reports 5 data hourly how can I ...
0
votes
0answers
52 views

Predict a subset from sequence of sets

I want to solve this data mining problem: There is a sequence $S(t) = [A_1,...,A_{t-1}]$ . Each $A_i \subset U$ are set. Current time is $t \in \mathbb{N}$. $B$ is a subset of $A_t$. ($B \subset A_t$)...
0
votes
0answers
19 views

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean?

In the context of PCA and random projection, what does $\tilde{X} \sim XR$ mean? The projection seems to generally be defined as $\tilde{X} = XR$, but what does the tilde do there?
0
votes
0answers
21 views

Does “more points” necessarily imply “smoother boundary”?

So in the context of Naive Bayes I encountered a postulate that the more "nearest neighbours" one includes (i.e. the larger the $k$ in $k$-NN), the smoother the boundaries of classes become. Is this ...
0
votes
0answers
29 views

Extrapolation of measurement data

I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it ...
0
votes
0answers
32 views

Can I use independence for bayesian network?

I'm struggling trying to understand bayesian network. Having trouble finding P(F&G) I know independence and conditional independence are different things. Conditional independence is when A ...
0
votes
0answers
42 views

Difference between Data Mining and a model?

I am pretty much confused with the terms between pattern mining and a model in data mining. For example, I have a number of sequential patterns like below: No. 1: A->B->C->A = 0.5 No. 2: B->C->D = 0....
0
votes
0answers
38 views

Things between Maximum Likelihood Estimation and Ordinary Least Squares?

What are the similarities and differences between Ordinary Least Squares and Maximum Likelihood Estimation? Are they both parametric methods? If we have Ordinary Least Squares, why do we need ...
0
votes
0answers
25 views

Build the conclusion with confidence

(if this question doesn't fit here, please point me to right direction. Thanks.) We need to check a sales data S’s accuracy to determine if it can be used in downstream calculations. Currently, it's ...
0
votes
0answers
269 views

Markov chain modelling?

I am working on a personal loan dataset. For each loan, we recorded its credit status monthly after it was drawn by the borrower. Let's say there were 6 status coded by A-F. My project is to use ...
0
votes
0answers
128 views

Number of clusters in $k$-means clustering for higher-dimensional data.

I've read the Wikipedia article and a lot of posts on stackexchange (like this really thorough one) on determining the number of clusters in a data set. Based on that, I am currently using the ...
0
votes
0answers
348 views
0
votes
0answers
23 views

Find repeated attribute patterns in collection of objects

I have objects $o_1,...,o_k$where every object has a set of properties $p_1,...,p_j$. I'm looking for property patterns that are repeated among these objects. It could be perhaps called correlation of ...
0
votes
0answers
21 views

Find all groups that meets the condition

I have $n$ elements, each of them have two unsigned int attributes $x$ and $y$. Now I'd like to find out all the groups that fit the following condition: $A.x \geq B.y$ and $B.x \geq A.y$. The ...