Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-analysis]

Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

21
votes
2answers
5k views

Where to start learning about topological data analysis?

I was wondering if anyone could help me out with finding a nice introductory introductory text for topological data analysis (I'm speaking as somebody who has two semesters of experience with topology,...
15
votes
4answers
7k views

Polynomial fitting where polynomial must be monotonically increasing

Given a set of monotonically increasing data points (in 2D), I want to fit a polynomial to the data which is monotonically increasing over the domain of the data. If the highest x value is 100, I don'...
13
votes
2answers
965 views

Roadmap for learning Topological Data Analysis?

I'm a math major who has recently graduated and I will be starting full time work in 'data analysis'. Having finished with decent marks and still being incredibly interested in mathematics, I was ...
12
votes
4answers
3k views

Mathematics base for data mining and artificial intelligence algorithms. [closed]

Could you give me some clarification about data mining and artificial intelligence algorithms? What mathematics base they used for? Could you give me starting point, in mathematics, to understand ...
9
votes
3answers
1k views

Removing noise when the signal is not smooth

Suppose we have (an interval of) a time series of measurements: We assume it can be explained as a "simple" underlying signal overlaid by noise. I'm interested in finding a good algorithm to estimate ...
7
votes
5answers
657 views

Finding the orientation of a noisy ellipse

This question comes from a neuroscience study which generates $12$ vectors. The vectors are evenly spaced, $30 n$ degrees for $n=0,\dots, 11$, each with their tail centered on the origin. I am ...
7
votes
2answers
285 views

Monitoring a data stream

You are monitoring a data stream which is delivering very many $32$-bit quantities at a rate of $10$ Megabytes per second. You know that either: $A$: All values occur equally often, $or$ $B$: Half ...
7
votes
2answers
99 views

How do I find the formula (or rules) that created a list of numbers with seemingly no pattern?

Newbie here, and I apologize if this is the wrong forum for this type of question... I have a group of 200 or so alphanumeric codes from an unknown source. Here's an example piece of the data set: <...
6
votes
2answers
41k views

Value range of normalization methods? min-max, z-score, decimal scaling

I am working my way through Normalization (data transformation) of data and was curious about four methods: min-max normalization, 2. z-score, 3. z-score mean absolute deviation, and 4. decimal ...
6
votes
2answers
31k views

Use a set of data points from a graph to find a derivative

I have a data logger that is recording the temperature readings from thermocouples at a specific interval. This gives me data points that I can graph where the x-coordinate is time and the y-...
6
votes
2answers
613 views

Chaos theory and fractal geometry: Constructing from data

I understand that fractal geometry represents behaviour of 'chaotic' system, if I am not wrong. And also, fractals are generated by a recursive function. But, lets say I have random data lying with me....
6
votes
0answers
136 views

Explicit definition of a sequence

Suppose there are 6 sequences $a=(a_n)_{n\geq 0}, b=(b_n)_{n\geq 0},c=(c_n)_{n\geq 0},d=(d_n)_{n\geq 0},e=(e_n)_{n\geq 0},f=(f_n)_{n\geq 0}$, the data can be seen here: Data. I found out by trial and ...
6
votes
0answers
697 views

Relationship between eigenvectors of correlation and covariance matrices

For the purpose of computing principal components of a dataset, represented as matrix $X$ of dimensions $n \times p$ with $n$ samples and $p$ features, we can compute sample covariance matrix $S$, and ...
5
votes
2answers
659 views

Lottery Ticket Question

A lottery ticket contains six different numbers, chosen from 1 to 39. The winning ticket will match all six numbers in the correct order, plus a bonus number, which may match the other six numbers. ...
5
votes
3answers
29k views

Find the median given a table of relative frequencies

I came across the following GRE question. I had no problem finding the mean. However, the answer for the median is given to be 1. I don't understand how they arrive at this. Find the mean and ...
5
votes
2answers
935 views

Difference between Topological Data Analysis and Graph Technology

I'm trying to understand the difference between Oracle's graph technology, which apparently has an inherent understanding of topology, and Ayasdi's Topological Data Analysis technology. Are these two ...
5
votes
2answers
247 views

Highest points in heart cycle graph

I'm working on an application that reads the heart cycle from a device, and I've aimed to get this image: Now, I need to get the highest points that appear in every cycle in order to calculate the ...
5
votes
4answers
642 views

Testing for chaos in data

I have data for 3 variables ,each with respect to the discrete time values. How do I check for the existence of chaos for this 3D discrete system?(I don't have the analytic eqs.,just the data). MY ...
5
votes
1answer
1k views

Abstract Algebra for Business Intelligence / Data Mining

Twitter released its Algebird library, an abstract algebra library for scala. At a high level, what are the maths involved in abstract algebra that have an impact on aggregating large datasets?
5
votes
0answers
79 views

Is persistent homology mathematically “interesting”? [closed]

Currently, I am occupying myself with (non-applied perspectives on) Persistent Homology, having a background in non-applied maths (representation theory). After having spent some time on the topic ...
4
votes
2answers
3k views

Way to compare three numbers with an algorithm.

Say I have values 10, 3, 6 What would be a way of finding a subjective difference between them. Or in other words, how close they are to each-other. So if i had 3, 2.5, 2.8; they would be close ...
4
votes
2answers
4k views

Power set of a set containing a set

I need some help understanding this concept a little better. I understand the general power sets, but only worked nice and easy examples where only the set consisted of only single elements like $\{a,...
4
votes
3answers
4k views

What would this curve be called?

In an resistor-capacitor circuit, the voltage stored is measured by taking the two values in Farads and Ohms - and the curve looks like this: What would this curve be called, logarithmic? exponential ...
4
votes
1answer
5k views

Mutual Information for clustering

I'm working on a document clustering application and decided to use Normalized Mutual Information as one of the measures of effectivenes. But I don't really understand how to implement this in that ...
4
votes
2answers
128 views

Are there 'serious' mathematical problems emerging from data science?

Are there any proper research-level mathematics problems that have come out of the activity known as "data science"? I put the quotes because it really is not so clear what "data science" is to me; I ...
4
votes
1answer
262 views

Is topological data analysis useful for low dimensional data?

I often read how Topological Data Analysis (TDA) is useful especially for highly dimensional data. But, what about (apparently) low dimensional ones? Example: consider measuring the resistance of a ...
4
votes
2answers
5k views

What is the most scientific way to assign weights to historical data?

This is a common question I usually face while processing historical data. I have year on year data of an event for the past N years.I would like to assign weights to the data of these N years so that ...
4
votes
2answers
420 views

libraries to plot topological data analysis persistence diagrams, complexes, and barcodes

I am starting to do some Topological Data Analysis and am looking for a good plotting library to produce the diagrams of complexes, barcodes, and persistence diagrams. The most popular TDA libraries ...
4
votes
1answer
364 views

Calculate trend and represent in text?

First off, I'm terrible at math. I'm writing a script that monitors transactions from clients daily over a 7 day period. Given a set of numbers like below, I would like to calculate a trend and ...
4
votes
2answers
5k views

How to see that K-means objective is convex?

I'm trying to proof that the objective of the K-means clustering algorithm is non-convex. The objective is given as $J(U,Z) = \|X-UZ\|_F^2$, with $X \in\mathbb{R}^{m\times n}, U\in \mathbb{R}^{m\...
4
votes
1answer
766 views

Normality test vs. Fitting a Gaussian curve

I have a set of one-dimensional data, and I suspect that the data is normally distributed. Before embarking in a normality test, I decided to fit a Gaussian curve to the histogram of relative ...
4
votes
3answers
2k views

Parameters estimation for gaussian function with offset

I've read the paper Least square fitting of a Gaussian function to a histogram by Leo Zhou on how to perform a Least Square Fitting of a gaussian function to a histogram. The Gaussian function used ...
4
votes
2answers
195 views

Detecting increasing pulse trains

I have a one dimensional point process representing the times of events which is also mixed in with lots of data that I regard as noise. The interval between the events in the point process are ...
4
votes
1answer
365 views

Fitting a surface to 2D measurements

I am looking for a way to fit a surface given a set of measured data $(x, y) \mapsto z$. A typical example would consist of anywhere between $10$ and $30$ measurements spread evenly over a disc. ...
4
votes
1answer
51 views

How do I determine a good function to fit a set of data?

I have a calculated some rates based on data, but I was wondering if there is a way to determine a good math function that might fit this data. For example, consider the following: ...
4
votes
0answers
75 views

Are the results from persistent homology complete?

Do the homology groups for each Betti number as produced by persistent homology completely capture all the topological characteristics of the space? If not, what topological features are not captured? ...
4
votes
0answers
2k views

When to use cumulative moving average vs a simple moving average?

After reviewing the Wikipedia page on moving averages, the difference between the simple moving average and cumulative moving average are clear: 1) Simple moving average only considers the last n ...
4
votes
0answers
75 views

What do the parameters of a multinomial logistic regression correspond to?

I've recently started learning about data science/statistics and learned how to derive such models as linear regressors and logistic regressors. What I don't understand, however, is what the ...
4
votes
1answer
589 views

Logistic functions - how to find the growth rate

We have the formule for a model with logistic growth: $$ N_t = N_{t-1} + g\, N_{t-1}\left( 1 -\dfrac{N_{t-1}}{K}\right)$$ where $g$ defines the growth rate and $K$ is the carrying capacity. Let's ...
3
votes
1answer
444 views

What is KL-Divergence? Why Do I need it? How do I use it?

I am currently studying KL Divergence. But It seems very confusing that I don't maybe understand why do I ever need it and what is that for? As I have been reading stuff about Mutual Information, it ...
3
votes
2answers
183 views

How do I use specific data points on a graph to determine an equation?

I need to find a function $f(x)$ such that the following data points would fit on it: $$f(1) = 0 \\ f(2) = 0.5 \\ f(4) = 1.0 \\ f(8) = 1.5 \\ \cdots $$ and so on. So the pattern is every time $x$ ...
3
votes
2answers
20 views

Number of distinct scatterplots from $p$ variables in a data set

Consider the following quote from the text An Introduction to Statistical Learning: In practice, we often encounter data sets that contain many more than two variables. In this case, we cannot ...
3
votes
2answers
157 views

Need to find function by the data

I have found interesting sequence, but I can't find its function. Here are the input and output data: ...
3
votes
1answer
252 views

Prerequisites for Gunnar Carlsson's Topology and Data

I am planning on doing a project on topological data analysis in the near future and intend to use Gunnar Carlsson's paper "Topology and Data" as my introduction to the field. I am familiar with point-...
3
votes
1answer
319 views

Link between PCA and the Eckart-Young theorem

We can find a really good explanation about the link between: finding the subspace upon which the projected data cloud has the maximal variance (or "inertie") finding the subspace upon which the ...
3
votes
1answer
605 views

Tropical geometry: practical applications?

In 1960, E. Wigner published a paper entitled "The Unreasonable Effectiveness of Mathematics in the Natural Sciences". Theoretical mathematical structures pave the way to further advances and ...
3
votes
1answer
128 views

Median of Medians

Given a set A with median Am = 10 and set B with median Bm = 20 is it true that the median of the combined set C is $10 \le$ Cm$\le 20$ ? My first thought was that this wasn't true so I tried to find ...
3
votes
2answers
87 views

What is the chance of a sample in white noise to be a local extremum?

I draw a long sequence of values from a uniform or normal distribution. Observing the resulting sequence, what is the chance of any point in the sequence (aside from the first and last where it is not ...
3
votes
1answer
18k views

Boxplots and bar graphs

I'm studying for the GRE and came across several questions that I was unable to answer in a practice booklet, even after looking at the answer and trying to work backwards, and searching google and ...
3
votes
1answer
33 views

Smooth a graph as if placing a rope across the data…?

I'm not sure how to correctly phrase this question, in fact if I knew exactly what I needed to ask I could probably work it out myself, so please bear with me. What I need to do is smooth out a line,...