Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-analysis]

Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

0
votes
1answer
20 views

What Conclusions Can I Draw from Percentiles About my Set of Data?

I have a specific survey question that I am analyzing for the topic Should High Schools Implement a Uniform Policy?. I am supposed to create biased survey questions ...
0
votes
0answers
7 views

Best Average to Represent Quantitative Data?

For a final assignment, my two group members and I have created a biased survey called Should Public Schools Implement a Uniform Policy? We have about 2-3 ...
0
votes
1answer
20 views

How to fit Poission distribution with large mean?

I have one set of discrete data plotted below. I suspect the data follows Poisson distribution with mean $\mu \simeq 400$ and standard deviation $\sigma \simeq \sqrt{400} = 20$. $$ n(k) = \frac{\mu ^...
0
votes
3answers
47 views

What Can Median Tell Me About my Data?

For a final assignment for math, I was required to take the "Yes!" side of 'Should Uniforms Be Implemented at School?' My job was to create a biased survey that would make people agree with our side ...
-1
votes
0answers
37 views

What does Cross-Validation do (Data Science)?

I am confused about what cross-validation is for: a) model optimization or b) model validation? Some good online courses mention that cross validation can be used to select the best model after ...
0
votes
0answers
19 views

Is there any exact definition for “imbalanced datasets” or “imbalanced learning”?

What is the exact definition for "imbalanced datasets" or "imbalanced learning"? Some article states it as, In such problems, almost all the instances are labeled as one class, while far fewer ...
-3
votes
0answers
22 views

Dimension reduction in Principal Components Analysis (PCA)

it's probably a dumb question but I can't find the answer to this anywhere. In PCA we want to reduce the dimension of data $x$ from $d \times 1$ to $k \times 1$ ; so we search a matrix A where $A^TA=...
0
votes
0answers
17 views

Extrapolation error of linear regression lines for a two-cluster data set

I am studying on some linear regression problems using the least-squares method and stumbled upon a problem regarding the error when extrapolating far-away datapoints. About the problem: For a point $...
0
votes
0answers
6 views

Calinski - Harabasz criterion for choosing optimal number of clusters

I've been using the CH criterion to determine the optimal number of clusters for some data I'm working with. I know that in general you would evaluate the criterion for a number of different choices ...
0
votes
0answers
26 views

How to quickly resolve a value from noise around a signal?

Kalman filter seems to be slow in resolving a steady signal. Is there a way to apply Ito calculus (stochastic calculus) to resolve the signal quicker? The data takes a minute or longer to ...
1
vote
1answer
30 views

Numerical integration of first order differential equation & fitting to experimental data

We are working on phase transformation kinetics in Advanced High Strength steels and we are trying to fit our experimental data to the following differential for recristallization, by Kulakov et. al ...
0
votes
0answers
24 views

Producing a range of 0-100 based on multiple variables with different units (percent, numerical)

I'm absolutely no statistics wizard and it is my first post here, so I hope you bear with me. I'm working in Excel on a project where I need to convert data from a number of variables into a "one ...
2
votes
1answer
30 views

Solving equations with 6 different gaussian function

I have a large dataset coming from absorption spectroscopy experiment and i'm facing a problem to extract relevant informations. My matrix look like this : $ S_n(x) = \left( \frac{a_n}{\sigma_1\sqrt{...
1
vote
2answers
31 views

How to calculate average on set of data points where recent data has more weight

Take for example this dataset: [1,1,1,1,1,1,1,1,1,200,1,1,1,1,....,1] I want to calculate a running average on this with a certain window size for avg, let's say ...
1
vote
1answer
18 views

Training vs validation set

I have a data set with which I am trying to find correlations. I split the data into a training set and a validation set. I also have a solver I built which finds the "best coefficients" to give me ...
1
vote
1answer
37 views

change detection formula most suited to analyse 3 vector changes

The following data is captured from the magnetic sensor of a device. This data has a time factor. The device is at times in motion and other times still How can one detect (using formula) if there ...
0
votes
0answers
15 views

Kernel trick in svm?

Let $\phi : \mathbb R^d \rightarrow \mathbb R^n$. Consider a kernel $K(x, y)$. Prove that for all $x, y \in \mathbb R^d, \| φ(x)−φ(y)\|_2 = \sqrt{K(x,x)−2K(x,y)+K(y,y)}.$ If I recall correctly $K(...
0
votes
0answers
13 views

Auto-correlation function for a time signal

For a time signal $x(t)= \cos^2(2\pi t)$, solve for the Auto-correlation Function. Basically I have been following a guide on how to compute the auto-correlation function and am getting hung-up on ...
3
votes
1answer
32 views

Smooth a graph as if placing a rope across the data…?

I'm not sure how to correctly phrase this question, in fact if I knew exactly what I needed to ask I could probably work it out myself, so please bear with me. What I need to do is smooth out a line,...
1
vote
0answers
34 views

Linear Regression Diagnostics

I am trying to determine if there is a relationship between a dependent variable y and independent variable x by fitting a least squares regression model. Scatterplot of data: Diagnostic plots: ...
1
vote
0answers
26 views

Linear Interpolation for scattered 3-D data

I have a dataset of scattered 3-D points (non-regular) that carry some variable and am trying to interpolate that variable to a new point. I have currently implemented a couple of methods, but don't ...
0
votes
1answer
22 views

When can i use a normal distribution to describe my data?

I have dataset which is markedly left-skewed, and I wonder if it will be inappropriate to use the normal distribution curve to analyses the data given it's highly skewed? Picture of my data
0
votes
1answer
18 views

Linearising an Equation to the Form y=mx+c

I have been working on some data analysis stuff and I have to linearise this equation so I can plot it as a straight line with form y=mx+c $$T=2\pi\sqrt{\frac{(k^2 + h^2)}{gh}}$$ Where, T will be ...
0
votes
1answer
14 views

Is there a name for this method of data consolidation / reduction?

Say I have a data series of N elements. I want to create second series that represents a rough approximation of the first series with a reduced set of elements M. Question 1: Is there a proper name ...
0
votes
1answer
36 views

Why is the number of non zero eigenvalues equal to $x^T \Sigma^{-1} x$

I've been reading this code and I found that the number of non-zero eigenvalues of the estimated covariance is equal to $x_i^T \Sigma^{-1} x_i$. I want to know how to arrive at this result. Some ...
1
vote
0answers
22 views

How to calculate smoothing spline coefficients

I am attempting to calculate smoothing spline coefficients based on the description in Reinsch's 1967 paper, but I'm having some trouble. The first derivative is not continuous. Here are the ...
1
vote
1answer
24 views

Appropriate number of significant figures to use in plot

I am trying to illustrate the convergence of a sequence of iterates by means of the following plot: convergence plot In my discussion I would like to state the fact that 1) the function is ...
0
votes
0answers
20 views

sampling from an existing sample

I have an existing data set with 8 000 observations. I want to check an association between two parameters in the data set. I also want to see if maybe for this purpose I do not need 8 000 ...
1
vote
3answers
55 views

Confidence interval of an average coefficient calculated from curve fits, with each curve fit using a parameter which has its own confidence interval

I am curve fitting a model to data sets in order to determine a coefficient for each curve fit/data set. I then calculate an average coefficient and the 95% confidence interval for this average. ...
1
vote
1answer
25 views

Data transformation: new min, max and mean

I have a dataset of 31 values, of which the $min = -0,8,\ max = 11, 1$ and $mean = 5,0$. Is there a way to transform these data to a $\text{new} \ minimum (0.3), \ \text{new} \ maximum (11.4)$ and $...
0
votes
1answer
63 views

Logic to choose a data point from a pool of data points based on two criteria.

My data points can be expressed in a vector form as: $X = [x(1), x(2), x(3), ..........,x(N)]$ Each element of $X$ has two criteria/characteristics, $P$ and $H$. $P$ is the posterior probability (...
0
votes
0answers
8 views

Fitting to a distribution in higher dimention $D \geq 2$

I have data related to $n$ number of variables $n\geq 2$, I need to fit it to one of the known distributions, I am looking for related literature. In a particular example, let the distribution be ...
1
vote
1answer
45 views

Can this be solved with Least Squares?

I have images (data is represented by grayscale values from 0 to 255) and the total sum of the grayscale values represent the mass of the object. I know that every grayscale is related to mass, given ...
1
vote
0answers
29 views

Average percentage overlap between two/more datasets

I am analyzing different schemes of mutual funds. Each scheme has many funds in its portfolio. I wanted to analyse the overlap (of funds) between these schemes. I can find the overlapping of scheme1 ...
0
votes
0answers
18 views

Running a Statistical test on an unbalanced number of samples

I would like to run a a statistical test, such as an ANOVA, on the following data: I have a variable z that depends on two independent variables x and y, where x and y can take on five possible ...
0
votes
1answer
31 views

What projection (transformation) is it?

Suppose I have a 3D scattered data $$(A_1,B_1,Z_1)$$$$(A_2,B_3,Z_3)$$$$(A_3,B_3,Z_3)$$$$...$$$$(A_n,B_n,Z_n),$$ shown schematically in the picture below (left) and there is no order in the data. Now,...
1
vote
1answer
34 views

Whats a good book on data analysis for someone with solid understanding of statistics?

In about four months I will start helping out in a bee research center. For this job, anyone with a background in science or lab-work can apply and its mainly to help doing the tedious little things ...
1
vote
0answers
28 views

Matlab and integrating under area of Gaussian curve for lab experiment

This is going to be a long post due to information that I am going to have to present. A brief outline in what I am asking is, that I need to integrate a Gaussian at a specific height above the x-...
0
votes
0answers
46 views

A dataset has skewness = 1 with missing data. Standard deviation around median is 1.5. How much data will be unaffected?

There is no other description about the data, if it is univariate, bivariate, etc. niether the type of distribution is given. I recently came across this question, I would like to know how skewness ...
1
vote
1answer
57 views

How to detect inflection points without function expression in a live time series scenario?

Most of the learnings and examples out there talks about finding inflection point given there is a function expression. What happens when I do-not know the function expression? The movement of the ...
0
votes
0answers
8 views

What is “data distriubtion”? Does it Belong to Probability Space?

While reading the paper BEGAN : Boundary Equilibrium Generative Adversarial Network the autor writes as following: "the generator $G(z)$, which maps a sample $z$ from uniform distirubiton to the data ...
0
votes
0answers
59 views

Does doing homework improve student's grades? (Statistical analysis)

I am currently measuring the progress on student's grades and their commitment in doing their homework. My goal here is to prove to them that doing homework will improve their grades. I am working ...
0
votes
0answers
26 views

Unable to understand mathematical expression for a given Data Analysis problem

Note: This question is part of my textbook and I am unable to copy the text and thus attaching the corresponding snips. I have one question (word problem) related to Data Analysis which i was able to ...
1
vote
1answer
93 views

Why is there so little data-driven fluid dynamics research?

With the insane growth of data science, I notice that there's hardly any data-driven fluid dynamics research out there. What could explain this phenomenon? I have heard that fluid dynamicists ...
0
votes
0answers
21 views

Mean difference

I have two sets of sample data with the first one having the mean of 100.1 and the second set having the mean of 192.1 ...
0
votes
1answer
16 views

How to calculate uncertanty based on the uncertanty of a data set?

Suppose you have a data set of experimental measurements were for each measurement you have the uncertainty associated to it. What is the correct way to calculate the uncertainty associated with the ...
0
votes
0answers
23 views

Dynamic data bounded normalization with bias towards mean

Input dataset: A sample of random number of random positive real numbers. This sample is collected periodically. Output: Normalization of those numbers normalized to between 0 and 1. Bias towards the ...
0
votes
2answers
46 views

Interpretation of the given box and whisker plot

I wish to understand whether i have interpreted below box & whisker plot correctly; this will also assert my understanding on the same. (I am learning basic statistics & measure of dispersion) ...
0
votes
1answer
17 views

How to create an accurate baseline mean when outliers exist in a large dataset

I have a relatively large database of 40,000+ entries, all with several data points per entry-line. The Data is collected on an ongoing basis. I would like to establish a baseline mean and standard ...
4
votes
1answer
50 views

How do I determine a good function to fit a set of data?

I have a calculated some rates based on data, but I was wondering if there is a way to determine a good math function that might fit this data. For example, consider the following: ...