Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-analysis]

Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

0
votes
0answers
25 views

cumulative average and normal average, what is interpretation differ?

Recently I ask a question, Delete it and post it from other view. I have some words: This report presents the data as the cumulative average value per user per the time increment you are using (...
0
votes
0answers
22 views

What kind of data visualization should I use? [closed]

I'm going to program a customized phone keyboard where some letters are larger than others, depending on how often I misstype them. For example, if I often pressed "w" instead of "e", I'd make the "e" ...
0
votes
1answer
32 views

Finding out what data is useful to use…

I'm trying to determine the best method for narrowing down data to use in a program I'm writing. I could use some help. Here is a use case... Sometimes we have bad data in our system and it can lead ...
0
votes
1answer
27 views

SVD and PCA - help

I have some data on cars: i have been given the Noise,size,speed,if its electric or not, if its a lorry or not. it looks something like this: $$ \begin{matrix} Noise & Size & Speed &...
0
votes
0answers
16 views

Interpolation on curvilinear grid

How could someone interpolate spatial data on a curvilinear grid, like the one in the picture shown?
1
vote
0answers
40 views

Limitations to using the method of least squares to fit an ellipse to experimental data?

I have used the method described below many times without issue to determine the major and minor axes of the "data ellipse". However, for this set of data (Fig. A) the method described below is ...
3
votes
0answers
33 views

Solving recurrences using substitution method

I have given $$ T(n)= \begin{cases} T(n/3)+T(2n/3)+n,\quad &n>1 \\ 1, \quad &n=1 \end{cases} $$ I tried it again and again but couldn't think beyond, $$ T(n)=T(n/27)+3T(...
0
votes
1answer
16 views

Similarity between observations and an ideal trend

I have a series of well known data showing, for example, the average weight by age in Europe. Let's say yesterday I called a lot of people and I asked for their age and weight. The data coming from ...
1
vote
0answers
19 views

How can I make a best fit line out of data where each point is weighted more heavily than the previous?

I'm thinking that there should be a linear regression technique that allows me to value recent data more that older data. (Maybe this difference in weight could be configurable through a constant). ...
0
votes
0answers
12 views

longitudinal study with repeated measures

I have a data-set which is collected using two devices (A and B) to measure the same variable, for 10 participants during 5 different types of activities . Each activity goes for 15 minutes and there ...
1
vote
2answers
29 views

In Support-Vector-Machine, why is the hyperplane given by $(p-1)$ dimensions?

In SVM, each feature vector is viewed as a data point in a $p$-dimensional plane and different labels are separated by a $(p-1)$ dimensional hyperplane. Please see the wikipedia entry: https://en....
0
votes
0answers
74 views

Sample mean and sample covariance canonical forms

Let $(x_1, x_2,\dotsc, x_n)$ be a sequence of vectors: ($\forall i=1,\dots,n$) \begin{pmatrix} x_i^1 \\ x_i^2\\ \vdots\\ x_i^m \end{pmatrix} In statistics, one often has to compute the ...
0
votes
1answer
24 views

Calculating error on slope of graph

I'm trying to find the rate of change and the error on that rate based on 7 measurements points and the assumption that the trend is linear. My calculations are below: $$ \begin{array}{|c|c|c|c|} \...
1
vote
0answers
31 views

Modeling distributed densities or is it?

Lets say that I am counting pixels over time. Each pixel is either the color red or blue. Let's say that I make up some threshold for the number of red pixels that I count and that when I reach this ...
0
votes
1answer
20 views

Understanding the lower and upper limit of a graph

I was recently presented the following graph: I was given very little context and I have to recreate it. I don't want it to be done for me, but some context would help. Any idea what these upper and ...
0
votes
0answers
8 views

Problem about multivaluable distributions. (Hotelling distribution)

From $n_1$ p-dimensional random vectors: $X_1,...,X_{n_1}$ independents with $X_i\sim N_p(\mu_1,\Sigma)$ and $n_2$ random vectors with the same dimension: $Y_1,...,Y_{n_2}$ independents with $Y_i\sim ...
0
votes
1answer
26 views

Quantile-Quantile (QQ) Plots

I understand how to assess whether an exponential/normal distribution is suitable to model a piece of data when the parameters are given, i.e. finding the theoretical quantiles and plotting against ...
0
votes
0answers
17 views

Regression analysis for Measurement values

I'm in a situation where I'm provided with a signal and a couple of measurement values for that signal. Usually the signal will drift a little when the measurement start (i.e. because of heat). I want ...
0
votes
0answers
14 views

Minimum and Maximum Data Points in a Spectrum

If I am collecting data that will be used to assemble a spectrum, how often should I take a data point to properly sample, but not oversample given the following information? I can control my data ...
1
vote
1answer
18 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
0
votes
0answers
17 views

Calculating an average of multiple data sets based on priority.

First and foremost I'm no mathematician, it may be a case of over complication. What I'm trying to do is that I have 2 sets of data, I want to calculate one average for both sets but I want the first ...
0
votes
2answers
35 views

Similarity between two lists

I am working with psychology, more precisely, the Big Five personality traits. I have a test which measures 5 variables, that is believed to describe personality traits of human beings. The variables ...
0
votes
0answers
12 views

Help with transforming a value so that it better fits within a recursive filter

I am working on an application where I need to evaluate a metric that determines how 'well distributed' a set of points ($X) are in the 2D plane that is a camera image. By evaluating this metric for ...
2
votes
1answer
41 views

Compare Two Values - Stardew Valley Food

In the game, Stardew Valley, you can pick up items through farming, fishing, foraging, cooking, etc. Most tasks use up your character's energy, such as swinging an ...
4
votes
2answers
119 views

Are there 'serious' mathematical problems emerging from data science?

Are there any proper research-level mathematics problems that have come out of the activity known as "data science"? I put the quotes because it really is not so clear what "data science" is to me; I ...
1
vote
0answers
16 views

Finding Objects Most Similar to Other Objects as a “Linear Combination”

I have a small collection of objects, $\{o_i\}_{i=1}^{20},$ that have several properties (all quantitative, or can be made to be quantitative) that differ among themselves. So $o_1$ has, say, $10$ ...
0
votes
1answer
24 views

What Conclusions Can I Draw from Percentiles About my Set of Data?

I have a specific survey question that I am analyzing for the topic Should High Schools Implement a Uniform Policy?. I am supposed to create biased survey questions ...
0
votes
0answers
18 views

Best Average to Represent Quantitative Data?

For a final assignment, my two group members and I have created a biased survey called Should Public Schools Implement a Uniform Policy? We have about 2-3 ...
0
votes
1answer
23 views

How to fit Poission distribution with large mean?

I have one set of discrete data plotted below. I suspect the data follows Poisson distribution with mean $\mu \simeq 400$ and standard deviation $\sigma \simeq \sqrt{400} = 20$. $$ n(k) = \frac{\mu ^...
0
votes
3answers
54 views

What Can Median Tell Me About my Data?

For a final assignment for math, I was required to take the "Yes!" side of 'Should Uniforms Be Implemented at School?' My job was to create a biased survey that would make people agree with our side ...
0
votes
0answers
20 views

Is there any exact definition for “imbalanced datasets” or “imbalanced learning”?

What is the exact definition for "imbalanced datasets" or "imbalanced learning"? Some article states it as, In such problems, almost all the instances are labeled as one class, while far fewer ...
0
votes
0answers
22 views

Extrapolation error of linear regression lines for a two-cluster data set

I am studying on some linear regression problems using the least-squares method and stumbled upon a problem regarding the error when extrapolating far-away datapoints. About the problem: For a point $...
0
votes
0answers
9 views

Calinski - Harabasz criterion for choosing optimal number of clusters

I've been using the CH criterion to determine the optimal number of clusters for some data I'm working with. I know that in general you would evaluate the criterion for a number of different choices ...
0
votes
0answers
29 views

How to quickly resolve a value from noise around a signal?

Kalman filter seems to be slow in resolving a steady signal. Is there a way to apply Ito calculus (stochastic calculus) to resolve the signal quicker? The data takes a minute or longer to ...
1
vote
1answer
32 views

Numerical integration of first order differential equation & fitting to experimental data

We are working on phase transformation kinetics in Advanced High Strength steels and we are trying to fit our experimental data to the following differential for recristallization, by Kulakov et. al ...
0
votes
0answers
26 views

Producing a range of 0-100 based on multiple variables with different units (percent, numerical)

I'm absolutely no statistics wizard and it is my first post here, so I hope you bear with me. I'm working in Excel on a project where I need to convert data from a number of variables into a "one ...
2
votes
1answer
30 views

Solving equations with 6 different gaussian function

I have a large dataset coming from absorption spectroscopy experiment and i'm facing a problem to extract relevant informations. My matrix look like this : $ S_n(x) = \left( \frac{a_n}{\sigma_1\sqrt{...
1
vote
2answers
40 views

How to calculate average on set of data points where recent data has more weight

Take for example this dataset: [1,1,1,1,1,1,1,1,1,200,1,1,1,1,....,1] I want to calculate a running average on this with a certain window size for avg, let's say ...
1
vote
1answer
19 views

Training vs validation set

I have a data set with which I am trying to find correlations. I split the data into a training set and a validation set. I also have a solver I built which finds the "best coefficients" to give me ...
1
vote
1answer
38 views

change detection formula most suited to analyse 3 vector changes

The following data is captured from the magnetic sensor of a device. This data has a time factor. The device is at times in motion and other times still How can one detect (using formula) if there ...
0
votes
0answers
15 views

Kernel trick in svm?

Let $\phi : \mathbb R^d \rightarrow \mathbb R^n$. Consider a kernel $K(x, y)$. Prove that for all $x, y \in \mathbb R^d, \| φ(x)−φ(y)\|_2 = \sqrt{K(x,x)−2K(x,y)+K(y,y)}.$ If I recall correctly $K(...
0
votes
0answers
13 views

Auto-correlation function for a time signal

For a time signal $x(t)= \cos^2(2\pi t)$, solve for the Auto-correlation Function. Basically I have been following a guide on how to compute the auto-correlation function and am getting hung-up on ...
3
votes
1answer
32 views

Smooth a graph as if placing a rope across the data…?

I'm not sure how to correctly phrase this question, in fact if I knew exactly what I needed to ask I could probably work it out myself, so please bear with me. What I need to do is smooth out a line,...
1
vote
0answers
35 views

Linear Regression Diagnostics

I am trying to determine if there is a relationship between a dependent variable y and independent variable x by fitting a least squares regression model. Scatterplot of data: Diagnostic plots: ...
1
vote
0answers
30 views

Linear Interpolation for scattered 3-D data

I have a dataset of scattered 3-D points (non-regular) that carry some variable and am trying to interpolate that variable to a new point. I have currently implemented a couple of methods, but don't ...
0
votes
1answer
22 views

When can i use a normal distribution to describe my data?

I have dataset which is markedly left-skewed, and I wonder if it will be inappropriate to use the normal distribution curve to analyses the data given it's highly skewed? Picture of my data
0
votes
1answer
23 views

Linearising an Equation to the Form y=mx+c

I have been working on some data analysis stuff and I have to linearise this equation so I can plot it as a straight line with form y=mx+c $$T=2\pi\sqrt{\frac{(k^2 + h^2)}{gh}}$$ Where, T will be ...
0
votes
1answer
14 views

Is there a name for this method of data consolidation / reduction?

Say I have a data series of N elements. I want to create second series that represents a rough approximation of the first series with a reduced set of elements M. Question 1: Is there a proper name ...
0
votes
1answer
49 views

Why is the number of non zero eigenvalues equal to $x^T \Sigma^{-1} x$

I've been reading this code and I found that the number of non-zero eigenvalues of the estimated covariance is equal to $x_i^T \Sigma^{-1} x_i$. I want to know how to arrive at this result. Some ...
1
vote
0answers
52 views

How to calculate smoothing spline coefficients

I am attempting to calculate smoothing spline coefficients based on the description in Reinsch's 1967 paper, but I'm having some trouble. The first derivative is not continuous. Here are the ...