Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-analysis]

Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

0
votes
0answers
11 views

Why is F1 score good for imbalanced classes?

Ok, suppose we have an extremely unbalance multiclass classification problem where most examples belong to class 0, and we want to decide a target metric of performance. We want to choose a metric ...
0
votes
0answers
15 views

To code for principal component in functional PCA [on hold]

I am trying to get the way to find eigenfunction from eigenvalues. As far as I understand from reading that I have to multiply eigenvalues by something to get principle components in functional PCA. ...
0
votes
0answers
9 views

Decomposition of posterior porbability

So, I'm currently working through "Data Analysis: A Bayesian Tutorial 2nd Ed." by D.S. Sivia and I am stuck an equation, which I am not able to reproduce (p.40 - 42 for those who might have a copy). ...
0
votes
1answer
25 views

I ran an ANN model and got an extremely low R2 but a pretty good MSE, what does this mean?

I ran an artificial neuron network on data with about 2,000 rows and 3 features. I got a R2 of .06 which is really low, but a good MSE of .41. Why are these performance evaluators of this model ...
0
votes
0answers
14 views

Linear advection equation with coefficient given at data points

I am working on a problem to solve some particle population balances. In the analysis of some experiments I got an equation of this type: $$\frac{\partial u}{\partial y} - \alpha(x, y) \frac{\partial ...
0
votes
0answers
6 views

Unit vector Identicon

Is there a way to visualize multiple n-dimensional ($n\approx300$) unit vectors with the property that large changes to the vector result in large visual changes, but small changes result in little to ...
0
votes
0answers
7 views

What is a lag-based correlation?

could anyone please advise as to what a lag-based correlation is? Is this the same as a cross-lagged analysis? This type of analysis was recommended to me by a reviewer to elucidate the ...
0
votes
1answer
21 views

Should i remove “know” faulty datapoints from dataset before doing statistical analysis or after importing to statistical analysis program? [closed]

I have a dataset of 21017 rows in excel, this has ~12 columns. At row 10013 i have i have one, (1), measurement that i know must be faulty (As it physically cant happen) I want to import this ...
2
votes
0answers
27 views

Is it appropriate to use clustering to partition the dependent variable into separate datasets for a home price prediction model?

I'm struggling to decide how to deal with a heteroskedasticity problem in a home price prediction model I'm developing. The training set residuals are normally distributed around zero, but they have ...
0
votes
0answers
26 views

How to fit models to data?

I am working on a project in which I am comparing how well and exponential, logistic and Gompertz model fit to tumour growth data. I have the equations for all three models but I don't know how to go ...
1
vote
2answers
79 views

Recursion $T(n)=5T(n/4)+n^3(\log\log(n))$ [closed]

I need help solve this: $\log$ is in base $4$ n=4^4^k $T(n)=5T(n/4)+n^3(\log\log(n))$ I tried to do something and I got stuck here: $T(4^4^k)=5T(4^(4^k-1))+((4^4^k)^3)k$ how can I continue from here I ...
2
votes
0answers
95 views

Compare euclidean distances from different dimensions

EDIT: I'd like to reformulate my primary question. I have a set of points in $\mathbb{R}^D$ and I reduce the dimensions of the points to some $\mathbb{R}^L$. I do this with multiple configurations ...
1
vote
1answer
30 views

How to find the causality of a periodic behaviour

If you use Google Trends (a tool to illustrate the frequency of search terms) and search for the word "war" with USA as geographic region, you will be presented with the graph below showing monthly ...
0
votes
1answer
27 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
0
votes
0answers
13 views

Can Kabsch's algorithm also provide the covariance of its solution?

Say you have a collection of data points $p_i \in \Bbb R^3$, and a collection of corresponding reference points $r_i \in \Bbb R^3$. Kabsch's algorithm, which relies on SVD decomposition, provides an ...
0
votes
1answer
32 views

Finding out what data is useful to use…

I'm trying to determine the best method for narrowing down data to use in a program I'm writing. I could use some help. Here is a use case... Sometimes we have bad data in our system and it can lead ...
0
votes
1answer
30 views

SVD and PCA - help

I have some data on cars: i have been given the Noise,size,speed,if its electric or not, if its a lorry or not. it looks something like this: $$ \begin{matrix} Noise & Size & Speed &...
0
votes
0answers
17 views

Interpolation on curvilinear grid

How could someone interpolate spatial data on a curvilinear grid, like the one in the picture shown?
3
votes
0answers
48 views

Solving recurrences using substitution method

I have given $$ T(n)= \begin{cases} T(n/3)+T(2n/3)+n,\quad &n>1 \\ 1, \quad &n=1 \end{cases} $$ I tried it again and again but couldn't think beyond, $$ T(n)=T(n/27)+3T(...
0
votes
1answer
16 views

Similarity between observations and an ideal trend

I have a series of well known data showing, for example, the average weight by age in Europe. Let's say yesterday I called a lot of people and I asked for their age and weight. The data coming from ...
1
vote
0answers
20 views

How can I make a best fit line out of data where each point is weighted more heavily than the previous?

I'm thinking that there should be a linear regression technique that allows me to value recent data more that older data. (Maybe this difference in weight could be configurable through a constant). ...
0
votes
0answers
14 views

longitudinal study with repeated measures

I have a data-set which is collected using two devices (A and B) to measure the same variable, for 10 participants during 5 different types of activities . Each activity goes for 15 minutes and there ...
1
vote
2answers
29 views

In Support-Vector-Machine, why is the hyperplane given by $(p-1)$ dimensions?

In SVM, each feature vector is viewed as a data point in a $p$-dimensional plane and different labels are separated by a $(p-1)$ dimensional hyperplane. Please see the wikipedia entry: https://en....
0
votes
0answers
76 views

Sample mean and sample covariance canonical forms

Let $(x_1, x_2,\dotsc, x_n)$ be a sequence of vectors: ($\forall i=1,\dots,n$) \begin{pmatrix} x_i^1 \\ x_i^2\\ \vdots\\ x_i^m \end{pmatrix} In statistics, one often has to compute the ...
0
votes
1answer
28 views

Calculating error on slope of graph

I'm trying to find the rate of change and the error on that rate based on 7 measurements points and the assumption that the trend is linear. My calculations are below: $$ \begin{array}{|c|c|c|c|} \...
1
vote
0answers
31 views

Modeling distributed densities or is it?

Lets say that I am counting pixels over time. Each pixel is either the color red or blue. Let's say that I make up some threshold for the number of red pixels that I count and that when I reach this ...
0
votes
1answer
21 views

Understanding the lower and upper limit of a graph

I was recently presented the following graph: I was given very little context and I have to recreate it. I don't want it to be done for me, but some context would help. Any idea what these upper and ...
0
votes
0answers
10 views

Problem about multivaluable distributions. (Hotelling distribution)

From $n_1$ p-dimensional random vectors: $X_1,...,X_{n_1}$ independents with $X_i\sim N_p(\mu_1,\Sigma)$ and $n_2$ random vectors with the same dimension: $Y_1,...,Y_{n_2}$ independents with $Y_i\sim ...
0
votes
1answer
28 views

Quantile-Quantile (QQ) Plots

I understand how to assess whether an exponential/normal distribution is suitable to model a piece of data when the parameters are given, i.e. finding the theoretical quantiles and plotting against ...
0
votes
0answers
17 views

Regression analysis for Measurement values

I'm in a situation where I'm provided with a signal and a couple of measurement values for that signal. Usually the signal will drift a little when the measurement start (i.e. because of heat). I want ...
0
votes
0answers
15 views

Minimum and Maximum Data Points in a Spectrum

If I am collecting data that will be used to assemble a spectrum, how often should I take a data point to properly sample, but not oversample given the following information? I can control my data ...
1
vote
1answer
18 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
0
votes
0answers
18 views

Calculating an average of multiple data sets based on priority.

First and foremost I'm no mathematician, it may be a case of over complication. What I'm trying to do is that I have 2 sets of data, I want to calculate one average for both sets but I want the first ...
0
votes
2answers
44 views

Similarity between two lists

I am working with psychology, more precisely, the Big Five personality traits. I have a test which measures 5 variables, that is believed to describe personality traits of human beings. The variables ...
0
votes
0answers
13 views

Help with transforming a value so that it better fits within a recursive filter

I am working on an application where I need to evaluate a metric that determines how 'well distributed' a set of points ($X) are in the 2D plane that is a camera image. By evaluating this metric for ...
2
votes
1answer
44 views

Compare Two Values - Stardew Valley Food

In the game, Stardew Valley, you can pick up items through farming, fishing, foraging, cooking, etc. Most tasks use up your character's energy, such as swinging an ...
4
votes
2answers
128 views

Are there 'serious' mathematical problems emerging from data science?

Are there any proper research-level mathematics problems that have come out of the activity known as "data science"? I put the quotes because it really is not so clear what "data science" is to me; I ...
1
vote
0answers
21 views

Finding Objects Most Similar to Other Objects as a “Linear Combination”

I have a small collection of objects, $\{o_i\}_{i=1}^{20},$ that have several properties (all quantitative, or can be made to be quantitative) that differ among themselves. So $o_1$ has, say, $10$ ...
0
votes
1answer
47 views

What Conclusions Can I Draw from Percentiles About my Set of Data?

I have a specific survey question that I am analyzing for the topic Should High Schools Implement a Uniform Policy?. I am supposed to create biased survey questions ...
0
votes
0answers
27 views

Best Average to Represent Quantitative Data?

For a final assignment, my two group members and I have created a biased survey called Should Public Schools Implement a Uniform Policy? We have about 2-3 ...
0
votes
1answer
23 views

How to fit Poission distribution with large mean?

I have one set of discrete data plotted below. I suspect the data follows Poisson distribution with mean $\mu \simeq 400$ and standard deviation $\sigma \simeq \sqrt{400} = 20$. $$ n(k) = \frac{\mu ^...
0
votes
3answers
60 views

What Can Median Tell Me About my Data?

For a final assignment for math, I was required to take the "Yes!" side of 'Should Uniforms Be Implemented at School?' My job was to create a biased survey that would make people agree with our side ...
0
votes
0answers
20 views

Is there any exact definition for “imbalanced datasets” or “imbalanced learning”?

What is the exact definition for "imbalanced datasets" or "imbalanced learning"? Some article states it as, In such problems, almost all the instances are labeled as one class, while far fewer ...
0
votes
0answers
30 views

Extrapolation error of linear regression lines for a two-cluster data set

I am studying on some linear regression problems using the least-squares method and stumbled upon a problem regarding the error when extrapolating far-away datapoints. About the problem: For a point $...
0
votes
0answers
11 views

Calinski - Harabasz criterion for choosing optimal number of clusters

I've been using the CH criterion to determine the optimal number of clusters for some data I'm working with. I know that in general you would evaluate the criterion for a number of different choices ...
0
votes
0answers
29 views

How to quickly resolve a value from noise around a signal?

Kalman filter seems to be slow in resolving a steady signal. Is there a way to apply Ito calculus (stochastic calculus) to resolve the signal quicker? The data takes a minute or longer to ...
1
vote
1answer
35 views

Numerical integration of first order differential equation & fitting to experimental data

We are working on phase transformation kinetics in Advanced High Strength steels and we are trying to fit our experimental data to the following differential for recristallization, by Kulakov et. al ...
0
votes
0answers
26 views

Producing a range of 0-100 based on multiple variables with different units (percent, numerical)

I'm absolutely no statistics wizard and it is my first post here, so I hope you bear with me. I'm working in Excel on a project where I need to convert data from a number of variables into a "one ...
2
votes
1answer
30 views

Solving equations with 6 different gaussian function

I have a large dataset coming from absorption spectroscopy experiment and i'm facing a problem to extract relevant informations. My matrix look like this : $ S_n(x) = \left( \frac{a_n}{\sigma_1\sqrt{...
1
vote
2answers
42 views

How to calculate average on set of data points where recent data has more weight

Take for example this dataset: [1,1,1,1,1,1,1,1,1,200,1,1,1,1,....,1] I want to calculate a running average on this with a certain window size for avg, let's say ...