Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-analysis]

Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

0
votes
0answers
12 views

Linear advection equation with coefficient given at data points

I am working on a problem to solve some particle population balances. In the analysis of some experiments I got an equation of this type: $$\frac{\partial u}{\partial y} - \alpha(x, y) \frac{\partial ...
2
votes
1answer
691 views

Mathematical model building with dependent and independent variables

I have been working with data and building models on data. I have developed models in regression using cubic and power series. It works fine for variables with one dependent and one independent ...
0
votes
0answers
6 views

Unit vector Identicon

Is there a way to visualize multiple n-dimensional ($n\approx300$) unit vectors with the property that large changes to the vector result in large visual changes, but small changes result in little to ...
2
votes
0answers
93 views

Compare euclidean distances from different dimensions

EDIT: I'd like to reformulate my primary question. I have a set of points in $\mathbb{R}^D$ and I reduce the dimensions of the points to some $\mathbb{R}^L$. I do this with multiple configurations ...
0
votes
0answers
6 views

What is a lag-based correlation?

could anyone please advise as to what a lag-based correlation is? Is this the same as a cross-lagged analysis? This type of analysis was recommended to me by a reviewer to elucidate the ...
0
votes
1answer
19 views

Should i remove “know” faulty datapoints from dataset before doing statistical analysis or after importing to statistical analysis program? [closed]

I have a dataset of 21017 rows in excel, this has ~12 columns. At row 10013 i have i have one, (1), measurement that i know must be faulty (As it physically cant happen) I want to import this ...
1
vote
1answer
1k views

Type 2 Error Question - How to calculate for a two tailed?

The modulus of rupture (MOR) for a particular grade of pencil lead is known to have a standard deviation of 250 psi. Process standards call for a target value of 6500 psi for the true mean MOR. For ...
1
vote
1answer
30 views

How to find the causality of a periodic behaviour

If you use Google Trends (a tool to illustrate the frequency of search terms) and search for the word "war" with USA as geographic region, you will be presented with the graph below showing monthly ...
2
votes
0answers
26 views

Is it appropriate to use clustering to partition the dependent variable into separate datasets for a home price prediction model?

I'm struggling to decide how to deal with a heteroskedasticity problem in a home price prediction model I'm developing. The training set residuals are normally distributed around zero, but they have ...
0
votes
0answers
22 views

How to fit models to data?

I am working on a project in which I am comparing how well and exponential, logistic and Gompertz model fit to tumour growth data. I have the equations for all three models but I don't know how to go ...
3
votes
3answers
1k views

Kalman filter with sensors having different sampling rate

I would like to implement Kalman filter scheme with two different kind of sensors - say 9DOF IMU (gyro/accelerometer/magnetometer) and GPS sensor (position/velocity/altitude). Problem is, as i ...
1
vote
2answers
79 views

Recursion $T(n)=5T(n/4)+n^3(\log\log(n))$ [closed]

I need help solve this: $\log$ is in base $4$ n=4^4^k $T(n)=5T(n/4)+n^3(\log\log(n))$ I tried to do something and I got stuck here: $T(4^4^k)=5T(4^(4^k-1))+((4^4^k)^3)k$ how can I continue from here I ...
1
vote
2answers
81k views

calculation of median of grouped data

While calculating the median of grouped data of total frequency $N$, in order to find the median class which value should be taken into consideration to match against cumulative frequency : $\frac N2$ ...
0
votes
1answer
27 views

Which similarity formula should I use?

I was studying Cosine Similarity and I have just seen this article. https://medium.com/@rahulkuntala9/cosine-similarity-and-handling-categorical-variables-29f907951b5 The author uses Cosine ...
4
votes
3answers
4k views

What would this curve be called?

In an resistor-capacitor circuit, the voltage stored is measured by taking the two values in Farads and Ohms - and the curve looks like this: What would this curve be called, logarithmic? exponential ...
0
votes
0answers
13 views

Can Kabsch's algorithm also provide the covariance of its solution?

Say you have a collection of data points $p_i \in \Bbb R^3$, and a collection of corresponding reference points $r_i \in \Bbb R^3$. Kabsch's algorithm, which relies on SVD decomposition, provides an ...
0
votes
1answer
32 views

Finding out what data is useful to use…

I'm trying to determine the best method for narrowing down data to use in a program I'm writing. I could use some help. Here is a use case... Sometimes we have bad data in our system and it can lead ...
1
vote
1answer
761 views

Relationship between subset medians and the median

Suppose we have a set of data $A = \{a_1, a_2, \dots, a_n\}$ and $B = \{b_1, b_2, \dots, b_n \}$. So there are $2n$ elements in total. Further suppose the median of $A$ and $B$ is $a$ and $b$, ...
0
votes
1answer
29 views

SVD and PCA - help

I have some data on cars: i have been given the Noise,size,speed,if its electric or not, if its a lorry or not. it looks something like this: $$ \begin{matrix} Noise & Size & Speed &...
0
votes
0answers
17 views

Interpolation on curvilinear grid

How could someone interpolate spatial data on a curvilinear grid, like the one in the picture shown?
0
votes
1answer
16 views

Similarity between observations and an ideal trend

I have a series of well known data showing, for example, the average weight by age in Europe. Let's say yesterday I called a lot of people and I asked for their age and weight. The data coming from ...
3
votes
0answers
46 views

Solving recurrences using substitution method

I have given $$ T(n)= \begin{cases} T(n/3)+T(2n/3)+n,\quad &n>1 \\ 1, \quad &n=1 \end{cases} $$ I tried it again and again but couldn't think beyond, $$ T(n)=T(n/27)+3T(...
0
votes
1answer
26 views

Calculating error on slope of graph

I'm trying to find the rate of change and the error on that rate based on 7 measurements points and the assumption that the trend is linear. My calculations are below: $$ \begin{array}{|c|c|c|c|} \...
1
vote
0answers
20 views

How can I make a best fit line out of data where each point is weighted more heavily than the previous?

I'm thinking that there should be a linear regression technique that allows me to value recent data more that older data. (Maybe this difference in weight could be configurable through a constant). ...
0
votes
0answers
13 views

longitudinal study with repeated measures

I have a data-set which is collected using two devices (A and B) to measure the same variable, for 10 participants during 5 different types of activities . Each activity goes for 15 minutes and there ...
0
votes
0answers
75 views

Sample mean and sample covariance canonical forms

Let $(x_1, x_2,\dotsc, x_n)$ be a sequence of vectors: ($\forall i=1,\dots,n$) \begin{pmatrix} x_i^1 \\ x_i^2\\ \vdots\\ x_i^m \end{pmatrix} In statistics, one often has to compute the ...
1
vote
2answers
29 views

In Support-Vector-Machine, why is the hyperplane given by $(p-1)$ dimensions?

In SVM, each feature vector is viewed as a data point in a $p$-dimensional plane and different labels are separated by a $(p-1)$ dimensional hyperplane. Please see the wikipedia entry: https://en....
0
votes
1answer
20 views

Understanding the lower and upper limit of a graph

I was recently presented the following graph: I was given very little context and I have to recreate it. I don't want it to be done for me, but some context would help. Any idea what these upper and ...
1
vote
0answers
31 views

Modeling distributed densities or is it?

Lets say that I am counting pixels over time. Each pixel is either the color red or blue. Let's say that I make up some threshold for the number of red pixels that I count and that when I reach this ...
0
votes
0answers
9 views

Problem about multivaluable distributions. (Hotelling distribution)

From $n_1$ p-dimensional random vectors: $X_1,...,X_{n_1}$ independents with $X_i\sim N_p(\mu_1,\Sigma)$ and $n_2$ random vectors with the same dimension: $Y_1,...,Y_{n_2}$ independents with $Y_i\sim ...
0
votes
1answer
28 views

Quantile-Quantile (QQ) Plots

I understand how to assess whether an exponential/normal distribution is suitable to model a piece of data when the parameters are given, i.e. finding the theoretical quantiles and plotting against ...
0
votes
0answers
17 views

Regression analysis for Measurement values

I'm in a situation where I'm provided with a signal and a couple of measurement values for that signal. Usually the signal will drift a little when the measurement start (i.e. because of heat). I want ...
0
votes
0answers
14 views

Minimum and Maximum Data Points in a Spectrum

If I am collecting data that will be used to assemble a spectrum, how often should I take a data point to properly sample, but not oversample given the following information? I can control my data ...
1
vote
1answer
18 views

Weighting function for a scatter plot of ratio and difference across several orders of magnitude

I am comparing straight line distances to the shortest discovered path. I have millions of points with a pair (straight line distance,...
3
votes
1answer
77 views

The number of nuts in a package of “Premium Cashews” is normally distributed with a mean of 433 and a standard deviation of 6 nuts.

The number of nuts in a package of "Premium Cashews" is normally distributed with a mean of 433 and a standard deviation of 6 nuts. Packages with fewer than 420 nuts or more than 445 nuts will be ...
0
votes
0answers
18 views

Calculating an average of multiple data sets based on priority.

First and foremost I'm no mathematician, it may be a case of over complication. What I'm trying to do is that I have 2 sets of data, I want to calculate one average for both sets but I want the first ...
0
votes
2answers
41 views

Similarity between two lists

I am working with psychology, more precisely, the Big Five personality traits. I have a test which measures 5 variables, that is believed to describe personality traits of human beings. The variables ...
0
votes
0answers
13 views

Help with transforming a value so that it better fits within a recursive filter

I am working on an application where I need to evaluate a metric that determines how 'well distributed' a set of points ($X) are in the 2D plane that is a camera image. By evaluating this metric for ...
2
votes
1answer
44 views

Compare Two Values - Stardew Valley Food

In the game, Stardew Valley, you can pick up items through farming, fishing, foraging, cooking, etc. Most tasks use up your character's energy, such as swinging an ...
4
votes
2answers
126 views

Are there 'serious' mathematical problems emerging from data science?

Are there any proper research-level mathematics problems that have come out of the activity known as "data science"? I put the quotes because it really is not so clear what "data science" is to me; I ...
1
vote
0answers
16 views

Finding Objects Most Similar to Other Objects as a “Linear Combination”

I have a small collection of objects, $\{o_i\}_{i=1}^{20},$ that have several properties (all quantitative, or can be made to be quantitative) that differ among themselves. So $o_1$ has, say, $10$ ...
0
votes
1answer
38 views

What Conclusions Can I Draw from Percentiles About my Set of Data?

I have a specific survey question that I am analyzing for the topic Should High Schools Implement a Uniform Policy?. I am supposed to create biased survey questions ...
0
votes
0answers
25 views

Best Average to Represent Quantitative Data?

For a final assignment, my two group members and I have created a biased survey called Should Public Schools Implement a Uniform Policy? We have about 2-3 ...
0
votes
3answers
55 views

What Can Median Tell Me About my Data?

For a final assignment for math, I was required to take the "Yes!" side of 'Should Uniforms Be Implemented at School?' My job was to create a biased survey that would make people agree with our side ...
0
votes
1answer
23 views

How to fit Poission distribution with large mean?

I have one set of discrete data plotted below. I suspect the data follows Poisson distribution with mean $\mu \simeq 400$ and standard deviation $\sigma \simeq \sqrt{400} = 20$. $$ n(k) = \frac{\mu ^...
0
votes
0answers
20 views

Is there any exact definition for “imbalanced datasets” or “imbalanced learning”?

What is the exact definition for "imbalanced datasets" or "imbalanced learning"? Some article states it as, In such problems, almost all the instances are labeled as one class, while far fewer ...
21
votes
2answers
5k views

Where to start learning about topological data analysis?

I was wondering if anyone could help me out with finding a nice introductory introductory text for topological data analysis (I'm speaking as somebody who has two semesters of experience with topology,...
9
votes
3answers
1k views

Removing noise when the signal is not smooth

Suppose we have (an interval of) a time series of measurements: We assume it can be explained as a "simple" underlying signal overlaid by noise. I'm interested in finding a good algorithm to estimate ...
0
votes
0answers
28 views

Extrapolation error of linear regression lines for a two-cluster data set

I am studying on some linear regression problems using the least-squares method and stumbled upon a problem regarding the error when extrapolating far-away datapoints. About the problem: For a point $...
0
votes
1answer
2k views

Find equation to fit data with plateau

I have a dataset with X and Y values. I would like to fit the data which plateaus at a certain value Y (want to extract the Y where it plateaus). The data includes (0 0) and for increasing x (no ...