Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

learn more… | top users | synonyms

3
votes
0answers
172 views

How does 2D kriging interpolation work?

I have a grid of points Example ...
2
votes
0answers
39 views

How can I infer order from partially ordered discrete sequences?

A really interesting problem that I can't stop thinking about! Have run in to this a couple of times but yet to find a smart approach to either solve or frame this problem. This is my try at ...
2
votes
0answers
43 views

How can I minimize the error from fitting two functions to different parts of data?

I have some data, and I expect that part of the data will fit to a power law, and part of the data to a line. Here's a made up example of some similar data: Where I've made the red part a quadratic ...
2
votes
0answers
54 views

Predicting profit with price variation

I am currently working on a high school project that aims to predict profit from X amount of items to Y amount of profit based off a deviated sale price. For instance: I sale 10 cookies for 10 ...
2
votes
0answers
42 views

Find spikes in data

I have some datasets and I need to find spikes in them. Imagine the data looks like trading data. If the spike is big enough, I need to log it, otherwise, proceed in the analysis. I tried with a ...
2
votes
0answers
202 views

curve fit but unknown function

I face a strange problem. I have data $[X,Y]$ with no noise. If I plot $Y$ as a function of $X$, it looks like a straight line and the regression coefficient is quite good. For the simplest model ...
2
votes
0answers
125 views

The correct probability distribution / way to analyze daily changes

I am working on a report which is being sent through to end users that should flag to them any "large changes" in the day-to-day values for the past 30 days for something we would assume the ...
2
votes
0answers
1k views

Exponent p-value generated in Excel

Excel gave me a p-value of 1.44909E-09 Notice is does not say .09 but 09 This is confusing me, I am trying to analyze my data but am stuck at this point. If it were E-9 it could be ...
1
vote
0answers
12 views

When to use the various mean functions (arithmetic, geometric, quadratic) appropriately?

In data analysis, when are the use of these different means appropriate? For example, I'm using data to infer the peak of waveforms at different points around a semicircle. If my data has intensity ...
1
vote
0answers
32 views

Data analysis: How did people beat the Great Hall game?

This is the game: There is a Great Hall with 102 doors. 100 of these doors lead to one of 100 different side rooms. The 101st door, at the end of the Great Hall leads to the Great Tower, where ...
1
vote
0answers
36 views

How can I interpolate the following data and get the equation?

I want to interpolate between three vectors which have 8 rows in each. For eg. X = [1 2 3 4 5 6 7 8], Y = [7 8 9 10 11 12 13 14 ], Z = [15 16 17 18 19 20 21 22] I want to find the equation Z = ...
1
vote
0answers
20 views

What is the name (if any) of this statistical analysis concept?

I have a list of random numbers. I sort them, and designate each one as belonging to a tier of some sort (quartiles, percentiles, or even arbitrary cutoff points by proportion) based on the ...
1
vote
0answers
26 views

What function should I use to fit this data? Logistic function is not good enough fit.

I'm using curve_fit from numpy to try an fit some data I have. The function I used was the logistic function defined http://en.wikipedia.org/wiki/Logistic_function and the parameters I picked were ...
1
vote
0answers
30 views

Noise Reduction

I'm working on a problem that involves production rate data from oil wells. These rates are recorded per month i.e. after a month of production the produced volume is calculated and reported as rate ...
1
vote
0answers
30 views

Is Principal Component Analysis applicable to this type of situation?

I'm trying to model the response of ant populations to pheromones in this way: The ants are simulated as Self Propelled Particles with internal energy. They undergo acceleration due to their internal ...
1
vote
0answers
28 views

Predictive models

Given a set of temperatures of different cities for a month, which prediction model should I use for a two day look ahead prediction? Regression models or Time series?
1
vote
0answers
94 views

Are there any programs like family echo that I can use to map mathematics?

Family echo is an online program that allows one to make a family tree, if nothing is clicked it shows most of the family tree as it is, but if one clicks a name one can see clearly all the ancestors ...
1
vote
0answers
58 views

How can non-binary data be tested for distinguishability from random?

Given a large string of Base-N digits (in other words, not necessarily binary), is there software or an algorithm that can output whether the data is reasonably pseudorandom or reveals a pattern? In ...
1
vote
0answers
16 views

Statistics linear modelling in R

Suppose I have a date set of the form: Test Subject/Sex/No. of mistakes made in the morning/ No. of mistakes made in the afternoon A / M / 2 / 5 B / F / 1 / 4 C / M / 3 / 5 D / F / 1 / 5 Suppose ...
1
vote
0answers
15 views

Curve 'anti-correlation' detection

If I have a data set, what algorithm can I use to detect when one curve moves in clear opposition to the majority of curves? For example: At point 15, the purple line moves up in opposition to the ...
1
vote
0answers
67 views

Analyzing/Modeling DynamoDB Usage

I'm seeking the correct mathematical tools to analyze/model a real world process for the purpose of optimizing resource allocation. I want to analyze consumed throughput of a AWS DynamoDB table and ...
1
vote
0answers
51 views

Mean value of discrete periodic signals

It is clear that a continuous periodic signal always takes at some point $x_m$ the absolute mean value. Then we could define the absolute mean value of the signal as the value that it takes at $x_m$. ...
1
vote
0answers
18 views

Find two clusters in an ordinated array

As the titles goes, I have an array of sorted real positive numbers, and some information on the structure of the data. I know that there are two constants $a>>b>0$ and two other numbers ...
1
vote
0answers
47 views

$y=\sqrt x \rightarrow \ln y = \ln x$ problem

So I had some raw data, and it looked like there was a square root relation. I plotted graphs of $y=\sqrt{x}$ They look kind of straight. So doing the algebra and taking the ln of both sides, we ...
1
vote
0answers
43 views

Form of a function I can fit like a polynomial that has an asymptote on x-axis and is always positive

So I have a small list of pairs of the form $(\mathbb{R}, \mathbb{R}_{\geq 0})$ and I want to fit a function to this data. Additionally I know that as $x$ grows large in either direction $y$ tends to ...
1
vote
0answers
39 views

Linear model for data that follow gaussian distribution

I have a question about linear regression. We have the linear regression of input data $(X,Y)=((x_1,y_1),(x_2,y_2)...(x_n,y_n))$ is $$F=aX+b$$ a,b are factors of the linear line, $y_i$ is {-1,1}. ...
1
vote
0answers
105 views

Implementation of Total Variation Regularization Algorithm (Lagged Diffusivity Algorithm)

I am trying to compute the derivative of an experimentally-measured quantity as a function of time. The data are fairly noisy, which causes problems. For instance, using finite differences (central ...
1
vote
0answers
29 views

Effect of Event on Another

I am having difficulty wrapping my mind around conceptualizing the impact of one event on another. For the scope of this question, let's define the following: Event A: An assault occurs; Event B: A ...
1
vote
0answers
35 views

Bivariate Correlation Unique solution

I have a set of data, let's say "score vs loss ratio", the score is an arbitrary number I gave to a team by looking at their past history. I want to prove my hypothesis which is higher the score, high ...
1
vote
0answers
23 views

How to compute a “weather” score for repeated trials over time, with a bias for recency

I'm doing some analysis on repeated software tests that can either pass or fail. For each test, I've collected a set of historical (pass/fail, timestamp) records, and I'd like to compute a score to ...
1
vote
0answers
214 views

Kalman filter with sensors having different sampling rate

I would like to implement Kalman filter scheme with two different kind of sensors - say 9DOF IMU (gyro/accelerometer/magnetometer) and GPS sensor (position/velocity/altitude). Problem is, as i ...
1
vote
0answers
44 views

Approximating a probability distribution given multiple datasets

I am beginner in mathematics/statistics and apologise in advance for my faulty use of language. I am working on a problem in statistical genetics. Given a dataset, $D$, I obtained probabilities for ...
1
vote
0answers
19 views

Correct sampling methods for this data set / requirements

I am just looking for a push in the right direction as to what kind of sampling methods I can use to fulfill this set of sampling criteria. I had thought stratified sampling, but I'm not sure if ...
1
vote
0answers
58 views

Why do candlestick plots seem to show a cyclical structure?

One oddity I notice is that if a random or quasi-random data series such as price data is plotted, it is similar to a random walk, but if the same data series is plotted using candlesticks ...
1
vote
0answers
93 views

Significance Tests & Confidence Intervals for Non-normal data

I have 2 nonnormally distributed samples of different sizes (N1~=N2). To evaluate whether there is a significant difference between these samples, I used the Mann_whitney U test (ranksum in MATLAB) ...
1
vote
0answers
24 views

“Sane” quantisation of moderate-dimensional analogue data

Warm-up problem (with solution): Suppose I have a measuring device that spits out a stream of real numbers, which I will model as i.i.d. random variables with range $\mathbb{R}$ and continuous ...
1
vote
0answers
19 views

Isolating clusters in data

This is a problem that I keep encountering in one form or another every so often. Given a large N dimension space with a sufficiently large (M) sample of points. [1] If I were to be asked to find a ...
1
vote
0answers
10 views

grouping of skewed dara

I have a set of data composed of the measurements of spaces. There are 2 distinct sizes represented in this data set. (distinct in that the larger size is larger by some unknown multiple) I am ...
1
vote
0answers
25 views

mutual information for power low distributed data?

I have a dataset with power-low distribution, I would like to measure some kind of correlation/mutual information between the data to its class,(for feature selection task), Can I use ...
1
vote
0answers
337 views

Can you compare data sets with percentages with the Chi Square test?

I’m confused about the Chi Square test.. Apparently you have to use it when comparing frequencies only, but what if both data sets are percentages? e.g. comparing the percentage of smokers in specific ...
1
vote
0answers
41 views

Unit Root Test interpretation

I am analyzing data collected hourly over $9$ years (around $80000$ hours) and I have following result for Unit Root Test : How can I interpret the data? Can I use the Autoregressive model?
1
vote
0answers
116 views

Least squares fit or decomposition of average points (from data)

I have a set of data consisting of the averages of a function $g(r)$ at regular intervals; i.e. my first datapoint is the average of the function from $0$ to $\Delta r$, the second point is the ...
1
vote
0answers
51 views

How to understand the task about bootstrap?

I have the following task. Let $X_1,\ldots,X_n$ be a sample (i.i.d.), $T_n = \overline{X}_n^2=\left(\sum\limits_{i=1}^nX_i\right)^2,$ and $\hat\alpha_k ...
1
vote
0answers
43 views

problem in dimensionality reduction

I am using multidimensional scaling to plot my data in R. However there is a hierarchy in my dataset which i want to exploit and I am using the delaunay triangulation to visualize the plot. So now I ...
1
vote
0answers
267 views

Classification: Why k-Nearest Neighbor method is more appropriate for a Mixture of Gaussians?

I'm reading a book named "The Elements of Statistical Learning" in which it states 2 scenarios when we are trying to predict the class label: Scenario 1: The training data in each class were ...
1
vote
0answers
29 views

Extrapolating parameters for a fit different from the one originally used for regression analysis (no longer having access to data)

Say I have some data, I assume a fit of the form: $\alpha_1 e^{\beta_1 k}$, and I extract some values $\alpha_1>0$ and $\beta_1<0$ for $k=[1,\ldots]$. Say this appears to be a "near perfect" ...
1
vote
0answers
135 views

How can I create a good subset of data proportional to the full data set

I am programming a neural network and I need to use some data to train it. I am following the steps of an experiment which tried to predict direction of stock price index movement using neural ...
1
vote
0answers
37 views

How to find BUE of N? where $X_i$ iid $ P(X_i=j)=1/N$, for $j=1, \cdots, N$

Let $X_1, \cdots, X_n$ be iid with $P(X_i=j)=1/N$, for $j=1, \cdots, N$, where $N$ is an unknown positive integer. I would guess that the order statistics $X_{(n)}$ is the complete sufficient ...
1
vote
0answers
48 views

Fitting multiple functions with shared parameters to data

consider a set of real-valued continuous functions $$f_i(t;a)$$ for $1\leq i\leq n$ with some shared non-negative parameters $a=(a_1,\ldots,a_d)\in\mathbb{R}^d$. Furthermore, assume that there exists ...
1
vote
0answers
93 views

Filter/Removal of periodic 'delta-peaks'

Currently I am measuring data (Counts over Time). Due to measurement problems I have some nasty peaks in this data. These peaks are periodical, very sharp (~3 datapoints over a range of 10000) and ...