Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

learn more… | top users | synonyms

2
votes
0answers
145 views

curve fit but unknown function

I face a strange problem. I have data $[X,Y]$ with no noise. If I plot $Y$ as a function of $X$, it looks like a straight line and the regression coefficient is quite good. For the simplest model ...
2
votes
0answers
93 views

The correct probability distribution / way to analyze daily changes

I am working on a report which is being sent through to end users that should flag to them any "large changes" in the day-to-day values for the past 30 days for something we would assume the ...
2
votes
0answers
316 views

Exponent p-value generated in Excel

Excel gave me a p-value of 1.44909E-09 Notice is does not say .09 but 09 This is confusing me, I am trying to analyze my data but am stuck at this point. If it were E-9 it could be ...
1
vote
0answers
36 views

Form of a function I can fit like a polynomial that has an asymptote on x-axis and is always positive

So I have a small list of pairs of the form $(\mathbb{R}, \mathbb{R}_{\geq 0})$ and I want to fit a function to this data. Additionally I know that as $x$ grows large in either direction $y$ tends to ...
1
vote
0answers
33 views

Linear model for data that follow gaussian distribution

I have a question about linear regression. We have the linear regression of input data $(X,Y)=((x_1,y_1),(x_2,y_2)...(x_n,y_n))$ is $$F=aX+b$$ a,b are factors of the linear line, $y_i$ is {-1,1}. ...
1
vote
0answers
23 views

Effect of Event on Another

I am having difficulty wrapping my mind around conceptualizing the impact of one event on another. For the scope of this question, let's define the following: Event A: An assault occurs; Event B: A ...
1
vote
0answers
32 views

Bivariate Correlation Unique solution

I have a set of data, let's say "score vs loss ratio", the score is an arbitrary number I gave to a team by looking at their past history. I want to prove my hypothesis which is higher the score, high ...
1
vote
0answers
16 views

How to compute a “weather” score for repeated trials over time, with a bias for recency

I'm doing some analysis on repeated software tests that can either pass or fail. For each test, I've collected a set of historical (pass/fail, timestamp) records, and I'd like to compute a score to ...
1
vote
0answers
36 views

Approximating a probability distribution given multiple datasets

I am beginner in mathematics/statistics and apologise in advance for my faulty use of language. I am working on a problem in statistical genetics. Given a dataset, $D$, I obtained probabilities for ...
1
vote
0answers
17 views

Correct sampling methods for this data set / requirements

I am just looking for a push in the right direction as to what kind of sampling methods I can use to fulfill this set of sampling criteria. I had thought stratified sampling, but I'm not sure if ...
1
vote
0answers
33 views

Data Envelopment Analysis

I have been given an example for a Linear Programming Model in Data Envelopment Analysis. The given data is (the last two columns are the ratios but so far I've only used them in a graphical method); ...
1
vote
0answers
40 views

Why do candlestick plots seem to show a cyclical structure?

One oddity I notice is that if a random or quasi-random data series such as price data is plotted, it is similar to a random walk, but if the same data series is plotted using candlesticks ...
1
vote
0answers
59 views

Significance Tests & Confidence Intervals for Non-normal data

I have 2 nonnormally distributed samples of different sizes (N1~=N2). To evaluate whether there is a significant difference between these samples, I used the Mann_whitney U test (ranksum in MATLAB) ...
1
vote
0answers
22 views

“Sane” quantisation of moderate-dimensional analogue data

Warm-up problem (with solution): Suppose I have a measuring device that spits out a stream of real numbers, which I will model as i.i.d. random variables with range $\mathbb{R}$ and continuous ...
1
vote
0answers
13 views

Isolating clusters in data

This is a problem that I keep encountering in one form or another every so often. Given a large N dimension space with a sufficiently large (M) sample of points. [1] If I were to be asked to find a ...
1
vote
0answers
7 views

grouping of skewed dara

I have a set of data composed of the measurements of spaces. There are 2 distinct sizes represented in this data set. (distinct in that the larger size is larger by some unknown multiple) I am ...
1
vote
0answers
249 views

Can you compare data sets with percentages with the Chi Square test?

I’m confused about the Chi Square test.. Apparently you have to use it when comparing frequencies only, but what if both data sets are percentages? e.g. comparing the percentage of smokers in specific ...
1
vote
0answers
37 views

Unit Root Test interpretation

I am analyzing data collected hourly over $9$ years (around $80000$ hours) and I have following result for Unit Root Test : How can I interpret the data? Can I use the Autoregressive model?
1
vote
0answers
93 views

Least squares fit or decomposition of average points (from data)

I have a set of data consisting of the averages of a function $g(r)$ at regular intervals; i.e. my first datapoint is the average of the function from $0$ to $\Delta r$, the second point is the ...
1
vote
0answers
45 views

How to understand the task about bootstrap?

I have the following task. Let $X_1,\ldots,X_n$ be a sample (i.i.d.), $T_n = \overline{X}_n^2=\left(\sum\limits_{i=1}^nX_i\right)^2,$ and $\hat\alpha_k ...
1
vote
0answers
39 views

problem in dimensionality reduction

I am using multidimensional scaling to plot my data in R. However there is a hierarchy in my dataset which i want to exploit and I am using the delaunay triangulation to visualize the plot. So now I ...
1
vote
0answers
163 views

Classification: Why k-Nearest Neighbor method is more appropriate for a Mixture of Gaussians?

I'm reading a book named "The Elements of Statistical Learning" in which it states 2 scenarios when we are trying to predict the class label: Scenario 1: The training data in each class were ...
1
vote
0answers
27 views

Extrapolating parameters for a fit different from the one originally used for regression analysis (no longer having access to data)

Say I have some data, I assume a fit of the form: $\alpha_1 e^{\beta_1 k}$, and I extract some values $\alpha_1>0$ and $\beta_1<0$ for $k=[1,\ldots]$. Say this appears to be a "near perfect" ...
1
vote
0answers
107 views

How can I create a good subset of data proportional to the full data set

I am programming a neural network and I need to use some data to train it. I am following the steps of an experiment which tried to predict direction of stock price index movement using neural ...
1
vote
0answers
33 views

How to find BUE of N? where $X_i$ iid $ P(X_i=j)=1/N$, for $j=1, \cdots, N$

Let $X_1, \cdots, X_n$ be iid with $P(X_i=j)=1/N$, for $j=1, \cdots, N$, where $N$ is an unknown positive integer. I would guess that the order statistics $X_{(n)}$ is the complete sufficient ...
1
vote
0answers
286 views

How should I calculate “fill rate”? People inquire about an item and then buy it days/months later?

So I have posts(people posting a want for an item) and then I have fills(the person actually getting the item) There are hundreds of posts per day but the fill for that item can be days or weeks away ...
1
vote
0answers
99 views

Jacobian approximation at given point without explicit derivatives expression

after solving a NLproblem with optimization method, I would like to compute confidence intervals, prediction bounds and standard deviation for these optimal parameters. Explicit formulas I have read ...
1
vote
0answers
33 views

How to symmetrically adjust data by removing certain value

Say I have a data set in the following format: {a:200, b: 100; c:400} The sum of a, b and c equals 700. Now, I have another value which is supposed to be the actual total say 500. How do I adjust ...
0
votes
0answers
14 views

Estimation of real estate

I'm working on a project on estimating the price value of real estate. First I have collected a lot of data (500 000 instances) with details such as postal code, number of bedrooms, build year, ...
0
votes
0answers
27 views

How do you use R to find the box counting dimension of a two dimensional set of data, or scatter plot?

I'm using the software R to do some analysis on some data sets for a graduate project. R has a package called "fractaldim", in this package is a function for finding the box counting dimension. The ...
0
votes
0answers
18 views

What is correlation kernel and compare with gaussian kernel

I read a paper that said about correlation kernel that defined: $$W(x-y)=(α/1+d(|y − x|))$$ where $α =  (\int(1+d(y − x)dy)^{-1}$, $(d(|y − x|))$ is spatial Euclidean distance from the central ...
0
votes
0answers
14 views

Get a set of points from Fourier plot on Matlab?

I have a program that collects raw data and also does a Fourier transform to get complex numbers. When I feed the raw data into Matlab to do its own FFT, I can get a nice plot of power on frequency. ...
0
votes
0answers
8 views

Is there any common formula to estimate the salience of peaks of a histogram?

I need to use some quantitative measure to estimate the salience of peaks and the flatness of valleys of a histogram which represents the distribution of lightness of an image. For example, if an ...
0
votes
0answers
8 views

One-class Support Vector Machine Sensitivity Drops when the number of training sample increase

I am using One-Class SVM for outlier detections. It appears that as the number of training samples increases, the sensitivity TP/(TP+FN) of One-Class SVM detection result drops, and classification ...
0
votes
0answers
40 views

How to interpret the indicator function?

I am reviewing a paper titled " Bayesian Sampling Approach to Decision Fusion" by Biao Chen and Pramod K Varshney. This paper uses an indicator function that I am not being able understand. The ...
0
votes
0answers
29 views

What type of equation matches these data points?

Can anyone tell me what equation will match these data points? 377, 384, 1113, 1120, 19861, 19868 Wolfram Plot
0
votes
0answers
20 views

Betfair Odds Percentage Movements & Hedging

I want to determine if the odds on Betfair have decreased by a certain percentage and then calculate the hedged profit when hedging on that percentage, but it's made tricky by the fact that decimal ...
0
votes
0answers
13 views

Implementation of Total Variation Regularization Algorithm (Lagged Diffusivity Algorithm)

I am trying to compute the derivative of an experimentally-measured quantity as a function of time. The data are fairly noisy, which causes problems. For instance, using finite differences (central ...
0
votes
0answers
28 views

On the truncation of spherical harmonics

Suppose there is a function $f(\theta,\phi)$ defined on the surface of a sphere, and $\theta$ and $\phi$ are the polar and azimuthal angles respectively. Similar with the fact that a function defined ...
0
votes
0answers
17 views

How to evaluate the difference between two classes of data which are highly overlapped

I’m trying to implement a classifier based on a dataset comprising two classes of high dimensional time-series observations (the values of the two classes of observations are highly similar). I ...
0
votes
0answers
23 views

What method should I use for this optimization / feature selection project

I'm going to describe a problem and I'm not sure how to best solve it. I will describe the situation. When answering please recommend a method and maybe a software library. I'm using Python for my ...
0
votes
0answers
21 views

Data normalization methods for RNA-seq

Not sure if this is the best place to ask such a question, please re-direct me if there is a more appropriate site. I am using bioinformatics software to look at gene expression levels in populations ...
0
votes
0answers
13 views

Well-behaved interpolation

I have a set of time-dependent scalar data. I would like to remove certain time intervals and fit the remaining data "well-behaved", which I cannot quite exactly define. The intervals may also be at ...
0
votes
0answers
11 views

Best multivariate approximation scheme for scattered data approximation

what is the best multivariate approximation scheme for scattered data approximation on a set E of points in $R^d$?
0
votes
0answers
30 views

Find spikes in data

I have some datasets and I need to find spikes in them. Imagine the data looks like trading data. If the spike is big enough, I need to log it, otherwise, proceed in the analysis. I tried with a ...
0
votes
0answers
11 views

Is Expectation Propagation (EP) affected by the prior?

I understand EP by reading Minka's thesis: http://research.microsoft.com/en-us/um/people/minka/papers/ep/minka-ep-uai.pdf I'm trying to apply it to solve a Bayesian inference problem. However, I'm ...
0
votes
0answers
30 views

Simple maths for an algorithm to searching a table to find out if one type of entry has a causal effect upon another type?

I asked this over on StackExchange and was sent over here... I have a database table which is recording two different types of events which seem to be related to one another. It records the date and ...
0
votes
0answers
25 views

Distance measures for binary data

I was wondering what are some good distance measures for binary data that have the following properties. I know that there are measures like the Jaccard index and the Dice Index, but they don't ...
0
votes
0answers
17 views

Fit Conditional Probability Given Continuous Response

Given a sample of data, all the techniques aimed to find conditional probabilities of the response variable require that this is dichotomous or at least categorical (see multinomial logistic ...
0
votes
0answers
119 views

Kalman filter with sensors having different sampling rate

I would like to implement Kalman filter scheme with two different kind of sensors - say 9DOF IMU (gyro/accelerometer/magnetometer) and GPS sensor (position/velocity/altitude). Problem is, as i ...