# Questions tagged [data-analysis]

Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

340 questions
136 views

### Explicit definition of a sequence

Suppose there are 6 sequences $a=(a_n)_{n\geq 0}, b=(b_n)_{n\geq 0},c=(c_n)_{n\geq 0},d=(d_n)_{n\geq 0},e=(e_n)_{n\geq 0},f=(f_n)_{n\geq 0}$, the data can be seen here: Data. I found out by trial and ...
695 views

### Relationship between eigenvectors of correlation and covariance matrices

For the purpose of computing principal components of a dataset, represented as matrix $X$ of dimensions $n \times p$ with $n$ samples and $p$ features, we can compute sample covariance matrix $S$, and ...
75 views

### Are the results from persistent homology complete?

Do the homology groups for each Betti number as produced by persistent homology completely capture all the topological characteristics of the space? If not, what topological features are not captured? ...
2k views

### When to use cumulative moving average vs a simple moving average?

After reviewing the Wikipedia page on moving averages, the difference between the simple moving average and cumulative moving average are clear: 1) Simple moving average only considers the last n ...
74 views

### What do the parameters of a multinomial logistic regression correspond to?

I've recently started learning about data science/statistics and learned how to derive such models as linear regressors and logistic regressors. What I don't understand, however, is what the ...
47 views

19 views

### What is the term for “merging complementing data streams”

I'm merging 2 data streams together. One of them is precise, the other accurate. Excuse if this is obvious, but let me be sure I'm clear: Accurate: Long term average is correct, but otherwise a ...
58 views

### Why is it so hard to find numbers with some palindromic properties?

Lets start by defining palindromes and then looking at some problems; We can define the set $P_d$ as set of all palindromes with $d$ digits. If $p\in P_d$, then $p=(x,y)$, where $x$ is the ...
35 views

### Which statistic test should I use?

For a task on data cleaning and analysis using SPSS I have two research questions for which I should pick an appropriate statistical test. It's been over a few years though since my last statistics ...
34 views

### Geometric reason for PCA eigen vectors being transposed

I was trying to think of the reason behind the PCA eigen vectors being transposed, that is, why $y= U_D^T x$. Geometrically, is it because since we are taking the projection of the point $x$ into a ...
33 views

### Does Beta distribution act more generally than I reckon?

In Bayes theory, Beta distribution is often introduced to make the assumption for parameter p which is from Binomial(N,p). My question is whether Beta distribution can model all probability or ratio ...
416 views

### Spectral relaxation of k-means clustering

I am working on a presentation on Spectral relaxation of k-means clustering (http://papers.nips.cc/paper/1992-spectral-relaxation-for-k-means-clustering.pdf) and I am a bit stuck. I understand ...
24 views

### Generating stastical model from taxi dataset

From the dataset available here of NYC taxi journeys I wish to create some kind of model that will allow me to estimate the probability that a request comes in in a given time interval (on the order ...
123 views

### Need help understanding the proof of the Johnson-Lindenstrauss lemma by Dasgupta and Gupta

The Johnson-Lindenstauss lemma is the following: Let $\varepsilon\in(0,1)$, $\mathcal{P}\subseteq\mathbb{R}^d$ be a finite subset and $k\in\mathcal{O}(\varepsilon^-2\log{n})$. Then there exists a (...
65 views

### Markov-Chain Monte-Carlo: Are transformations on the inputs valid?

The problem: I am trying to solve a high dimensional (up to ~50) class of data fitting & modelling problems. The user specifies the problem, so I would like to make the configuration as easy as ...
26 views

### Methods for Extrapolating Data Close To Observations

I am aware of many different ways to go about interpolating between the values of known data points. However, whenever I come upon (I work in Quantitative Finance) the need to extrapolate data I find ...
56 views

### How can I infer order from partially ordered discrete sequences?

A really interesting problem that I can't stop thinking about! Have run in to this a couple of times but yet to find a smart approach to either solve or frame this problem. This is my try at ...
110 views

### How can I minimize the error from fitting two functions to different parts of data?

I have some data, and I expect that part of the data will fit to a power law, and part of the data to a line. Here's a made up example of some similar data: Where I've made the red part a quadratic ...
105 views

### Predicting profit with price variation

I am currently working on a high school project that aims to predict profit from X amount of items to Y amount of profit based off a deviated sale price. For instance: I sale 10 cookies for 10 ...
483 views

### curve fit but unknown function

I face a strange problem. I have data $[X,Y]$ with no noise. If I plot $Y$ as a function of $X$, it looks like a straight line and the regression coefficient is quite good. For the simplest model ...
307 views

### The correct probability distribution / way to analyze daily changes

I am working on a report which is being sent through to end users that should flag to them any "large changes" in the day-to-day values for the past 30 days for something we would assume the day-to-...
7k views

### Exponent p-value generated in Excel

Excel gave me a p-value of 1.44909E-09 Notice is does not say .09 but 09 This is confusing me, I am trying to analyze my data but am stuck at this point. If it were E-9 it could be 1.44909/...
20 views

### How can I make a best fit line out of data where each point is weighted more heavily than the previous?

I'm thinking that there should be a linear regression technique that allows me to value recent data more that older data. (Maybe this difference in weight could be configurable through a constant). ...
31 views

### Modeling distributed densities or is it?

Lets say that I am counting pixels over time. Each pixel is either the color red or blue. Let's say that I make up some threshold for the number of red pixels that I count and that when I reach this ...
20 views

### Finding Objects Most Similar to Other Objects as a “Linear Combination”

I have a small collection of objects, $\{o_i\}_{i=1}^{20},$ that have several properties (all quantitative, or can be made to be quantitative) that differ among themselves. So $o_1$ has, say, $10$ ...
35 views

### Linear Regression Diagnostics

I am trying to determine if there is a relationship between a dependent variable y and independent variable x by fitting a least squares regression model. Scatterplot of data: Diagnostic plots: ...
36 views

### Linear Interpolation for scattered 3-D data

I have a dataset of scattered 3-D points (non-regular) that carry some variable and am trying to interpolate that variable to a new point. I have currently implemented a couple of methods, but don't ...
64 views

### How to calculate smoothing spline coefficients

I am attempting to calculate smoothing spline coefficients based on the description in Reinsch's 1967 paper, but I'm having some trouble. The first derivative is not continuous. Here are the ...
62 views

### Average percentage overlap between two/more datasets

I am analyzing different schemes of mutual funds. Each scheme has many funds in its portfolio. I wanted to analyse the overlap (of funds) between these schemes. I can find the overlapping of scheme1 ...