Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-analysis]

Questions on the use of mathematical techniques to extract properties from given data. Consider if your question might be more suited for Cross Validated (stats.SE) instead.

6
votes
0answers
136 views

Explicit definition of a sequence

Suppose there are 6 sequences $a=(a_n)_{n\geq 0}, b=(b_n)_{n\geq 0},c=(c_n)_{n\geq 0},d=(d_n)_{n\geq 0},e=(e_n)_{n\geq 0},f=(f_n)_{n\geq 0}$, the data can be seen here: Data. I found out by trial and ...
6
votes
0answers
695 views

Relationship between eigenvectors of correlation and covariance matrices

For the purpose of computing principal components of a dataset, represented as matrix $X$ of dimensions $n \times p$ with $n$ samples and $p$ features, we can compute sample covariance matrix $S$, and ...
4
votes
0answers
75 views

Are the results from persistent homology complete?

Do the homology groups for each Betti number as produced by persistent homology completely capture all the topological characteristics of the space? If not, what topological features are not captured? ...
4
votes
0answers
2k views

When to use cumulative moving average vs a simple moving average?

After reviewing the Wikipedia page on moving averages, the difference between the simple moving average and cumulative moving average are clear: 1) Simple moving average only considers the last n ...
4
votes
0answers
74 views

What do the parameters of a multinomial logistic regression correspond to?

I've recently started learning about data science/statistics and learned how to derive such models as linear regressors and logistic regressors. What I don't understand, however, is what the ...
3
votes
0answers
47 views

Solving recurrences using substitution method

I have given $$ T(n)= \begin{cases} T(n/3)+T(2n/3)+n,\quad &n>1 \\ 1, \quad &n=1 \end{cases} $$ I tried it again and again but couldn't think beyond, $$ T(n)=T(n/27)+3T(...
3
votes
0answers
372 views

Getting a feel for the Normal-Inverse-Wishart conjugate prior to multivariate normal distribution

I am trying to get a feel for the Normal-Inverse-Wishart conjugate prior, which I have started to use, sparingly, in my work, where I am trying to cluster multivariate normal data. As Wikipedia ...
3
votes
0answers
798 views

How does 2D kriging interpolation work?

I have a grid of points Example ...
3
votes
0answers
147 views

Find spikes in data

I have some datasets and I need to find spikes in them. Imagine the data looks like trading data. If the spike is big enough, I need to log it, otherwise, proceed in the analysis. I tried with a ...
2
votes
0answers
26 views

Is it appropriate to use clustering to partition the dependent variable into separate datasets for a home price prediction model?

I'm struggling to decide how to deal with a heteroskedasticity problem in a home price prediction model I'm developing. The training set residuals are normally distributed around zero, but they have ...
2
votes
0answers
93 views

Compare euclidean distances from different dimensions

EDIT: I'd like to reformulate my primary question. I have a set of points in $\mathbb{R}^D$ and I reduce the dimensions of the points to some $\mathbb{R}^L$. I do this with multiple configurations ...
2
votes
0answers
48 views

Interpreting the constant when performing a fixed effect panel data regression in Stata

Dear Stackexchange community, I am running a panel data regression on 20 years of monthly historical excess returns of the stocks in the S&P 500 at 31/12/2017. I like to test the effectiveness of ...
2
votes
0answers
22 views

Data analysis over geometric distribution sample with R (statistics pack)

Problem : The following observations are the number of tries $21$ football players had to make until they succeeded in scoring a penalty kick. Consider that all players have equal skills. $$3 \...
2
votes
0answers
19 views

What is the term for “merging complementing data streams”

I'm merging 2 data streams together. One of them is precise, the other accurate. Excuse if this is obvious, but let me be sure I'm clear: Accurate: Long term average is correct, but otherwise a ...
2
votes
0answers
58 views

Why is it so hard to find numbers with some palindromic properties?

Lets start by defining palindromes and then looking at some problems; We can define the set $P_d$ as set of all palindromes with $d$ digits. If $p\in P_d$, then $p=(x,y)$, where $x$ is the ...
2
votes
0answers
35 views

Which statistic test should I use?

For a task on data cleaning and analysis using SPSS I have two research questions for which I should pick an appropriate statistical test. It's been over a few years though since my last statistics ...
2
votes
0answers
34 views

Geometric reason for PCA eigen vectors being transposed

I was trying to think of the reason behind the PCA eigen vectors being transposed, that is, why $y= U_D^T x$. Geometrically, is it because since we are taking the projection of the point $x$ into a ...
2
votes
0answers
33 views

Does Beta distribution act more generally than I reckon?

In Bayes theory, Beta distribution is often introduced to make the assumption for parameter p which is from Binomial(N,p). My question is whether Beta distribution can model all probability or ratio ...
2
votes
0answers
416 views

Spectral relaxation of k-means clustering

I am working on a presentation on Spectral relaxation of k-means clustering (http://papers.nips.cc/paper/1992-spectral-relaxation-for-k-means-clustering.pdf) and I am a bit stuck. I understand ...
2
votes
0answers
24 views

Generating stastical model from taxi dataset

From the dataset available here of NYC taxi journeys I wish to create some kind of model that will allow me to estimate the probability that a request comes in in a given time interval (on the order ...
2
votes
0answers
123 views

Need help understanding the proof of the Johnson-Lindenstrauss lemma by Dasgupta and Gupta

The Johnson-Lindenstauss lemma is the following: Let $\varepsilon\in(0,1)$, $\mathcal{P}\subseteq\mathbb{R}^d$ be a finite subset and $k\in\mathcal{O}(\varepsilon^-2\log{n})$. Then there exists a (...
2
votes
0answers
65 views

Markov-Chain Monte-Carlo: Are transformations on the inputs valid?

The problem: I am trying to solve a high dimensional (up to ~50) class of data fitting & modelling problems. The user specifies the problem, so I would like to make the configuration as easy as ...
2
votes
0answers
26 views

Methods for Extrapolating Data Close To Observations

I am aware of many different ways to go about interpolating between the values of known data points. However, whenever I come upon (I work in Quantitative Finance) the need to extrapolate data I find ...
2
votes
0answers
56 views

How can I infer order from partially ordered discrete sequences?

A really interesting problem that I can't stop thinking about! Have run in to this a couple of times but yet to find a smart approach to either solve or frame this problem. This is my try at ...
2
votes
0answers
110 views

How can I minimize the error from fitting two functions to different parts of data?

I have some data, and I expect that part of the data will fit to a power law, and part of the data to a line. Here's a made up example of some similar data: Where I've made the red part a quadratic ...
2
votes
0answers
105 views

Predicting profit with price variation

I am currently working on a high school project that aims to predict profit from X amount of items to Y amount of profit based off a deviated sale price. For instance: I sale 10 cookies for 10 ...
2
votes
0answers
483 views

curve fit but unknown function

I face a strange problem. I have data $[X,Y]$ with no noise. If I plot $Y$ as a function of $X$, it looks like a straight line and the regression coefficient is quite good. For the simplest model $$...
2
votes
0answers
307 views

The correct probability distribution / way to analyze daily changes

I am working on a report which is being sent through to end users that should flag to them any "large changes" in the day-to-day values for the past 30 days for something we would assume the day-to-...
2
votes
0answers
7k views

Exponent p-value generated in Excel

Excel gave me a p-value of 1.44909E-09 Notice is does not say .09 but 09 This is confusing me, I am trying to analyze my data but am stuck at this point. If it were E-9 it could be 1.44909/...
1
vote
0answers
20 views

How can I make a best fit line out of data where each point is weighted more heavily than the previous?

I'm thinking that there should be a linear regression technique that allows me to value recent data more that older data. (Maybe this difference in weight could be configurable through a constant). ...
1
vote
0answers
31 views

Modeling distributed densities or is it?

Lets say that I am counting pixels over time. Each pixel is either the color red or blue. Let's say that I make up some threshold for the number of red pixels that I count and that when I reach this ...
1
vote
0answers
20 views

Finding Objects Most Similar to Other Objects as a “Linear Combination”

I have a small collection of objects, $\{o_i\}_{i=1}^{20},$ that have several properties (all quantitative, or can be made to be quantitative) that differ among themselves. So $o_1$ has, say, $10$ ...
1
vote
0answers
35 views

Linear Regression Diagnostics

I am trying to determine if there is a relationship between a dependent variable y and independent variable x by fitting a least squares regression model. Scatterplot of data: Diagnostic plots: ...
1
vote
0answers
36 views

Linear Interpolation for scattered 3-D data

I have a dataset of scattered 3-D points (non-regular) that carry some variable and am trying to interpolate that variable to a new point. I have currently implemented a couple of methods, but don't ...
1
vote
0answers
64 views

How to calculate smoothing spline coefficients

I am attempting to calculate smoothing spline coefficients based on the description in Reinsch's 1967 paper, but I'm having some trouble. The first derivative is not continuous. Here are the ...
1
vote
0answers
62 views

Average percentage overlap between two/more datasets

I am analyzing different schemes of mutual funds. Each scheme has many funds in its portfolio. I wanted to analyse the overlap (of funds) between these schemes. I can find the overlapping of scheme1 ...
1
vote
0answers
38 views

Matlab and integrating under area of Gaussian curve for lab experiment

This is going to be a long post due to information that I am going to have to present. A brief outline in what I am asking is, that I need to integrate a Gaussian at a specific height above the x-...
1
vote
0answers
109 views

Intersection of a Line and Surface Point Cloud

I am trying to determine where a given straight line intersects a small cluster of points that define an arbitrary surface in 3-D space. My approach has been to create a surface defined as $z = a + ...
1
vote
0answers
27 views

How to find the point estimate for a question with given mean and s.d.

I'm trying to go through a question and I am rather confused because after I compare the question to lecture slides, it seems to me that the answer has already been given but I assume that isn't the ...
1
vote
0answers
71 views

How to normalize noisy data?

I'm a computer science student and for a project I'm taking sensor data and normalizing it into a 0 - 255 range. Problem is when I get the data I don't know the minimum and maximum (it varies very ...
1
vote
0answers
39 views

Data fitting/simulating by mathematical techniques

I'm recently dealing a set of market data where it had the following properties: it does not blow up but has unique signature of removable singularities/discounity, the one that usually appear in ...
1
vote
0answers
19 views

Matching Metric - How to Normalize for different Amounts in Equation

I'm a non-mathematician. I'm trying to find if tweets are on topic with news articles algorithmically. Part of this involves taking each word from every tweet and seeing if it's in the news articles ...
1
vote
0answers
22 views

Post-hoc transformation a weighted average

For a data project, I calculated a weighted average of $r$ based on the inverse weighting of $d$, a vector where all $d_i > 1$. I am interested in figuring out if it is possible to transform the ...
1
vote
0answers
19 views

Percentage Scaling when using data in excel

I have data in percentage, and the smallest value is 18.96% and largest is 95.85. What I'm trying to do is scale the data in a way that makes 18.96% to be 40% and 95.85% to be 100%.
1
vote
0answers
43 views

How to assess the fit of a horizontal trend line?

I have two trend-lines, both with a slight negative slope. The first has a greater slope, but significant scatter of the data points, while the second is nearly horizontal with an obvious better fit. ...
1
vote
0answers
42 views

Talk about continuity of discrete data

It is not difficult to say the continuity of the simple function. It is also not difficult to say the continuity of the geometry. For example, we can say the following function is $C^0$ continuous. ...
1
vote
0answers
27 views

What conditions of normality must be met for paired/unpaired t-tests?

We assume that $X_i \sim N(μ_1, σ_1^2)$ and $Y_i \sim N(μ_2, σ_2^2)$ [i.e. that they are normal distributions with a finite (or known?) mean and variance] [...] -- Tamhane & Dunlop, ...
1
vote
0answers
35 views

If I divide one column of a matrix by another, do I get new “information”?

Sorry in advance for my poor math terminology. Does dividing one column of a data-frame by another column give new "information" or help with classification? I am collecting data on treadmill ...
1
vote
0answers
28 views

When extrapolating for projections, how do you know which function-form to use?

When you have a set of raw data, there are many models to extrapolate a projection. Linear, Quadratic, Logarithmic, Etc. How does one know which to use? I'm assuming its by the nature of the data ...
1
vote
0answers
39 views

How to measure distribution of high-dimensional data

I have to methods of projecting random samples in $R^n$ onto a manifold defined by $C(q)=0$, which is a lower-dimensional subset. Now, samples in $R^n$ are uniformly distributed. However, when ...