For classification problems, we may use cross entropy as our loss function, while for regression problems, we may choose mean square error.
However, here I'd like to find a proper loss function for my regression problem with many outliers. Since these outerliers may greatly affect mean square error, MSE is not proper for this problem.
I did some search and found the huber loss, but it seems outerliers still affect the final loss, though alleviated.
I'd like to find some loss function, with give target $\hat{y}$, and estimated $y$, the loss may be calculated like the follows
$$ Loss(y, \hat y) = \begin{cases} \frac{1}{\theta}|y-\hat y| & (|y-\hat y| \le \theta )\\ 1& (|y-\hat y| > \theta) \end{cases} $$
Here $\theta$ is a known parameter. For example we may plot the loss as follows (with $\theta = 0.6$).
And do the optimization like
$$ \min \sum_{i=1}^N Loss(f(x_i), \hat y_i) $$
Here $f$ may be some certain model, for example linear model or some more complex nonlinear models (like the nn etc)
However, when directly using this loss, there are several problems
- First, it'not convex, but this may not be a big problem, since some local optimum (if is's close to the global optimum if possible) may still be ok.
- Second, a large area of its gradient is zero, which means, it's easily to "get stuck", and cannot optimize any more.
So I'm looking for some approximate loss, which looks like the expected one I posted, but can be optimized easier.
Thank you for any suggestions.