if the support of a bivariate random variable is not rectangular, we immediately know that the two r.vs are not independent. But if the support is rectangular, then we can't say for sure right? Is the only way then to check these 2 formulae$$ f_{X|Y}(x|y)= f_X(x)$$
$$ f_{(X,Y)}(x,y)=f_X(x)f_Y(y) $$
Yes. Continuous random variables are independent if and only if their joint p. d. function equals the product of their marginal p. d. functions, for all points in the joint support. [Which is equivalent to: the conditional p.d.f. always equalling the marginal p.d,f. , as you have.]
That is basically the definition, after all.
You are correct.
Consider the joint density $f_{X,Y}(x,y)=3(x-y)^2+\frac12$ for $0<x<1,0<y<1$ so supported on the unit square. $X$ and $Y$ are not independent, as given $X=x$ you have $Y$ more likely to be far away from $x$ than close to $x$.
It would of course fail your tests: for example $f_X(x)f_Y(y)=(3x^2-3x+\frac32)(3y^2-3y+\frac32)$, which is not the same as the joint density
Is the only way then to check these two formulae
No, you do not need to check the formulae to test independence.
Let $X$ and $Y$ be two random variables with a joint probability density function $f(x,y)$. Then, X and Y are independent if and only if:
$f(x,y) > 0$ on a rectangular region (i.e. its support is rectangular.)
$f(x,y) = g(x)h(y)$ for some functions $g(x)$ and $h(y)$.
