As mentioned in the comment, we need five points to determine a conic.
Since a point together with a direction is equivalent to two points,
two points plus two directions is not enough to uniquely determine a conic.
we need at least one more point or some other condition.
I can imagine in some application, one will prefer to find an ellipse as
"round" as possible. In following paragraphs, I'll outline how to find the
one with minimal eccentricity.
Let $P_1$ and $P_2$ be the two points we want the ellipse to pass through. Let $\ell_1$ be the line we want the ellipse to have as a tangent line at $P_1$. Define $\ell_2$ in a similar manner. We will assume $\ell_1 \not\parallel \ell_2$. As a result, $\ell_1$ and $\ell_2$ intersect at some unique point $O$. We will choose the coordinate system such that $O$ is the origin.
Let $\vec{p}_1$ and $\vec{p}_2$ be the vectors $\overrightarrow{OP}_1$ and $\overrightarrow{OP}_2$. Since $\ell_1 \not\parallel \ell_2$, $\vec{p}_1$ and $\vec{p}_2$ form a basis of $\mathbb{R}^2$. For any point $X$ in the plane, we can represent the
corresponding vector $\vec{x} = (x,y) \stackrel{def}{=} \overrightarrow{OX}$ as
$$\vec{x} = u \vec{p}_1 + v\vec{p}_2$$
As mentioned in the comment, there are infinitely many ellipses passing through $P_1$ and $P_2$ having $\ell_1$ and $\ell_2$ as tangent lines. In terms of $u, v$, this family of
ellipses all have the form
$$(u-1)^2 + (v-1)^2 + 2\alpha u v = 1\quad\text{ with }\quad |\alpha| < 1\tag{*1}$$
To convert this to an equation in $\vec{x}$ and hence $(x,y)$, let $\vec{q}_1, \vec{q}_2$ be the dual basis for the basis $\vec{p}_1$, $\vec{p}_2$ of $\mathbb{R}^2$. More precisely,
$\vec{q}_1$ and $\vec{q}_2$ are the two vectors defined by the relation:
$$\vec{p}_1 \cdot \vec{q}_1 = \vec{p}_2 \cdot \vec{q}_2 = 1
\quad\text{ and }\quad
\vec{p}_1 \cdot \vec{q}_2 = \vec{p}_2 \cdot \vec{q}_1 = 0
$$
It is easy to see
$$\vec{x} = u \vec{p}_1 + v \vec{q}_1 \quad\iff\quad
\begin{cases}u = \vec{q}_1\cdot \vec{x},\\ v = \vec{q}_2\cdot\vec{x}\end{cases}$$
and $(*1)$ can be rewritten as
$$(\vec{q}_1\cdot \vec{x} - 1)^2
+ (\vec{q}_2\cdot \vec{x} - 1)^2
+ 2\alpha(\vec{q}_1\cdot \vec{x})(\vec{q}_2\cdot \vec{x}) = 1\tag{*2}$$
The LHS is a quadratic polynomial in $x$ and $y$. It is well known the center of
corresponding ellipse, $\vec{x}_c$, coincides with the critical point of this quadratic polynomial.
Taking gradient of LHS with respect to $\vec{x}$, we get
$$
\vec{q}_1 ( \vec{q}_1 \cdot \vec{x}_c + \alpha \vec{q}_2\cdot \vec{x}_c - 1 ) +
\vec{q}_2 ( \vec{q}_2 \cdot \vec{x}_c + \alpha \vec{q}_1\cdot \vec{x}_c - 1 ) = \vec{0}$$
As a result,
$$
\vec{q}_1 \cdot \vec{x}_c = \vec{q}_2 \cdot \vec{x}_c = \frac{1}{1+\alpha}
\quad\implies\quad
\vec{x}_c = \frac{1}{1+\alpha}(\vec{p}_1 + \vec{p}_2)
$$
Let $M$ be the mid-point of $P_1P_2$. Independent of $\alpha$, the center $\vec{x}_c$ always lies on the line joining $O$ and $M$.
Let's see how the ellipses look like in various limits.
If $\alpha \sim -1$, the center $\vec{x}_c$ will be far away.
The corresponding ellipse will be large and long with semi-major axis roughly in the direction of $\vec{p}_1 + \vec{p}_2$. The eccentricity of the ellipse $e$ will be very high. If we increase $\alpha$ away from $-1$, the eccentricity start to drop.
If $\alpha \sim 1$, the center $\vec{x}_c$ will be close to the mid-point $M$ of $P_1P_2$. The corresponding ellipse will be short and looks like a line segment between $P_1$ and $P_2$.
Once gain, the eccentricity will be very high. As we decrease $\alpha$ away from $1$, the eccentricity drop again.
If $\alpha = 0$, $\vec{x}_c = \vec{p}_1+\vec{p}_2$. The center now coincides to the point $N$ where $OP_1NP_2$ forms a parallelogram. The eccentricity of this ellipse is usually
moderate.
Based on this observation, the decision to use a particular $\alpha$ is equivalent to picking a point on a ray starting at $M$ pointing in the direction of $N$ and use
that point as the center $\vec{x}_c$ of the ellipse.
Among all these ellipses, there is one who eccentricity $e$ is minimized.
To determine the corresponding $\alpha$, let us first
rewrite $(*2)$ to a from symmetric with respect to $\vec{x}_c$.
$$(\vec{q}_1\cdot ( \vec{x} - \vec{x}_c) )^2
+ (\vec{q}_2\cdot ( \vec{x} - \vec{x}_c)^2
+ 2\alpha(\vec{q}_1\cdot (\vec{x} - \vec{x}_c) )(\vec{q}_2\cdot ( \vec{x} - \vec{x}_c) ) = \frac{1-\alpha}{1+\alpha}$$
We will make a further assumption that
$$\begin{cases}\vec{p}_1 = (1,0),\\ \vec{p}_2 = (r,s)\end{cases}
\quad\iff\quad
\begin{cases}\vec{q}_1 = (1, -\frac{r}{s}),\\ \vec{q}_2 = (1, \frac{1}{s})\end{cases}
$$
and $s > 0$. If we let $\vec{x} - \vec{x}_c = (\tilde{x}, \tilde{y})$, we get
$$\left(\tilde{x} - \frac{r}{s} \tilde{y}\right)^2 + \left(\frac{\tilde{y}}{s}\right)^2 + 2\alpha
\left(\tilde{x} - \frac{r}{s} \tilde{y}\right)\left(\frac{\tilde{y}}{s}\right)
= \frac{1-\alpha}{1+\alpha}
$$
We can recast this into a matrix form
$$\begin{bmatrix}\tilde{x}\\ \tilde{y}\end{bmatrix}^T
\begin{bmatrix}
1 & \frac{\alpha-r}{s}\\
\frac{\alpha-r}{s} & \frac{r^2-2\alpha r+1}{s^2}\end{bmatrix}
\begin{bmatrix}\tilde{x} \\ \tilde{y}\end{bmatrix}
= \frac{1-\alpha}{1+\alpha}
$$
Let $\Delta$ be the $2\times 2$ matrix in above expression.
Its characteristic polynomial has the form
$$\det(\lambda I_2 - \Delta ) =
\lambda^2 - (1 + \frac{r^2-2\alpha r+1}{s^2}) \lambda + \frac{1-\alpha^2}{s^2}
$$
Let $\lambda_1 > \lambda_2$ be the two eigenvalues of $\Delta$.
Let $a$ and $b$ be the semi-major axis and semi-minor axis for the ellipse.
It is easy to see
$$
\lambda_1 b^2 = \lambda_2 a^2 = \frac{1-\alpha}{1+\alpha}
\quad\implies\quad
a^2 : b^2 = \lambda_1 : \lambda_2
\quad\iff\quad
e^2 = \frac{a^2 - b^2}{a^2} = 1 - \frac{\lambda_2}{\lambda_1}
$$
To minimize $e$, we need to maximize the ratio $\displaystyle\;\frac{\lambda_2}{\lambda_1}\;$. This is equivalent to maximizing the expression
$$\frac{\lambda_1\lambda_2}{(\lambda_1+\lambda_2)^2} = \frac{\det(\Delta)}{\text{Tr}(\Delta)^2}
= \frac{\frac{1-\alpha^2}{s^2}}{\left( 1 + \frac{r^2-2\alpha r+1}{s^2}\right)^2}
$$
Taking logarithm and differentiate, this condition for maximizing above expression
becomes
$$\frac{-2\alpha}{1-\alpha^2} = 2\left(\frac{-2r}{r^2 - 2\alpha r + 1 + s^2}\right)
\quad\implies\quad
\alpha_{min} = \frac{2r}{r^2+s^2+1}
$$
We can rewrite the last formula only using vector operations
$$
\bbox[8pt,border:1px solid blue;]{
\alpha_{min} = \frac{2\vec{p}_1\cdot\vec{p}_2}{|\vec{p}_1|^2 + |\vec{p}_2|^2}
}
$$
It is now in a form independent of choice of coordinates and remain valid even when the points $P_1$ and $P_2$ live in $\mathbb{R}^3$.
As an example, consider the special case $\ell_1 \perp \ell_2$. We have
$$\vec{p}_1 \cdot \vec{p}_2 = 0
\quad\implies\quad
\alpha_{min} = 0
\quad\implies\quad
\vec{x}_c = \vec{p}_1 + \vec{p}_2.$$
The center of the minimal ellipse is the point $N$ we encountered before.
Up to ordering, the semi-major and semi-minor axis of the ellipse is parallel to $\ell_1$ and $\ell_2$.
