I think the total number of edges for a graph with $8$ vertices would be:
$n(n-1)/2$ which would yield $28$.
total number of set with $28$ elements is $2^{28}$.
But I'm not sure how I can limit the number of edges to be maximum of $6$....
HINT: There are indeed $\binom82=28$ possible edges. You simply need to pick $6$ of them. How many ways are there to choose $6$ elements from a set of $28$ elements? (This answers the question in your title. If you want the graphs with at most $6$ edges, you’ll have to count the number of ways pick $0,1,2,3,4$, and $5$ of them, as well, and add the results.)