Does convex optimization belong to linear programming or nonlinear programming?
Is convex optimization an undergraduate topic or a graduate topic?
Convex optimization involves minimizing a convex objective function (or maximizing a concave objective function) over a convex set of constraints.
Linear programming is a special case of convex optimization where the objective function is linear and the constraints consist of linear equalities and inequalities.
Nonlinear programming concerns optimization where at least one of the objective function and constraints is nonlinear.
(Adapted from Mathematical optimization: Major subfields on Wikipedia.)
Therefore, convex optimization overlaps both linear and nonlinear programming, being a proper superset of the former and a proper subset of the latter. However, note that nonlinear programming, while technically including convex optimization (and excluding linear programming), can be used to refer to situations where the problem is not known to be convex (see Boyd and Vandenberghe, p. 9, below). Hence, it may be more useful in practice to think of a hierarchy: linear - convex - nonlinear. Another useful view is given by the following quote, kindly supplied by littleO: "The great watershed in optimization isn't between linearity and nonlinearity, but convexity and nonconvexity." -- R. Tyrrell Rockafellar, in SIAM Review, 1993
Whether convex optimization is an undergraduate topic or a graduate topic is not easily answered: it could be either advanced undergraduate or graduate, depending on the institution and department. For instance, Convex Optimization by Boyd and Vandenberghe has been used for a graduate course, but according to the authors, the required background is linear algebra and advanced calculus. Thus, it would appear that it could be used at the advanced undergraduate level with the proviso, mentioned in the preface, that knowledge of mathematical analysis and probability is beneficial to follow some of the details.