syllogism in mathematics Syllogism is defined as:
"All A are B", and "All C are A", thus "All C are B".
For example:
"All birds have features", "Penguins are birds", thus "Penguins have feather"
How do we represent this logic in mathematics? To which mathematical branch it belongs?
 A: I think it's best to think of it as part of propositional or first order logic (both parts of mathematical logic). You are talking about a particular rule of inference called (perhaps unsurprisingly) hypothetical syllogism. It says that if $P$ implies $Q$ and $Q$ implies $R$, then $P$ implies $R$. You have the following premises:
$$\begin{array}{ll}
\forall x(P(x)\rightarrow B(x)) &\text{for all }x,\text{ if }x\text{ is a penguin, then }x\text{ is a bird.}\\
\forall x(B(x)\rightarrow F(x))&\text{for all }x,\text{ if }x\text{ is a bird, then }x\text{ has features.}
\end{array}$$
And from these two premises, the standard rules of propositional first order logic allow you to conclude that
$$\forall x(P(x)\rightarrow F(x))\qquad\text{for all },x\text{ if }x\text{ is a penguing, then }x\text{ has features.}$$
You can also model it easily in naive set theory. Taking 
$$\begin{align*}
\mathcal{B}&=\{ x\mid x\text{ is a bird}\}\\
\mathcal{F}&=\{x \mid x\text{ has features}\}\\
\mathcal{P}&=\{x \mid x\text{ is a penguin}\},
\end{align*}$$
then your premises are that
$$\mathcal{B}\subseteq\mathcal{F}\qquad\text{and}\qquad \mathcal{P}\subseteq \mathcal{B}$$
(everything which is a bird has features; everything which is a penguin is a bird). The properties of set inclusion immediately yield that if these two inclusions hold, then so does
$$\mathcal{P}\subseteq \mathcal{F};$$
i.e., everything whic his a penguing has features.
I should note that mathematical logic would be the area of mathematics that studies these things directly; however, they are used in mathematics all the time, just like modus ponens or de Morgan's laws.
A: As Arturo Magidin says, the basic kinds of formal logic studied today are propositional logic and first-order logic, and these can be used to model various aspects of syllogistic reasoning.
However, the actual study of syllogistic logic (also called traditional logic or term logic, the logic studied by Aristotle) is not very common in mathematics or mathematical logic today. 
One reason for this is that there was a very complex set of conventions for when a statement in this logic is true, relating to which statements imply the existence of objects with the stated properties, and these conventions are not the ones that we use today.
For example, there was a common convention which can be called "universal affirmatives have existential import". In this convention, a statement "All S are P" is only true if (1) there is at least one thing that satisfies S and (2) every thing that satisfies S satisfies P.  In particular, "All S are P" was taken to imply "Some S is P"; these two phrases are called "subalterns" of each other. 
The relationship between the four claims "All S are P", "All S are not P", "Some S is P", and "Some S is not P" was laid out, in traditional logic, in the "square of opposition" (see e.g [1]).
In contemporary mathematics, we do not follow the same conventions, and some of the terminology from the square of opposition is completely lost. For example, no mathematical logic book I have ever seen uses the term "subaltern". And our modern convention is that "All S are P" does not imply the existence of any object satisfying S. Thus we accept that "All S are P" and "All S are not P" are both true if no S exists. In the square of opposition, those quoted statement are taken to be contradictory; if there are no S then "All S are P" is false and "All S are not P" is true. 
Another reason that Aristotle's logic is not directly studied today is that it had no way to handle nested quantifiers. For example, the $\epsilon$-$\delta$ definition of a continuous function involves three levels of quantifiers. We want the logic we study to be able to handle that sort of statement. 
1: http://plato.stanford.edu/entries/square/
A: Your question here concerns a representation of a traditional logic not Aristotelian logic nor propositional calculus nor first-order predicate calculus.  Traditional logic, deals with syllogisms like "All A are B, All C are A, therefore, all C are B."  Aristotle's logic deals with implications like "If all A are B, and all C are A, then all C are B."  As Jan Lukasiewicz writes on p. 21 of his Aristotle's Syllogistic From the Standpoint of Modern Formal Logic 
"The difference between the Aristotelian and the traditional syllogism is fundamental.  The Aristotelian syllogism as an implication is a proposition, and as a proposition must be either true or false.  The traditional syllogism is not a proposition, but a set of propositions which are not unified so as to form one single proposition.  The two premises written usually in two different lines are stated without a conjunction, and the connexion of these loose premisses with the conclusion by means of 'therefore' does not give a new compound proposition.  The famous Cartesian principle, 'Cogito, ergo sum', is not a true principle, because it is not a proposition. It is an inference, or, according to a scholastic terminology, a consequence.  Inferences and consequences, not being propositions, are neither true nor false, as truth and falsity belong only to propositions.  They may be valid or not.  The same has to be said of the traditional syllogism.  Not being a proposition the traditional syllogism is neither true nor false; it can be valid or ivalind.  The traditional syllogism is either an inference, or a rule of inference, when stated in variables."
Now, to represent these sorts of statements, we can't actually do such in a logical way, but rather a metalogical way since rules of inference don't qualify as propositions.  I'll use the single turnstile "|-" to mean "yields".  Let Axy stand for "all x is y."  So, this ""All A are B", and "All C are A", thus "All C are B"" can get represented as:
Aab, Aca |- Acb.
By contrast, if we use -> to represent implication and write in infix notation, the Aristotleian syllogism can get written in a logical way as:
|-(Aab->(Aca->Acb)) 
