Why we have mild hemolytic reaction in ABO incompatibility
Both Rh and A and B antigens are proteins. They are antigens capable of inducing immune response.
The issue is not in the chemical structure of these antigens.
The reason why ABO incompatibility is milder in comparison to Rh incompatibility is this;
The Rh negative mother will produce IgG antibodies when sensitized by an Rh positive fetus. IgG molecules can cross the placenta and cause hemolysis in the baby.
While in the case of ABO incompatibility the antibodies produced by the mother are mainly IgM in type and so they don't cross the placenta. The exception to this rule is in 1% of mothers who are O antigen blood type and they have an A or B blood group fetus. In this subset of mothers they do produce IgG and so hemolytic disease of the newborn may develop.
However, in this situation the ABO incompatibility is still milder than Rh incompatibility because it's thought that A and B antigens are widely spread throughout the tissues and they'll quickly neutralize the transplacentally transfused maternal antibodies and thereby no enough time to bind fetal RBCs.
In addition, fetal RBCs appear to have less surface expression of A or B antigen, resulting in few reactive sites. This results in hyperbilirubinemia as a predominant manifestation of incompatibility (rather than anemia), and peripheral blood film frequently reveals a large number of spherocytes and few erythroblasts, unlike what is seen in Rh incompatibility (erythroblastosis fetalis), in which blood film reveals a large number of nucleated RBCs and few spherocytes.