Near x=0, xcosx−ln(1+x)=2x2 +O(x3). Therefore, for the quotient to have a finite limit, both the constant and linear terms of the numerator must vanish.
The numerator expands as (c+1)+(a−c+1)x+ [2(a−1)2 − b2+2(c−2)]x2 +O(x3). The constant term gives c=−1; the linear term then gives a=−2.
With these values, the coefficient of x2 in the numerator is 29 − b2 − 23=3− b2. The limit is consequently (1/2)(3− b2)=2, so b2=2.
Thus a2+b2+c2=4+2+1 =7, which is option (2).