A linear stability analysis is conducted for the onset of natural convection driven by a concentration gradient in a horizontal layer of a near-critical binary fluid mixture. The problem is regarded as a limiting case of double-diffusive convection. The governing equations for small perturbations after normal-mode expansion are solved numerically with finite difference discretization to obtain the critical concentration Rayleigh number. It is found that, when the height of the fluid layer is small, the initial density stratification is negligible and the theoretical criterion developed under Boussinesq approximation with the modified Rayleigh number is accurate even extremely close to the critical point. However, for a large height, the initial density stratification makes the fluid layer become more unstable, and deviations from theoretical predictions are observed. We further propose a method to estimate these deviations, which can be used to check the applicability of the theoretical criterion. As the second part of the study, we apply the criterion to interpret the onset of convection for a transient problem: a near-critical binary fluid mixture confined in a two-dimensional cavity submitted to concentration increases at the bottom wall. The numerical results demonstrate four typical behaviours of the concentration boundary layer: onset of convection, collapse of the concentration boundary layer, return to stability, and remaining stable. Comparisons between numerical results and the stability criterion are made, where consistencies are found except for the behaviour of return to stability. We attribute the inconsistency to the existence of lateral walls, whose stabilizing effect is strong when the return to stability happens.