"R-squared is only valid for mean equation, whereas GARCH models deal with variance equation. Therefore, you should not worry about its value no matter what the estimation yields. Convergence properties, significance of coefficients, correlogram of residuals and squared residuals, etc. can be used for model diagnostics."