We propose a new algorithm for numerical solutions of constrained nonlinear optimal control problems, based on constrained LQ problems. The proposed algorithm is described as follows. First, we approximate the constrained nonlinear optimal control problems by the Taylor expansion technique, resulting in the standard LQ problems, but with linearized constraints. Then, by making use of penalty function methods, we construct the augmented LQ problem, which is one of unconstrained optimal control problems, and therefore we can easily obtain the optimal solution of the augmented LQ problem by Riccati transformation. Finally, repeating the above procedure with a certain type of filter, we ventually obtain the numerical solutions for constrained nonlinear optimal control problems. The effectiveness is demonstrated through simulation.