Reliability designers often try to achieve a high reliability level of systems. The problem of system reliability optimization where complex system is considered. The system reliability maximization subject to component’s criticality and cost constraints is introduced as reliability optimization problem (ROP). A procedure, which determines the maximal reliability of non series–non parallel system topologies is proposed. In this procedure, system components are chosen to be maximized according to it’s criticalities. To evaluate the systems reliability, an adapting approach is used by the ant colony algorithm (ACA) to determine the optimal system reliability. The algorithm has been thoroughly tested on bench mark problems from literature. Our numerical experiences show that our approach is promising especially for complex systems. The proposed model proves to be robust with respect to its parameters.