A Video encoder performs video data compression by having combination of three main modules such as Motion estimation and compensation, Transformation, and Entropy encoding. Among these three modules, transformation is the module of removing the spatial redundancy that exists in the spatial domain of video sequence. Discrete Cosine Transformation (DCT) is the defacto transformation method in existing image and video coding standards. DCT is a most popular transformation technique for DPCM/DCT-based video coding. Even though the DCT has very good energy preserving and decorrelation properties, it suffers from blocking artifacts. This problem affects the performance of other subsequent modules involved in the video encoder. To overcome this problem, a hybridization method has been incorporated in transformation module of video encoder. This paper presents an hybridization in the transformation module by incorporating DCT as transformation technique for inter frames and a combination of wavelet filters for intra frames of video sequence. This proposal is also applied in the existing H.264/AVC standard. Extensive experiments have been conducted with various standard CIF and QCIF video sequences. The results show that the proposed hybrid transformation technique outperforms the existing technique used in the H.264/AVC considerably.