It was originally part of a great feast on January 6 called "Epiphany" that celebrated several "manifestations" of Christ - his birth, his recognition by the Magi, his baptism in the Jordan, and his miracle at Cana.
In about the fourth century, Christians in the western world pulled Christmas out of the Epiphany celebration and gave it a feast of its own on December 25. The time between this new feast on December 25, and the feast of the Epiphany on January 6 became the Christmas season - the "12 days of Christmas."
But the Christmas season no longer ends on January 6. That's because eventually an "octave" was added to the feast of the Epiphany - an extension of the celebration for a week after the feast.
When this happened, the Christmas season no longer ended on Epiphany but continued through the octave. The feast of the Baptism of the Lord was eventually placed at the end of the octave ... and now it is the feast that brings the Christmas season to a close.
- Little Blue Book, Diocese of Saginaw