Colossians 2:16 by itself leaves room for two possibilities.My position is found here:
However, those who choose to practice Sabbath-keeping should not judge those who do not keep the Sabbath (Colossians 2:16). Further, those who do not keep the Sabbath should avoid being a stumbling block (1 Corinthians 8:9) to those who do keep the Sabbath. ref
1.) The Colossians were not keeping God's holy days, they were being judged by Jews because they were not keeping them, and Paul was encouraging them not to let anyone judge them for not keeping them.
2.) The Colossians were keeping God's holy days, they were being judged by pagans because they were keeping them, and Paul was encouraging them not to let anyone judge them for keeping them.
In Colossians 2:16-23, Paul described the people who were judging the Colossians as promoting human teachings and precepts, self-made religion, asceticism, and severity to the body, which means that they were being judged by pagans and that #2 is the case. We should be careful not to mistake things that were only said against obeying the teachings of man as being against obeying the commands of God.