Previous climate changes that were driven by natural means took place over long periods of time, but even then it's not like species had time to adapt. Humans could not easily adapt to changes over a short period of time. We're not already. Climate change combined with continued human population growth already kill millions each year. Many people die due to famine and disease, with famine sometimes due to extreme droughts. With sea levels rising some island countries will go completely under water. Coastlines will be completely changed. Where now you have at least a billion people, if not billions, they would need to move inland. Ocean currents would also change. Ports would need to be in new locations. That involves massive infrastructure moves, abandoning cities or else putting massive walls around them. More extreme climate changes could bring prolonged drought to important growing regions, meaning not enough food could be grown for the ever growing human population, which again, in parts of the world there is already not enough food due to various reasons.

There's a massive amount of animal and plant species that are being lost and will continue to be lost due to climate change. There's further elimination of forests (which also accelerates CO2 rise), usually due to deforestation, and pollution on both land and in the water. Pollute the water and fish stocks in the ocean can suffer. These things make it more difficult for humans to survive. We're not talking about simply having to adapt to climate changes, but other changes due to our actions.

As for a new ice age, you are still talking about climate change that will have great impact. At least countries would not be going under water, but some infrastructure would need to move toward the receding oceans. Hundreds of millions or more would not be able to adapt to the new environment, needing to move further from the poles. There would also be a shift of food growing regions.

Either way we are talking about the need for massive changes for humans to adapt. Species before us did not have our intelligence to realize the problems we are facing and work towards solving them. They also didn't do as much damage to our environment. Species may have migrated, but they didn't have our ability to look further ahead and come up with a plan. Humans can work together to solve these problems. Previous species couldn't do that on the scale we can.

Why would we want to make life so much more difficult for future generations? We are already having an impact, but the impacts would escalate. If we can do something, why not? Rather than changes due to natural means over a much longer time scale that humans could slowly adapt too, why necessitate extreme adaptation and the loss of so much life?