In the old days, they were called "profiteers," and it wasn't unusual to see them hanging from lampposts when an enraged and exploited populace finally figured it out.
As the brilliant Michael Lewis wrote once, everyone who is in a market is trying to manipulate the price. That is the whole point of being in the market.
The reason speculators can make a profit is because the price that sellars can sell for, today, is lower than the price that buyers can be forced to pay in the future. If there were no arbitrage opportunity there could be no speculation.
Of course they have an impact on prices. That is how speculators (anyone who is not, simplistically, a producer or consumer of the commodity) make money. That is the point of the game. The reason why they are allowed in the game is that they provide some value in providing liquidity and making it possible for producers and consumers to hedge risk.
I contend that because of 50 years of easy living and easy money, helped along by the rest of the world taking fiat in exchange for valuable goods and services, we have a very very large structural problem in the US economy. Before we can slim down, and live within our means, we have to cut out a lot of parasitical activities, including a legal system that takes 6 or 7 years to settle simple disputes, a governmental system where untold trillions go to beltway bandits to produce papers and viewgraphs to "inform" the policy and regulatory process, a regulatory scheme that encourages fee seeking and rent-seeking behaviors even though little or no value is provided in return for those rents and fees.
Of daily life in the average corporation, or especially daily life in a governmental or regulatory agency or law firm, what fraction of your time is spent doing something that provides a real direct benefit to a customer, client or taxpayer. Of your purported income, what fraction do you have voluntary control over and freely spend to obtain wanted goods and services, as opposed to services and fees that you are forced into willy-nilly.