Broadly speaking, there are 4 categories of wealth in the Bible: godly poor, ungodly poor, godly rich, ungodly rich.
Main issue isn't if you're rich or poor but if you're godly. This passage is focused in ungodly rich. James says ungodly rich are using their power/influence to persecute believers, especially the poor. James doesn't try to flatter or appease the rich, instead he speaks the truth in a warning. James is acting as a voice for the voiceless and encourages the church to do the same.
James has the 2nd Coming if Christ and the Judgment Day in mind as he warns the ungodly rich about their behaviour. They prefer hoarding wealth and letting it rot instead of helping someone in need. To be godly, share with the needy and support those in advancing the kingdom of God.
God pays attention to how we get our wealth as well as how it is used. These people took advantage of others. The godly rich pay back debts in a fair and timely manner.
James is addressing the abuse of wealth as these people also try to influence the court to have their way.
Even if you do not feel rich, take the time to evaluate how you are using what you have. Do you value money more than your neighbour?