If you find it hard to build muscle in your upper body, it might not be your upper body training that is slowing you down.
Working the biggest muscle groups in your body will help to create more muscle and a better hormone profile for building muscle.
If you look at lower body exercises lots of them challenge all the big leg muscles and the big muscles in your back, so training with these exercises will have a bigger impact on your hormone profile than bicep curls for example that only hit a small set of muscles. Build your legs and your back first and then building your arms and shoulders will be a lot easier.
Try all variations of deadlifts, squats and lunges and try to focus on increasing your total volume lifted.