Ideally, no, but because you are acquiring foreign sounds at a later stage of your life, it has become a must. However, you will need the IPA only in the first stage of your learning as you will have to visually recognize the sounds as opposed to the natural way of learning where a toddler learns to speak by listening to the sounds in his surroundings before imitating them and finally formulating a word. You will have to do it the other way around that’s why it is often difficult, but it is not impossible.
Visual recognition is the first step to training your ears to hear the sounds and the nuances between each and every sound. Once your ears are capable of listening for sounds, you will no longer need the IPA. You can start the process of learning the natural way, like a toddler would. But until then, you will have to keep the IPA as close to you as possible.