Is a panic attack something to be worried about? Can't I just let it go away and pretend it never happened?

No. If you have a panic attack, you must seek a doctor's advice. This is because you're at risk of:

Anticipatory anxiety - This is triggered by just thinking about the possibility of having another panic attack. This can cause you to become reclusive, avoiding people.

Avoidance - If you have panic attacks, you may feel like not doing activities that seem to trigger panic attacks, such as driving, being inside elevators, or going to work. While this may temporarily help you with the fear of the attack, it makes it very difficult for you to have a normal working or social life. It also does not stop the attacks from occurring.

Agoraphobia - This is quite common with panic disorders. Agoraphobia is the fear of being in places or situations from which escape might be difficult or in which help might not be easily available. So because you are afraid of panic attacks, you might be too afraid to go to public places, crowds, or even to travel by bus or plane. You may be too afraid to leave your home!