That was the case for me. I never had a panic attack until years into my DPDR. My first DPDR experiences were after smoking marijuana, it only felt like I had not completely come back to normal after the trip. As if I was still tripping a tiny bit days and days after smoking. And it started to grow from that.
There was anxiety in some of my trips, but usually not so much and it was still worth it to continue I thought. The negative feelings while smoking got worse as DPDR was progressing, until it was making me feel bad systematically and I quit.
I never even smoked a lot. The shortest time I have ever had between two joints was one week, but usually it could be more than a month, up to a year. So it didn't take too much.