in the first gear, or 2nd gear, if I accelerate and then release throttle a little faster (just a little little faster), I can hear some clunk noise from the rear end of the car

I'm wondering if anything wrong with the rear suspension or shaft, but car drives normally.

it happens in 1st gear more often, and minimum in 2nd gear, in other gear I cannot hear, maybe just because it's covered by other noise.

I see a lot of people has the same thing but only hear the noise from the front which highly suggested a engine mount issue, but this is not my case, the noise actually comes from the rear end of the car.

can it also be just a exhaust pipe noise?

 

It could be. Can you get under and look and test for loose slack/parts. Bit hard to work out the problem from here.