First a little history on our car. It's an 05 Cavalier 85k miles had been wrecked when we bought it. We fixed the damage mostly cosmetic and a new radiator and it needed a couple wheels. The guy we purchased the car from buys fixes and resales wrecks. After all the work was done and the car still on his lift we pulled a tape measure and everything was straight and even.
Now the problem we are having is when driving on a loose gravel road or a rain soaked road the back of the car feels very unsettled almost like it wants to drift. I've checked everything I can think of just looking for opinions of what I should check

I'd recommend comparing it with another un wrecked rear end. But, it may also be normal. The rear is a solid rear axle, which is a bit easier to get it to spin on the rear. That may not be much help, but hope it helps some.

this is a lot late but may help a future member.
if the rear toe is not set correctly, the rear end will wander on slippery road conditions. On dry pavement, the tire scrubs and rolls...with low traction, the rear end wanders. This may be confirmed visually looking at the rear tires for toe wear ( a feather edge pattern on either edge of the tire; but not both). If the tires look ok...have an alignment check. if it is out you will need alignment readings for the correction.

since there is no toe adjustment on your cav; it uses a beam rear axle, the fix is installation of shims behind the spindle. If the readings are out more than the shim can correct; a new axle will be required.

Using the alignment readings, an adjustable shim will correct both camber and toe at the same time. time for install is 1 to 1.5 hours per side. depending on the alignment readings; just 1 or both sides may need to be done.