Not knowing what kind of pitless you have, it would be hard to tell you how to fix the problem. We use epoxy to plug the holes in the well seal here in Florida and used to epoxy the spools in the Whitewater Pitless adaptors in Michigan years ago. The idea is to first thread the wires through the pitless spool, then when the epoxy sets up, put the spool back in by splicing the wires top and bottom. Once back in place, the water can't seep back up into the upper pipe.
For a quick fix, you could simply drill a hole in the upper pipe so the water runs out before it get all the way to the top.