Usually, it stays where it is but is marked as not there by the drive. Drives don't like to write data, so we we came up with the idea of imaginary file locations on the physical level.

Imagine the following scenario: You have 2 closed and sealed boxes, left and right box. Both boxes contain an identical ball.

You ask me to move the ball in the left box, to the right box. What can I do? I can physically open the boxes, remove the ball from the right box, move the left ball into the right box and seal both boxes again. This is slow and damages the boxes, even if minimally.

Alternatively, I can just go 'done' and you can immediately find the identical ball in the right box.

Can you still find the ball in the left box? Yes. However, if you asked me if there's something in the left box, I'll say no, because I do not care about it's content anymore, therefore it is open for further use. You can decide to ignore my answer and check inside nonetheless, and that is exactly what recovery software does. It checks all the boxes to see if there's anything in there.

Computers use something called a pointer. Which is basically an address book.

Imagine a special street: alphabet street, houses A-Z. The pointer is an address book. If your file John lives in house A. When you want to find John you look in the address book to find house A to get John.

Now it's too much work to kick John out and replace John with another Jane (Overwriting). So Jane moves into house B and the computer in the address book replaces John's house address with 'vacant' and Jane gets the address house A. (Not exactly but ELI5)

If you want to find Jane you can go to John's old address or house A (which is actually house B). once the street is full John is then kicked out and replaced.

Tldr: your file still exists on the computer, but the computer points to a different file with the same name on the computer. A forensic scientist (or program) could still likely recover the original file.

Forget about replacing and think about just deleting the old file. The fact that the new one has the same name is irrelevant. The old file can be retrieved the same way a deleted file can be retrieved (there are countless ELI5) posts about that.

The same thing that happens when you delete the old file.

And what is that, you might ask? When a hard drive "deletes" a file, what it actually does is mark that file's "space" as available. Kind of like listing a property for sale; that space might not get used right away, but as time goes on, it gets more and more likely that it will get overwritten by something else eventually.

The new file over writes the old file, there are not two files. If you notice, there should be a message on your computer that says something’s like, “there is already a file with this name do you want to overwrite the file or change the name”. If you change the file name then a new file will be created. For example when I’m working in my resume, the file name is “resume”. If I make changes to the file and I can save (replace) to the same file called “resume” and there is still just one document. But I can also save my changes to a new file called “resume1” Now I have two files, my original file (resume) and the file with changes (resume1) Hope that helps a little.

"Overwriting" is a purely logical operation in this case. It does not actually overwrite the physical file - it just forgets where the previous file was located, and instead remembers the new one with the same name.

If you use a file undelete tool that accesses the disk (or under some cases the file system) directly you can still read the old file. 

No it does not (not physically anyway). The old name just starts pointing at the new file.