The Delete button for projects is in the Library panel in the right-hand column. So technically to rename a Project you could choose Save As, rename it, change any other details, click Save, and then delete the old project.
One thing that stinks is you currently cannot directly save an image to a project from the Save As dialog. (Bug submitted)
I'm having no problem saving an existing project as a new one. If you have a problem there, then something is probably wrong.