You can delete custom folders. You cannot delete system folders.
When you delete a folder, its contents are moved to the Trash. You can recover contents from the Trash by dragging and dropping them to another folder.
The Trash folder is purged on a regular basis, so items in the Trash do not stay there forever.
Either drag the folder to the Trash or right-click the folder and choose Delete.
If there are messages in the folder that you want to save, move them to another folder.