To delete the default emoji when you create a new bucket:

Click on the 3 dot menu and select Customize bucket.

Then click save, without doing anything else.

You can of course fully customize the bucket here as well, by adding color, images & patterns, icons and emojis in any combination you would like.