You can add/edit the Alert by going to Content > Sections > Alert.  If this section has content it will display the alert button and corresponding alert lightbox with the section content.

To turn off the alert, simply make sure there is no content in the section OR you can re-name the section (eg. "Alert - Off") and then you can label the section back to "Alert" when you would like to turn it on.