The Developer's Guide to WordPress/Optimal Hosting

Hosting for speed
Not all hosts are created equal, and this can impact page load times. Page load times greatly impact the user experience. In general look for the following in a host:
 * Good bandwidth
 * Close to your users
 * Serves content from fast storage, such as an SSD instead of an HDD.

Hosting for reliability and resiliance
Reliability is a key, and often overlooked attribute. Some services allow you to take advantage of features which help improve reliabiliy or at least help improve resiliance to failure.
 * Automatic backups (Not a replacement for doing your own manual backups, but can be useful in a pinch)
 * Geographically distributed

Take the load off with a cache
Wordpress is a dynamic CMS, which means it does significant backend processing. For pages which are commonly accessed but do not change, a common practice is to use a cache to speed retrieval. One common solution is to use the open source varnish cache, which has a numbler of existing plugings.