The build server is usually a pretty important machine for the organization. Building software is processor-intensive, as well as memory- and disk-intensive. Builds shouldn't take too long, so you will need a server with good specifications for the build server with lots of disk space, processor cores, and RAM.
The build server also has a kind of social aspect: it is here that the code of many different people and roles integrates properly for the first time. This aspect grows in importance if the servers are fast enough. Machines are cheaper than people, so don't let this particular machine be the area you save money on.
