The server file automatically updates when you run the game, so deleting them would just force the game to redownload the IPs. I believe you could remove servers from the list in earlier versions, but I doubt there's a way of doing so in 3.1.
Note that a means of quickly joining a server is to learn what number it corresponds to on the server list. For example, I know that the Jousting server is server 6, and themy clan server is server 44. From singleplayer I can type
/join 6
And instantly join the Jousting server. In the same way,
/join 44
Would get me to my clan server.