The problem with implementing elo and ladders at the same time when not having players face other players from the same ladder is that the amount of elo lost to someone who is not on similar level as you (if you rank a lot higher) reduces your elo points way too much and vice versa, the workaround here would not work (as in dota, sc2, lol etc) where you rank people in different ladders and have them face opponent in the same "vicinity of skill" and thus rank.
Now with toribashs' large community (erm...) such a solution does not seem viable, and random games (random opponent) should also ONLY count towards your rank (erm...).
So practically this becomes a no-no within the current setup of toribash, and remaking the system to support such a ladder system would not work due to the low amount of players, so it really becomes catch 22 to implement such a system here.
EDIT, SC2, DOTA2, and LOL utilises systems similar or identical to the elo system (with a few modifications through "voting" on performances of other players)