I'm thinking of making a C++ library to provide an API, and then making a compression algorithm using it.

I'm pretty sure it's just a text file with a custom extension.

As for the actual text in the file, have a look at parts 1 and 2 of the Advanced Replay Hacking (link) tutorial.

If any more questions arise, please ask and I hope that I can answer them.

Also I don't fully understand what the sentence in your post body means, but code on brother

i think its a javascripts or something like that

It's a text format record of game commands. I don't know that there's a complete specification available anywhere, but it should be relatively explanatory. It's space, semicolon, and new-line delimited, and is otherwise human readable.

It is not Javascript.

As far as compression goes, they're quite compressible already, but toribash.exe doesn't have any decompression built in to handle compressed replays. Without that, not much to be done. If your replay folder really is too big for your tastes though, turn on filesystem compression.