Having a key for closing pop up such as mod selector and game rules UI would be convinient since pressing Esc key would opening menu instead of closing pop up (or maybe esc key should close pop up then opens menu when all pop up closed)

You can close mods / gamerules / shaders menus by pressing the corresponding hotkeys (CTRL + M / CTRL + G / CTRL + H respectively).
The way it works at the moment is done on purpose and there are currently no plans to change that.

Ohh i didnt know about this closing