It was until today, when it started to take more than one minute to properly load that modal window of IMCE upload.

IMCE upload window
I quickly realized, IMCE folder was using about 1GB of memory (default upload folder -/sites/default/files) which stored more than 7,000 images - each time IMCE had to do a check up on all those files. I decided to look on IMCE settings, can I change default upload folder and start fresh? Luckily I was able to that:
From IMCE config - admin/config/media/imce/profile/edit/1 - create a new upload directory for IMCE

IMCE upload directory
Woila - after applying new settings - IMCE was blasting fast again. Hope this helps.