Don't worry.
You don't want unused RAM on your server, and Linux knows that. So, it uses RAM at its maximum before swapping and/or garbage collecting.
In other words, it's expected to have as much RAM as possible in use at any time. So, I can't see any real issue on what you say.