The behavior you are seeing is normal and expected.
SQL Server does its own memory management. What that means is that, as it needs more and more memory, it will grab the required memory from the operating system upto the maximum allowed. But then, SQL OS will not release the memory back to the Windows OS when it is no longer required. It holds on to that memory and will use it as and when required.