Please understand/know that I don’t want to worry you.
Don’t get me wrong, I just don’t want to worry you.