The answer to this thought-provoking question relies on your ideas about exactly what a language is. If you consider a language to be a means of transmitting data about the external world from one person to another, then music doesn't seem to fit the bill. One can't find any information about the outside world purely from the music, although one can infer certain ideas if one knows some of the history behind the creation of a piece of music.
But if you think of language only as a way to share information in a way that many people can understand, then yes, music is a language. That is, written music is a language; the actual sounds of the music are purely artistic and abstract. (Composers can certainly evoke certain images or emotions through their music, but those evocations aren't universal and rely on the shared history and knowledge of the listeners.) Musical notation can be considered as much a language of music as mathematics can be considered a language of science.
Consider this: If a scientist in, say, outer Mongolia doesn't speak English, he won't understand what you mean if you tell him, "The square of the hypotenuse is the sum of the squares of the other two sides." But if you show him, "a2 + b2 = c2," he'll recognize the Pythagorean Theorem and know that it applies to right triangles. That's because you're speaking in a language that he understands: mathematics.
In the same way, if you describe a piece of music in your own language, someone from the other side of the Earth might not understand a lick of it. But if you write out the music using standard musical notation — with staves, time signatures, key signatures, clefs, dynamics, and all that — people around the world can instantly understand your "data," which, like mathematics, is essentially a set of instructions.
And like mathematics, there are many different levels of comprehension of musical notation, based on levels of training. A professional musician may be flummoxed by mathematical equations when Greek letters start to appear, but she can instantly read the notes in a C-clef, understand the markings for pizzicato, and build a fully diminished seventh chord without a second thought. A master mathematician, on the other hand, may be just the opposite — she may be able to deduce the speed of an electron with only a little information but just barely be able to read the notes in the treble clef. Nonetheless, both the musician and the mathematician have some shared vocabulary in both mathematics and musical notation.