First, to answer the rhetorical question posed in the title of this post, no, we don’t all worship the Goddess. Many of us, particularly historical reconstructionists and hard polytheists, don’t have a single “Goddess” or “God” in our cosmology. Freyja is Freyja, Isis is Isis, and Ceres is Ceres. They are not “aspects of the Goddess”, or “emanations of the ultimate female life-force” or anything like that. To us, they are distinct and unique beings with their own personalities, qualities, and complex natures. Still others see the Gods as Jungian archetypes, or merely as useful mythological creations with no objective reality.
Does this mean that dualists, or Goddess-worshippers, are “wrong”? Not at all. It merely means that there is no consensus among those often found under the “Pagan” umbrella as to the nature or number of the Gods, and thus it is insufficient to form a definition of “Pagan” on that basis.
So, too, mere polytheism is not enough to call someone a “Pagan”, either. Many religions are polytheist in nature to one degree or another; Shinto, Hinduism, and even Mormonism have been thus described, but most people would never think to include them in any contemporary definition of “Pagan”.
So just having more than one deity isn't enough. We must perforce look elsewhere for some sort of commonalities that could be used to come up with a definition of “Pagan”.
Is it a lifestyle choice? Again, while it may be for some people, it’s definitely not for all. Some “Pagans” wear pentagrams and Ren Faire garb, are vegetarians who belong to a food co-op, and flit from barista to bookstore cashier to au pair to pay the bills, but others wear suits to their corporate jobs and come home and grill a t-bone while watching football. (Many don’t even use the pentagram as a symbol!) The “Pagan” tent admits people from all lifestyles, and many of those people don’t feel particularly comfortable around people who don’t share their own particular lifestyle. You don’t get to say “you shouldn't feel uncomfortable around people who go skyclad” (or who are polyamorous, or who are omnivorous, or who hunt, or who drive SUV’s, or whatever it happens to be). Some people are, and that attitude needs to be recognized as a legitimate choice.
Is there some sort of commonality in ritual, perhaps? Again, the answer is a resounding “no”. Many people follow a calendar based on the solstices, equinoxes, and cross-quarter days. Many others follow a lunar calendar, doing ritual on the full or new moons (or both). Still others follow an historical calendar, only doing major rituals three or four times a year, at the beginning of winter, in the middle of winter, and at the beginning of spring, as recorded in the Heimskringla and other sources. Still others follow the ritual calendar of ancient Athens, or Rome, or Canaan.
As for the content of the rituals, that forms an even greater tangle. Some “Pagans” form a circle, invoke the guardians of the cardinal directions, and raise energy in ritual. Others invoke Gods and Goddesses directly, sometimes even inviting spirit possession as an integral part of ritual. Still others don’t involve “magic” at all in their rituals; they are purely votive offerings made to the Gods or land-spirits. Some have lengthy rituals with multiple parts, scripts, and elaborate props. Others do their rituals off the cuff. Some do ritual dramas. Some dance. Some share sacred meals, or drinking, as ritual, while others have a more symbolic sharing of food and drink, while others don’t have anything of the sort in ritual. Some use the four elements, some five, some two, and many don’t invoke any elements at all. Some perform animal sacrifices, and others vehemently oppose the practice. It’s the Wild West when it comes to ritual.
Organizationally, the question is just as open. There are formal covens whose members are bound by oaths, there are less formal covens, study groups, individuals (often called “solitaries”), Druid groves, Ásatrú kindreds, fellowships, etc. There’s no common structure; some have strong leaders, some practice pure democracy, some require consensus, others elect leaders. So there’s nothing organizationally that makes a group “Pagan”.
We’re starting to run out of possibilities.
If the definition of “Pagan” isn't based on cosmology or theology (or theology), it’s not based on specific lifestyle choices, it’s not based on the ritual activities or calendars, it’s not based on organization, what the heck is it based on?
Or, to ask the question another way, what do a Druid, an Ásatrúar, a Dianic Wiccan, a practitioner of the Religio Romana, and a Gardnerian Wiccan have in common? If all they have in common is that “they’re all Pagan”, then the term itself becomes meaningless, as it describes nothing more than an arbitrary grouping of individuals and faiths that would otherwise have nothing in common.
Many have already realized this, and have moved to distance themselves from the “Pagan” label, as it carries with it all sorts of connotations that they don’t want to be associated with. Most Ásatrúar, Théodsmen, etc. consciously use the term “Heathen” rather than “Pagan” precisely to avoid association with people whom they see as very different from themselves. Others avoid such umbrella terminology altogether and simply use a more specific label; they are not “Pagan”, but Khemetic Orthodox, or Théodish, or Religio Romana, or Celtic Reconstructionist, or Druid, or whatever.
And, given the lack of commonality to define what it means to be "Pagan", perhaps they're onto something.