We are both correct.

Male penis envy
I worked with Freud in Vienna. We broke over the concept of penis envy. He thought it should be limited to women – Woody Allen in Zelig

While not the same kind of penis envy as that typically referred to in psychoanalysis, the phrase "penis envy" or "small penis syndrome"[1] is also sometimes used to describe the envy of a male over another male's penis. Although this subconscious or conscious envy may solely be based on the idea that a larger penis is universally more satisfying and appealing to a sexual partner, other implications arise from the fact that a large penis has been seen in many cultures as a symbol of high masculinity, dominance and power. While this whole matter has probably always been a part of human psychology, recent developments have made the issue slightly more public in the western world.

The media attention given to penis size and some women being vocal in their opinions of penis size have led some men to state their envy of others with larger penises. Television shows such as Sex and the City and Ally McBeal popularised the penis size issue when characters in these TV shows stated their preference for well-endowed men over more modestly-endowed men. Also, in the 1977 film Annie Hall, Woody Allen's character, upon hearing the question asked by the title character about penis envy, replied that he "was one of the few males that suffered from it". This conception of Freud's theory is usually the explanation behind the term penis envy.

Men can underestimate the size of their own penises, see Perceptions of penis size.