His most controversial work, Portrait of Madame X (Madame Pierre Gautreau) (1884) is now considered one of his best works, and was the artist's personal favorite; he stated in 1915, "I suppose it is the best thing I have done." When unveiled in Paris at the 1884 Salon, it aroused such a negative reaction that it likely prompted Sargent's move to London. Sargent's self-confidence had led him to attempt a risque experiment in portraiture—but this time it unexpectedly back-fired. The painting was not commissioned by her and he pursued her for the opportunity, quite unlike most of his portrait work where clients sought him out. Sargent wrote to a common acquaintance:

I have a great desire to paint her portrait and have reason to think she would allow it and is waiting for someone to propose this homage to her beauty. ...you might tell her that I am a man of prodigious talent.

It took well over a year to complete the painting. The first version of the portrait of Madame Gautreau, with the famously plunging neckline, white-powdered skin, and arrogantly cocked head, featured an intentionally suggestive off-the-shoulder dress strap, on her right side only, which made the overall effect more daring and sensual. Sargent repainted the strap to its expected over-the-shoulder position to try to dampen the furor, but the damage had been done. French commissions dried up and he told his friend Edmund Gosse in 1885 that he contemplated giving up painting for music or business.

Writing of the reaction of visitors, Judith Gautier observed:

Is it a woman? a chimera, the figure of a unicorn rearing as on a heraldic coat of arms or perhaps the work of some oriental decorative artist to whom the human form is forbidden and who, wishing to be reminded of woman, has drawn the delicious arabesque? No, it is none of these things, but rather the precise image of a modern woman scrupulously drawn by a painter who is a master of his art."

Prior to the Madame X scandal of 1884, Sargent had painted exotic beauties such as Rosina Ferrara of Capri, and the Spanish expatriate model Carmela Bertagna, but the earlier pictures had not been intended for broad public reception. Sargent kept the painting prominently displayed in his London studio until he sold it to the Metropolitan Museum of Art in 1916 after moving to the United States, and a few months after Gautreau's death.