Leaving aside the question of the paternity of Edward of Lancaster (as we've seen, there's no proof that anyone other than Henry VI was the boy's father), what of the various men that popular historians have romantically linked with Margaret of Anjou?
As I should have made clear in the earlier version of the post, there were indeed contemporary rumors that Edward of Lancaster was a bastard and that Margaret was unfaithful to her husband--a ballad written by a Yorkist sympathizer refers to "fals heryres fostered," for instance, and Pope Pius II quoted Warwick as saying that Margaret and "those who defile the king's chamber" had taken over the government. The contemporary gossips, however, were reticent about naming names; modern writers have been less so.
Bertram Fields, for instance, in his book Royal Blood, a defense of Richard III, baldly asserts that Margaret "had been known for years to have dallied with her favorites, notably the dukes of Suffolk and Somerset." Like too many admirers of Richard III, Fields, while bemoaning the various myths that have grown up around that king, is quite content to perpetuate myths about other historical figures. The claim that Margaret and William de la Pole, Duke of Suffolk, were lovers was not made until the sixteenth century, and even then only in a throwaway remark by the chronicler Edward Hall that Suffolk was "the queen's darling." It was Shakespeare, not contemporaries of Margaret and Suffolk, who gave us the story of a full-blown love affair between Suffolk and Margaret.
As for the Duke of Somerset, there were three such dukes associated with Margaret: Edmund Beaufort, Duke of Somerset (died 1455) and his sons Henry (died 1464) and Edmund (died 1471). Though Paul Murray Kendall and others have stated that the elder Edmund was suspected of fathering Margaret's child, I have yet to find any contemporary English source alleging that he and Margaret were lovers--although in his youth, Edmund was linked romantically with the widowed Catherine of Valois, Henry V's queen. Hall states that when Somerset was arrested in 1453, he was in Margaret's great chamber; assuming that this noncontemporary account is true, it's notable that Somerset was not said to have been in Margaret's bedchamber. Margaret did grant the elder Somerset an annuity of 100 marks in 1451. It is recorded as having been paid at Michaelmas 1453, at which time it was noted that it was being paid for past and future services as well as "for the great good will and kindness that he will show [her] in her urgent affairs." Helen Maurer has suggested that the "urgent affairs" referred to Henry's recent mental breakdown, which would make sense. There is no reason, however, to assume that the annuity was prompted by a love affair.
The elder Edmund was killed at St. Albans in 1455; his eldest son, Henry, took up the family dukedom and the Lancastrian cause. Henry Beaufort is linked suggestively with Margaret in one contemporary rumor: on March 15, 1461, Prospero di Camulio, Milanese Ambassador in France, wrote from Brussels to Cicho Symonete, Secretary to the Duke of Milan: "They say here that the Queen of England, after the king had abdicated in favour of his son, gave the king poison. At least he has known how to die, if he did not know what to do else. It is said that the queen will unite with the Duke of Somerset. However these are rumours in which I do not repose much confidence." Henry VI, of course, had not abdicated in favor of his son, nor had he been poisoned. The rumor that Margaret intended to "unite" with Somerset, then, should inspire no more confidence in us than it did in Camulio. I do, however, confess to finding it plausible that this Somerset could have been Margaret's lover--he was young, handsome, and charismatic and had done her the great service of defeating the Duke of York at Wakefield--but there is no evidence that he actually did play such a role in Margaret's life.
Henry Beaufort was executed at Hexham in 1464. (Being a Beaufort during this period did not auger well for one's future.) Late in that year, his younger brothers, Edmund and John, joined Margaret of Anjou in exile in France. There is no evidence that either of these men, or any of the other men who shared Margaret's exile, were romantically involved with her. Indeed, the younger Beauforts and the Duke of Exeter, who was also exiled abroad, spent most of their exile in the service of the Duke of Burgundy, far away from Koeur Castle in France, where Margaret was lodged.
Another man who occasionally is named by modern writers as Margaret's lover (and a possible father of Edward of Lancaster) is James Butler, Earl of Wiltshire; Kendall, for instance, claims that he was rumored to be the prince's father, though he doesn't cite a source for his assertion. Wiltshire, who was executed after the Battle of Towton, was the royal treasurer of the Lancastrian government and was particularly disliked by the Yorkists, but there is nothing to indicate that he was unusually close to Margaret. I have found no contemporary allegation that he and Margaret were lovers.
Philippe Erlanger, a French writer who wrote a very fanciful biography of Margaret, patriotically hinted that Pierre de Breze, Seneschal of Normandy, was Margaret's lover, and he even attributed Henry Beaufort's brief defection to Edward IV as having been prompted by jealousy over Margaret's intimacy with Breze. The latter, who was twenty years Margaret's senior, lent her military assistance and traveled with her in the early 1460's, but he had been closely associated with Margaret's family for decades and was likely motivated by those ties rather than any carnal passion. If Margaret was suspected of having an affair with Pierre during this time, it seems odd that Edward IV's government failed to exploit the propaganda possibilities that such a relationship presented.
Proving a negative is notoriously difficult, and it can't be shown that Margaret was not having sexual intercourse with any of these men or with anyone else besides her husband. (Contrary to one historical novelist's assertion, however, Suffolk can be positively eliminated as a possible father of Edward of Lancaster, having been murdered over three years before the boy was born.) As it hasn't been proven either, however, that Margaret was having an affair with any of these men, confident assertions like that of Fields quoted above should have no place in nonfiction unless supported by contemporary evidence.