They tell me that the Visa Waiver scheme has now been replaced by ESTA and completion of the I-94w card has been scrapped.
The ESTA replaces the I-94W for arrivals by air and sea. The VWP itself has not been replaced.

They tell me that the Visa Waiver scheme has now been replaced by ESTA and completion of the I-94w card has been scrapped.
The ESTA replaces the I-94W for arrivals by air and sea. The VWP itself has not been replaced.
Having read what you have written:
"The ESTA replaces the I-94W for arrivals by air and sea. The VWP itself has not been replaced."
As you say VWP still exists with its original rules and have now decided that we cannot rely on our getting through Chicago with 100% certainty despite what the London Embassy said.
So it's a case of chance it or get the visa.