What temp did you primary it at? With English ale yeasts you can get noticeably different tastes with only a few degrees of difference.
Sorry, should have mentioned that: primary @68° F (S04 lists optimal range between 15-20°C) for 4 days and diacetyl rest for 3 days @72°F. Secondary @64°F.
Don't get me wrong: my self-criticism is not a matter between bad and good; it's a matter between good and great.