The problem

A financial platform built on Django traded foreign exchange through FXall, an electronic trading network. The connection started failing with an error that read: "could not convert field: 20231215-05:13:05ùùù".

The development team couldn't reproduce the fault locally or on test, and while the link was down, the platform couldn't trade through FXall.

Finding the cause

If a fault only happens on live, the answer is probably on live. The investigation moved to the production server, and the cause turned up there: a corrupt timestamp stored in the FIX session file.

That would explain why it couldn't be reproduced elsewhere. Removing the bad value brought the connection to FXall straight back.

Stopping it coming back

Clearing the fault restored trading, but it didn't explain why it had happened. So the work continued:

  • The backend and DevOps engineers helped trace how the timestamp was being corrupted, focusing on the QuickFIX library.
  • The QuickFIX maintainers were contacted, and they pointed to a likely fault in its SSLSocketInitiator.
  • Monitoring was set up, and the trading users were asked to report any repeat immediately.
  • A scheduled job was added to check the session file and remove invalid values automatically.

The result

Trading through FXall was restored. The platform could then detect and clear this fault by itself, rather than silently stop trading.

Lessons for any live system

  • Look at what differs on live. When you can't reproduce a fault, compare the live data, files and state with your test setup.
  • Read the error closely. The odd characters at the end of that timestamp were the clue.
  • Fix it twice. First restore the service, then deal with why it broke.
  • Ask the people who maintain your tools. Library maintainers may have seen the problem before.
  • Automate the safety net. If you can't remove a cause right away, you can often detect it and correct it automatically.

This article draws on work by Syed Quadri, SMART5's director, while working at another organisation. It wasn't a SMART5 client project, so the organisation isn't named.