That's a nice one, eh?! When working on it last week, I got knots in my brain.
First of all, I set a narrow scope: I decided that I wanted to fix the calc engine, not the calc(out) record. To achieve that, I based the main tests on setting fields to double values before doing the operation, as this gets as close as possible to what the calc record does.
The other tests (using literals and setting fields to integers) were added later when I saw different failure patterns between them and the main tests.
On plain 3.14, the operations completely depend on sizeof(long). On 64bit things work fine (as setting bit31 is *not* setting the sign bit), on 32bit things fail.
To get a more reproducible situation, I first backported the change in the calc engine that changes all operations to work on fixed 32bit size types. With that change, I had the tests consistently failing in a pattern that matched Dirk's report.
After applying Dirk's patch, all tests passed. That's the state I pushed to LaunchPad.
I never tested in a real IOC using Dirk's database.
Having a test producing the same failure pattern that Dirk saw using his database, and his fix passing all my tests and (as per Dirk's report) his database testing seemed good enough to me.
That's a nice one, eh?! When working on it last week, I got knots in my brain.
First of all, I set a narrow scope: I decided that I wanted to fix the calc engine, not the calc(out) record. To achieve that, I based the main tests on setting fields to double values before doing the operation, as this gets as close as possible to what the calc record does.
The other tests (using literals and setting fields to integers) were added later when I saw different failure patterns between them and the main tests.
On plain 3.14, the operations completely depend on sizeof(long). On 64bit things work fine (as setting bit31 is *not* setting the sign bit), on 32bit things fail.
To get a more reproducible situation, I first backported the change in the calc engine that changes all operations to work on fixed 32bit size types. With that change, I had the tests consistently failing in a pattern that matched Dirk's report.
After applying Dirk's patch, all tests passed. That's the state I pushed to LaunchPad.
I never tested in a real IOC using Dirk's database.
Having a test producing the same failure pattern that Dirk saw using his database, and his fix passing all my tests and (as per Dirk's report) his database testing seemed good enough to me.