Fixing Novopay (and other “Big IT” projects)

TL;DR: How do you get five elephants into a mini?

I said last week that there’s no way I could build a school payroll system without error. Neither could you. Together though, we could get pretty damn close.

The elephant thing is funny, but strikes at the root of the issue: we need to ignore the elephant. Please excuse me while I flog a dead metaphor for a little while:

  • We had a pretty cool elephant. It was big and hairy, occasionally left giant steaming turds around the place, and was really handy at moving logs around. 
  • Someone decided the elephant was a bit old, even though it still got the jobs done.
  • A replacement elephant was created, but the creators thought the trunk was for squirting water, not shifting logs, so it wasn’t strong enough to do the jobs we need it to do.  They then set about engineering a cybernetic augmented trunk system to strengthen the trunk.

A key issue with Novopay, and the school payroll system in general, along with many other huge black-box IT projects (yeah, I’m looking at you, $1.5bn+ IRD replacement system), is the approach of replacing the system as a whole. Forget about the elephant, let’s look for a system of interconnected components that – as a whole – gets us to where we need to be. Perhaps they end up looking like an elephant, or maybe a forklift, but that’s irrelevant.

In the case of Novopay, what are the core problems that need to be solved? Rather than tender for a complete system, we would be much better off if the Ministry of Education tendered for a “Teacher’s Payroll Calculation Component”. This would be defined as taking a set of inputs (tenure, contract details, tax period, hours worked), and emitting outputs (gross and net pay, tax, Kiwisaver, holidays accrued). The inputs and outputs of course have to be implemented in a transparent, open manner: REST would make developer’s lives easiest, but as long as it’s well-defined and not some proprietary binary format, who cares?

Those of you immunised against pragmatism will be having convulsions right now. “But, but, where does the data come from? Where is it stored? What about security and archiving? This is only a tiny piece of the problem!”.

I’m not saying calculating a single pay run is an easy task. On the contrary it sounds incredibly complex. But by breaking this one component out separately we can do so many things:

  • TEST this component against the existing systems. Throw some really hairy combinations at it and make sure the outputs are correct and identical to the existing system. Create those as automated test cases that we run daily and verify output.
  • INTEGRATE with this component in any way we think is appropriate. Let 1,000 data entry systems bloom. Want to write a pay calculation front-end for Palm Pilot? Here’s the API and security model, go to town.
  • SECURE this component by ensuring that only teachers and schools (and other components of the larger system) with appropriate access can get at it.
  • IMPLEMENT this component against the existing system by delegating the single task of calculation to this component using its API. Ding! You’ve just removed a dependency on the huge project.
  • REPLACE this component if it sucks. You know the API, you have the test cases you need to achieve: if you think you can do better, go write me a new one.

But wait! I hear you saying, what about storing teacher details, historical pay runs, transferring money to bank accounts? Ben! You’re lying to us. We need to spend more money on Big IT!

It’s ok, just relax and keep slicing away at that elephant. Contract to build a “Teacher Information Storage System”. Define the fields you need to store, the security with which it needs to be stored, and the interfaces it needs to expose (surprise! one of those probably matches perfectly with the payroll calculation inputs).

The other obvious upside is you can approach each component as a best-of-breed system. Rather than using fucking Oracle Forms for user input, you could contract the front-end data entry to a talented web development studio, safe in the knowledge that they won’t be out of their depth having to code payroll calculation.

What’s more, if one particular implementation (or implementer) really sucks (ahem Talent2), then the damage is limited to the one component they are working on, not the entire system.

You get my drift? At no stage have I said this will save money. It probably will, but even if it doesn’t the end result is a well-defined set of interconnected components, all of which could be replaced individually.

There is of course a gigantic problem with my approach: everyone working on the systems, teachers, the Ministry, all the way down to coders, would need to collaborate openly and honestly. Bummer.

16 Replies to “Fixing Novopay (and other “Big IT” projects)”

  1. Why don’t they give each School a Xero.com accounting software and then just run it like a normal payment system.

    1. My understanding (possibly very incorrect) is for a couple of reasons.

      1. The final pay is conducted in bulk from MoE funds, so schools can’t pay direct?

      2. The pay rules are so complex and archane that an individual system couldn’t cope.

      However: my solution wouldn’t stop Schools from running Xero and integrating the outputs with a component of the “New Novopay”.

      1. Also you have relievers who work across multiple schools and need tax, kiwisaver, leave etc balanced across all the work they do.

  2. “The secret to building large apps is never build large apps. Break your applications into small pieces. Then, assemble those testable, bite-sized pieces into your big application”

    – Justin Meyer of JavaScript MVC

    Katrina Owen’s Theraputic Refactoring presentation does a great job of showing how you can take legacy code, build a test suite against it and then use that test suite to refactor the code base to a better implementation.

  3. Nice note Ben. A few comments:
    1. But old systems that aren’t modular to start with are very hard to unpick in the way you suggest.
    2. That said, Datacom were progressively rebuilding the system and they had started with a new web based front end that was already rolled out to 100’s of schools. It would be very very hard for a 3rd party without inside knowledge of the existing system to take this approach. One presumes the MOE wanted something faster.
    3. Oracle forms was a great tool in its time – in the 1990’s. I haven’t seen anyone use it since then for a significant fresh development. If the MOE thought they were buying a modern system in Novapay they have been duped. Forms is orphan technology for which it is already difficult to find experienced developers. I’ve heard Novopay described as an old system with a new brochure.
    4. Ironically one of the reasons Oracle forms became unpopular, other than the licensing model which became too expensive, was that it didn’t allow code to be well structured and easily reused which is necessary if you think that at a later date you might want to do as you suggest.

    1. I was waiting for someone to bring up point 1. My answer for that is “if not now, then when?”

      There are ways and means to unpick any system, be it screen-scraping or even dirty old CSV import/export over FTP.

      If we don’t overcome this problem here and now, we are forever doomed to these broken monolithic systems.

      1. Exactly. When you see stories about report totals not matching payments you just know this is not engineered to modern standards. How could this occur if the code was modular and a particular calculation was only coded once???

  4. All good common sense Ben, but as with every prior attempt to ‘fix’ big (or any size) projects, common sense really isn’t that common – I say this from 20 years on mostly enterprise or ministry/govt projects.

    Beginning with imprecise/missing requirements, then fear and greed in the RFP stage, mixed with overly rigid or overly fluid design in the build phase that does not address future system complexity – *and* occasional abject incompetence (then arrogance and lies to cover up the failings) is what ruins projects.

    I’ve worked with some amazing architectural and design talents – and sound & robust systems have been dreamed up by these people, however, not all architects and designers are created equal – and like the captain of a ship, a bad one will put everyone on the rocks sooner rather than later. Incompetent/arrogant BAs, architects, designers and the PMs and/or stakeholders who employ, then back them, when all but those in the ivory tower know the project will fail because of them, have been the problem on every over budget/time project I’ve been on – that and outsourcing development to god knows where “to save costs”…

    If there was an *independent* audit on every ministry IT project over the last 10 years (for a start…) in NZ, the taxpayer would call for floggings as millions of dollars are blown and continue to be blown across the board. 99% of them just haven’t got the exposure of novopay.

    You can blame the vendor of course – and rightly so in some instances, but those that employed the vendor or ran the project independently, must also take responsibility. The problem is not and never has been the methodology or the technology, it’s always the ‘knowledge’ people running the show who get offended when their precious way of working and/or design is questioned by the people doing the actual work, because petty egos are hurt or they think it’ll make them look bad to their higher ups – or not enough margin will be made and corners are cut.

    Ask any lower echelon PM, BA, dev or tester how a project is going and you’ll get the real answer, not the political one spewed from the jealously protected single line of reporting to the stakeholders. How many times have you heard of key people leaving or being ‘removed’ from a project? I’ll bet cents to dollars these are always the people who question the status quo, get frustrated when shot down and leave or have their contract terminated by an offended petty tyrant. The flow on effects from this are a less knowledgeable team, lower morale and then more people leave…

    If you want to fix an IT (or any large) project, the process must be open, independently auditable and those with the ‘power’ must be open to change and not be afraid of being wrong… Perhaps an onsite psychologist would be a good idea? Or psychometric testing (or the like) to make sure individuals can work together? Also, learn why previous projects fail, there’s rafts of info out there…

    At the end of the day, everyone on the project wants it to work, so listening to history and to the team doing the actual work in an environment of open discourse – without repercussions – is what gives projects (or any endeavour) the best chance of success. Which, as you quite rightly conclude, is a “Bummer” and only happens on rare occasions.

    I don’t see anything changing any time soon, because the way things work currently is, those that fuck it up get promoted or are moved sideways because they can’t be fired. Either way, more of the same awaits, whether it makes the news or not.

    1. All so depressingly true Jo. The ass-covering from the Minister down is largely to blame here.

      A fixed-price project allows everyone to bury their heads in the sand and spend public money like it’s no big thing.

  5. Brilliant stuff. You have it exactly correct. I work for a large organisation that is drowning in systems that ‘mostly’ do what’s expected, but being all-integrated, fault finding is, in the extreme, impossible. Your solution would work in every case. I love it.

  6. Good points!
    I’m an infrastructure guy, not a coder, and I’m decades away from what little Systems Analysis training I’ve had, but surely the FIRST thing, before a single line of code is cut, is to get the data model nailed and signed off?
    That way everyone is coding to the same schema and you can modularize more effectively.
    You don’t want to be tacking on tables and attributes more than you need to and you don’t want hanging dependencies.

    One big criticism of Novopay has been that that referential integrity is handled in code. That’s nuts, considering it runs on Oracle. It means that every line of code written to modify an attribute of a record has to be validated against a CRUD matrix unless an API has been written to take care of it.

    I’ve seen this kind of thing in th
    e past with apps using ancient flat-file DB’s. Any API’s were often hideously proprietary and loaded with so many caveats that basically you tried to integrate at your own risk.

    Get the data structure right on a proper open DBMS with built-in referential integrity and it doesn’t matter if you have an elephant or an antigrav-platform

  7. Thing is, if it was simple, they’d just devolve payroll to the schools and they get a payroll app or contract a provider individually.

    The problem (as I understand it) is that if you teach 2 days a week at school A and 3 days at B, it counts as a single employment (for tax, seniority, holiday and much else). So it has to go through one system. But also, you report to school A *and* B, so they both need to control data on your hours and so on – and do so without breaking things for the other.

    This is not congruent with most other employments and does not admit of a particularly simple solution – and devolution might well make it worse.

  8. I heartily agree with what you are describing…

    The challenge is getting people in government/public service to change their mindsets, realise the depth of talent that is resident in New Zealand, and stop pretending that the big boys (IBM, Oracle, etc) have credibility.

    The company I work for had a measure of success with the Ministry of Defence a couple of years ago getting them to sign up to an agile contract. That being said, we’ve been dealing with another state-funded entity, and have had a very long period of contract negotiation getting them to go agile.

    Not that what you’re proposing is necessarily agile, but it is different to the status quo, and the status quo dies hard.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.