CityTime was launched in 2003 at a budget of $63 million, but costs swelled dramatically as the project stumbled along for nearly a decade.
From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.
In these types of projects it seems like no one wants to point the finger. Once costs have doubled, or tripled, everyone is afraid of taking the blame for wasting all that money. So they keep 'working with' the vendor.
Clearly fraud and kickbacks kept this stumbling along much longer than a normal failing IT project would. I'm not sure why or how an expense like that on an overage like that on a major payroll system would not be on the mayor's daily agenda. The right people must have been getting those kickbacks.
I've been on the wrong end of a couple of projects like this, and in my experience (not necessarily indicative of this case) the customer can be VERY at fault.
I've been dragged into 'feature acceptance meetings' that literally dragged out for weeks when testing of a new chat component devolved into a days-long discussion on whether the graphic in the header (provided by the customer) was appropriate. The unspoken issue proved to be that they didn't want us to use 'barren trees' in the picture of their building. We switched the photo used from one taken in winter to one taken in summer. This took something like 400 man-hours to resolve, after which we achieved signoff, but at the expense of having met budget.
Another instance on the same product was feature creep to the extreme. We had originally won a firm-fixed contract, meaning we would build products a, b and c for x dollars, and if it cost us more, it came out of our pocket. If it cost us less, we pocketed the profits. The customer decided, on day 1, that they actually wanted a lot more features than were described in the project scope defined, and that we won a legally binding contract to perform. Out of accomodation, we changed their product to a time-and-materials contract, meaning we had a fixed markup on labor and a minimal markup on hardware / tangibles. This then involved them dragging the project out for years longer than anticipated and signing invoices all the way, then griping at us on the budget near project completion and asking for a bunch of free labor (which is illegal under their new contract). The problem in that case was that the project management and senior execs changed repeatedly. We were at first working with their Office of Science and Technology, then with Information Services, then with Public Affairs. Each of these department changes involved swapping our equipment out so that we could have appropriately charged-back equipment to the appropriate department, which meant weeks of non-work. Each department had a different spin on what they wanted out of the project. We spent 8 months and thousands of man-hours implementing a very elegant services oriented architecture that could surface all of their legacy systems to a single web portal. Then we changed departments, and they literally told us to turn that feature off, and instead focus on a visual redesign. We implemented a single sign on service that we integrated with our product, but could be extended to work with literally every system in their enterprise. It was abandoned due to a department change.
During the first phase of the project, we'd worked with the customer to build out their data center. They wanted a new data center because the division that owned the project was going to be adding a bunch of other programs to integrate with what we were building. They spent over a year on construction, converting a service garage into a data center. They bought millions of dollars worth of EMC storage hardware and Cisco routers. Then they realized that they didn't have any budget planned for out-year maintenance, so they moved the venue. That was somewhere in the realm of $50 million dollars that was literally just thrown away, as the last I heard, they were re-converting the data center back into a garage.
In short, government contractors catch a lot of flack, because they control the visibility of these things. The contractors often can't speak to details of these projects due to security clearances or NDAs, and at the end of the day, the government is always happy to blame the vendor. It's well practiced, and designed with a variety of scapegoat clauses built in.
... it's just the article notes a subcontractor has already been convicted of fraud and kickbacks, so this might not have been a project quite like the projects you describe (though if this is a Terry Childs-type case, I'd love to hear about it).
Sorry, I meant to imply that that's just the way things are, however, after the realization that he'd been convicted, it's likely that whether he's done anything wrong on this particular project, he's drawn the scapegoat straw.
I meant to say those things, but got lost in typing so much other nonsense and trailed off. Thanks for bringing it up.
I wouldn't be so sure. One thing that working in government has taught me is that the bad contractors seem to be equal or greater proportion to the good.
They would bid on something if they weren't going to find a way to make more than what the original contract was for because in the bid wars they often lower the bids so much they have to recoup somehow. In this case they probably did not build the recoup method into the contract and did it illegally.
Everywhere I go I end up on SSO projects. I always swear I'll avoid them, but they seem to jump out at me when I least expect it.
Your assessment of the software and services seems fairly accurate to me (there are a couple of exceptions: CAS is simple enough to be easily understandable and works ok, and Atlassian Crowd looks ok too).
From personal experience I can confirm that this is exactly how it works when governments try to do things. There's such a compulsive need to negate risk that everything is all about the process instead of the results - and inevitably this leads to huge amounts of wasted time and money while people try to decide the right thing to do.
On a previous project for a very large automaker, it took me 5 full days plus a 1 week trip to 4 cities in China to change a single graphic. (The rational was that they wanted the client to talk directly with the engineers instead of having to go through all the corporate bureaucracy. So much for that. ;)
Replacing the logo itself took all of about 30 minutes (receive new picture, digitally retouch, overlay onto background image, resize, play around with transparency, save to png.)
Also, bear in mind, the 400 hours is 400 man-hours, which is a cumulative number based on the number of people involved. If you've never done business with government, let me assure you, it was a lot more people than necessary.
Absolutely. The man hours can easily pile up when you do a conf call with a half dozen folks where majority of them are on mute 90% of the time.
I recently did contract work for an agency and had instances where a simple spacing or image resizing took days of work because the client or the agency would understand and then forget that I must crop and/or resize images for them all to be of one size.
This is a common scenario. Government picks the lowest bidder, but the price can go up every time there is a change in specs. Then every government agency imaginable wants to decide what color to paint the shed and the whole thing bogs down, sometimes permanently. It's amazing that on this project they eventually managed to deliver a working system.
That's only about a 30% growth per year though, which may not seem so ridiculous on a year-over-year basis. I can see how such a mistake could be made, though it is pretty crazy.
From $63M to $600M? I would have thought that ppl may have noticed when the costs doubled or tripled the budget. Almost 10x the original budget? That type of oversight must go pretty high up.