xignite

The 10 Commandments of Web Service Change Management

Web service governance covers many topics one of which is change management. How do you change and enhance existing web services without breaking the code of your existing clients? After more than five years experience delivering web services to some pretty mission-critical applications, Xignite has learned a thing or two about what good business practices are in that domain. We use those rules internally when managing our services and we communicate those to our clients as well. These rules have worked well in helping prevent problems. We compiled those into the Ten Commandments below.

But let's first remind what those Commandments seek to achieve:

1. Never break existing code. It's pretty clear that clients that have already integrated their apps with your web service don't like it when things stop working. They will be quite upset when they find out it is because you made a change without telling them (this happened to us with code we wrote against the famed API of a just-as-famous Saas CRM vendor).
2. Eliminate the need to update Existing code. Ideally, once clients have built something in, they should never have to change it. If that is impossible, the frequency and complexity of those changes should be kept to a minimum and clients should be given a time-frame to made those changes and multiple environments to test those changes against.

Ten Commandments of Web Service Change Management

1. Thou shalt never remove or rename a web service. This is pretty basic. If you change the location of the web service, all hell breaks loose. If you are going to rename a service, make sure you leave a proxy of the new one where the old one was.
2. Thou shalt never remove or rename a web service operation. With SOAP, the operation defines the SOAP action. With REST, the operation is part of the resource url. In either case, if your remove or rename the operation, the code will break. Because of the nature of web services, keeping "old" operations around for compatibility purpose is typically not a huge issue since only those clients using those operations are affected. We found it is a good practice to do so.
   
3. Thou shalt never rename a web service operation parameter. With SOAP, the operation parameters are part of the SOAP message. With REST, the operation parameters make up the resource url. In either case, the client code will break.
4. Thou shalt never add a web service operation parameter. Adding a new parameter to an operation will break your client's code since that code does not currently pass that parameter. If the client uses REST, you may be able to add logic inside your web service to provide a default value for that new parameter, if the client uses the tighter-controlled SOAP, the code will break right away since it will never get passed the server and not even enter your code.
5. Thou shalt never remove a web service operation parameter. Removing an operation parameter is one of the lesser sins on this list. If the client uses REST, their code will most likely continue to work since the extra parameter will be ignored by your new service. If the client uses SOAP, the result will depend on the behavior of the client's SOAP toolkit. Some may fail, some may ignore the additional parameters (For instance, client code written over .Net would continue working).
6. Thou shalt never remove an enumeration value on an operation parameter. Some values used as operation parameters are enumerations (i.e. a list of valid string values like 'Daily', 'Monthly', and 'Annually' could be the enumeration values of a member called PeriodType). Whether the client uses SOAP or REST, if the client code passes one of those values after you removed them from the valid list, their code will break.
   
7. Thou shalt never remove or rename a returned object member. Just like for #2 and #5, this is a very evil sin. If the client uses SOAP, their parser will break when not finding the member it is looking for. If the client uses REST, their code will only break if they use that parameter in their code. Either way, this one sends you straight to customer service hell.
   
8. Thou shalt never change the type of a returned object member. Although all data is exchanged over the wire as strings, changing the type of an object could have bad consequences based on the logic the client built on its side. For example, if you change type type of a field from Integer to Double, the code on the client side will probably choke on those decimals.
9. When adding new enumeration values on a web service member, make sure they are not returned by old operations. This is one of the trickiest rules on the list (and we found it the hard way). Note that adding new enumeration values is not a Commandment on the list. The reason is that new enumeration values are typically not returned by old operations.  For example if an operation value helps you identify a given type of interest rate or currency, if your client's "old code" does not know about this value, it will never request it and never expose the problem. But if an "old operation" can return "new enumeration values", the client's SOAP parser will fail on trying to interpret those new unknown values. If this is functionality you need to have, you'd better turn those enumerations into strings.
10. Thou shall pray a lot before adding a new web service member. This is the only Commandment I wish did not make the list. The other ones make sense in a technical kind of way. This one really should not be. You should be able to add new web service members to your heart delight without anguishing over the consequences. After all, if a web service returns a new data element, the code on the other side does not know about it and should be able to simply ignore it. But the cold hard truth is that some parsers (Axis to be specific) will throw an exception when parsing new values returned by a service. The good news is that Axis plans to address this problem. The bad news is until then you will have to pray and notify your Axis users.

Why SOAP Rocks!

SOAP has had a little bit of bad rap lately. Some people think it is too complicated compared to its cousins REST and POX (plain old XML). The truth is that it all depends on the development environment you use. To prove our point, Xignite made this video that shows how you can integrate a web service in your application in less than 60 seconds and 3 lines of code. Watch it on YouTube:

   

Here is a higher quality Flash Version.

Twitter For Transparency

A few weeks ago, I was part to a discussion with Frank Kenney from the Gartner Group. He was talking about the need of providing greater governance transparency with web services. He recollected the enormous backlash SalesForce.com experienced last year when their system went down for a long period of time. As a remedy, and as as way to regain lost client confidence, SalesForce created a new web site (trust.salesforce.com) focused on providing its clients with complete transparency on the state of its systems.

Frank commented that this approach not only enabled SalesForce to yet again set the standard on how Saas should be done, but it built significant goodwill with its clients. SalesForce clearly showed it had nothing to hide.

This issue is equally significant for Xignite as more and more clients run more and more mission-critical business process on our services. We have historically achieved great levels of availability. We have nothing to hide. This is why we are busy at work providing a similar functionality to our enterprise clients.

Meanwhile, I was wondering if we could not do something simple that would help our clients easily answer the "are you having problems right now?" question which we get once in a while when a client has a problem accessing our services. 99% of the time the problem has to do with configuration changes on their side but I don't blame fir first wondering if the problem is with us.

Here comes Twitter. Can Twitter do to web service governance what it has done for individual: bring total transparency on the availability of our systems?

You be the judge. Our support pages now boast a twitter-powered System Status widget that provide 5 minutes reports on system availability. If our primary web service farms runs into problem then the status is instantly be updated.

The great thing about twitter is of course that it lets any of our clients subscribe to our twitter in case they want to be updated in real-time of any changes to our system. Now that's transparency.

Private Service Oriented Networks: The Enterprise's Answer to Web 2.0

I will be attending the Web 2.0 / Enterprise 2.0 in the Capital Markets Industry conference in NYC on 9/17 where I will moderating a panel on Enterprise Web 2.0 in Financial Services. This is a small event but it attests to the growing interest the financial services industry is giving to the new wave of technologies that have hit the enterprise over the last 2 years.

One of the most interesting ideas that I have seen emerging around this subject is a concept I call Private Service Oriented Networks. Here is how it goes:

Financial service providers (FSPs) have long provided their institutional clients (money managers, pension plans, etc.) with technology layered on top of their financial services. For example, with custody services, they provide analytics on top of the clients portfolio holdings. For execution services, FSPs provide services and analytics covering the pre-trade and post-trade processes.

The point of providing this technology is to increase the firm's differentiation and the "stickiness" of the services--which are often very commoditized. In the "old world", such technology took some pretty basic form: daily data feeds made available to clients--mostly through FTP, at best some web applications. This method of delivery made sense because the clients themselves were running standalone client/server applications. Clients needed files to upload data to their own systems.

Although primitive, those technologies delivered quite a bit of stickiness. During my days at Advent Software, I remember talking to some clients  ready to move several hundred million dollars to a new custodian solely based on the availability of an interface between that custodian and the firm's portfolio accounting system.

But under the new Service Oriented model, this old way of doing business falls apart and many FSPs know it. I have been talking to more and more firms who are trying to find ways to "upgrade" the way they deliver technology to their clients for the brave new service-oriented world.

What they want is provide their clients with a collection of unique, private web services allowing them to build more powerful, more responsive applications. Forget about the old daily downloads, we are talking intra-day information here, whether it's real-time portfolio or trading analytics, the goal is to enable the institutional client to easily access all the data and functionality they to build their new generation applications.

So imagine such a Private Service Oriented Network providing you--the institutional client IT executive- with market data, account information, execution services, accounting services, analytics services. All through an open web service (SOAP/WSDL or REST/WADL) interface. Suddenly, your ability to deploy service-oriented business applications greatly increases. And to you, the benefits are significant: rather than building everything in house, you can start treating your FSP as an extension of your own IT organization while preserving complete flexibility and control about what makes your business unique.

This is a bit similar to what SalesForce.com has done by exposing its internals through web services. You could say that the SF.com API was the first and largest of the Private Service Oriented Networks. Now thousands of clients have built deep integration between their internal systems and SalesForce.com. This makes switching CRM system hard.

I foresee that over the next several years FSPs large and small will be deploying their own web service networks. And just like in the old days having or not having an interface to your clients system could make or break a deal, tomorrow's institutional assets are likely to flow in the direction of the FSPs with the best networks in place.

 

All You Need Is A Good Clean SOAP Toolkit

I am a late comer to the SOAP vs. REST debate, mainly because this blog started after the battle began raging. But a recent post by Raymond Yee--who is writing a book about mashups--suggested a reaction.

Raymond's main point is that SOAP requires a developer to have deep knowledge of the protocols (WSDL and SOAP) to use web services effectively and that it is "so much  easier to get started with REST."

I disagree violently. The knowledge of SOAP and WSDL you need all depends on the SOAP toolkit you use.

I'll take an analogy; Do we require internet users to know HTML to navigate the web? No we don't. On the client side, the browser takes care of turning HTML into beautiful usable web pages. If end users had to read HTML pages in plain text they would find the web very obscure indeed.

On the server side, if you are writing your HTML by hand, you will definitely need to know the details of the protocol. But if you use a good advanced IDE that lets you write pages in a WYSIWYG fashion, you can build a web site without knowing anything about HTML.

So it all depends on the tools (client and server side) and how well they hide the protocol from you.

If you want to write a web service by hand, you will have to know SOAP and WSDL. If you use a good toolkit, all you have to do is write a function call and you have a web service. If you want to consume a web service without a good SOAP toolkit, you will have to learn the standards but not otherwise.

We have written dozens of web services which collectively have served billions of SOAP requests to hundreds of firms around the world yet we have never read the SOAP specs and we can barely read a WSDL by hand. All we had to do was to write meaningful function calls (which is an art in itself).

Our clients who use good SOAP toolkit can go from pointing to our WSDLs to consuming data in their application in less than 60 seconds without a hint of what the protocols are about. This is not something you can do with REST yet (WADL might be able to change that).

Clients who come to us with more primitive languages will tend to spend more time working out the details of the integration.

It seems that SOAP's bad rap comes from those who just have not wanted to use more advanced SOAP toolkits.

It's very sad indeed. It is as if back in 1996 we had abandoned the promise of the web and stopped building web pages because a bunch of people insisted in using a text editor instead of a browser to open HTML pages and complained about how unreadable this HTML was. Come on.

To his credit, Raymond is very open to being convinced. He wants to "be shown where my thinking is wrong  and how I can start using SOAP and WSDL with ease in PHP, Python, etc."

Unfortunately, and with due respect to the fantastic tools PHP and Pythons are, they are not on the top of my list of the "better SOAP toolkits"--at least in their editor form. We had to handhold too many developers using those to have them make the cut. I suppose that if you use a nice IDE on top of those, things are easier. But they don't require you to know SOAP and WSDL to make them work.

Web Services Availability: What's Right

In a recent post, Dan Farber and Larry Dignan talked about the fact that there is still much ground to cover before all major Internet properties will provide the "five nines" (99.9999% availability). Their key point was that today's Internet services are not "ready" for the massive wave of utility computing ahead and that more money and intelligence needed to be thrown at those problems.

Their post talks about "web services" in more general consumer sense than ours (where we mean XML web services), but the comment certainly applies. With more than 300 clients in 25 countries and upwards of 500 million web service requests a months, our web services are powering more and more mission-critical business processes buried deep inside the core fabric of our clients' business. This of course raises the question of the availability of those services and what's right given the maturity of the market.

Web services as a "computing utility platform" for business are at least 10 years behind the web as a utility platform for consumers. That wave started in 2005 while the web goes back to 1995. So it's normal to expect a lower level of maturity. Our experience to date has been that clients and prospective users have been more and more sensitive to availability issues but that guaranteeing "three nine" (99.9% availability) meets all current business needs. I expect that the market will be requesting another "nine" every year for the next 4 years (2008->99.99%, 2009->99.999%, 2010-> 99.9999% and maybe more).

There is one key reason for that. Few of our client business processes run themselves in a "fine nine" mode. Many processes are mission critical but they are mission-critical once or several times a day, giving plenty of time for retries and recovery. For instance, one of the world's largest corporate lenders gets some interest rates information from us. This info is used to feed some homegrown systems that decide how much interest corporate clients gets charge for tens of billions of dollars in corporate loans. That's mission critical. But the process runs once a day (replacing an old unreliable manual process). As long as our system is up within a 1/2 hour timeframe every day, all is well. In truth, we achieve more than 99.99% availability, with many monthly periods at 100% (we have been publishing availability data on our site since 2004. You can find it here). But it's almost irrelevant to this client.

The key is to map the service level to the needs. Why going to the trouble of providing 99.999% availability if your clients don't care at all or don't care yet? Running redundant data centers is extremely expensive and difficult to do for an emerging company. Your clients probably want you to spend your money doing other things firsts. But once they begin asking for more "nines", you'd better get moving.

And so we are planning for our redundant data center--which we will need to get to the "five nines".

Making Composite Apps Work

I recently had the opportunity to meet with the senior IT management team of one of one of the countries’ largest commercial banks. This was an open, round table discussion involving multiple Silicon Valley startup companies who—like us—try to woo similar-sized enterprises into using their latest and coolest wares. The CTO started saying that in the old days, standard software was supposed to meet 80% of a firm’s need, with the remaining 20% requiring customizations to fit the firm’s process. “With packaged software these days”, he said “it is more like 50%/50%. This forces us to spend a lot of time customizing things, while we are actually trying to be more agile and more responsive to changing market conditions.”

This comment struck a chord with me. Many IT executives now understand that re-usable web services finally offer the promise to address those changing needs. Instead of buying a standard app that only does 50% of the job, the firm can build a custom application from re-usable web services. If you can buy most of the services you need, you get the best of both worlds: the fit of a custom app for less than the price of a standard app.

That’s all nice and good, but there is a big assumption built in: you must be able to purchase most of the services you need without having to customize them much. Otherwise you are back to where you started. And there lies the problem with the state of the web service industry today. Dana Gardner made it clear on his ZDNet blog last month: “There are either not many services available or the services are too general and not specific to a specialized vertical industry or a niche function.”. Dana further states that 80% of those services need to be customized. This type of problem could seriously hamper the success of composite applications in IT America.

Since we have been delivering web services for a while, we saw that problem coming a while back. This is why we not only offer a broad and deep set of integrated web services sharply focused on a specialized industry and function (financial market and industry data--in case you did not notice), but we also engineer those services with the needs of the composite application designer and developer in mind. Our goal is to make those re-usable components so integration-ready that you just have to drop them “as-is” in your composite applications. We still have ways to go, but we will get there.

And what’s the payoff you say? Gardner again puts it best: “The higher [developers] can drive the percentage of reuse, the more scale and productivity they gain. They become the go-to organization for cost-efficient applications in a specific industry, or for specialized business processes.”

What's This Blog About?

This is the official blog of Xignite (pronounced "X-Ignite").

About This Blog

Because we are one of the few "pure-play" providers of financial web services, this blog will cover any of the following topics: web services, web 2.0, enterprise mashups, XML, SOAP, REST, POX, AJAX, RIA, web services standards, Microsoft.Net, financial services, market data, financial applications, industry data and many more.

The blog is mainly written by Stephane Dubois, our CEO but may also include guest contributors from the rest of the firm.

About Xignite

We provides financial web services for mission-critical corporate applications. Our mission is to power and inspire the next generation of service-oriented financial applications.        

We operate on the new frontier of the $7 billion financial market data industry. We lead the market for “financial web services” – a term we coined when introducing our first offering in 2003 to describe a new way of delivering business-critical financial market information to applications.

We offer the broadest integrated suite of financial web services available by subscription on an on-demand platform. With more than 50 solutions covering market quotes, news, corporate data, industry information, analytics, statistics and more, we empower clients who prefer to avoid the costs and integration pains common to traditional approaches and outdated technologies.