Hello again! How have you been?
I for one have had an eventful year and a half, to say the least, but let's not get into that here. I'll see if I can hack the whole story into a slideshow next Fanfest, worst case you buy me a beer and I spill my guts.
Anyway. We just want to give you some information about the coming release, Incarna, where we'll be deploying a few updates to the API. These updates are likely to affect your applications, my apologies for how late you're getting this information as we have been extremely busy little bees lately, and writing this up kinda fell through the cracks.
Bigger Data
As you probably know from reading recent devblogs, our data-hamsters have been changing datatypes here and there in order to extend the capacity of various datasets. This is of course something we have to follow up on in the API, and you too if you're working in a strongly typed environment or your application has anything limiting these variables' sizes, e.g. validation logic. So here are the changes by pages:
MarketOrders - orderID is now a 64bit integer instead of 32, and typeID is now a 32bit integer instead of 16.
WalletTransactions - transactionID is now a 64bit integer instead of 32. This applies to both char/ and corp/ pages, .csv and .xml.
Smaller Data
MarketOrders will now only return a list of active and non-expired orders. This call was fetching way too much information from the datastore and we decided to alter the query in a simple way, filtering out all expired and processed orders. This is potentially a big change but we figured most people are calling this to watch their active orders and for historic information they get their transactions. We're also quite sure that you'll let us know if we're wrong, as you always so diligently do.
Transaction Walking
As you may or may not know, we've refactored the inner workings of the transaction walker which is used on all the WalletTransactions pages to page back through transactions. The new implementation had a problem where getting the latest transactions had a "speed limit". If you or your corporation produced more than X transactions between cache refreshes then the API would only give you the oldest X of those. Now, on the other hand, the API will give you the latest X and then you can page back to get the rest of the latest transactions.
Error Throttling
We are constantly looking at how we can make the API run better and ensure a consistent quality of service. Usually we look inside the API at how it queries and caches data, but this time we took a look at it from the outside, specifically how erroneous API traffic is affecting the API's performance. Turns out there is a substantial amount of calls creating errors that can be avoided so we came up with a way to minimize them. It is quite simple and works like this: If your IP produces X errors in Y minutes the API will block you for Z minutes. If your IP gets blocked then you'll get an error message with the new error code 904, it would look something like this:
2011-06-21 13:18:52
Your IP address has been temporarily blocked because it is causing too many errors. See the cacheUntil timestamp for when it will be opened again. IPs that continually cause a lot of errors in the API will be permanently banned, please take measures to minimize problematic API calls from your application.
2071-06-21 13:21:50
We are going to be pretty strict about this but we'll ease up on it if the forums catch fire. The basic theory we used here is that if nobody is complaining then we're not strict enough. It's the same thing governments use when introducing new taxes - to find a good balance you need to be able to measure! So we're going to start with: if your IP passes 3 errors a minute your IP gets blocked for 3 minutes. We decided against 60 year blocks as the example above suggests, we're not sure our servers have that kind of an up- and/or life-time, although they are doing wonderful things with hot-swapping these days.
Note that the IP will NOT be blocked in the game itself or on other websites. The throttling only applies to the API.
The Error Throttling may very well affect your application, especially if it is fetching a lot of data from the API for many different players and corporations. By doing this we aim to protect the API from erroneous traffic that can be avoided by forcing 3rd party developers to design their applications in a way that minimizes repeated calls that return errors. This means better quality of service for everyone so we hope this will be well received by the masses.
At the time I'm writing this, we've deployed this to the SISI API so that you can test the changes. Note that this version will NOT support Customizable API Keys because we are not deploying that in this release. After the release we'll integrate the CAK back into the API and set that up on SISI so you should have plenty of time to play around with it until its late August release.
I think that's it for now. Again, my apologies for the tardiness of this blog, the Error Throttling alone may take time to tackle but that's something we can easily tune down for a while if you're having problems running your applications.
Happy coding!
- CCP Elerhino
More...
I for one have had an eventful year and a half, to say the least, but let's not get into that here. I'll see if I can hack the whole story into a slideshow next Fanfest, worst case you buy me a beer and I spill my guts.
Anyway. We just want to give you some information about the coming release, Incarna, where we'll be deploying a few updates to the API. These updates are likely to affect your applications, my apologies for how late you're getting this information as we have been extremely busy little bees lately, and writing this up kinda fell through the cracks.
Bigger Data
As you probably know from reading recent devblogs, our data-hamsters have been changing datatypes here and there in order to extend the capacity of various datasets. This is of course something we have to follow up on in the API, and you too if you're working in a strongly typed environment or your application has anything limiting these variables' sizes, e.g. validation logic. So here are the changes by pages:
MarketOrders - orderID is now a 64bit integer instead of 32, and typeID is now a 32bit integer instead of 16.
WalletTransactions - transactionID is now a 64bit integer instead of 32. This applies to both char/ and corp/ pages, .csv and .xml.
Smaller Data
MarketOrders will now only return a list of active and non-expired orders. This call was fetching way too much information from the datastore and we decided to alter the query in a simple way, filtering out all expired and processed orders. This is potentially a big change but we figured most people are calling this to watch their active orders and for historic information they get their transactions. We're also quite sure that you'll let us know if we're wrong, as you always so diligently do.
Transaction Walking
As you may or may not know, we've refactored the inner workings of the transaction walker which is used on all the WalletTransactions pages to page back through transactions. The new implementation had a problem where getting the latest transactions had a "speed limit". If you or your corporation produced more than X transactions between cache refreshes then the API would only give you the oldest X of those. Now, on the other hand, the API will give you the latest X and then you can page back to get the rest of the latest transactions.
Error Throttling
We are constantly looking at how we can make the API run better and ensure a consistent quality of service. Usually we look inside the API at how it queries and caches data, but this time we took a look at it from the outside, specifically how erroneous API traffic is affecting the API's performance. Turns out there is a substantial amount of calls creating errors that can be avoided so we came up with a way to minimize them. It is quite simple and works like this: If your IP produces X errors in Y minutes the API will block you for Z minutes. If your IP gets blocked then you'll get an error message with the new error code 904, it would look something like this:
2011-06-21 13:18:52
Your IP address has been temporarily blocked because it is causing too many errors. See the cacheUntil timestamp for when it will be opened again. IPs that continually cause a lot of errors in the API will be permanently banned, please take measures to minimize problematic API calls from your application.
2071-06-21 13:21:50
We are going to be pretty strict about this but we'll ease up on it if the forums catch fire. The basic theory we used here is that if nobody is complaining then we're not strict enough. It's the same thing governments use when introducing new taxes - to find a good balance you need to be able to measure! So we're going to start with: if your IP passes 3 errors a minute your IP gets blocked for 3 minutes. We decided against 60 year blocks as the example above suggests, we're not sure our servers have that kind of an up- and/or life-time, although they are doing wonderful things with hot-swapping these days.
Note that the IP will NOT be blocked in the game itself or on other websites. The throttling only applies to the API.
The Error Throttling may very well affect your application, especially if it is fetching a lot of data from the API for many different players and corporations. By doing this we aim to protect the API from erroneous traffic that can be avoided by forcing 3rd party developers to design their applications in a way that minimizes repeated calls that return errors. This means better quality of service for everyone so we hope this will be well received by the masses.
At the time I'm writing this, we've deployed this to the SISI API so that you can test the changes. Note that this version will NOT support Customizable API Keys because we are not deploying that in this release. After the release we'll integrate the CAK back into the API and set that up on SISI so you should have plenty of time to play around with it until its late August release.
I think that's it for now. Again, my apologies for the tardiness of this blog, the Error Throttling alone may take time to tackle but that's something we can easily tune down for a while if you're having problems running your applications.
Happy coding!
- CCP Elerhino
More...