The purpose of the API is to allow automated testing to be tracked and reported in testlink much like manual tests are. In addition it could be used to provide a hook into an automated build system so testers are aware of new work asap. Testlink tracks test sets in two ways. First is test suites. These represent your entire pool of available tests. Second is test plans. Test plans represent some set of tests you need executed by your testers or your automation system and are taken from the test suites. Test plans are executed against builds of your software on one or more platforms (eg windows and Mac)
So with that background we can come up with some ideas for what we need in an API and compare with what testlink actually provides. Let's assume that you already have a working automated build and test system but need better reporting because there is too much data to work with without a tool to help. Since this system is already running and more or less fully automated it's pretty unlikely that anyone would be willing to manually add anything major to the database such as test case information and such.
Given that... Here's what I would expect a reasonable API to provide.
- Create a test case within a suite which may be part of another suite. Also create the suites if they don't exist.
- Create a test plan containing a list of pre-created test cases. Update as appropriate.
- Assign results to tests in a test plan run against a build for a platform.
- Create a build and associate it with a test plan (or several)
Not too much eh? The nice thing is that this hides a great deal of complexity which is handled manually in normal testing. Hiding complexity is always a major goal for a good API. Another advantage is that this concept keeps requests larger and less frequent. When dealing with distributed systems you pay a fixed time cost for every request made. That cost goes into negotiating connections, security, protocol headers and so on. So, fewer requests to a web API is usually faster.
Sadly testlink makes a very easy mistake in its API design. The developer chose to expose every simple step used in creating test plans and recording results except for a few cases where the API is more difficult to use than necessary. The list of actions look something like this
- Create a single test suite optionally belonging to another (bad stuff happens if the suite already exists)
- Create a test case belonging to a previously created test suite (ditto)
- Create a test plan
- Create a build
- Create a platform
- Assign a platform to a test plan
- Assign a test case to a test plan
- Record a result for a test case in a test plan run against a build for a platform
- Yadda Yadda blah blah.
So. A user of this API needs to jump through the same hoops as they would when doing things manually. While this saves mental effort in designing the API you end up paying for it every time you have to write code to check that a suite (and the suites it's contained in) exists and create everything that's missing via a dozen or so function calls. Not fun.
Think of an API as a time investment. The more menial bullshit your API does for you the less you have to do when using it.
One last example. A test case in testlink has a number of properties such as its author, a description and the steps that are taken to carry out the test. When you use the testlink API you provide a key (a bunch of gibbrish) to get access. The key is unique to you. Now when calling an API to create a test case using your key the API ought to be smart enough to assume you are the author unless told otherwise. The version of testlink I'm using fails to make this leap. This means you need to modify the API (I did this, small win for Open Source) or have both a user name and their key to use the API for just this one function. Again if an API can reasonably do something for a user it should.
No comments:
Post a Comment