Skip to content

Instantly share code, notes, and snippets.

@smsohan
Last active July 31, 2016 21:19
Show Gist options
  • Save smsohan/40f979cf06141fb578b8c232f13a6412 to your computer and use it in GitHub Desktop.
Save smsohan/40f979cf06141fb578b8c232f13a6412 to your computer and use it in GitHub Desktop.
A Case Study of Automated Example Driven REST API Documentation at Cisco using SpyREST
Abtstract
what is the problem? -> REST API documentation
why case study? -> to see how it can be used in practice
what was done -> used in a commercial setting
how does it help -> how it could be used in other place, and implications for the design of automated documentation tool
Introduction
Why we need automated REST API Documentation
The Problem at Cisco -> Background information
SpyREST
Adopted as a case study
Contributions
Layout of the rest of the paper
Related work
API documentation
Augmenting API Documentation with Insights from Stack Overflow: Robillard: Auto relate API documentation with Stack Overflow Q & A. " Some of the most
severe obstacles faced by developers learning a new API are related to its documentation [32], in particular because of scarce information about the API’s design, rationale [31], usage scenarios, and code examples [32]."
REST API documentation
Example driven documentation
Case studys on API documentation in practice
Flow:
Studies to understand API usability -> documentation
Research to combine formal API documentation -> Usage examples
Research to find usage examples from StackOverflow, open source repositories
Research to extend learnings into Web APIs
MDA, hRESTS, Swagger, RAML, WADL -> Synthetic concept retrofit vs. work as is.
Microservices
Extracting Models from Web API Documentation: Aims to define a structure to represent non-SOAP Web APIs. The structure contains opertations, parameters, input and output formats. No mention of grouping of operations under resources, and the use of HTTP headers, Verbs etc.
REST API Management and Evolution Using MDA: Versioning + structure.
A Maturity Model for Semantic RESTful Web APIs: JSON-LD(HATEOS), HYDRA
Shortfall
SpyREST
Core concept
System architecture overview
Case study
The API
The team
The process
SpyREST adoption
Example code
Screenshots
Overview/explanation
Auto generated documentation
Feedback from API developers
Process
Informal
Limitation that the API developers are co-workers
Semi-Structured questionnaire / Focused groups -> sharing and collaboration
How long they used
Years of experience with REST APIs
Things worked well
Efficiency
Effectiveness
Ease of use
Things that didn't work well
Troubles
Usage Summary
Discussion
Lessons learned:
1. Auto generated REST API documentation from functional test code work well in practice.
- Developers write test code anyway.
- With real examples, meaningful data is captured for documentation.
- The structure of REST APIs payloads do not always follow strict schema.
2. Custom content is used to provide overview.
- Such as API access information, rate limits, error handling and error codes etc.
- Changelog
3. Continuous update helps API evolution, before and after release.
- Documentation helps dev and QA teams work in parallel.
- Late changes before release automatically reflect on the documentation.
- Documenting a new version only needs to run the same tests against the new version for API elements that aren't affected.
4. API documentation needs same release pipeline as the API.
- Dev, Staging environment(s), production environment(s)
5. Consistently repeatable API examples are needed.
- Need seed data for API objects that aren't accessible through the API.
- Need a setup/teardown for API examples that change state.
6. Version controlling custom content is challenging.
- Custom content is edited on the wiki, outside of the code.
7. Confidential information need to be obfuscated from documentation.
- API credentials
- API example data such as internal URLs, emails etc.
8. Manual changelog can be time consuming.
- Done based on the test code.
Generalizability:
The concept of generating REST API documentation with API structure and usage examples by intercepting API calls:
1. Can be used to document other REST APIs.
2. Can be applied to non REST APIs.
Conclusion
Future work
Parts to decribe documentation
A Case Study of Automated Example Driven REST API Documentation at Cisco using SpyREST
Key reviewing criteria are listed below. Note that not all criteria are appropriate for every submission — e.g., improvement on the state-of-the-practice may be irrelevant for an experience report — and that we will adjust criteria to fit the given type of submission.
Relevance to ICSE SEIP audience
The core concepts of the work either originate in research, either at ICSE or a related conference, or are novel ICSE-appropriate topics.
The core concepts originated in research as described in the ASE papers. Also, API and API documentation are topics of interest to the ICSE audience.
Improvement on the state-of-the-practice
The amount of improvement that the work achieves above and beyond the state-of-the-practice.
1. Automated REST API documentation.
2. A novel approach using HTTP proxy servers to intercept example API calls.
3. Qualitative feedback from developers.
Impact of tech transfer activity
The scale of the impact (e.g., individual vs team vs several teams) of the tech transfer work (only relevant for tech transfer activities).
Backstory
1. Commercial API for accessing and controlling the features of a cloud based cyber security application.
2. API is developed and maintained by a team of engineers
3. API is used by 100+ customers
4. 70K+ API calls/day
5. SpyREST has been used for 1.5 years.
6. Documents two APIs, and 2 versions for each API.
7. In total 58 API endpoints, 100+ examples.
8. A team of engineers involved.
9. Cloud and on-premise options.
Generality of results
The probability that the work, approach, or lessons learned are applicable to developers outside of the studied group.
Clarity of lessons learned
The clarity in which the lessons learned are presented and how well they are supported with data and discussion.
1. Auto generated REST API documentation from functional test code work well in practice.
- Developers write test code anyway.
- With real examples, meaningful data is captured for documentation.
- The structure of REST APIs payloads do not always follow strict schema.
2. Custom content is used to provide overview.
- Such as API access information, rate limits, error handling and error codes etc.
- Changelog
3. Continuous update helps API evolution, before and after release.
- Documentation helps dev and QA teams work in parallel.
- Late changes before release automatically reflect on the documentation.
- Documenting a new version only needs to run the same tests against the new version for API elements that aren't affected.
4. API documentation needs same release pipeline as the API.
- Dev, Staging environment(s), production environment(s)
5. Consistently repeatable API examples are needed.
- Need seed data for API objects that aren't accessible through the API.
- Need a setup/teardown for API examples that change state.
6. Version controlling custom content is challenging.
- Custom content is edited on the wiki, outside of the code.
7. Confidential information need to be obfuscated from documentation.
- API credentials
- API example data such as internal URLs, emails etc.
8. Manual changelog can be time consuming.
- Done based on the test code.
Overall quality of the manuscript
While we intentionally do not limit submissions to certain categories, we do encourage potential authors to view ICSE SEIP 2016 list of papers for examples of appropriate submissions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment