CSC 405 Lecture Notes Week 4
Some Project Specific Testing Details
Relevant Testing Theory



  1. Mapping Testing Terminology onto 405 and 406
    1. Consider last week's testing activities.
    2. What follows are notes on how they'll be accomplished in 405 & 406.
    3. These are notes, not the answers.


  2. Unit Testing
    1. For the OCU, CppUnit looks like a good choice.
      1. Originally a port of JUnit to C++.
      2. Start at sourceforge site.


    2. For the kernel-level code, unit testing may not be done independently.
      1. See the "Linux Test Project" for a start.
      2. It may provide some unit testing.
      3. It also is at sourceforge.


  3. Module Testing
    1. CppUnit provides test fixtures.
    2. It provides other class-level testing features.
    3. See the CppUnit "Cookbook".

    4. LTP provides support for kernel module testing.
    5. There is also the scalable testing platform.


  4. Integration Testing
    1. OCU and kernel-level independently tested.
    2. OCU teams write stub for kernel model.
    3. Not clear what kernel stubs may be possible.


  5. System Testing
    1. The issue of kernel builds needs to be worked out.
    2. Also, the issue of how far to go with competing 802-11 and Batman needs to be determined.


  6. Acceptance Testing
    1. Once OCU user-level spec is done, write acceptance spec from it.
    2. If possible, have actual human OCU users conduct acceptance tests (or Julie as customer rep).
    3. STP claims to provide kernel acceptance testing.


  7. Black Box Testing
    1. The OCU may be black-box testable in its entirety.
    2. For kernel-level, black-box testing is driven by specs, as possible.


  8. White box
    1. OCU may not need explicit white-box tests.
    2. For kernel, it depends on what LTP has to offer, or comparable testing frameworks.


  9. Testing Design
    1. To be determined based on chosen frameworks.
    2. For the physical layout, I suggest standard UNIX directories, a la the CSC 309 testing repository.


  10. Test Plan
    1. I strongly recommend a common, uniform structure for these
    2. C/C++ Function comment for unit test plans.
    3. C++ .h file comment for module test plans.
    4. Source-resident .html for integration plan.
    5. Use doxygen (or equiv) for generating browsable test plans.


  11. Top-down Testing
    1. Test OCU top-down, with MESH stub.
    2. OCU model unit tests may be done bottom up, as necessary.


  12. Bottom-up Testing
    1. Test kernel-level bottom up / inside out.


  13. Test Case
    1. Again, I recommend having a standard test design convention for how this is represented.


  14. Testing Oracle
    1. For OCU, given absence of formal(ized) spec, the oracle will be a human domain expert.
    2. For kernel-level, the extant specs are used, with human interpretation as necessary.


  15. Testing Stub
    1. As noted above, major stub is for OCU testing.


  16. Test Driver
    1. Provided by the chosen frameworks.
    2. Also, if dynamic testing feature is implemented, the OCU is a live run-time test driver.


  17. Regression Testing
    1. CppUnit is both programmatic and file based.
    2. Kernel-level is most likely log file based.
    3. STP may provide regression support.


  18. Test Coverage
    1. gcov is a widely-used C coverage tool.
    2. LTP has lcov, a graphical front-end to gcov.


  19. Test Subsumption
    1. OCU is simple enough so as not to need this.
    2. LTP may have some support, but not likely.


  20. Test Automation
    1. Done with make, per the requirements of the chosen frameworks.


  21. Mutation Testing
    1. Most likely not performed in 405.
    2. We may do some in 406


  22. Testing Harness
    1. Covered above, particularly under Test Drivers.


  23. Testing Framework
    1. To summarize ...
    2. CppUnit for OCU testing.
    3. LTP for kernel-level testing.
    4. Other possibilities are CxxTest, DejaGnu.


  24. Resources (see 405 main index for links)

  25. Design and testing best practices.
    1. Concrete structure in Fisher's 308,309 Standard Operating Procedures (SOPs):
      1. Vol 1 -- Repository Structure
      2. Vol 2 -- Design & Imple'n Conventions
      3. Vol 3 -- Testing Conventions


    2. Widely-used tools:
      1. CppUnit
      2. Gcov
      3. Linux Test Project, Lcov


    3. An exemplary site for testing "best practices"
      1. Check out the SQLite Testing site at www.sqlite.org/testing.html
      2. Its testing design and infrastructure look a lot like what we'll need for 405 and 406 project testing.