tP: DeliverablestP: Practical Exam


tP: Practical Exam Dry Run

tP: Practical Exam Dry Run (PE-D)

PE-D Overview

What: The product version released by the v1.5 iteration is subjected to a round of peer acceptance/system testing, also called the Practical Exam (PE) Dry Run as this round of testing will be similar to the graded Practical Exam that will be done at v1.6.

When, where: uses a 60-minutes slot at the start of week 12 → 13 transition point (i.e., Fri, Nov 8th 1600) lecture slot. It will be conducted via Zoom. If you can't find another suitable location to join the Zoom meeting, you may do so from the lecture venue.

Grading: The PE dry run affects your grade in the following ways.

  • You can earn up to 2 marks (in your final tP score) for your performance in the PE-D.
    Graded based on the usefulness and the quality of your bug reports, as evaluated by the receiving team.
  • PE dry run is a way for you to practice for the actual PE.
  • Taking part in the PE dry run will earn you participation points.
  • There is no penalty for bugs reported in your product. Every bug you find is a win-win for you and the team whose product you are testing.

Why:

  • To train you to do manual testing, bug reporting, bug bug fixing, communicating with users/testers/developers, evaluating products etc.
  • To help you improve your product before the final submission.

PE-D Preparation

  • Ensure that you have accepted the invitation to join the GitHub org used by the course. Go to https://github.com/nus-cs2103-AY2425S1 to accept the invitation.

  • Ensure you have access to a computer that is able to run course projects e.g. has the right Java version.

  • Ensure you can use CATcher on your computer. You should have done this when you smoke-tested CATcher earlier in the semester.

  • Have a good screen grab tool with annotation features so that you can quickly take a screenshot of a bug, annotate it, and post in using CATcher.
    You can use Ctrl+V to paste a picture from the clipboard into a text box in a bug report.

  • [Optional] Have a good screen recording tool if you plan to use screen recording clips as part of your bug reports. Ensure that your screen recording tool can create small files as CATcher doesn't allow files bigger than 10Mb.
    As the CATcher support for uploading screen recordings is new and limited, use it only if strictly necessary -- use screenshots for other cases.

  • Download the product to be tested.

    PE Dry Run (at v1.5)
    • After you have been notified which team to test (likely to be in the morning of PE-D day), wait until you are no more than 2 hours from the PE-D start time, and download the JAR file and the UG/DG PDF file from the team's latest release. Downloading the files any earlier runs the risk of you testing an outdated version of the product.
    PE (at v1.6)
    • After you have been notified of the download location, download the zip file that bears your name. The password required to unzip it will be given to you at the start of the PE.

    Testing tips

    Use easy-to-remember patterns in test data. For example, if you use 12345678 as a phone number while testing and it appears as 2345678 somewhere else in the UI, you can easily spot that the first digit has gone missing. But if you used a random number instead, detecting that bug won't be as easy. Similarly, if you use Alice Bee, Benny Lee, Charles Pereira as test data (note how the names start with letters A, B, C), it will be easy to detect if one goes missing, or they appear in the incorrect order.

    Go wide before you go deep. Do a light testing of all features first. That will give you a better idea of which features are likely to be more buggy. Spending equal time for all features or testing in the order the features appear in the UG is not always the best approach.

    PE-D During the session

    Use the CATcher Web app for reporting bugs. More instructions will be given during the PE-D e.g., which session to use.

    Use MS Teams (not Zoom) to contact prof if you need help during the session. Use Zoom chat only if you don't get a response via MS Teams.

    How many bugs to report?

    Report as many bugs as you can find during the given time. Take longer if you need (unlike the PE, PE-D is not timed strictly). If you can't find many bugs at this stage when the product is largely untested, you are unlikely to be able to find enough bugs in the better-tested final submission later. In that case, all the more reasons to spend more time and find more bugs now.

    For reference, here are what we consider as qualities of a good bug report:

    • has a descriptive title
    • has enough details e.g., steps to reproduce, expected, actual, and screenshots.
    • severity/type labels chosen are not too far off
    • is written in a non-confrontational tone
    • points out a potentially problematic behavior (or a good way to improve the product)

    Evaluation rubric:

    • Meets expectations (i.e., good enough to earn participation points): 3-5 good bug reports
    • Exceeds expectations (i.e., good enough for 1 mark): more than 6-8 good bug reports
    • Greatly exceeds expectations (i.e., good enough for 2 marks): more than 9 or more good bug reports

    As you can't be sure which of your bug reports will be considered as good (the last criterion in particular) by the receiving team, we recommend that you aim to submit at least 8-10 bug reports in total if you aim for 1 mark, more if you are aiming for 2 marks. The median bug reports count in the last round was 9.

    PE and PE-D are manual testing sessions. Using test automation tools or scripting is not allowed. But you may use such techniques when testing your own product.

    Test the product and report bugs as described below, when the prof informs you to begin testing.

    Testing instructions for PE and PE-D
    a) Launching the JAR file
    • Get the jar file to be tested:
      PE Dry Run (at v1.5)
      • Download the latest JAR file from the team's releases page, if you haven't done this already.
      PE (at v1.6)
      • Download the zip file from the given location (to be given to you at least a few hours before the PE), if you haven't done that already.
      • The file is zipped using a two-part password.
        • We will email you the second part in advance, via email (it's unique to each student). Keep it safe, and have it ready at the start of the PE.
        • At the start of the PE, we'll give you the first part of the password (common to the whole class). Use combined password to unzip the file, which should give you another zip file with the name suffix _inner.zip.
        • Unzip that second zip file normally (no password required). That will give you a folder containing the JAR file to test and other PDF files needed for the PE. Warning: do not run the JAR file while it is still inside the zip file.
          Ignore the padding_file found among the extracted files. Its only purpose is to mask the true size of the JAR file so that someone cannot guess which team they will be testing based on the zip file size.
        • Recommended: Try above steps using the this sample zip file if you wish (first part of the password: password1-, second part: password2 i.e., you should use password1-password2 to unzip it).
          Use the JAR file inside it to try the steps given below as well, to confirm your computer's Java environment is as expected and can run PE jar files.

      Steps for testing a tP JAR file (please follow closely)

      1. Put the JAR file in an empty folder in which the app is allowed to create files (i.e., do not use a write-protected folder).
        In rare cases, the team could have submitted a ZIP file instead of a JAR file. In that case, unzip that file into the target folder.
      2. Open a command window. Run the java -version command to ensure you are using Java 17.
        Do this again even if you did this before, as your OS might have auto-updated the default Java version to a newer version.
      3. Check the UG to see if there are extra things you need to do before launching the JAR file e.g., download another file from somewhere
        You may visit the team's releases page on GitHub if they have provided some extra files you need to download.
      4. Launch the jar file using the java -jar command rather than double-clicking (reason: to ensure the jar file is using the same java version that you verified above). Use double-clicking as a last resort.
        We strongly recommend surrounding the jar filename with double quotes, in case special characters in the filename causes the java -jar command to break.
        e.g., java -jar "[CS2103-F18-1][Task Pro].jar"
        Windows users: use the DOS prompt or the PowerShell (not the WSL terminal) to run the JAR file.
        Linux users: If the JAR fails with an error labelled Gdk-CRITICAL (happens in Wayland display servers), try running it using GDK_BACKEND=x11 java -jar jar_file_name.jar command instead.

      If the product doesn't work at all: If the product fails catastrophically e.g., cannot even launch, or even the basic commands crash the app, do the following:

      1. Check the UG of the team, to see if there are extra things you need to do before launching the JAR.
        Confirm that you are using Java 17 and using the java -jar command to run the JAR, as explained in points above.
      2. Contact our head TA via MS Teams (name: Kim Hyeongcheol, NUSNET: dcskh) and give him
        (a) a screenshot of the error message, and
        (b) your GitHub username.
      3. Wait for him to give you a fallback team to test.
        Contact the prof (via MS Teams) if you didn't get a response from him within 5 minutes.
      4. Delete bug reports you submitted for the previous team (if any), using CATcher.
      5. You should not go back to testing the previous team after you've been given a fallback team to test.
      b) What to test

      • Test based on the Developer Guide (Appendix named Instructions for Manual Testing) and the User Guide PDF files. The testing instructions in the Developer Guide can provide you some guidance but if you follow those instructions strictly, you are unlikely to find many bugs. You can deviate from the instructions to probe areas that are more likely to have bugs.
        If the provided UG/DG PDF files have serious issues (e.g., some parts seem to be missing), ask prof for permission to use the Web versions of UG/DG instead.

      • The DG appendix named Planned Enhancements (if it exists) gives some enhancements the team is planning for the near future. The feature flaws these enhancements address are 'known' -- reporting them will not earn you any credit.
        However, you can report type.FeatureFlaws bugs if you think these enhancements themselves are flawed/inadequate.
        You can also report type.DocumentationBug bugs if any of the enhancements in this list combines more than one enhancement.

      • You may do both system testing and acceptance testing.

      • Focus on product testing first, before expanding the focus to reporting documentation bugs.
        Reason: If there are serious issues with the jar file that makes product testing impossible, you need to find that out quickly (within the first 10 minutes) so that you can switch to a different product to test. If you find yourself in such a situation much later, you will not be able to get the full allotted time for testing.

      • Be careful when copying commands from the UG (PDF version) to the app as some PDF viewers can affect the pasted text. If that happens, you might want to open the UG in a different PDF viewer.
        If the command you copied spans multiple lines, check to ensure the line break did not mess up the copied command.

      c) What bugs to report?
      • You may report functionality bugs, feature flaws, UG bugs, and DG bugs.




        PE Dry Run (at v1.5)
        • You can also post suggestions on how to improve the product.
          Be diplomatic when reporting bugs or suggesting improvements. For example, instead of criticising the current behavior, simply suggest alternatives to consider.
        PE (at v1.6)
        • Do not post suggestions but if the product is missing a critical functionality that makes the product less useful to the intended user, it can be reported as a bug of type Type.FeatureFlaw. The dev team is allowed to reject bug reports framed as mere suggestions or/and lacking in a convincing justification as to why the omission or the current design of that functionality is problematic.

        d) How to report bugs
        • Post bugs as you find them (i.e., do not wait to post all bugs at the end) because bug reports created/updated after the allocated time will not count. Even minor updates (such as changing a label) outside the allowed time window will invalidate that bug.
          Using CATcher
          • Launch CATcher, and login to the correct profile (when CATcher asks, consent to creating a new repo):
            • PE Dry Run: CS2103/T PE Dry run
            • PE: CS2103/T PE
          • Post bugs using CATcher.
          • If you encounter a 'Failed to fetch' error during CATcher login, try again. If it still fails, try a different network e.g., your mobile phone's hotspot.
          • If GitHub prompts you to 're-authorize' CATcher (this happens if GitHub detects a rush of login requests from the same account), just re-authorize as requested.
          Not using CATcher

          Issues created for PE-D and PE need to be in a precise format for our grading scripts to work. Incorrectly-formatted responses will have to discarded. Therefore, you are not allowed to use the GitHub interface for PE-D and PE activities, unless you have obtained our permission first.

          If you 'warmed up' CATcher for the PE earlier, you should already have a repo named pe in your GitHub account, created by CATcher during that warming up. If that is not the case, create a repo to post your bug reports as given in the panel below:

          • Post bug reports in the pe repo.
          • The whole description of the bug should be in the issue description i.e., do not add comments to the issue.
          • Choose exactly one type.* label and exactly one *.severity label.
          e) Bug report format
          • Each bug should be a separate issue i.e., do not report multiple problems in the same bug report.
            If there are multiple bugs in the same report, the dev team will select only one of the bugs in the report and discard the others.
          • When reporting similar bugs, it is safer to report them as separate bugs because there is no penalty for reporting duplicates. But as submitting multiple bug reports take extra time, if you are quite sure they will be considered as duplicates by the dev team later, you can report them together, to save time.
          • Write good quality bug reports; poor quality or incorrect bug reports will not earn credit.
            Remember to give enough details for the receiving team to reproduce the bug. If the receiving team cannot reproduce the bug, you will not be able to get credit for it.

          • Assign exactly one severity.* label to the bug report. Bug reports without a severity label are considered severity.Low (lower severity bugs earn lower credit)

          Bug Severity labels:

          • severity.VeryLow : A flaw that is purely cosmetic and does not affect usage e.g., a typo/spacing/layout/color/font issues in the docs or the UI that doesn't affect usage. Only cosmetic problems should have this label.
          • severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
          • severity.Medium : A flaw that causes occasional inconvenience to some users, but they can continue to use the product.
          • severity.High : A flaw that affects most users and causes major problems for users. i.e., only problems that make the product almost unusable for most users should have this label.

          When applying for documentation bugs, replace user with reader.

          • Assign exactly one type.* label to the issue.

          Type labels:

          • type.FunctionalityBug: A functionality does not work as specified/expected.
          • type.FeatureFlaw: Some functionality missing from a feature delivered in v1.6 in a way that the feature becomes less useful to the intended target user for normal usage. i.e., the feature is not 'complete'. In other words, an acceptance-testing bug that falls within the scope of v1.6 features.
            These issues are counted against the product design aspect of the project. Therefore, other design problems (e.g., low testability, mismatches to the target user/problem, project constraint violations etc.) can be put in this category as well.
            Features that work as specified by the UG but should have been designed to work differently (from the end-user's point of view) fall in this category too.
          • type.DocumentationBug: A flaw in the documentation e.g., a missing step, a wrong instruction, typos
          • If you need to include < or > symbols in your bug report, you can either use \ to escape them (i.e., use \< and \> e.g., x \< y instead of x < y) or wrap it inside back-ticks.
            Reason: CATcher and GitHub strips out content wrapped in < and >, for security reasons.

          PE-D After the session

          • The relevant bug reports will be transferred to your issue tracker within a day after the session is over. Once you have received the bug reports for your product, you can decide whether you will act on reported issues before the final submission v1.6. For some issues, the correct decision could be to reject or postpone to a version beyond v1.6.
            Reminder: There is no penalty for any of the bugs you received in the PE-D.

          Dealing with "What the h___ the tester was thinking?" type bug reports

          Some bug reports will make you angry because they seem baseless, wrong, rude etc. It's still possible to get value from such bug reports though:

          • After you got over the initial indignation, dig deeper to see if there's even the slightest possibility that there is a bug. For example, consider this scenario:
            1. The tester claims a certain command doesn't work.
            2. All your team members tried the exact same command and it works as advertised. What the h___ the tester is trying to pull here?
            3. In reality, the error is actually caused by a duplicate entry in the database resulting from a previous command; the tester didn't mention that command in the bug report (because s/he didn't realize the two are connected).
          • The reported bug might be non-existent but the tester's screw up can indicate other areas to improve. For example, the tester reports a missing feature that is clearly mentioned as 'not implemented' in the UG, but perhaps the UG can be improved to make that fact harder to miss?
          • What exactly about the bug report that makes you angry? Remind yourself not to do the same offence when you report bugs yourself in the future.

          Use the pain of dealing with this kind of bug reports as an opportunity to develop the following mindset:

          • The product is guilty until proven innocent: If the bug report has even a hint of something amiss with the product, it's your (not the tester's) responsibility to try and prove if it is really a problem or not. Why? because finding a bug is a win for you -- as you can then fix it and thereby avoid the embarrassment of releasing a buggy product.

          • A crappy bug report is better than no bug report: If the bug actually exists, it is better to have some indication about it than none at all. In a real project, a tester that fails to find bugs can cause more harm to your career than a tester who finds bugs but doesn't report them well.

          • I used to get angry at bug reports, but not anymore: If you work hard, take pride in the quality of your work, it's no wonder that you get angry when others find faults with your work incorrectly. Aim to move past that phase where you take bug reports personally. The sooner you can tackle any sh*tty bug report calmly and objectively, the sooner you'll rise to the 'professional' software engineer level.

          • If you have received stray bug reports (i.e., bug reports that don't seem to be about your project), do let us know ASAP (email the prof).
          • You can navigate to the original bug report (via the back-link provided in the bug report given to you) and post in that issue thread to communicate with the tester who reported the bug e.g. to ask for more info, etc. However, the tester is not obliged to respond. Note that simply replying to the bug report in your own repo will not notify the tester.
            • Do not argue with the tester to try to convince that person that your way is correct/better. If at all, you can gently explain the rationale for the current behavior but do not waste time getting involved in long arguments. If you think the suggestion/bug is unreasonable, just thank the tester for their view and discontinue to discussion.
          • Aim to do a systematic triaging of issues received. Some suggestions:
            • Close duplicate issues.
            • Use labels (create new labels if necessary) to,
              • differentiate bugs from the rest (e.g., feature suggestions/flaws).
              • indicate priority of the bugs that need fixing.
            • Assign each bug to the person who should fix it.
          • You may ignore type/severity.* labels given by the tester. They will not affect you or the tester either way -- they were there just for the testers to practice. You may apply your own type/severity labels if you wish.
            In particular, beware of simply following the type.* given by the tester; it is your job to decide the correct type of the issue. e.g., What the tester labeled as a bug might actually be a feature flaw.
          • If a bug report is simply a feature suggestion, you can take note of it and close it (to reduce clutter in the issue tracker, and to make it easy for the teaching team to track your progress on dealing with PE-D issues). Similarly, you can close PE-D issues not relevant to v1.6.

          Note that listing bugs as 'known bugs' in the UG or specifying unreasonable constraints in the UG to make bugs 'out of scope' will not exempt those bugs from the final grading. That is, PE testers can still earn credit for reporting those bugs and you will still be penalized for them.
          However, a product is allowed to have 'known limitations' (e.g., a daily expense tracking application meant for students is unable to handle expenses larger than $999) as long as they don't degrade the product's use within the intended scope. They will not be penalized.

          Even bugs inherited from AB3 need to be fixed. As mentioned in a previous week, even bugs you inherited from AB3 need fixing (because "we inherited it from the previous dev team" is not a valid excuse to leave a bug unfixed). If you are unsure if something is such a bug that need fixing, please post in the forum.

          Identify bugs you missed in the PE-D: Visit the issue tracker of the team you tested and see bugs reported by others who tested the same product. Identify bugs you missed (if any). That knowledge might help you find similar bugs in your own product as well as find more bugs during the PE.


          tP: DeliverablestP: Practical Exam