
[[jakarta-tck-appeals-process]]
= Jakarta TCK Test Appeals Process

Jakarta has a well established process for managing challenges to its
TCKs. Any implementor may submit a challenge to one or more tests in the
Jakarta EE TCK as it relates to their implementation.  Implementor
means the entity as a whole in charge of producing the final certified release.
*Challenges filed should represent the consensus of that entity*.

== Valid Challenges
Any test case (e.g., test class, @Test method), test case configuration (e.g., deployment descriptor), test beans, annotations, and other resources considered part of the TCK may be challenged.

The following scenarios are considered in scope for test challenges:

* Claims that a test assertion conflicts with the specification.
* Claims that a test asserts requirements over and above that of the specification.
* Claims that an assertion of the specification is not sufficiently implementable.
* Claims that a test is not portable or depends on a particular implementation.

== Invalid Challenges
The following scenarios are considered out of scope for test challenges and will be immediately closed if filed:

* Challenging an implementation’s claim of passing a test.  Certification is an honor system and these issues must be raised directly with the implementation.
* Challenging the usefulness of a specification requirement.  The challenge process cannot be used to bypass the specification process and raise in question the need or relevance of a specification requirement.
* Claims the TCK is inadequate or missing assertions required by the specification.  See the Improvement section, which is outside the scope of test challenges.
* Challenges that do not represent a consensus of the implementing community will be closed until such time that the community does agree or agreement cannot be made.  The test challenge process is not the place for implementations to initiate their own internal discussions.
* Challenges to tests that are already excluded for any reason.
* Challenges that an excluded test should not have been excluded and should be re-added should be opened as a new enhancement request

Test challenges must be made in writing via the {TechnologyShortName} specification project issue tracker
as described in <<tck-test-appeals-steps>>

All tests found to be invalid will be added to the Excluded Tests for that version of the {TechnologyShortName} TCK.


[[tck-test-appeals-steps]]
== TCK Test Appeals Steps

1. Challenges should be filed via the Jakarta EE Platform specification project’s issue tracker using the label `challenge` and include the following information:
* The relevant specification version and section number(s)
* The coordinates of the challenged test(s)
* The exact TCK version
* The implementation being tested, including name and company
* The full test name
* A full description of why the test is invalid and what the correct behavior is believed to be
* Any supporting material; debug logs, test output, test logs, run scripts, etc.

2. Specification project evaluates the challenge. +
Challenges can be resolved by a specification project lead, or a project challenge triage team, after a consensus of the specification project committers is reached or attempts to gain consensus fails.
Specification projects may exercise lazy consensus, voting or any practice that follows the principles of Eclipse Foundation Development Process.
The expected timeframe for a response is two weeks or less.
If consensus cannot be reached by the specification project for a prolonged period of time, the default recommendation is to exclude the tests and address the dispute in a future revision of the specification.

3.  Accepted Challenges. +
A consensus that a test produces invalid results will result in the exclusion of that test from certification requirements, and an immediate update and release of an official distribution of the TCK including the new excluded tests. The associated `challenge` issue must be closed with an `accepted` label to indicate it has been resolved.

4.  Rejected Challenges and Remedy. +
When a`challenge` issue is rejected, it must be closed with a label of `invalid` to indicate it has been rejected.
There appeal process for challenges rejected on technical terms is outlined in Escalation Appeal.
If, however, an implementer feels the TCK challenge process was not followed, an appeal issue should be filed with specification project’s TCK issue tracker using the label `challenge-appeal`.
A project lead should escalate the issue with the Jakarta EE Specification Committee via email (jakarta.ee-spec@eclipse.org).
The committee will evaluate the matter purely in terms of due process.
If the appeal is accepted, the original TCK challenge issue will be reopened and a label of `appealed-challenge` added, along with a discussion of the appeal decision, and the `challenge-appeal` issue with be closed.
If the appeal is rejected, the `challenge-appeal` issue should closed with a label of `invalid`.

5. Escalation Appeal. +
If there is a concern that a TCK process issue has not been resolved satisfactorily, the
https://www.eclipse.org/projects/dev_process/#6_5_Grievance_Handling[Eclipse Development Process Grievance Handling] procedure should be followed to escalate the resolution. Note that this is not a mechanism to attempt to handle implementation specific issues.