Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions app/docs/dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

This simple evaluation function checks if the supplied response is within a tolerance range defined in `params`. Works exactly like the [numpy.isclose](https://numpy.org/doc/stable/reference/generated/numpy.isclose.html#numpy.isclose) function.

Valid params include `atol` and `rtol`, which can be used in combination, or alone. As the comparison made is the following:
Valid params include `absolute_tolerance` and `relative_tolerance`, which can be used in combination, or alone. As the comparison made is the following:

```python
is_correct = abs(res - ans) <= (atol + rtol*abs(ans))
is_correct = abs(res - ans) <= (absolute_tolerance + relative_tolerance*abs(ans))
```

## Inputs
Expand All @@ -15,17 +15,17 @@ is_correct = abs(res - ans) <= (atol + rtol*abs(ans))
"response": "<number>",
"answer": "<number>",
"params": {
"atol": "<number>",
"rtol": "<number>"
"absolute_tolerance": "<number>",
"relative_tolerance": "<number>"
}
}
```

### `atol`
### `absolute_tolerance`

Absolute tolerance parameter

### `rtol`
### `relative_tolerance`

Relative tolerance parameter

Expand All @@ -42,7 +42,7 @@ Relative tolerance parameter
Real difference between the given answer and response

### `allowed_diff`
Allowed difference between answer and response, calculated using the supplied `atol` and `rtol` parameters
Allowed difference between answer and response, calculated using the supplied `absolute_tolerance` and `relative_tolerance` parameters


## Examples
16 changes: 8 additions & 8 deletions app/docs/user.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,20 @@ Use this evaluation function to check if a student's response is within a tolera
A response is accepted if:

```
|response - answer| ≤ atol + rtol × |answer|
|response - answer| ≤ absolute_tolerance + relative_tolerance × |answer|
```

The left-hand side is the absolute difference between the student's response and the correct answer. The right-hand side is the total allowed difference, made up of a fixed part (`atol`) and a part that scales with the size of the answer (`rtol × |answer|`). A response is marked correct whenever the actual difference does not exceed the allowed difference.
The left-hand side is the absolute difference between the student's response and the correct answer. The right-hand side is the total allowed difference, made up of a fixed part (`absolute_tolerance`) and a part that scales with the size of the answer (`relative_tolerance × |answer|`). A response is marked correct whenever the actual difference does not exceed the allowed difference.

## Parameters

Both parameters default to `0` (exact match required) and can be used individually or together.

### `atol` — Absolute tolerance
### `absolute_tolerance` — Absolute tolerance

Specifies a fixed margin around the answer, regardless of its magnitude. Use this when you know the acceptable error in the same units as the answer.

### `rtol` — Relative tolerance
### `relative_tolerance` — Relative tolerance

Specifies an acceptable error as a fraction of the answer's magnitude. Use this when the answer is very large or very small and a percentage-based margin makes more sense than a fixed one.

Expand All @@ -31,26 +31,26 @@ No params needed. The student must enter exactly `42` (floating-point precision
### Absolute tolerance

```json
{ "atol": 0.05 }
{ "absolute_tolerance": 0.05 }
```

With answer `9.81`, accepts any response in the range **9.76 – 9.86**. Good for physical measurements where the acceptable error is known in the same units.

### Relative tolerance

```json
{ "rtol": 0.01 }
{ "relative_tolerance": 0.01 }
```

With answer `6.674e-11`, accepts any response within **1%** of the answer. Good for very large or very small values where a fixed margin would be impractical.

### Combined tolerances

```json
{ "atol": 0.01, "rtol": 0.005 }
{ "absolute_tolerance": 0.01, "relative_tolerance": 0.005 }
```

Both tolerances contribute: with answer `9.81`, the allowed difference is `0.01 + 0.005 × 9.81 ≈ 0.059`. Useful when you want a minimum floor (`atol`) plus a proportional allowance (`rtol`).
Both tolerances contribute: with answer `9.81`, the allowed difference is `0.01 + 0.005 × 9.81 ≈ 0.059`. Useful when you want a minimum floor (`absolute_tolerance`) plus a proportional allowance (`relative_tolerance`).

## Notes

Expand Down
31 changes: 15 additions & 16 deletions app/evaluation.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,31 +21,30 @@ def evaluation_function(response, answer, params) -> dict:
to output the grading response.
"""

rtol = params.get("rtol", 0)
atol = params.get("atol", 0)

is_correct = None
real_diff = None
relative_tolerance = params.get("relative_tolerance", params.get("rtol", 0))
absolute_tolerance = params.get("absolute_tolerance", params.get("atol", 0))

if not (isinstance(answer, int) or isinstance(answer, float)):
raise Exception("Answer must be a number.")

real_diff = None
allowed_diff = atol + rtol * abs(answer)
allowed_diff = absolute_tolerance + relative_tolerance * abs(answer)
allowed_diff += spacing(answer)
is_correct = False
feedback = ""

if not (isinstance(response, int) or isinstance(response, float)):
feedback = "Please enter a number."
else:
real_diff = abs(response - answer)
allowed_diff = atol + rtol * abs(answer)
allowed_diff += spacing(answer)
is_correct = bool(real_diff <= allowed_diff)
return {
"is_correct": False,
"real_diff": None,
"allowed_diff": allowed_diff,
"feedback": "Please enter a number.",
}


real_diff = abs(response - answer)
is_correct = bool(real_diff <= allowed_diff)

return {
"is_correct": is_correct,
"real_diff": real_diff,
"allowed_diff": allowed_diff,
"feedback": feedback,
"feedback": "",
}
30 changes: 30 additions & 0 deletions app/evaluation_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -244,5 +244,35 @@ def test_response_is_not_number(self):
self.assertEqual(response.get("is_correct"), False)
self.assertEqual("Please enter a number." in response.get("feedback"), True)

def test_full_param_correct(self):
body = {
"response": 1e6,
"answer": 1e7,
"params": {
"relative_tolerance": 1e-1,
"absolute_tolerance": 8.2e6
},
}

response = evaluation_function(body['response'], body['answer'],
body.get('params', {}))

self.assertEqual(response.get("is_correct"), True)

def test_full_param_incorrect(self):
body = {
"response": 1e6,
"answer": 2e7,
"params": {
"relative_tolerance": 1e-1,
"absolute_tolerance": 8.2e6
},
}

response = evaluation_function(body['response'], body['answer'],
body.get('params', {}))

self.assertEqual(response.get("is_correct"), False)

if __name__ == "__main__":
unittest.main()
2 changes: 1 addition & 1 deletion readme.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# IsSimilar

This function checks whether a student's numeric response is within an acceptable tolerance of the correct answer, using absolute (`atol`) and relative (`rtol`) tolerance parameters. The comparison follows the formula: `|response - answer| ≤ atol + rtol × |answer|`. By default both tolerances are 0, requiring an exact match (within floating-point precision).
This function checks whether a student's numeric response is within an acceptable tolerance of the correct answer, using absolute (`absolute_tolerance`) and relative (`relative_tolerance`) tolerance parameters. The comparison follows the formula: `|response - answer| ≤ absolute_tolerance + relative_tolerance × |answer|`. By default both tolerances are 0, requiring an exact match (within floating-point precision).

For more information, look at the docs in `app/docs/`.

Expand Down
Loading