Instructions for Using the Human Notes Evaluator

This document provides detailed instructions for using the Human Notes Evaluator application.

Contents

1. Overview
2. Evaluating Documents
3. Viewing Results
4. Preparing Documents for Evaluation
5. Understanding Evaluation Criteria
6. Troubleshooting

Overview

The Human Notes Evaluator is designed to systematically assess documents (such as clinical notes or reports) across multiple quality criteria. The application:

• Presents documents to evaluators in a random order
• Collects ratings across 9 quality dimensions on a 1-5 scale
• Tracks which documents have been evaluated
• Displays results in a table format

Evaluating Documents

1. Enter your name in the "Evaluator Name" field
2. For each criterion, select a rating from 1-5 (Not at all to Extremely)
3. All criteria must be rated before submission
4. Click "Submit Evaluation" to record your assessment
5. The system will then present another random document for evaluation
6. Continue until all documents have been evaluated

Viewing Results

1. Click the "View Results" button in the header
2. Results are displayed in a table showing:
   • Timestamp of evaluation
   • Document identifier
   • Document description (if provided)
   • Evaluator name
   • Scores for each criterion

Preparing Documents for Evaluation

To add your own documents for evaluation:

1. Download the template file (sample_documents_template.csv) from the application
2. Open the template in a spreadsheet program (Excel, Google Sheets, etc.)
3. Add your documents using the following format:
   • filename: A unique identifier for each document (not shown to evaluators)
   • description: (Optional) A brief description of the document context
   • note: The full text of the document to be evaluated
4. Save the file as CSV format
5. Upload the completed CSV file using the file upload on the home page

Example CSV Format:

filename,description,note
doc1.txt,Patient with heart failure,This is the full text of the first document...
doc2.txt,Emergency room visit,This is the full text of the second document...
doc3.txt,,This is a document without a description...

Understanding Evaluation Criteria

Documents are evaluated on a 1-5 scale (Not at all = 1, Extremely = 5) across these criteria:

1. Up-to-date
   The note contains the most recent test results and recommendations.
2. Accurate
   The note is true. It is free of incorrect information.
3. Thorough
   The note is complete and documents all of the issues of importance to the patient.
4. Relevant
   The note is extremely relevant, providing valuable information and/or analysis.
5. Well-organized
   The note is well-formed and structured in a way that helps the reader understand the patient's clinical course.
6. Clear
   The note is clear, without ambiguity or sections that are difficult to understand.
7. Concise
   The note is brief, to the point, and without redundancy.
8. Thoughtful
   The note reflects the author's understanding of the patient's status and ability to develop a plan of care.
9. Internally consistent
   No part of the note ignores or contradicts any other part.

Troubleshooting

Common Issues:

1. No documents available for evaluation
   • Ensure the CSV file you uploaded exists and is properly formatted
   • Check that not all documents have already been evaluated
2. Unable to submit evaluation
   • Verify that all criteria have been rated
3. Results not showing
   • Results will only appear after at least one document has been evaluated

For additional support, please contact your administrator.