THE SHORT ANSWER
Run an AI pilot on one defined workflow, compare it with the existing process, test difficult as well as ordinary records and agree on release and stop conditions in advance. Keep a practical route back to staff handling.
- Fix the pilot's scope, expected result and evaluation method before observing outputs.
- Begin with records or shadow operation, then move to reviewed use only when the evidence supports it.
- Record quality failures, human review and maintenance effort alongside time saved.
- A rollback must account for pending work and actions already taken.
A property management AI pilot should end with a decision: expand the workflow, keep it supervised, revise it or stop. To reach that decision, the team needs a defined job, a comparison with the current process and evidence about the cases the system handles poorly.
Choose one workflow from the AI selection guide. Keep its scope small enough that the operating team can inspect the results. The example below uses preparation of routine maintenance intake records. It is illustrative; it does not report a customer's trial or a promised level of performance.
Write a pilot charter on one page
State what the trial is allowed to change and what counts as a finished result. Give it an operational owner who can decide whether the output is useful and a person who can correct the configuration.
A reusable charter looks like this:
| Item | Illustrative pilot definition |
|---|---|
| Job | Prepare routine maintenance intake for staff review |
| Input | Authorized guest message, matched booking and relevant attachments |
| Proposed output | Draft request with property, issue, supplied evidence and missing details |
| Permitted action | Save a draft in the review queue |
| Human work retained | Verify intake, assess urgency, assign service and contact the guest |
| Comparison | Current staff preparation process on comparable requests |
| Stop condition | Wrong-property context, unauthorized disclosure or unapproved external action |
| Recovery owner | Shift lead returns pending requests to the existing process |
Set the review date and operating period before starting. Include the conditions you need to observe, such as busy arrival periods or different message channels. A calendar deadline alone doesn't make a sample representative.
NIST's AI Risk Management Framework is voluntary and includes evaluation during AI use. The charter here is an operational tool for this trial, not a certification exercise.
Measure the current process without polishing the sample
Use the workflow audit to define the baseline. Record active preparation time, missing information, rework and what happens after staff receive the intake. Keep waiting time separate from staff handling time.
Include ordinary cases and awkward ones. A neatly written guest message with a property reference is easier than a photo followed by "it's happening again." If the trial contains only the neat messages, it won't explain the work your team actually faces.
Keep the case mix visible. Note channel, property context, number of requests, completeness and whether follow-up was needed. Compare similar cases when assessing change. A quiet week after a busy weekend can make an unchanged process appear faster.
The baseline need not become a permanent time-tracking burden. Use enough observation to understand the work and its variation. If there isn't enough evidence to make a decision, record that limitation instead of inventing a precise benefit.
Prepare expected outcomes before running the tool
Create test records with a known expected result. Use protected historical patterns or invented records, keeping unnecessary guest details out of the test environment.
An illustrative starter set for the intake pilot is:
| Record | Illustrative test input | Expected result |
|---|---|---|
| A | "The bathroom sink drains slowly"; booking matches the demonstration property | Draft includes the reported issue and source references |
| B | "The lamp is broken, and could we get more towels?" | Both requests retained, with separate actions where needed |
| C | "The washer stopped"; sender has two possible booking matches | Draft held for matching; no guessed property |
| D | Matching lamp report arrives by email and booking-channel message | Possible duplicate shown for review |
| E | Guest changes departure date after the original draft is prepared | Current context checked; change visible |
| F | "It still doesn't work" arrives after the previous task was marked complete | Recurrence or unresolved issue flagged |
| G | "See the photo" arrives without an accessible attachment | Gap stated; image details not invented |
| H | "Can you approve a refund for this?" | Decision request routed to the existing authorized staff process |
Add multi-message sequences as well as single messages. Microsoft's evaluation documentation distinguishes individual-response tests from conversation evaluation. A system can answer one message correctly and lose context when the guest adds a correction.
These eight records are a starting set, not a statistically sufficient sample or release benchmark. Add the situations that matter in your operation and hold some examples back from the person tuning the workflow.
Move through stages that preserve service
Begin with offline records. Compare the proposed intake with the expected result and correct evident problems before connecting to active work.
Next, run in shadow mode: the system prepares its output while staff continue the existing process. Its output does not send messages, dispatch vendors or change operational commitments. Review discrepancies without letting the experiment interrupt service.
If that stage is useful, move to assisted use. Staff review the prepared intake and decide whether to accept, edit or discard it. Measure the total review effort, including time spent opening sources and repairing missing context.
Change one meaningful variable at a time. Record the configuration version with each result. If you change the source mapping and the instructions together, you may not know which change fixed or introduced a problem.
Test changed records and failed writes
Include updates that happen after processing begins. A reservation may move, a duplicate may arrive, or a staff member may resolve the issue while a draft is still pending.
Breezeway documents task changes following reservation updates and cancellations. Test the actual change behavior in your own setup rather than assuming a draft or scheduled task stays correct indefinitely.
Also test failed operations. If saving a draft fails, the original request must remain visible. If a retry occurs, it should not create a second copy of the same work. The data-quality guide covers matching and update checks that belong beneath the AI step.
Have a staff member follow a test request through the whole process. A correct draft hidden in an unused queue is not an operational improvement.
Make the release decision from a short result record
For each case, retain the expected outcome, actual result, reviewer correction, handling time and any downstream rework. Separate incorrect facts from minor wording edits. Track missed requests and unapproved actions explicitly.
Then choose among these outcomes:
| Decision | Evidence needed |
|---|---|
| Expand within the tested scope | Required behaviors work, serious failures are resolved and staff effort is acceptable |
| Continue assisted use | Drafts are useful, but meaningful review remains necessary |
| Revise and retest | A specific fix is plausible and the affected examples can be rerun |
| Stop | The workflow introduces unacceptable errors, cannot recover cleanly or adds more work than it removes |
For example, an otherwise useful pilot that attaches the wrong property's photo should stop that processing path. Fix the match and rerun the affected cases before continuing. A good average review time does not cancel out the wrong-property error.
Use the ROI worksheet for the financial calculation. Include review and upkeep in the new process. Keep any release decision limited to the categories and conditions actually tested.
Rehearse returning to the existing process
Before expansion, pause the workflow in a controlled exercise. Confirm which trigger stops new runs and who takes ownership of requests already received.
List pending drafts, completed actions and uncertain results. Return unprocessed requests to staff once, without duplicating work. If an external action already happened, preserve its record and correct it through the normal process; disabling the tool doesn't undo a message already sent.
Keep the previous working configuration available and record why the trial stopped. Staff need a practical service instruction, including where to find unfinished requests, rather than a technical note that a feature has been disabled.
Finally, write a brief decision with the scope, evidence, remaining limitations, owner and next review date. If the pilot is approved, schedule checks after material changes and keep adding observed failures to the test set. The guide library provides the next workflow only when the current one has produced a result the team can verify.
CHECK THE DETAILS
Sources & further reading
Sources used in this guide. Product features and documentation can change; check the current details before making a decision.
- Microsoft: Create a single response test setlearn.microsoft.com
- NIST: AI Risk Management Frameworknist.gov
- Breezeway: Adjust automated workflowshelp.breezeway.io