Migrating Data from a Legacy System Without Losing History

In every system switch there is a moment when someone says: just move the data. That sentence hides the hard part. Data is not files that move from place to place; it is meaning preserved in a particular structure, and meaning can be lost even when every row was copied in full.

First: what even needs to move

The assumption that everything must move is expensive and usually unnecessary. A practical split into three groups:

Group	What it includes	Approach
Mandatory, structured	Active patients, allergies, medications, diagnoses, future appointments	Full conversion into the new system’s structure
Important, as text or a document	Historical visit summaries, letters, old results	Usually enough as a readable document linked to the patient and the date
Left behind	Old financial history, system logs, drafts, internal system data	Read access in the old system for a defined period

The distinction between the first row and the second is the meaningful one. Structured data is data the system can act on — an allergy that triggers a warning against a prescription, a medication that appears in an active list. The same information as free text inside a summary looks identical to a human eye and is worthless to the system. Allergies and medications therefore get top priority in structured conversion, even if they are a small amount of data.

The traps that recur almost every time

1. A national ID with a leading zero

An ID number stored as a number rather than as text loses its leading zeros. A patient with a number that starts with zero becomes an eight-digit number, and then check-digit validation fails, the match against the health fund breaks, and duplicates are created. This is the most common failure in migrations in Israel, and it is easy to check in advance.

2. Dates in a mixed format

A file in which some of the dates were written as day-month-year and some as month-day-year will convert without error, and will produce wrong birth dates in every case where the day is less than 13. The check: count how many birth dates fall in an implausible range, and examine the distribution by month.

3. Character encoding

Hebrew stored in an old encoding and read as a different encoding turns into gibberish. Sometimes this is obvious, and sometimes it touches only certain characters such as an apostrophe or punctuation marks, and then it is discovered late.

4. Hidden control characters

Systems that display Hebrew alongside numbers sometimes add invisible directional control characters. If they were saved together with the data and not only in the display, they will pass through the conversion and break search and exact matching: a name will look identical on screen but not be found in a search. It is worth cleaning them out explicitly.

5. Value lists that do not match

Statuses, appointment types, document categories — every system has its own lists. Mapping them is manual work there is no way around. Better to do it early, because it usually reveals that the old list grew without order and includes duplicate values.

6. Duplicate patients

Almost every legacy system has duplicate patients: once with a name in Hebrew and once in Latin, once with a passport and once with a national ID. A migration carries the duplicates over as they are. The moment before the migration is the best opportunity to merge them.

How it works in practice

1. An early first export. Even before a final decision, ask for a sample export. It reveals what can be extracted at all, and that affects the decision.
2. A trial conversion on a copy. Run the conversion in full into a test environment, and go over it. Do not skip this.
3. A manual sample check. Choose twenty patients — including edge cases: a long name, a patient with a passport, an infant, a patient with a long history — and compare screen against screen.
4. Count checks. How many patients, how many visits, how many documents, how many allergies — before and after. A gap requires an explanation.
5. A final conversion in a defined window. Over a weekend, with a predefined rollback point and a decision on who declares success.
6. Read access to the old system. For a defined period. This is the safety net that lets you sleep at night.

A check worth doing every time: take five patients and verify that the sum of charges, the number of visits and the allergy list are identical in both systems. If something does not add up in five, it will not add up in five thousand.

Who does what

A successful migration requires three clear roles. The outgoing vendor provides the data and explains its structure. The incoming vendor performs the conversion and is responsible for the technical result. And the clinic — this is the part people tend to skip — decides the questions of meaning: what counts as an active patient, how to map an old status that has no equivalent, and what to do with records that contradict themselves. These are not technical questions, and no vendor can answer them for you.

It is worth allocating explicit time for this. One working day of someone who knows the data, early in the process, saves weeks of after-the-fact corrections.

What to do with data that cannot be converted

Every migration leaves a tail: records with missing fields, values that have no equivalent, corrupted documents. The temptation is to discard them quietly. The right approach is to separate them into three piles. What can be fixed manually in a reasonable quantity — fix. What is not critical — record and leave. And what is critical and cannot be converted — bring it over as an attached document with a note, so that at least the information is accessible to a human eye.

What matters is that the third pile be known and documented. The dangerous gap is not a datum that is missing, but a datum everyone assumes exists.

What to document

At the end of the migration it is worth leaving a short document describing what was moved, what was not, how the value lists were mapped, and what is known about data that could not be converted. Two years from now, when someone asks why documentation from a certain year is missing, this document will save a working day and misunderstandings.

And finally, what cannot be transferred

Some of the knowledge in a clinic is not in the system but in people’s heads: who the sensitive patients are, why a certain appointment is held, how a certain process is run. A move to a new system is a good opportunity to write some of this down. Not to document for documentation’s sake, but because during the transition every process is being reexamined anyway — and this is the only time in a decade that this happens.

Back to the blog