Auto data migration

[jackycute]

It's a bit pain that if we need to migrate data (not schema).
The package we're using now (sequelize) might able to do this (correct me if wrong).
So we need to add some internal mechanism to auto detect version and run corresponding migration scripts.

[SISheogorath]

Definitely yes :D I saw the migrate-to-0.6.0 repo and thought "Oh no... not again!" :D

If there is something more specific you need help with feel free to ask/provide a task. #automateallthethings

[SISheogorath]

This looks very useful: https://github.com/sequelize/umzug

In case it's not what we are searching for we should maybe include the migrations in the main repository and add a version in database. This can used to stop runtime in case you jump between versions and automatic migrate in case a new minor version exists. This way the migration works smooth and we don't have to ship too much code in the main repository. More or less it's always the same base structure in the migration repos. So why source it out?

[jackycute]

@SISheogorath Hey, thanks for looking for us.
Actually we already using sequelize built-in migration feature
https://github.com/hackmdio/hackmd/tree/master/lib/migrations
And I bet it's using umzug to do these.

It could help us add/remove columns or add/remove tables.
But it can't help us to update or migrate data which need to insert or decode in our own way.
I've search for a while, maybe we have to do this by ourselves.

[jackycute]

Based on this issue https://github.com/sequelize/sequelize/issues/4417
I think we are on our own.

[jackycute]

And as @SISheogorath and I mentioned above.
The sequelize migration tool - sequelize-cli is using the umzug to do the migration.
So maybe we could learn form it.
https://github.com/sequelize/cli/blob/master/package.json

[a60814billy]

When first startup HackMD and postgreSQL didn't have any table schemas, It will cause some migration error.

hackmd_1          | 2017/02/21 17:09:20 Connected to tcp://hackmdPostgres:5432
hackmd_1          |
hackmd_1          | Sequelize [Node: 6.9.5, CLI: 2.5.1, ORM: 3.30.2]
hackmd_1          |
hackmd_1          | Parsed url postgres://hackmd:*****@hackmdPostgres:5432/hackmd
hackmd_1          | == 20160515114000-user-add-tokens: migrating =======
hackmdPostgres_1  | ERROR:  relation "public.Users" does not exist
hackmdPostgres_1  | STATEMENT:  ALTER TABLE "public"."Users" ADD COLUMN "accessToken" VARCHAR(255);
hackmdPostgres_1  | ERROR:  relation "public.Users" does not exist
hackmdPostgres_1  | STATEMENT:  ALTER TABLE "public"."Users" ADD COLUMN "refreshToken" VARCHAR(255);
hackmd_1          | Unhandled rejection SequelizeDatabaseError: relation "public.Users" does not exist
hackmd_1          |     at Query.formatError (/hackmd/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
hackmd_1          |     at Query.<anonymous> (/hackmd/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
hackmd_1          |     at emitOne (events.js:96:13)
hackmd_1          |     at Query.emit (events.js:188:7)
hackmd_1          |     at Query.handleError (/hackmd/node_modules/pg/lib/query.js:131:8)
hackmd_1          |     at Connection.<anonymous> (/hackmd/node_modules/pg/lib/client.js:180:26)
hackmd_1          |     at emitOne (events.js:96:13)
hackmd_1          |     at Connection.emit (events.js:188:7)
hackmd_1          |     at Socket.<anonymous> (/hackmd/node_modules/pg/lib/connection.js:121:12)
hackmd_1          |     at emitOne (events.js:96:13)
hackmd_1          |     at Socket.emit (events.js:188:7)
hackmd_1          |     at readableAddChunk (_stream_readable.js:176:18)
hackmd_1          |     at Socket.Readable.push (_stream_readable.js:134:10)
hackmd_1          |     at TCP.onread (net.js:548:20)

Although sequelize migration fail. but this does not affect HackMD running.

Maybe we'll solve this problem with this issue together.

[SISheogorath]

For those who want to try this one in Hacktober or afterward:

A useful reference is: https://github.com/abelnation/sequelize-migration-hello