I’ve been using Remember the Milk (RTM) for a decade and was going to switch to Everdo. RTM allows users to export to JSON which I should be able to alter and later import into Everdo.
It’s going to take a bit of work to understand and transform the formatting and I could (attempt) to write this as a Python tool for others to reuse but I was unsure of interest. Making it with the intention of sharing would change how I write this and require extra work but would others find it of value?
I think it might help someone eventually, but also the RTM export format could change in the future, so it would require tweaking the script.
Let me know if you have questions regarding the import format or similar.
Finally coming back to this. Is it safer to work with API calls instead of directly importing JSON? I note the warnings about possible corruption on the import page
The API endpoint is extremely limited - it’s not really designed for importing data.
Corruption via JSON is possible because it’s too flexible. To avoid corruption, it’s wise to backup the database and verify that the the application works afterwards. Specifically, the navigation to all lists should work.
Thanks for suggesting how to test the JSON import was successful.
Any hints on tracking down corrupted objects? I did back things up and can roll back but that doesn’t help me in finding the problem as I do want to import the entries. I can navigate to all lists but the UI only renders item counts for two of the six. It’s not replicating the problem entries to the master and I don’t mind getting my hands dirty to tidy items up. Are there any debug logs I can enable on the desktop client? I couldn’t spot anything interesting in renderer.log.
The UI only rendering item counts on some lists is expected behavior, I just hadn’t noticed it before.
My problem is that the data isn’t replicating; I import the ~280 items and ~40 tags on a client, can browse to the six lists and filter by tags but items aren’t sync’d back to the server. If I create an item using the UI after the import then that is sync’d.
Is there a limit I’m exceeding? Should I be importing on the server?
After performing an import, you could try to trigger a manual push to the server and see what happens.
I’ve got three clients and one server, I’ll call the client I’m importing on clientA
I did a manual push from clientA and the items did appear on the server but
a) They were not distributed to other clients (clientB and clientC)
b) Updating imported entries on the server did not result in an update on clientA.
My gut says my is being imported by clientA but not distributed as Everdo thinks it’s invalid. Do you have any logging/debugging suggestions?
Choosing an entry at random, they seem valid but maybe I’ve overlooked something:
{
"id": "BCE6E026B0AA47D78AF8752312C61B07",
"type": "a",
"list": "i",
"title": "settle XXXXX",
"created_on": 1587183282,
"is_focused": 0,
"due_date": 1612011600,
"tags": [
"58A592FB4D174EE1B5D0DEAC626428A1",
"B3E60E4CC5624FAD9D7C361D58C286AA",
"59ACB12B0AB340489F336FED81BB4523"
],
"note": "Repeating config: FREQ=WEEKLY;INTERVAL=6;WKST=SU \nRTM ID: 770524877"
},
I didn’t bother mapping the repeating rules, the note text is just to remind me what to do.
and a matching list i’m creating
{
"id": "B3E60E4CC5624FAD9D7C361D58C286AA",
"title": "auto-import",
"type": "l"
},I note my created_on date is going to be years younger then my Everdo installation, maybe this is leading to them being ignored?
Also what is the deletions section of the json output? I have five invalid entries in this on my server side:
{
"sync_id": "4442424636433746",
"sync_id_string": "DBBF6C7F",
"ts": 1609371246,
"entity_type": "t"
},
{
"sync_id": "DBBF6C7FAB4F4AB2AE40D230E5266009",
"sync_id_string": "DBBF6C7FAB4F4AB2AE40D230E5266009",
"ts": 1609371246,
"entity_type": "t"
},
{
"sync_id": "67B1E84622404CB5A9A59793DC9157C1",
"sync_id_string": "67B1E84622404CB5A9A59793DC9157C1",
"ts": 1609371391,
"entity_type": "t"
},
{
"sync_id": "3143414634434432",
"sync_id_string": "1CAF4CD2",
"ts": 1609371439,
"entity_type": "t"
},Yes, this is the reason incremental sync doesn’t work. You should push / pull these items manually to all devices. Then incremental updates will begin to work.
Deletions represent deleted objects. If you are migrating data from a different system, you don’t need to populate this array.
[2021-01-05 10:33:27.115] [error] {"ts":"2021-01-04T23:33:27.115Z","message":"this.services.syncController.initiateSync is not a function","stack":"TypeError: this.services.syncController.initiateSync is not a function\n at Wi.initiateSync (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:536:1671210)\n at nt (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:9:15415)\n at Proxy.i (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:9:18853)\n at nt (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:9:15415)\n at Proxy.e.$emit (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:9:44015)\n at click (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:536:1561851)\n at nt (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:9:15415)\n at HTMLButtonElement.i (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:9:18853)\n at HTMLButtonElement.Dr.a._wrapper (file:///C:/Users/Username/AppData/Local/Programs/Everdo/resources/app.asar/bundle.js:9:58773)"}
Does the 2-way sync work without errors before the import?
Sync mode client says everything is fine but Manual Client - Sync Now gives the same error.
I have just tested it and it looks like the Sync button in the settings got broken in one of the recent updates. The button is rarely used, so the issue went unnoticed for a while. But this bug probably created a lot of confusion here.
After completing the Push/Pull cycle, try setting the sync mode to “Client” instead of “Manual Client”. This will avoid the error and sync will run automatically. Meanwhile the button defect will be fixed in the next release.
Sync status is success in Client mode but changes to items after the first push are not propagated, I was testing by changing tags.
Out of interest I rolled back, set the created_on for all new entries to 1609919008 (~now) and imported on ClientA but the entries were still not synced to the server. I didn’t do a manual push as I’d expect new entries to be handled like any other.
I’ve been able to reproduce my issue using random alphanumeric strings, is there a support email address I can post the json file to?
As the human language elements are removed it’s also easier to troubleshoot, without any manual push/pull it appears one item is syncing but it’s somehow being duplicated.
[2021-01-07 12:52:36.801] [debug] SyncLogger.ts: Sync request created. Changes since last sync @ 1609984351: 2 items, 0 tags, 0 deletions.I think I found the issue. You need to set sync_id on each item in order for it to sync properly. This is really a flaw in the import logic - it should at least try to populate sync_id equal to id.
If you try to import something like this:
"items": [
{
"id": "4E7F84093DA741EBB9032323D03E0517",
"sync_id": "4E7F84093DA741EBB9032323D03E0517",
"type": "a",
"list": "i",
"title": "test-123",
"created_on": 1609986274,
"is_focused": 0,
"time": 15,
"tags": []
}
]
you will see that it works after a push/pull.
Also, if you populate both created_on and changed_ts using the current timestamp, the items will sync incrementally without a push.
Many thanks for investigating, apologies for the delay in the response but I didn’t get an email alert and I didn’t want to bother you. I’ll test this tomorrow.