To verify that no data is lost with issues like, #7. There should be a script to test partial posts/comments & likes between the sincereDB and crawledDB. I.e. if a post is marked as done in crawledDB it should also exist in sincereDB. If not perhaps it should be marked crawled-not-parsed or similar.
To verify that no data is lost with issues like, #7. There should be a script to test partial posts/comments & likes between the sincereDB and crawledDB. I.e. if a post is marked as done in crawledDB it should also exist in sincereDB. If not perhaps it should be marked crawled-not-parsed or similar.